Some Challenges for Intelligent Tutoring Systems
Proceedings of the Tenth International Joint Conference on Artificial Intelligence. Panels and Invited Talks. Pages 1166-1167.
© 1987 International Joint Conferences on Artificial Intelligence
Abstract
- Brown, J. S. & Burton, R. R. (1978). Diag- nostic models for procedural bugs in basic mathematical skills. Co~nitive Science. 2.. pp 155-192.
- Cronbach, L J & Snow, R E (1977). Al2li llHks lllli1 Instructional Methods. New York: Irvington.
- Johnson, W L (1986). Intension-based Pia~nosis Qf ~ Pro~mmin~ Ermr.s... London: Pitman (Research Notes in AI series).
- Langley, P & Ohlsson, S (1984). Automated Cognitive Modelling. ~ ~ Qf ~ AAAI .:. .8.4 Conference. pp 193-197.
- Sleeman, D (1982a) Assessing aspects of competence in basic algebra. In Intelli~ent Tutorin~ ~ (eds D Sleeman & J S Brown) London: Academic Press, pp 185 - 199.
- Sleeman, D (1982b). Inferring (mal)rules from pupils' protocols. In the Proceedin~s Q{ ~ 12.82 European AI Conference. pp 160-164.
- Sleeman, D (in press). PIXIE: A shell for developing Intelligent Tutoring Systems. In lntelli~ent CAl (M. Yazdani & R. Lawler, eds). New Jersey: ABLEX. 1168
- PANELS AND INVITED TALKS Sleeman, D, Hirsh, H B & InYung, Kim (in prep). Expanding an Incomplete Domain Theory: Two Case Studies.
References were extracted automatically from the PDF and may contain errors.
Full text extracted automatically from the PDF; figures, tables and equations may differ. The PDF is the version of record.
D Sleeman
Department of Computing Science, The University, Aberdeen, AB9 2UB, Scotland, UK
In this paper, I will outline several critical issues which will have to be addressed before ITSs (Intelligent Tutoring Systems) can become truly responsive to the problems experienced by individual stu dents. Firstly, such systems need the abil ity to analyse student responses and infer the nature of previously unencountered behaviour patterns. Secondly, even if one has an accurate and highly detailed student model, ITSs need to communicate the nature of the student's errors to the indivi dual student. This turns out to be a difficult problem. These issues will be dis cussed in detail in my presentation; an out line is given below:
Explainin~: Previously Unencountered Behaviour
One of the contributions of recent work by workers in the ITS and Cognitive Science fields has been to show that student errors that were said by teachers to be the result of careless work are often due to slightly buggy procedures.
Brown & Burton (1978) report such bugs for subtraction:
| 72 | 123 |
| 28 | 78 etc |
| 56 | 155 |
where the smaller number is always sub tracted from the larger irrespective of its position in the column. Similarly, Sleeman (Sleeman, 1982a) has reported a number of common errors in algebra:
&
Builders of ITSs have encoded such errors, thus making their systems capable of diagnosing them, (Brown & Burton, 1978; Sleeman, 1982a). Similarly, PROUST is able to analyse deviant student programs by having variants of program plans explicitly represented (Johnson, 1986). The great challenge is to build systems that are able to identify previously unencountered rules. Langley and Ohlsson (Langley & Ohlsson, 1984) address the issue of the student using a known operator inappropriately. This research group has addressed the issue of creating new operators, or at least operators which are variations of known operators. We have evolved two quite distinct approaches:
- The INFER* system attempts to work forward from the initial task, backwards from the student's answer, and infers new rule(s) to fill the "gaps" created by this bi-directional search. (Note the backward search uses a series of heuristics to focus the search.)
The MALGEN system takes a correct rule set, applies a series of perturba tions to the rule-set and "filters" the resulting rules to remove those which are thought to be totally unreasonable. (For example, algebra students appear to know the form of an acceptable answer and so we have never seen m x = n transformed to x = n/ or x = /m.)
Below we give a trace of the INFER* program as it attempts to infer a new mal rule for the task 3 + 4 x = 9, given the stu dent answer x = 2:
Comment on steps
A to B: This is a simple algebraic rearrangement.
- B to C: This uses knowledge of the target equation (3 + 4 x = 9) to instantiate n to be a number on the right hand side of the target equation. (This is a heuristic.)
- C to D: This uses a domain rule in the backward direction. [Note there are 2 ways this can fire.]
D to E: This step is analogous to the A to B step.
E to F: This step is analogous to the B to C step, but here uses knowledge of the coefficient on the left hand side of the equation to instantiate the variable.
- G to F: The Inference Step is applied as none of the heuristics or domain rules are applicable.
The above trace shows a successful path. Often in trying to find a new rule in a protocol, the algorithm generates a large number of paths (as of course there are a sizeable number of heuristics). An initial form of this work was described in Slee man (l982b); a more sophisticated algo rithm is discussed in Sleeman, Hirsh & Kim (1987). This current algorithm is able to determine complex numerical relation ships between a series of coefficients, and is able to progressively resolve rules once an additional new rule has been encoun tered.
PIXIE illli1 Model-based Remediation
It has long been assumed by workers in ITS that once one has a complete model of a student's problem solving on a range of tasks, then remediation of the student would be straightforward, (Brown & Burton 1978). Recent experiments with the PIXIE project have attempted to demonstrate this.
As a result of earlier analyses of stu dents' errors in solving algebra tasks we were able to formulate a number of errors or mal-rules. These, together with correct rules, and a series of tasks formed a data base for the PIXIE Intelligent Tutoring Sys tem (Sleeman, in press). PIXIE is able to infer a model of the student's problem solv ing expressed in terms of these rules and mal-rules. In our earlier experiments, we merely provided these analyses to the teacher. Subsequently, we watched a teacher remediate individual students based on these analyses, and after interviewing a number of other teachers, implemented a remedial system based on this experience. (Virtually all the teachers we encountered said that they taught algebra in a pro cedural, or rule-based, way:)
The form of the remediation now provided by PIXIE is:
- highlight each of the incorrect transi tions in the student's protocol (e.g., "you appeared to change 3x+5=9 to x+ 3+5=9"), explain why that was wrong and then say what the change should have been. (This step is repeated for ~ error in the proto col.)
- show the student with an appropriate commentary ~ he should have solved the complete task.
When this was first tried out with a small number of students, we were told:
- the comments were too verbose.
- certain mathematical terms used in our explanations were not understood. [We have now provided a flash card.]
- there were difficulties because PIXIE's standard approach was to ask the stu dent to work a series of tasks, then, if necessary to start remediating from the first task; by which time the student had forgotten how he had worked the first task. [We now remediate, immediately after an error, when necessary.]
These changes were made to PIXIE and we ran an experiment to determine whether simply RETEACHING a topic was as effective as TARGETED (or MODEL BASED REMEDIATION) ie. the remedial sequence discussed above. As this result was inconclusive we decided that we should attempt a series of experiments using human-tutors to probe these phenomena. A summary of these experi ments is:
- no difference has been found between the effectiveness of RETEACIDNG and TARGETED remediation - even when we modified the TARGETED approach to include Cognitive Disso nance (i.e., demonstrating to the pupil his solution was wrong) and Cognitive Engagement (involving the student actively in explaining what he had done).
the errors with this population appear to be unstable. That is errors, noted on a pretest often have disappeared by the time of the interview several days later.
Sum mazy
Getting a system to recognise previ ously unencountered behaviour patterns, is a very open-ended problem which is, in general, not-solvable.
Even though PIXIE has a model for a student's problem solving, it has not so far proved possible to remediate very effectively. There appears to be two rea sons for this. Firstly, the errors shown by this student population on this topic appear to be unstable. (Thus an important pre condition for remediation is invalidated.) Secondly, we now believe that remediation is very complex - even when one knows what one wishes to communicate. After all, much of our social know-how is decid ing how to say something to another person so we can be understood, and secondly so that the listener will do what the speaker wishes him to do. (Frequently, we have to go to extreme lengths to achieve this. Such as asking a known competitor to do the task, thereby the person himself wishes to do the task.) Cronbach and Snow (1977) have already demonstrated the difficulty of deciding an effective treatment for a student on the basis of a student's aptitudes. Their work was based on post-hoc analyses of the experiments of a series of investigators; thus ITS workers had some justification for believing those conclusions were unduly pessimistic. However, the recent experi ments with PIXIE suggest that their conclu sions are probably worryingly realistic!
@inproceedings{ijcai1987p237,
title = {Some Challenges for Intelligent Tutoring Systems},
author = {Sleeman, Derek H.},
booktitle = {Proceedings of the Tenth International Joint Conference on
Artificial Intelligence, {IJCAI-87}},
publisher = {International Joint Conferences on Artificial Intelligence},
editor = {John McDermott},
pages = {1166--1167},
year = {1987},
note = {Panels and Invited Talks},
url = {https://ijcai.org/proceedings/1987/237},
}
TY - CPAPER TI - Some Challenges for Intelligent Tutoring Systems AU - Sleeman, Derek H. T2 - Proceedings of the Tenth International Joint Conference on Artificial Intelligence, IJCAI-87 PB - International Joint Conferences on Artificial Intelligence PY - 1987 SP - 1166 EP - 1167 UR - https://ijcai.org/proceedings/1987/237 ER -Copied
Machine-readable versions of this paper, for tools and AI agents:
Welcome
Future Conferences
AI Journal Division
IJCAI Proceedings
About IJCAI
IJCAI Awards