Cognitive Modelling for Predicting Examinee Performance

1 Introduction

Recently, there are a lot of researches for better education, e.g., on massive open on-line course (MOOC) [Anderson et al., 2014], intelligent tutoring systems [Burns et al., 2014] and cognitively diagnostic assessment [Nichols et al., 2012].

A crucial step of these educational researches is the cognitive modelling of examinees, which aims at discovering the latent factors/characteristics of examinees. Usually, the effectiveness of cognitive analysis is validated by predicting the possible scores from these examinees to each problem by collaboratively modelling a group of examinees, i.e. predicting examinee performance (PEP). Since PEP could be further applied to numerous applications, such as the personalized remedy recommendation and the teaching plan improvement, massive efforts of both data mining and educational psychology have been made to the solutions of cognitive modelling for PEP: In terms of data mining, Matrix factorization (MF) [Koren et al., 2009] is a classical prediction technique, which is widely used to model examinees [Toscher and Jahrer, 2010; Thai-Nghe et al., 2010; Desmarais, 2012]. In educational psychology, most of the existing studies focus on cognitive diagnosis. With the cognitive diagnosis models (CDMs) [DiBello et al., 2006], examinees are characterized by the proficiency on specific skills (e.g. problem-solving skills like calculation), where a Q-matrix [Tatsuoka, 1984] is previously given as the prior knowledge from education experts for denoting which skills are needed for each problem.

∗Corresponding author.

In spite of the importance of previous studies, there are still some limitations in existing methods. For instance, the latent factors in MF are unexplainable for describing the knowledge state of examinees' cognition. Comparatively, the results of CDMs could lead to a better interpretation. However, CDMs can only analyse examinees based on the simple objective problems, and the information of the subjective problems is largely underexplored. As shown in Figure 1, the objective (e.g. chosen) problems have dichotomous scores with either right or wrong response and the subjective (e.g. free-response) problems have polytomous scores with correct, incorrect or partially correct response. Obviously, for subjective problems, it is harder for examinees to get a correct response by guessing an answer or get a wrong response by carelessness (e.g. a slip of a pen). Thus, these subjective problems measure the examinees much better, and it is of significant importance to extract information from both objective and subjective problems for cognitive modelling rather than simply ignore the subjective problems or treat them as objective ones [Samejima, 1972]. Towards this goal, there are several challenges: How to handle the dichotomous scores of objective problems and the polytomous scores of subjective problems simultaneously? How to get both precise and interpretable cognitive analysis? How to predict the examinee performance based on this cognitive modelling?

To conquer these challenges, in this paper, we propose a fuzzy cognitive diagnosis framework (FuzzyCDF) for examinees' cognitive modelling. Since the response to a subjective problem may be partially correct, we adopt a solution inspired by fuzzy systems. Specifically, we first fuzzify the skill proficiency of examinees using fuzzy set theory. Then we suppose the skill interactions on objective and subjective problems satisfy two different hypotheses: conjunctive and compensatory [Pardos et al., 2008], and fuzzify the problem mastery of examinees based on these two hypotheses. Next, we model the generation of problem scores (predict examinee performance) by considering two exceptions: slip and guess. In this way, FuzzyCDF is represented as a four-tier hierarchical model and we propose a Monte Carol Markov chain (MCMC) sampling algorithm to estimate parameters, and thus we could predict examinee performance (i.e. scores). The main contributions of this paper are outlined as follows:

• To the best of our knowledge, this is the first comprehensive attempt for extracting information from both objective and subjective problems to get more precise and interpretable cognitive analysis for PEP task.
• We propose a four-tier hierarchical cognitive model FuzzyCDF, which combines educational assumptions and fuzzy theories to redefine skill proficiency and problem mastery, and it predicts the examinee performance by considering the slip and guess factors.
• We design a simple but effective MCMC sampling algorithm for parameter estimation and conduct extensive experiments on real-world datasets to demonstrate the effectiveness of FuzzyCDF.

2 RelatedWork

We introduce existing modelling methods for PEP from two aspects: data mining methods and cognitive diagnosis.

Data Mining Methods. Recently, more and more studies demonstrate the effectiveness of MF for examinee modelling and predicting examinee performance by factorizing the score matrix. For instance, [Toscher and Jahrer, 2010] utilized singular value decomposition (SVD) and other factor models to model examinees. In [Thai-Nghe et al., 2010], MF technique was compared with regression methods for PEP. Besides, [Desmarais, 2012; Sun et al., 2014] applied nonnegative matrix factorization to infer the Q-matrix. However, the latent factors inferred by the MF model are usually unexplainable, i.e. each dimension of the factor vector cannot correspond to a specific skill. In this work, we will adopt cognitive diagnosis on exam data to obtain interpretative results.

Cognitive Diagnosis. In educational psychology, many cognitive diagnosis models (CDMs) [DiBello et al., 2006] have been developed to mine examinees' skill proficiency, which can be used for PEP. Figure 2 shows a toy process of cognitive diagnosis. CDMs can be roughly divided into two categories: continuous ones and discrete ones. The fundamental continuous CDMs are item response theory (IRT) models [Rasch, 1961; Birnbaum, 1968; Embretson and Reise, 2013], which characterize examinee by a continuous variable, i.e. latent trait, and use a logistic function to model the probability that an examinee correctly solves a problem. For discrete CDMs, the basic method is deterministic inputs, noisy "and" gate model (DINA) [Haertel, 1984; Junker and Sijtsma, 2001; De La Torre, 2011]. DINA describes an examinee by a latent binary vector which denotes whether she masters the skills required by the problem, and a given Q-matrix is used to guarantee the interpretation of the diagnosis results. Though discrete CDMs can model examinees interpretatively, their diagnosis results are usually not accurate enough. Furthermore, existing methods cannot handle the subjective problems.

3 Fuzzy Cognitive Diagnosis

In this section, we will introduce our fuzzy cognitive diagnosis framework (FuzzyCDF). As shown in Figure 3 (from up to bottom), our proposed method is a generation process which starts with examinees' latent trait, then determines exminees' skill proficiency, next computes examinees' problemmastery and generates examinees' observable scores on problems by considering slip and guess factors. And we propose a MCMC sampling algorithm to infer the unobservable parameters of FuzzyCDF. For better illustration, Table 1 shows some math notations and each step of FuzzyCDF will be specified in the following subsections.

3.1 Fuzzifying Skill Proficiency

In this subsection, we show the way to get the proficiency of examinees on specific skills (e.g. problem-solving skills like calculation), which is the first step in cognitive diagnosis models. Here, the challenge is that we have to handle both objective problems and subjective problems, simultaneously.

In DINA-based CDMs, the examinees' skill proficiency is assumed to be absolutely mastered (i.e. 1) or nonmastered (i.e. 0) so that this modelling can fit objective problems with absolutely right or wrong responses, and meanwhile, this skill proficiency is influenced by the high-order latent trait of the examinee and the properties of the skill [De La Torre and Douglas, 2004]. However, for a subjective problemwhich has a partially correct response, the above "absolutely" modelling on skill proficiency cannot fit well as shown in the experiments. Thus, to address this issue, we adopt fuzzy systems [Jantzen, 2013; Chrysafiadi and Virvou, 2014] in cognitive modelling and redefine the proficiency of an examinee on a skill by the following assumption:

Assumption 1 The proficiency of an examinee on a skill is the grade of membership of the examinee in a fuzzy set which is related to the skill.

Here, we assume that a skill k is related to a fuzzy set (J, µk), where J is the set of examinees and µk : J → [0, 1] is the membership function. Then, for each j ∈ J, we define the proficiency of examinee j on skill k, αjk, as the grade of membership of j in (J, µk), µk(j). Thus, if examinee j masters skill k to some extent then the element j is a fuzzy member of the fuzzy set, i.e. 0 ≤ αjk = µk(j) ≤ 1. In this way, we can fuzzify the proficiency of an examinee on a skill into a continuous variable with a value in [0,1]. For instance, as shown in Figure 4(a), examinee j is fully included, fully excluded and partially included by the fuzzy sets which is related to Skill 1, Skill 2 and Skill 3, respectively. It also means, examinee j has completely mastered, completely nonmastered and partially mastered Skill 1, Skill 2 and Skill 3, respectively. Formally, following an IRT-like high-order logistic model, αjk and µk(j) is defined as:

The implication of this definition is that the proficiency of an examinee on a specific skill (αjk) depends on the difference between the examinee's high-order trait (θj) and the properties of the skill: difficulty (bjk) and discrimination (ajk) of skill k for examinee j. Here, the coefficient 1.7 is an empirical scaling constant in logistic cognitive models [Hulin et al., 1983; Camilli, 1994]. Generally, the major difference between this definition and the traditional models [De La Torre and Douglas, 2004] is that we redefine the skill proficiency from binary variables to the probabilistic ones.

3.2 Fuzzifying ProblemMastery

Based on the fuzzified skill proficiency in Section 3.1, we can further fuzzify the problem mastery of examinees (i.e. the probability of being able to solve the problem). CDMs assume that problemmastery is a result of interaction of examinees' proficiency on required skills of this problem [Pardos et al., 2008]. Thus, we will first discuss the skills' interaction on objective and subjective problems, then introduce a method to model the two kinds of interactions according to fuzzy logic and fuzzify the mastery of an examinee on a problem.

The skill's interaction on problems can be mainly categorized into conjunctive and compensatory [Pardos et al., 2008]. Conjunctive means that an examinee must master all the required skills to solve a problem, while compensatory means that an examinee is possible to solve a problem as long as she masters any skill required by the problem. As for examinations, an objective problem has a unique standard answer and cannot be answered correctly unless the examinee masters all the required skill without any omission. Thus the skill's interaction on objective problems is usually assumed to be conjunctive [Pardos et al., 2008]. In contrast, a subjective problem is a free-response one and the examinees can write not only the final answers but also the solving process including writing equations, deducing, calculating and so on (a toy example is shown in the right part of Figure 1). That is, given a subjective problem and the required skills, the more skills an examinee masters, the higher score she will get on this problem. Therefore in this research we assume that the skill's interaction on subjective problems is compensatory, the effectiveness of which can be demonstrated experimentally. To sum up, we propose an assumption about skills' interaction on the problems as the following:

Assumption 2 The skills' interaction on objective (or subjective) problems is conjunctive (or compensatory).

Now, we model these two kinds of interactions in a fuzzy way to fuzzify the problem mastery. Specifically, given the set of examinees J, suppose we have a problem i requiring Skill 1, Skill 2 and Skill 3 with their fuzzy sets (J, µ1) , (J, µ2) and (J, µ3). As shown in Figure 4(b), examinee j1 is fully included by the fuzzy sets related to Skill 1, Skill 2 and Skill 3 and examinee j2 is not included by the fuzzy set related to Skill 3, which means that if we adopt the conjunctive assumption then examinee j1 has mastered problem i because she has mastered all the skills needed by problem i and examinee j2 has not; as shown in Figure 4(c), examinee j1 is fully included by the fuzzy sets related to Skill 1 and Skill 2 and examinee j2 is not included by any of the fuzzy set. Here, if we adopt the compensatory assumption then examinee j1 is possible to master problem i because she has mastered at least one of the skills needed by problem i. In summary, the set of the examinees who master all and any of the skills required for problem i is the intersection and union of the fuzzy sets related to the skills, respectively. Thus, we propose an assumption to fuzzify the problem mastery as follows:

Assumption 3 If the skills' interactions between each other on a problem is conjunctive (or compensatory), the mastery of an examinee on this problem is the grade of membership of this examinee in the intersection (or union) set of the fuzzy sets related to the skills required by the problem.

Formally, given a Q-matrix with K skills, the mastery of an examinee j on a problem i, ηji, is defined as the following equation under conjunctive assumption:

And ηji for subjective problems is defined as the following equation under compensatory assumption:

Here, qik from the Q-matrix indicates whether problem i requires skill k (1 means required and 0 means non-required). Without loss of generality, we adopt the simplest fuzzy intersection and union operation 1 (standard fuzzy intersection and union [GEORGE J and Bo, 2008]) as follows:

In this way, we could get the mastery of each examinee on every problem (ηji), either objective or subjective problem.

3.3 Modelling Generation of Examinees' Scores

With the problem mastery defined in Section 3.2, we can now determine examinees' scores on problems. Specifically, we take two exceptions, slip and guess [d Baker et al., 2008], into account and adopt two distributions to simulate the generation of scores of objective and subjective problems, respectively.

In a real-world examination, the score of an examinee on a problem not only depends on the examinee's problem mastery. For instance, an examinee who is unable to solve the problem can get a correct response by guessing an answer (e.g. choosing "C" as the final answer somehow); Meanwhile, she who is able to do it right may get a wrong response as a consequence of carelessness (e.g. a slip of a pen) [Embretson, 1985]. Here, we assume that each problem has its own slip and guess factors and consider these two exceptions to model the generation of examinees' scores.

Meanwhile, we handle the different score patterns of objective and subjective problems. With either right or wrong response, the score of an examinee on a objective problem can be coded to a binary variable with a value in {0,1}. Thus, we adopt a Bernoulli distribution to model the scores of examinees on objective problems. Considering different score scales of subjective problems, we normalize the scores on a subjective problem by dividing the full score of the problem into a continuous variable with a value in [0,1]. Then we assume that the score of examinees on subjective problems follow a Gaussian distribution, which is widely used in the literature [Mnih and Salakhutdinov, 2007].

1 The effects of different kinds of fuzzy set operation are out the scope of this paper.

Formally, combining the problem mastery of the examinees and the exceptions of slip and guess, we simulate the generation of the scores as follows:

Equations (5) and (6) stand for objective problems and subjective problems, respectively. Here, Rji denotes the score (normalized score for subjective problem) of the examinee j on problem i, ηji is the mastery of an examinee j on the problem i, si and gi denote the slip and guess factors of problem i, and σ2 is the variance of the normalized score of an examinee on a subjective problem. Thus, (1-si)ηji means this examinee masters the problem and answers it successfully (i.e. without carelessness), while gi(1-ηji) represents the examinee guesses a right response without mastering. That is, these are the two ways for an examinee to give a correct response.

Summary. To better understand our proposed FuzzyCDF method, we represent it using a graphic model as shown in Figure 5. Here, what we can observe are the score matrix R with M examinees, No objective and Ns subjective problems and the Q-matrix with K skills (if problem i requires skill k, then qik = 1). An examinee j is related to skill proficiency αjk, k = 1, 2, · · · ,K, which depends on high-order latent trait θj and skill parameters ajk, bjk, k = 1, 2, · · · ,K. A problem mastery ηji is determined by required skill proficiency αjk|qik = 1 and a problem score Rji is influenced by ηji and problem parameters si, gi. It is noted that σ is only used for modelling the generation of the score of an examinee on a subjective problem.

3.4 MCMC Training Algorithm

In this section, we will introduce an effective training algorithm using MCMC method for the proposed FuzzyCDF model, i.e. to infer the unshaded variables in Figure 5. Specifically, we assume that the prior distributions of the parameters in FuzzyCDF as the following:

where Beta(v, w, min, max) is a four-parameter Beta distribution which has two shape parameters v and w and is supported on the range [min,max]. The functional forms of the prior distributions are chosen out of convenience, and the associated hyperparameters are selected to be reasonably vague within the range of realistic parameters. Then, the joint posterior distribution of θ, a, b, s, g and σ2 given the score matrix R is as follows:

where L is the joint likelihood function of FuzzyCDF:

where Lo and Ls denote the joint likelihood functions of objective and subjective problems respectively, and are defined as the following equations:

where, Xji = (1 - si)ηji + gi(1 - ηji). Then, the full conditional distributions of the parameters given the data and the rest of parameters as the following:

Finally, we propose a Metropolis-Hastings (M-H) based MCMC algorithm [Hastings, 1970] for parameter estimation by Algorithm 1. To be specific, we first randomize all the parameters as the initial values. Then, using observed score matrix R and the Q-matrix, we compute the full conditional probability of skill discrimination a, skill difficulty b, examinee latent trait θ, problem slip factor s and guess factor g and the variance of normalized score of subjective problem σ2 . Next, the acceptance probability of samples can also be calculated based on M-H algorithm. In this way, we could estimate the parameters with the MCMC formed through sampling.

Predicting Examinee Performance. After the training stage, we can easily obtain fuzzified proficiency of the examinees on each skill based on estimated latent trait, skill discrimination and difficulty (Equation (1)). Then, we can further compute fuzzified mastery of the examinees on each problem according to Q-matrix and problem type (objective or subjective) by Equations (2) and (3). Combining the estimated slip and guess factors of each problem, we can now complete the PEP task, i.e. predict examinees' performance (i.e. score) on each problem, by Equations (5) and (6). Note that, the output of Equations (5) is continuous, and we can discretize them for predicting on objective problems.

Algorithm 1 Sampling algorithm for FuzzyCDF.

Input: score matrix R, experts' knowledge Q Output: samples of θ,a,b,s,g,σ2

• 1: Initialize θ0, a0, b0, s0, g0, σ2 0 with random values
• 2: for t = 1, 2, · · · , T do
• 3: Draw at ∼ U(at-1 - δa, at-1 - δa), bt ∼ U(bt-1 - δb, bt-1 - δb), and accept at, bt with the probability: min{1, L(st-1,gt-1,σ2 t-1 ,θt-1,at ,bt)P (at)P (bt) L(st-1,gt-1,σ2 t-1 ,θt-1,at-1,bt-1)P (at-1)P (bt-1) }.
• 4: Draw θt ∼ U(θt-1 - δθ, θt-1 + δθ), and accept θt with the probability:

• 5: Draw st ∼ U(st-1 - δs, st-1 + δs), gt ∼ U(gt-1 - δg, gt-1 + δg), and accept st, gt with the probability: min{1, L(st ,gt ,σ2 t-1 ,θt ,at ,bt)P (st)P (gt) L(st-1,gt-1,σ2 t-1 ,θt ,at ,bt)P (st-1)P (gt-1) }.
• 6: Draw σ2 t ∼ U(σ2 t-1 - δσ, σ2 t-1 + δσ), and accept σ2 t with the probability:

• 7: if convergence criterion meets then
• 8: return
• 9: end if
• 10: end for
• 11: return

4 Experiments

We first compare the performance of FuzzyCDF against the baseline approaches on the PEP task; then, we conduct a case study to assess the interpretation of the cognitive diagnosis.

Experimental Setup. The public dataset in our experiment comprises of scores of middle school students on fraction subtraction objective problems [Tatsuoka, 1984; DeCarlo, 2010]. The two private datasets 2 are collected from two final mathematical exams for high school students including both objective and subjective problems. We denote the three datasets as FrcSub, Math1 and Math2. Each of the datasets is represented by a score matrix and a given Q-matrix by education experts. The brief summary of these datasets is shown in Table 2. Further, Figure 6 shows the preview of these three datasets, where each column for each subfigure stands for a problem and each row above and below represents an examinee and a skill, respectively. Specifically, the three subfigures in the above show FrcSub's score matrix with only dichotomous scores, Math1's and Math2's normalized score matrix with both dichotomous and polytomous scores; The three subfigures in the below show the three Q-matrices.

2 The two private datasets we use have been publicly available on http://staff.ustc.edu.cn/%7Eqiliuql/data/math2015.rar.

For the prior distributions of parameters in FuzzyCDF, we set the hyperparameters as follows:

In this experiments, we set the number of iterations of Algorithm 1 to 5,000 and estimate the parameters based on the last 2,500 samples to guarantee the convergency of the Markov chain. Both our FuzzyCDF and other baseline approaches are implemented by using Matlab on a Core i5 3.1Ghz machine with Windows 7 and 4 GB memory.

PEP Task. To demonstrate the effectiveness of FuzzyCDF, we conduct experiments on PEP task, i.e. predicting the scores from the examinees on each problem. To observe how the methods behave at different sparsity levels, we construct different sizes of training sets, with 20%, 40%, 60% and 80% of score data of each examinee, and the rest for testing, respectively. We use root mean square error (RMSE) and mean absolute error (MAE) as the evaluation metrics. Then, we consider baseline approaches as follows:

• IRT:[Rasch, 1961; Birnbaum, 1968] a cognitive diagnosis method modelling examinees' latent trait and the parameters of problems like difficulty and discrimination.
• DINA:[Junker and Sijtsma, 2001] a cognitive diagnosis method modelling examinees' skill proficiency and the slip and guess factors of problems with a Q-matrix.
• PMF:[Mnih and Salakhutdinov, 2007] probabilistic matrix factorization is a latent factor model projecting examinees and problems into a low-dimensional space.
• NMF:[Lee and Seung, 2001] non-negative matrix factorization is a latent non-negative factor model and can be viewed as a topic model.

For the purpose of comparison, we record the best performance of each algorithm by tuning their parameters, and Figure 7 shows the PEP results of our FuzzyCDF and baseline approaches on three datasets. Here, we consider two implementations of the matrix factorization methods, PMF and NMF. That is, PMF-5D and PMF-10D (NMF-5D and NMF10D) represent the PMF (NMF) with 5 and 10 latent factors, respectively. Thus, there are totally seven results in each split.

From this figure, we observe that, over all the datasets, FuzzyCDF performs the best. Specifically, by combining educational hypotheses it beats PMF and NMF, and by quantitatively analysing examinees from a fuzzy viewpoint, it beats IRT and DINA. More importantly, with the increasing of the sparsity of the training data (training data ratio declines from 80% to 20%), the superiority of our FuzzyCDF method becomes more and more significant. For instance, when the training data is 20% and under the metric of MAE, the improvement of FuzzyCDF compared to the best baseline method IRT could reach 19%, 10%, and 8% on each data, respectively. In summary, FuzzyCDF captures the characteristics of examinees more precisely and it is also more suitable for the real-world scenarios, where the data is sparse and the examinees/problems are cold-start.

Fixing the training data ratio equal to 80%, Figure 8(a), 8(b) and 8(c) show the prediction performance for each specific problem in these three datasets. For simplicity, we only give the results of FuzzyCDF and four baselines that have better performance: PMF-5D, DINA, IRT and NMF-5D. From each subfigure, we can observe that FuzzyCDF outperforms almost all the baselines on all the problems. Specifically, in Math1 and Math2 datasets (the last five and four problems are subjective problems, respectively), FuzzyCDF method can obtain the best performance for both objective and subjective problems, which in turn proves the reasonability of our Assumption 2 about skill interaction on objective and subjective problems. However, matrix factorization methods (PMF and NMF) cannot fit the scores of objective problems very well and the normal psychometric methods (IRT and DINA) are unsuitable for subjective problems. Moreover, Table 3 shows the runtime of each method under this setting.

Case Study. Here, we present an example of the diagnosis results of an examinee on each skill in Dataset FrcSub using DINA and FuzzyCDF, and the results are shown in Figure 9. We can observe that both FuzzyCDF and DINA can obtain interpretatively meaningful diagnosis results with the well-designed Q-matrix. However, DINA can only distinguish whether an examinee masters a skill (1 or 0) while our FuzzyCDF can tell the extent to which the examinee masters a skill. Thus, based on our diagnosis results, an examinee can find out the true strength and shortcomings of hers at present, and furthermore, educators or tutoring systems can give her targeted remedy plans for improvement. We should note that traditional IRT and matrix factorization methods describe an examinee with latent variables, which cannot provide intuitive and interpretative results for each examinee.

Discussion. From the experimental results of PEP task, we can observe that FuzzyCDF outperforms the baselines on both objective and subjective problems. The case study demonstrated that FuzzyCDF could obtain interpretative cognitive analysis results for examinees, which can be used for composing a detailed and human-readable diagnosis report.

On the other hand, there is still some room for improvement. First, FuzzyCDF currently confronts the problem of high computational complexity, and we will try to design an efficient sampling algorithm in the future. Second, we can test more fuzzy set operation functions. Third, there may be some other problem types beyond objective and subjective problems that should be considered for cognitive modelling.

5 Conclusion

In this paper, we designed a fuzzy cognitive diagnosis framework, FuzzyCDF, to explore the scores of both objective and subjective problems for cognitive modelling. Specifically, we first fuzzified the skill proficiency of examinees based on a fuzzy set assumption, then fuzzified the problem mastery by mapping conjunctive and compensatory interactions into the fuzzy set operations, and next modelled the generation of the two kinds of problems with different distributions by considering slip and guess factors. Finally, extensive experimental results demonstrated that FuzzyCDF could quantitatively and interpretatively analyse the characteristics of each examinee and thus obtained a better performance for the PEP task. We hope this work could lead to more future studies.

Acknowledgements

This research was partially supported by grants from the National Science Foundation for Distinguished Young Scholars of China (Grant No. 61325010), the Natural Science Foundation of China (Grant No. 61403358) and the Fundamental Research Funds for the Central Universities of China (Grant No. WK2350000001). Qi Liu gratefully acknowledges the support of the Youth Innovation Promotion Association, CAS.