DGCD: An Adaptive Denoising GNN for Group-level Cognitive Diagnosis

1 Introduction

Cognitive diagnosis endeavors to assess students' mastery of knowledge concepts by predictively analyzing their accuracy in answering exercises, stemming from their interactions with exercises. With the advancements in deep learning [Qin et al., 2024; Lin et al., 2017; Wang et al., 2021; Sun et al., 2019], cognitive diagnostics research has received increasing attention. Abundant researches have focused on modeling the abilities of individual students and diagnosing their understanding of various concepts [De La Torre, 2011; Wu et al., 2015] through methods such as Item Response Theory (IRT) [Embretson and Reise, 2013], multidimensional IRT (MIRT) [Ackerman et al., 2003], matrix factorization (MF) [Ackerman et al., 2003], and neural cognitive diagnosis (NeuralCD) frameworks [Wang et al., 2020a]. However, students often participate in collaborative learning activities, including tests, exams, and group assignments, which offer additional benefits[Hammar Chiriac, 2014]. Therefore, evaluating group abilities is of paramount importance.

Currently, insufficient research has been undertaken in the realm of group-level cognitive diagnosis. Figure 1(a) illustrates the process of group-level cognitive diagnosis.

Traditional approaches consider the group as a unified entity and then apply individual cognitive diagnosis directly to the group, overlooking the intricate relationship between the group and the individual. The other approaches derive group competence by aggregating the competencies of individuals. For instance, the latest group diagnosis framework (MGCD) [Huang et al., 2021] utilizes deep neural networks to handle the group-level cognitive diagnosis task and attention mechanisms to learn group representation. However, previous methods failed to consider higher-order connectivity among groups, students, and exercises, resulting in suboptimal outcomes. To address this limitation, we construct the group-student-exercise graph to establish higher-order connectivity among groups, students, and exercises, and employ GNN to learn higher-order representations of groups.

In the real world, the data about student-exercise interactions exhibits noise [Zhang et al., 2023; Ma et al., 2024]. Previous models neglected to consider the impact of this noise on the overall performance of the model, rendering them less robust. As illustrated in Figure1(b), the interaction data involving student s2 is noisy. The RMSE for MGCD-D is less than that of MGCD, indicating the adverse influence of noise on the model's effectiveness. Meanwhile, although GNN can model higher-order relationships between groups, students, and exercises, it is susceptible to interaction noise due to its reliance on message passing mechanisms [Wu et al., 2021]. As evident in Figure1(b), MGCD exhibits superior performance compared to GNN in the presence of noisy data. To enhance the higher-order representation of the group, it is imperative to mitigate the impact of noisy interactions on GNN. Furthermore, the existence of relational imbalance in the GSE graph introduces bias in different relation information during message passing, further undermining GNN's effectiveness. For instance, in the message passing, a student node has a group neighbor node and multiple exercise neighbor nodes. The relationship between student and group differs from that between student and exercise.

To this end, in this paper, we propose an adaptive denoising GNN for group-level cognitive diagnosis, namely DGCD. Specifically, we first construct the GSE graph and employ GNN to model higher-order connectivity among groups, students, and exercises. Subsequently, to address the influence of noisy interactions, we introduce an adaptive denoising method to drop the noisy edges based on the reliability of the student-exercise edges, thus alleviating the impact of the noisy edges on the learning of subsequent representations. Additionally, to tackle the bias introduced by the unbalanced relations in the GSE graph during the message passing, we propose an entropy-weighted balance module to reweight the information of unbalanced relations, balancing the information aggregation across various relations. Finally, Extensive experiments conducted on four real-world datasets demonstrate the effectiveness of our DGCD.

2 RelatedWork

2.1 Cognitive Diagnosis

In the domain of educational psychology, diverse cognitive diagnosis models (CDMs) have been formulated to provide comprehensive insights into students' cognitive abilities [DiBello et al., 2006; Li et al., 2020; Yang et al., 2024; Zhang et al., 2024]. Previous research mainly focuses on the individual cognitive diagnosis, with DINA [De La Torre, 2011; Junker and Sijtsma, 2001] and IRT [Embretson and Reise, 2013]standing out as notable contributions that profile students based on inherent attributes. Additionally, MIRT [Reckase, 2009], an extension of IRT, enables the representation of students' cognitive abilities through multidimensional latent traits. Moreover, certain methodologies utilize matrix factorization (MF) to reveal underlying eigenvectors of students and exercises via score matrix decomposition [Toscher and Jahrer, 2010; Thai-Nghe and Schmidt-Thieme, 2015]. Diverging from traditional approaches representing student-exercise interactions using linear functions, the neural cognitive diagnostic framework (NeuralCD) [Wang et al., 2020a] employs neural networks to capture complex interactions between students and exercises. ReliCD [Zhang et al., 2023] proposes a Bayesian method to explicitly estimate the state uncertainty of different concepts of knowledge for students and enable the confidence quantification of diagnostic feedback.

In recent years, group-level cognitive diagnosis has gained significant prominence across diverse domains [Liu et al., 2023; Huang et al., 2021; Yu et al., 2024]. The initial strategy involves extending traditional CDMs and adapting them to assess group abilities. Traditional group-level cognitive diagnostic models primarily build upon Group IRT (GIRT) [Reise et al., 2006; Mislevy, 1983; Birenbaum et al., 2004], an extension of IRT tailored for group-level analysis. The second approach focuses on initially modeling individual abilities and subsequently assumes that a group's collective abilities are equivalent to the average proficiency of its members [Agrawal et al., 2014; Liu et al., 2016]. The recently introduced approach MGCD [Huang et al., 2021] utilizes an attention mechanism to calculate individual student weights and aggregates student ability states into group ability states. Nevertheless, existing methods fail to consider higher-order connectivity among groups, students, and exercises, while disregarding noise in interaction data. In this paper, we construct the GSE graph to establish higher-order connectivity between groups, students, and exercises, and we propose an adaptive denoising method to alleviate the impact of noisy interactions on model performance.

2.2 GNN-based Representation Learning

GNN-based representation learning methods have gained increased attention in recent years. In contrast to conventional representation learning methods, GNN-based approaches leverage graph structures to model node representations. Current GNN-based representation learning methods fall into two main categories: homomorphic graph-based [Kipf and Welling, 2016; Hamilton et al., 2017; Brody et al., 2021] and heteromorphic graph-based [Schlichtkrull et al., 2018; Wang et al., 2019; Hu et al., 2020]. Homogeneous graph-based representation learning methods do not account for the distinctions among node and edge types during the aggregation of information from neighboring nodes. Heterogeneous graphbased representation learning methods [Yang et al., 2021;

Yang et al., 2023; Liu et al., 2021; Wu et al., 2022] take into account various node types and relationship types during the aggregation of neighbor information, allowing for the capture of diverse and complex relationships between nodes.

Although GNN-based representation learning [Qin et al., 2023; Shen et al., 2021a] has demonstrated notable success, its neighbor aggregation scheme enlarges the impact of interactions on representation learning, rendering the process more susceptible to interaction noises. Furthermore, GNN may be influenced by relation unbalance. Several methods [Qiu et al., 2020; You et al., 2020; Wu et al., 2021; Ye et al., 2023] alleviate the impact of interaction noise through self-supervision, incorporating data augmentation and contrast learning to enhance node representation learning. However, these methods cannot be directly applied to the group-level cognitive diagnosis. In this paper, we introduce a novel denoising method to eliminate the noise edges in the GSE graph. Furthermore, to address the relational imbalance, we propose an information entropy-based approach that aims to balance the information of different relations in the message passing.

3 ProblemFormulation

We define the group-level cognitive diagnosis task and establish the group-student-exercise graph to utilize GNN within the broader context of group-level cognitive diagnosis.

Suppose there are m students S = {s1, s2, . . . , sm}, n group G = {g1, g2, . . . , gn}, h exercises E = {e1, e2, . . . , eh} and t knowledge concepts K = {k1, k2, . . . , kt}. The l-th group gl ∈ G consists of a set of students, i.e., group members with student indexesKl = {kl,1, kl,2, . . . , kl,|gl|}, where skl,∗ ∈ S and |gl| is the size of the group. In addition, there is an exercise-concept correlation matrix Q = {Qij}m×n. If exercise ei requires knowledge concept kj then Qij = 1. We employ a set of triplet (gi, ej, yij) to denote the response logs H, where gi ∈ G, ej ∈ E and yij is the correct rate that group gi got on exercise ej.

Group-Student-ExerciseGraph To establish higher-order connectivity between groups, students, and exercises, we construct the group-student-exercise (GSE) graph, denoted as Gr = (V, R). Here, V = {Vg, Vs , Ve } represents the set of three node types, with Vg ⊆ G, Vs ⊆ S, and Ve ⊆ E, where G, S, and E denote the sets of groups, students, and exercises, respectively. The set of relationships among the nodes is denoted as R = {r1, r2, r3}, where r1 indicates the inclusion relationship between groups and students, r2 and r3 denote the correct and incorrect relationships between students and exercises. Each graph comprises a group node, all student nodes within the group, and the exercise nodes the group has interacted with. We specify that edges between nodes are categorized into different types based on distinct relationships.

ProblemDefinition. Given group-exercise response records H, student-exercise response records F (the student-exercise response records in F here are the student-exercise interaction records contained in the group-exercise interaction records H), group-student-exercise (GSE) graphs, and exercise-concept correlation matrix Q, our task aims to utilize the GSE graphs to conduct group-level cognitive diagnosis and obtain more accurate and robust diagnostic results.

4 Methodology

This paper introduces an adaptive denoising graph neural network approach for group-level cognitive diagnosis. The following sections provide a concise overview of our method and a detailed presentation of individual modules.

Figure 2 illustrates our method comprising four primary modules: the representation learning module, the adaptive denoising module, the entropy-weighted balance module, and the group-level cognitive diagnosis module. The representation learning module captures higher-order connectivity among groups, students, and exercises. The adaptive denoising module seeks to alleviate the impact of noisy interactions. The entropy-weighted balance module aims to equalize information aggregation across various relationships and the group-level cognitive diagnosis module is employed to predict scores for interactions between the group and the exercises. The following will introduce their details.

4.1 Representation LearningModule

In practical settings, students within a group collaboratively interact with exercises, leading to intricate relationships among groups, students, and exercises. To capture intricate higher-order connectivity, we construct the group-studentexercise graph. Each group, along with its associated students and exercises they interact with collectively, serves as nodes, while the connections between them represent the edges. We then utilize a graph neural network to learn the higher-order representation of the group. First, this embedding layer takes the group one-hot vector xgi , student one-hot vector xsj and exercise one-hot vector xek as its inputs, and obtains the corresponding initial embedding vectors g0 i ∈ R1×d , s0j ∈ R1×d and e0k ∈ R1×d as follows:

where WG ∈ Rn×d , WS ∈ Rm×d , and WE ∈ Rh×d are trainable matrices, n, m, and h denote the number of groups, students, and exercises, while d denotes the embedding size.

Afterward, we categorize groups, students, and exercises as three distinct types of nodes, classifying the edges between them into three different types based on their respective relationships. Then, we aggregate information from neighboring nodes based on different node types and relationship edges, respectively. We denote the embedding of node i (it can denote either a group node, a student node, or an exercise node.) at the l-th GNN layer as oli . We formally define the message passing process from the (l - 1)-th layer to the l-th layer as:

where W l-1 o and W l-1 r ∈ Rd×d are trainable matrices, R denotes the set of relationships between groups, students, and exercises. Nr(i) denotes the set of neighboring nodes that have relationship r with node i. After stacking L GNN layers, the layer-aggregation mechanism is adapted to generate the final representation oLi .

4.2 Adaptive DenoisingModule

In the GSE graph, the relationship between groups and students is fixed, resulting in no noise in the group-student edge. However, noise is present in the edges between students and exercises. This noise is amplified during the message passing of the GNN, subsequently affecting the learning of the representation. To mitigate the impact of noise on representation learning, we propose an adaptive denoising strategy. This strategy aims to model the reliability of student-exercise edges and calculate the sampling probability p for each edge which denotes the retention probability of an edge.

Specially, we model that the reliability of edge vjk between student sj and exercise ek as a Gaussian distribution. To obtain the unique reliability distribution for edge vjk, We concatenate the vector representations of student sj and exercise ek and subsequently apply distinct transformation matrices to acquire the mean and variance parameters, respectively, i.e.,

where µjk and σjk ∈ R1 represent mean and variance parameters for the reliability of edge vjk, respectively. Wµ and Wσ ∈ R1×2d are different trainable matrices.

We proceed to sample the reliability distribution of edge vjk to obtain the reliability wjk of edge vjk. To facilitate gradient backpropagation during training, we employ the resampling technique as follows:

where random variable ϵ ∼ N(0, 1). A larger wjk indicates that the edge vjk is more important and should be retained, while a smaller wjk suggests that the edge vjk is more likely to be a noisy edge. Therefore, we drop the edge vjk based on wjk. To make the edge dropping procedure differentiable and enable an end-to-end optimization process, we apply a Relaxed Bernoulli distribution to obtain edge retention probability pjk through sampling. Specifically, the probability pjk is calculated by:

where ρ is a sigmoid activation function, t is temperature parameter of relaxation, and u ∼ Unif[0,1]. Then, we can obtain a new denoising view based on the retention probabilities of the edges. In the denoising view, the student-exercise correct answer adjacency matrix Adt and the student-exercise incorrect answer adjacency matrix Adf as follows:

where atjk , afjk are the elements of Adt and Adf , respectively.

4.3 Entropy-Weighted Balance Module

The number of neighboring nodes varies for nodes in the GSE Graph across different relationships. Unbalanced message passing can adversely affect the update of node representations. For instance, each student has only one group node neighbor and multiple exercise node neighbors, diminishing the influence of the group's representation in updating the student's representation during message passing. Hence, it is essential to balance information from different relations during the message passing.

Specifically, we employ the entropy weighting approach to balance information from different relationships by reweighting the information during the message passing. First, we normalize the aggregation of neighboring nodes for different relationships, and subsequently, we calculate the information entropy corresponding to different relationships using Eq.( 9). The specific process is as follows:

where hj,l st represents the normalized embedding regarding the exercise neighbors correctly answered by student sj in the l-th GNN layer. dst j is the sum of the j-th row of Adt ,and det k is the sum the k-th column of Adt . Similarly we can obtain hj,l sf , hj,l sg , hk,l et and hk,l ef . After normalization, we computed the information entropy for each relation:

where hi denotes the value of the i-th position in the vector h, and d is the dimension of the vector h. Hj,l st is the information entropy of hj,l st . Similarly we can obtain Hj,l sf , Hj,l sg , Hk,l et and Hk,l ef . A higher information entropy indicates a smaller amount of information. Therefore, to balance the information from different relationships, we define the weights of information corresponding to different relationships as:

Then we can get λj,l sf , λj,l sg , λk,l et and λk,l ef . Using the weights derived from information entropy, we redefine the message passing as follows:

where N s i denotes the set of students in group gi. N st j and N sf j represent the sets of exercises that student sj answered correctly and incorrectly, respectively. N et k and N ef k represent the sets of students that answered exercise ek correctly and incorrectly, respectively.

4.4 Group-Level Cognitive Diagnosis Module

After obtaining a group's representation gL i through the Llayer of GNNs, we can diagnose the cognitive state of the group through the diagnosis layer. Here, we utilize a multilayer perceptron (MLP) as a diagnostic layer. The first layer of the interaction layer is represented as:

where WB ∈ Rh×d is a trainable matrix, the edisc k is the differentiation of exercises, Qe is a matrix of knowledge concepts, and ϕ is an activate function.

Then we can obtain the correct rate of the group gi's answer on exercise ek through multiple linear layers:

Finally, we can obtain the prediction score ˆ yik via:

To optimize the model parameters and obtain the ability level of all groups, we maximize the likelihood p(Y ;Θ), which indicates the correct probability that all groups answer the exercises. We optimize the parameters with the evidence lower bound (ELBO):

where p(w) is the prior distribution for the reliability of the edge. q(w|X) is the posterior distribution we constructed for the edge. X is the set of representations, X = {S,E}. logp(Y |w) measures the likelihood that groups answer exercises correctly based on the reliability w of the student-exercise edge. We follow the variational autoencoder (VAE) [Kingma and Welling, 2014; Zha et al., 2023; Qin et al., 2022; Shen et al., 2021b] and calculate the two terms in Equation (19) separately:

where n represents the number of group, and Hs i , He i denote all students of group gi and exercises responded by group gi.

Finally, we define the final loss function as follows:

where Θ represents all the learnable parameters, and λkl, λreg are hyper-parameters controlling the effect strength of Lkl and the regularization.

5 Experiments

5.1 Dataset Description

We conduct the experiments on four public education benchmarks, including ASSIST12 [Feng et al., 2009], NIPS Edu [Wang et al., 2020b], SLPbio [Lu et al., 2021], and SLPmath [Lu et al., 2021]. All of these datasets contain group labels, with students from the same group belonging to the same class. Each dataset contains common interaction records for groups: group-exercise responses. Specifically, for each group-exercise response, we calculated the percentage of students in the group who answered the exercise correctly as the response outcome, and we used the student responses from the group-exercise response as the studentexercise responses. Detailed statistics of these datasets are presented in Table 1.

5.2 Experimental Setup

To verify the effectiveness of our DGCD, each dataset of group-exercise responses is divided randomly into two subsets: 80% for training and 20% for testing. As group-level cognitive diagnosis is a regression task to predict the correct rate of group responses to exercises, we selected root mean square error (RMSE) and the mean absolute error (MAE), to evaluate model performance.

In our DGCD, we set the dimension d of the vector to be the number of knowledge concepts. The number of the GNN layers in the representation learning module is set to 2. The number of diagnostic layers is set to 3. Additionally, the hyper-parameter λkl was searched in [1,1e-1,1e-2,1e-3,1e-4,1e-5,1e-6,1e-7]. And, t was optimized over the values [0.57,0.67,0.77,0.87].

5.3 Baseline Approaches

To validate the effectiveness of DGCD, we compare it against two baseline types. Initially, we assess its performance in contrast to cognitive diagnosis-based approaches as follows: IRT [Embretson and Reise, 2013], MIRT [Reckase, 2009], NeuralCDM [Wang et al., 2020a], MGCD [Huang et al., 2021] and Relicd [Zhang et al., 2023]. Then, we compared the graph-based approaches as follows: RGCN [Schlichtkrull et al., 2018], GATv2 [Brody et al., 2021], SGL [Wu et al., 2021] and RocSE [Ye et al., 2023].

5.4 Performance Comparison

To validate the effectiveness of DGCD, we conducted a comparative analysis utilizing the above four datasets. Table 2 presents the experimental results of our DGCD and the baseline models. We can find that our model consistently surpasses all baseline models by a substantial margin. Compared with IRT, MIRT, NCD ReliCD, and MGCD, our model shows better performance. This implies that the modeling of highorder connectivity among groups, students, and exercises using graph neural networks results in a more accurate group ability level. Our model exhibits noteworthy improvement across all four datasets compared to GAT, RGCN, SGL, and RocSE. This implies that our model outperforms in identifying noisy edges in student-exercise interactions, facilitated by the adaptive denoising module. Furthermore, in terms of the reliability of randomly deleting edges (for SGL) and calculating edges based on similarity (for RocSE), our approach demonstrates greater flexibility and adaptation to diagnostic tasks. Additionally, our entropy-weighted balance module ensures the equilibrium of information related to distinct relationships during aggregation. This mitigates bias in the aggregated information resulting from variations in the number of neighbors associated with different relationships.

5.5 Ablation Analysis

To study the impact of our key components, we introduced different variants of DGCD from two perspectives and analyzed their effects. "w/o AD" and "w/o EB" means removing the adaptive denoising module and the entropy-weighted balance module, respectively. As shown in Figure 3, our DGCD consistently achieves the best performance, demonstrating the contribution of each component. Meanwhile, it is evident that the adaptive denoising module significantly enhances model performance. This improvement can be attributed to the adaptive module's ability to eliminate noise edges, thereby mitigating the impact of noise on the model.

5.6 Effect of Hyperparameters

We study the impact of the hyperparameters t and λkl in Figure 4. t is a temperature parameter that affects the loose Bernoulli sampling process of the edge dropout. λkl is the weight of the Lkl that affects the modeling of student-exercise edge reliability. It has been observed that varying values of t yield different outcomes. Specifically, the model achieves optimal performance on ASSIT12 when t is equal to 0.67 and on SLPbio when t is equal to 0.77. Notably, different datasets are associated with distinct values of t. Similarly, optimal performance is observed for the model on ASSIST12 when λkl is set to 1e-3 and on SLPbio when λkl is set to 1e-6.

5.7 Case Study

To assess our model's capability against noise, we systematically introduced noisy data by replacing a certain percentage (5%, 10%) of population-exercise interactions in the training set, while keeping the test set unchanged. Figure 5(a) shows the result on the ASSIST12 dataset. We observed that the performance of MGCD, RocSE, and DGCD is influenced as the percentage of noise increases. In particular, DGCD consistently surpasses MGCD and RocSE, indicating its proficiency in mitigating the impact of noise interactions through adaptive denoising. Additionally, after introducing 10% noise to the ASSIST12 dataset, the performance of DGCD exceeds that of MGCD on the noise-free dataset. This further supports the superior performance and robustness of our DGCD in comparison to MGCD.

Meanwhile, we analyzed the retention probability of the edges learned by the adaptive denoising module on ASSIST12. As shown in Figure 5(b), student s547 answered incorrectly on exercises e7 and e22 but correctly on e20. Exercises e7, e20, and e22 share identical knowledge concepts and difficulty levels. Consequently, the interaction between s547 and e20 is deemed potentially noisy data, with our model learning a retention probability of 0.2956 for the edges connecting s547 and e20. This value is lower than the retention probability for the edges of s547 with e7 and e22. This observation demonstrates our model's ability to identify noisy edges and mitigate their impact on representation learning.

6 Conclusion

In this paper, we introduced a denoising graph neural network approach for group-level cognitive diagnosis, namely DGCD. The method considers higher-order connectivity among groups, students, and exercises, effectively alleviating the impact of noisy interactions on diagnostic results. To be specific, We initially constructed the group-studentexercise graph and subsequently utilized Graph Neural Networks to capture intricate higher-order connectivity among groups, students, and exercises, thereby obtaining advanced representations of the groups. Then, we introduced an adaptive denoising module to address the impact of noise interactions on diagnostic results. This module selectively removes noisy edges based on their reliability, resulting in a denoised view for subsequent representation learning. Furthermore, we designed an entropy-weighted balance module that utilizes information entropy reweighting to alleviate information bias resulting from the relational imbalance in the message passing. Finally, we conducted extensive experiments on four public education benchmarks and the experimental results demonstrate the effectiveness of our DGCD.

Acknowledgments

This work was supported in part by the National Natural Science Foundation of China under Grant 62107001, 62302009 and 62202006.