Hierarchical Temporal Multi-Instance Learning for Video-based Student Learning Engagement Assessment

1 Introduction

The emergence of Massive Open Online Courses (MOOC) has attracted wide attention and great expectation from the broad education community. Despite the promising potential of the new education avenue, poor student retention rates are recognized as one of its major caveats. To combat the deficiency, dynamic assessment of individual students' engagement during their online learning activities can offer justin-time instructional intervention to improve retention rates and personalized learning outcomes, whose effectiveness has been abundantly supported in the education literature [Dewan et al., 2019; Zeng et al., 2020]. Due to the large number of students frequently seen in a MOOC environment, manually conducting such assessment is prohibitively expensive. Therefore, it is strongly desirable to develop automated technologies capable of assessing student learning engagement on the fly.

∗ Jiayao Ma and Xinbo Jiang contribute equally.

†Xueying Qin and Songhua Xu both are correspondence authors.

The research on the automatic assessment of learning engagement has the following problems: 1) Due to the timeconsuming and laborious work of annotating clip by clip, most previous methods [Osokin, 2019; Wang et al., 2019] only solve the problems of assessing the learning engagement of the whole video, although it makes more sense to assess each short clip. 2) A course in distance education usually lasts for tens of minutes or even an hour, making it difficult for assessment. How to obtain effective features that can represent the entire video and each short clip has become an urgent problem to be solved.

In response to the problems mentioned above, we construct a novel hierarchical multiple instance learning (MIL) engagement assessment model as shown in Fig. 1. Based on the features extracted from video frames, a hierarchical neural network that consists of a clip-level MIL module and a videolevel MIL module is trained with only video-level labels. The network can order short clip-level features with their engagements, which is achieved by constraining on a global loss for the top module and a local loss for the bottom module, to ensure that the trained network can accurately and reliably infer local clip-level labels for a collection of temporallysequenced clip-level instances under the supervision of global video-level labels. In summary, the main contribution of this paper are threefold as follows:

• We propose a novel temporal multiple instance learning framework which is trained with video-level labels only but can both assess the learning engagement of the fulllength video and its constituent short clips.
• We construct a hierarchical neural network which comprises a bottom module and a top module with their tailor-designed loss functions to learn the latent relationship in the frame-clip-video structure.
• A new data set about online course studies is collected and some ablation experiments and comparative experiments are conducted on it to prove that the proposed method can effectively assess the learning engagement of both the whole video and each video clip.

2 RelatedWork

Learning Engagement Assessment Based on Visual Clues When a learner is studying, the camera can easily capture the learner's images and advanced features associated with learning engagement can be effectively extract from these video frames. Some methods assess learning engagement based on RGB videos due to their easy accessibility. Studies [Whitehill and Movellan, 2008; Whitehill et al., 2014] have found that facial expressions and head poses are directly related to learning engagement. [Frank et al., 2016] proposes a framework for engagement assessment, which uses facial expressions, sound, body poses and movements acquired through multiple sensors, and then a SVM classifier is used to classify engagement. [Chang et al., 2018; Yang et al., 2018] uses features such as facial expressions, eye parameters, gestures and head poses extracted from image sequences to recognize the learner's cognitive state. Our work only uses body poses, head poses and face landmarks as the input features.

Multiple Instance Learning The multiple instance learning(MIL) method is a solution to the problem of weakly supervised learning, which trains models based on weakly labeled data. The MIL is widely used in medical image diagnosis and detection [Li et al., 2019; Campanella et al., 2019; Rony et al., 2019]. In this problem, the training set consists of bags labeled as 0 or 1, and each bag contains many instances without labels. Taking cancer prediction as an example here, each image with a cancer/non-cancer label forms a "bag", and each pixel or block of pixels that constitutes the image is called an "instance". We divide the MIL works into three categories: the global MIL, which can determine whether there is a disease [Campanella et al., 2019; Dov et al., 2019]; the local MIL, which can mark diseased cells on the image [Xu et al., 2019; Liang et al., 2018]; the global and local MIL, which can detect whether there is a disease and indicate the diseased cells at the same time [Wang et al., 2020; Huang and Chung, 2019]. Compared with the above problems, learning engagement assessment is the same that the bag composed of multiple instances has a label, but each instance has no label. But the difference is that there are structural and temporal correlations between the instances, and the label of the bag is not limited to 0 or 1 but any real number within the label range.

3 Method

3.1 ProblemFormulation

In this video-based learning engagement assessment task, we first divide the video evenly into N clips, and each clip Xi is composed of l video frames, so the whole video can be represented as its sample clip sequence X = {X1,X2, ...,XN}, where the video is a bag and its sample clip sequence is a instance sequence. Furthermore, each clip Xi can be represented as Xi = {xi,1, xi,2, ..., xi,l} and xi,j is the jth frame of the ith clip, where the clip is a bag and its frames are instances. Based on this frame-clip-video structure, we use the hierarchical multiple instance learning module from bottom to top to gradually use frame-level features to aggregate the clip-level descriptors and then aggregate the video-level descriptor based on them. We first use the model g to extract the feature sequence Fi = {fi,1, fi,2, ..., fi,l} from the frames of ith video clip, where fi,j = g (xi,j). Then the bottom module Db uses Fi as the input to learn the clip-level feature Ci. The clip-level feature sequence C = {C1, C2, ..., CN} is used as the input of top module Dt to learn the video-level feature V. The video-level feature V and the clip-level feature Ci are respectively sent to the regression function Reg to obtain the video-level learning engagement ˆ Y and the clip-level learning engagement sequence ˜ Y = {˜ y1, ˜ y2, ..., ˜ yN}. The process can be modeled as follows:

and

where ϕ, θ, µ are trainable parameters. The only supervision information we have is the video-level label Y , and Y is a decimal between 0 and 1. The framework is shown in Fig. 2.

3.2 Preprocessing

Down-sampling Learners' head poses and facial landmarks tend to change gradually and slowly during his/her online studies. Therefore, we down-sample every raw video by keeping one frame every few frames for more efficient computational processing. In our experiments, 3000 frames for each video are reserved for assessment.

Segmentation Because the feature has little effect on the assessment of learning engagement at the time far away from it, and the network is usually difficult to deal with lengthy feature sequences, we segment an input video into short video clips as the basic analysis objects. We set the length of a video clip to be 30 frames, denoted as l = 30 in all our experiments to yield an empirically optimized trade-off between computational efficiency and accuracy of the proposed approach. The number N of video clips extracted from each video is 100.

. .

Feature Extraction According to the previous works [Wang et al., 2019; Wu et al., 2019], body languages and facial expressions have a strong correlation with a learner's learning engagement. Therefore we use head pose features, body pose features and facial landmarks as the input to improve the accuracy and robustness of our model. For video frame xi,j, we use methods proposed by [Yang et al., 2019; Guo et al., 2019; Osokin, 2019] to extract head pose features ei,j, body pose features bi,j and facial landmarks mi,j respectively. Then for video clip Xi, we can get head pose sequence Ei = {ei,1, ei,2, ..., ei,l}, body pose sequence Bi = {bi,1, bi,2, ..., bi,l} and facial landmark sequence Mi = {mi,1, mi,2, ..., mi,l}.

3.3 Hierarchical Temporal Multiple Instance Learning Module (H-TMIL)

Based on the frame-clip-video structure only with video-level labels, we propose the hierarchical temporal multi-instance learning framework composed of a bottom temporal multiinstance learning module (B-TMIL) and a top temporal multiinstance learning module (T-TMIL) which respectively dedicated to learning the latent relationship between a clip and its constituent frames and that between a video and its constituent clips. Through this framework, we build the connection between the underlying video frames and the video-level label, and can implicitly learn the expression of the middleware, namely the clip-level features which are useful for assessing clip-level learning engagement.

Bottom Temporal Multiple Instance Learning (B-TMIL)

The B-TMIL module acts on sampled video frame sequence which composes a short video clip, where the frames are instances and the clip is the bag. We need to obtain the bag's valid representation in order to accurately obtain its label. However, unlike the traditional MIL instances, for a short-time frame sequence, there is a strong timing correlation between frames. Some methods [Wang et al., 2019; Huynh et al., 2019; Wu et al., 2019] use LSTM to capture the temporal association and use the hidden state of the last layer to express the sequence, which will cause the loss of early information. In response to this problem, we use a selfattention multi-instance module to act on all hidden layer features of the bidirectional LSTM built on instances, and adaptively obtain the representation of the bag through trainable parameters.The body pose feature is taken as an example.

First, we input the body pose feature sequence Bi into the Bi-LSTM to obtain the hidden state sequence Hi = {hi,1, hi,2, ..., hi,l}. Inspired by the paper [Ilse et al., 2018], the clip-level aggregated feature ˜Bi can be computed as a weighted sum of h⊤i,j after dimension reduction:

Here δ refers to the ReLU function and ˜W1 refers to the full connection operation which are used to reduce dimension. The weight ˜αi,j is computed by:

where ˜ scorei,j is calculated as followed:

where ˜W1, ˜W2, ˜W3, ˜W4 are the weight matrices, ⊙ is an element-wise multiplication, σ is the Sigmoid function and τ is the Tanh function. Among them, Tanh function is used to obtain the correlation between features, and Sigmoid function acts as a gate mechanism. Similar to the aggregation process of clip-level body pose feature ˜Bi, we use the B-TMIL module to extract clip-level head pose feature sequence ˜Ei and clip-level facial landmark feature sequence ˜Mi.

Top Temporal Multiple Instance Learning(T-TMIL)

Similar to B-TMIL, T-TMIL acts between video clips which can be thought as instances and the full-length video composed of clips is the bag. However, due to the long duration of each video clip, we believe that there is no longer a strong temporal relationship between video clips. Previous work [Wang et al., 2019; Wu et al., 2019] generally believes that the video-level feature can be expressed as the average of all clip-level features. These methods treat all video clips equally, which adversely affects the final assessment accuracy. In order to get more robust and flexible representation of video, we still apply MIL module on the basis of clip-level features. And the top-level module becomes more similar to the traditional multi-instance structure. Here we also use body pose features as an example to describe the process of the T-TMIL.

The dimension of the clip-level feature is reduced through the fully connected operation to generate more efficient embedding representation. We construct the weighted combination of video clips to represent a video which can be calculated as follows:

where ˆ scorei is calculated as followed:

where ˆW1, ˆW2, ˆW3, ˆW4 are the weight matrices of TTMIL. Similar to the above process, we use the T-TMIL module to aggregate video-level head pose feature sequence ˆE and video-level facial landmark feature sequence ˆM.

3.4 Feature Fusion

We use three types of features to assess learning engagement, while our hierarchical module processes each type of feature separately. In order to achieve complementary advantages between different features and increase judgment information, we propose a weighted feature fusion method as shown in Fig. 3 and then obtain the clip-level and video-level fusion features for the assessment. The weight matrix has the same size as the feature matrix composed of the three features and is composed of trainable parameters. Each column of the weight matrix is first normalized by Softmax function to obtain different proportions of different features in this dimension, and then weighted summed to obtain the weighted fusion feature.

3.5 Loss Function

In order to train and optimize the proposed framework, we design two loss functions: the global and the local supervision loss function. The global supervision loss function uses the mean square error method to measure the difference between video-level learning engagement result and the ground-truth. The local supervision loss function also uses the mean square error method but measures the difference between the mean of clip-level learning engagement result and the label of the video. Through this joint loss function and our hierarchical temporal multi-instance learning module, we can constrain both video-level embedding features and clip-level embedding features. The formula of loss is as follows:

where α and β are two hyper-parameters. Each component of the loss can be formulated as follows:

and

where ˜ yki is the clip-level assessment result of i-th clip of kth video, ˆ Yk is the video-level assessment result of k-th video, and Yk is the video-level label of k-th video.

4 Experiments

Due to the lack of annotated long video data sets publicly available for our study on learning engagement assessment, we collect and annotate one in-house data set about online course studies. Using the data set, we explore the performance of the proposed network against that of multiple stateof-the-art methods.

4.1 Online Course Studies (OCS) Data set

This data set carries 236 videos captured of 59 subjects while they are watching four healthcare topic courses. In order to obtain more diverse data for the learning engagement, we have carefully selected courses on four topics: cancer prevention, respiratory diseases, renal system and sleep problems. The length of each course is about 30 minutes, and all subjects are required to watch the entire content of each course. Except for a 14-year-old junior high school student, all subjects are undergraduate or graduate students, aged between 20 and 32, majoring in software engineering and digital media. During the learning period, the performances of all subjects are according to personal preferences and status. We will not place any restrictions on the postures and behaviors of subjects.

Each learning process of the subjects will be given a score from 0 to 1 to indicate the level of the learning engagement. The larger the score, the higher the engagement. The score of the engagement is based on three factors: the subject's self-scoring, the annotator's score and the testing score. First, we require each subject to self-score according to the actual learning state of himself. Secondly, the annotator is required to carefully watch each learning video and gives the score of the engagement. We invite three annotators to score the same video from 0 to 8 at the same time. If the difference between any two labels is less than 2, the final label is the average of the three labels. Otherwise, the three labelers negotiate and determine the final label. Thirdly, each subject is asked to finish all the questions about the courses. Regarding whether to use test scores, we will make a decision based on the relationship between score distribution and engagement. All three scores will be normalized between 0 and 1. Then the arithmetic average of the three scores is the score of the engagement. Table. 1 shows the distribution of our data set.

In addition, we have obtained the consent of the subjects and can use the collected data for the research of this project. And we will not disclose anyone's identity information, and everyone's eye position will be covered with a black mask to protect his personal privacy.

4.2 Implementation Details

Data Processing Based on the trained model, we first obtain head pose, body key points and facial key points from original videos. As shown in Fig. 2, the head pose is composed of yaw, roll and pitch angles and the coordinates of the center point of head (5D vector). Only the upper body of each subject appears in the video, thus the body pose features only need to be composed of the upper body key points(24D vector). The face landmarks are 23 key points (46D vector) extracted from the original face model with 98 key points. The models for feature extraction are light-weight and unstable, whose outputs are sometimes missing. Therefore we perform linear interpolation over N(N ≤ 15, assumed that only the missing data within 0.5s is caused by the instability of the model) adjacent vectors with missing values. Then we evenly extract 3000 frames from the final data as input to the model. In addition, in order to eliminate the influence of the subjects at different positions in the screen to the model, we use the coordinates of the neck as the center point for the body pose, and subtract the coordinates of the center point from the coordinates of the remaining points. And for face landmarks, we use the mean value of the coordinates of the key points on the nose as the center point and also subtract the coordinates AMLM

MR-GRU

Figure 4: Performance comparison among our and state-of-the-art peer methods.

of the center point from other points. In addition, we found that the number of different types of data in our data set is quite different. To make the distribution of our data set more uniform, we randomly add noise to the features and resample the videos whose type is in a small number in different ways. And we randomly select three quarters of videos in the OCS data set as our train set and the remaining quarter as test set.

Training Setting We establish the project based on Pytorch and train it on Tesla K80G GPU. Our training epoch is set to 200. We initialize the learning rate to 0.001, and use Adam with a momentum of 0.9 and a weight decay of 1e-4 as the optimizer. When the training epoch is 60/100/160, we multiply the learning rate by 0.1.

Speed Our assessment model's speed is about 0.014s per video clip and the feature extraction models whose speeds are at least 20 FPS can be done in parallel.

4.3 Ablation Studies

TMILModule We explore the effectiveness of TMILmodule by replacing the TMIL module used for aggregating frame-level and clip-level features with the methods proposed by the previous work [Wu et al., 2019], such as the last hidden state vector of LSTM and simply mean pooling. The experimental results are shown in the Table. 3. When processing frame-level features, only using the last hidden state vector of LSTM performs the worst, which shows that only the final result of LSTM cannot effectively aggregates enough and effective information; as for the result using mean pooling is worse than which using the B-TMIL module, we believe that it is due to the use of self-attention aggregation which enables the model to adaptively find more important features and solve the problem that LSTM can't deal with long-range dependence effectively. In addition, when using B-TMIL module only for frame-level features, it performs better than that when using T-TMIL module only for clip-level features. This can be easily explained as that model can be more effective only when the basic features, that is, clip-level features are accurately aggregated. Based on this, we obtain more valid video-level features. To further prove the effectiveness of our proposed module, we choose some key frames and visualize the results of the B-TMIL module acting on the underlying video frames on the test set. Table. 2 shows the visualization results of our module on the three features. Since the subjects' facial expression has not changed much, the attention weights are almost in the same order of magnitude. As for the head pose and body pose features, the weights of attention increase significantly (as shown in the bold part in the table) when subject's head and body pose changes. It can be seen that our attention module has effectively paid attention to the parts that are important for obtaining results.

Local and Global Supervision The assessment results of the learning engagement from the entire video can be obtained from a predictor that takes the characteristics of the entire video as input, or the result of each video clip can be obtained first and be averaged as the final result. In the model, in addition to using the loss function for the final video-level result to achieve global supervision, we also add local supervision to constrain the mean of the clip-level results. The experimental results are shown in the Table. 4. We obtain the best results with using global supervision and local supervision at the same time. The result of using global supervision only is not as good as the result of using local supervision only. This is consistent with our previous description that more effective advanced features can be obtained only when effective basic features are obtained.

4.4 Comparison with State-of-the-Art

To explore the advantages of the proposed approachs in the engagement assessment task, we conduct a series of experiments where benchmarked performance of the new approach is compared with that of several peer methods on the OCS, such as BOOT [Wang et al., 2019], ENS-MODEL [Huynh et al., 2019], MFMI [Wu et al., 2019], MR-GRU [Zhu et al., 2020] and AMLM [Wu et al., 2020]. In addition, we also optimize MR-GRU and AMLM with our TMIL module. Since there is already an attention mechanism in MR-GRU, we only add the T-TMIL module to it, while for AMLM, we add the complete H-TMIL module. All peer methods are reimplemented due to the lack of open source alternatives.

For the task of assessing video-level learning engagement, Table. 5 shows the viedo-level mean absolute error (MAE) and mean squared error (MSE) on OCS. The original MRGRU is better than the original AMLM, but the AMLMwith our complete H-TMIL module has better performance than the MR-GRU with only the T-TMIL module, which further proves the effectiveness of our proposed module and the importance of aggregating effective underlying features. Fig. 4 shows the absolute error (AE) between algorithmically generated results and their corresponding labels using boxplots.

One of our most important contributions is that the proposed network can assess the learning engagement of video clips relatively accurately only using video-level labels. In order to demonstrate the effectiveness of our model on the task of assessing clip-level results, we annotate some videos clip by clip. Fig. 5 (1), (2) and (3) respectively represent the cliplevel results and corresponding labels of the five models in three videos with different learning engagement scores. The result curve of our method and the label curve basically maintains the same trend which fully demonstrates that our model can effectively learn clip-level features only with video-level labels provided. Table. 6 further presents the performance of the proposed approach with that of other five peer methods and two models improved with our modules in terms of the mean absolute error (MAE), mean squared error (MSE). Our model achieves remarkable results except on the video (2) where the range of labels has a relatively large float. Although the MFMI model has some advantages in overall statistical results, it does not well reflect the drastic changes of the engagement, as shown in the pink curve of Fig. 5 (2). All these results demonstrate the superiority of our approach compared with all peer methods in assessing clip-level learning engagement.

5 Conclusion

To solve the problem of assessing the changes of learning engagement of students in the learning process with only videolevel labels, a novel hierarchical temporal multi-instance learning network is introduced to automatically assess both video-level and clip-level learning engagement from video streaming data in an end-to-end learning fashion composed of a bottom and a top module which can respectively learn the latent relationship between a video and its constituent clips and that between a clip and its constituent frames.

FutureWork Our futureworkmainly consists of two parts. 1) More effective structure. Unify feature extraction and assessment into an end-to-end model and introduce more advanced structures to get high-level semantic information more effectively. 2) Practicality. Build a system that can assess learning engagement online and prove the versatility of model in other scenarios, such as evaluating the user's interest of advertisements or products and the degree of fatigue in driving.

Acknowledgements

This work was supported by Zhejiang Lab under Grant(No.2020NB0AB02), NSFC of China under Grant(No.61907026) and Shandong Province Higher Educational Science and Technology Program under Grant(No.J18KA392).