Adversarial Generation of Real-time Feedback with Neural Networks for Simulation-based Training

1 Introduction

Supporting the learning process through interactive feedback is important [Billings, 2012]. Appropriate and timely feedback intervention increases learning motivation, facilitates skill acquisition/retention, and reduces the uncertainty of how a student is performing [Davis, 2005]. With the development of virtual reality techniques, simulation-based training (SBT) has become an effective training platform in a range of applications including surgery [Wijewickrema et al., 2017; Ma et al., 2017], military training [Cosma and Stanic, 2011], and driver/pilot training [de Groot et al., 2011]. However, it still requires the presence of human experts so that real-time feedback can be provided during training to ensure that relevant skills are learned. This has been one of the obstacles to the spread of SBT systems [Lateef and others, 2010]. As such, it is important to automate the generation of real-time feedback in SBT.

Feedback generation is a classical problem in artificial intelligence (AI) systems. Intelligent tutoring systems are one such class of AI systems that aims to provide immediate instruction or feedback to learners [Billings, 2012]. Another example is autonomous driving systems that take the surrounding environment as input and output feedback to the car to adjust the steering wheel [Chen et al., 2015]. In reinforcement learning systems such as mobile robot navigation, the hardware or software agent learns its behaviour based on reward feedback from the environment [Sutton and Barto, 1998].

When compared to the above mentioned applications, SBT focuses more on educational gains such as the acquisition of proper skills [Steadman et al., 2006; Lateef and others, 2010]. As such, SBT requires a higher degree of "hands-on" experiential interaction. Figure 1 shows an example of such a SBT system: The University of Melbourne Virtual Reality Temporal Bone Surgical Simulator [Wijewickrema et al., 2016]. Rule-based feedback tutoring methods that work in domains such as algebra and physics are not flexible enough for SBT that requires a high level of user interaction and complex user behaviour. Autonomous driving systems and reinforcement learning systems mainly focus on the outcomes and mostly deal with cognitive tasks. Therefore, feedback generation methods in these systems are not directly transferable to SBT, especially for non-cognitive SBT scenarios. The aim of this paper is to develop an automated feedback generation method that can be used in SBT via supervised learning.

Feedback generation in SBT has three challenges. First, feedback should be generated in a timely manner as delayed feedback can lead to confusion or even cause fatal consequences in reality. An acceptable time-limit is 1 second after inappropriate action is detected. This is because feedback should be provided before the learner makes the next move [Rojas et al., 2014]. Second, feedback should be actionable instructions that can be followed by the trainee to improve skills or correct mistakes. This is because SBT tasks often consist of a series of delicate operations that require precise instructions. Third, feedback should be simple, referring to only a few aspects of the skill, as practically people cannot focus on many things at a time. Also, this reduces distractions to the trainee and decreases cognitive load, thus increasing the usefulness of the feedback [Sweller, 1988].

In this paper, we make the following contributions:

• We demonstrate how the adversarial technique can be

used to generate actionable knowledge or feedback with neural networks.

• We propose a novel neural network based feedback generation method that works with a L1 regularized loss to control the simplicity of feedback and a bounded update to ensure the generated feedback has practical meaning.
• We show that the proposed method has high effectiveness as well as high efficiency when compared to existing methods, making it possible to be used for real-time feedback generation in SBT.

The structure of the paper is as follows. Section 2 introduces related work in this field. Section 3 illustrates the realtime feedback process and the formal definition of the problem. The proposed feedback generation method is described in Section 4 and evaluated along with existing methods in Section 5. Section 6 concludes the paper.

2 RelatedWork

The simplest way to provide feedback in SBT is the rulebased approach. The "follow-me" approach (ghost drill) [Rhienmora et al., 2011] and the "step-by-step" approach [Wijewickrema et al., 2016] in surgical simulation are examples of this approach. However, it may be hard for a novice who has limited experience to follow a ghost drill at his own pace, and step-by-step feedback will not respond if the trainee does not follow the suggested paths.

Other works utilize artificial intelligence techniques to generate feedback that can change adaptively in response to the novice's abilities and skills. One example is the use of Dynamic Time Warping (DTW) to classify a time series of surgical data and support feedback provision in lumbar disk herniation surgery [Forestier et al., 2012]. However, this approach is less accurate at the beginning of a procedure when not much data is available. A supervised pattern mining algorithm was used in temporal bone surgery to identify significant behavioural patterns, classified as novice or expert, based on existing examples [Zhou et al., 2013b]. Here, when a novice pattern is detected during drilling, the closest expert pattern was delivered as feedback. However, it is very difficult to identify significant expert/novice patterns as novices and experts often share a large proportion of similar patterns.

A similar attempt used a prediction model to discriminate the expertise levels using random forests, and then generate feedback directly from the prediction model itself [Zhou et al., 2013a]. Here, the generated feedback was the optimal change that would change a novice level to an expert, based on votes of the random forest (Split Voting (SV)). Decision trees and random forests were used in other research areas as well to provide feedback. For example, a decision tree based method was used in customer relationship management to change disloyal customers to loyal ones [Yang et al., 2003; 2007]. Generating feedback from additive tree models such as random forest and gradient boosted trees is NP-hard, but the exact solution can be found by solving a transformed integer linear programming (ILP) problem [Cui et al., 2015].

In this paper, we propose a neural network based method to generate feedback using the adversarial technique. One intriguing property of neural networks is that the input can be changed by maximizing the prediction error so that it moves into a different class with high confidence [Szegedy et al., 2013]. This property has been used to generate adversarial examples from deep neural nets in image classification [Goodfellow et al., 2014]. An adversarial example is formed by applying small perturbations (imperceptible to the human eye) to the original image, such that the neural network misclassifies it with high confidence. Although the adversarial example has similarities to the feedback problem in that they both change the input to a different class, they are not synonymous. First, the adversarial example is formed by adding intentionally-designed noise that may result in states that do not exist or have practical meaning in a real-world dataset such as that of the feedback problem. Second, only a few changes to inputs are recommended for feedback, to make it useful to follow. These considerations lead to the formal problem definition below.

3 ProblemDefinition

In this section, we discuss the real-time feedback process, and show how skill/behaviour level is defined in SBT applications. We then formally define the feedback generation problem as applied to SBT.

3.1 Feedback Process Overview

Figure 2 illustrates the real-time feedback process in SBT. It operates in two steps: 1) offline training and 2) real-time feedback provision. In the offline stage, a feedback generation method is trained via supervised learning on labelled (novice/expert) skill samples. In real-time, when a trainee is practising on the simulator, novice skill will be captured and input into the feedback generation method to obtain feedback about where improvement is required. Technically, feedback is the suggested action that can improve novice skill to expert skill. Finally, the feedback will be delivered immediately to the trainee to improve behaviour. The focus of this paper is the feedback generation method as highlighted in grey.

3.2 Definition of User Skill

SBT often works with multivariate time series data. This is because a SBT task often consists of a series of steps over a period of time. The skill level is usually defined over a period of time, based on the values of certain skill metrics.

In general, skill metrics are: 1) motion-based, 2) timebased, 3) position-based, or 4) system settings. Motion-based metrics are often signals captured from haptic devices or sensors, for example, the speed and engine rpm in a driving simulator. Time-based metrics measure quantities such as reaction time in military training. Position-based metrics relate to the location of the current procedure and include quantities such as position coordinates and distance to landmarks. System settings refer to measures that affect the environment such as the magnification level in surgical simulation.

Definition 1. In simulation-based training, user skill is a feature vector summarizing user behaviour over an arbitrary period of time and annotated with class labels.

For example, consider a SBT environment for surgery. Here, the level of skill can be defined by the quality of a stroke, a continuous motion of the drill with no abrupt changes in direction. The period of time here, over which the user behaviour is summarised, is the time interval of the complete stroke. Metrics that define the quality of a stroke within this time interval include measures such as stroke length, speed, acceleration, duration, straightness, and force [Zhou et al., 2015]. We denote such a skill metric as a feature. A vector of feature values that defines user skill is an instance and is associated with a class label that denotes the skill level (expert or novice).

3.3 Feedback Generation Problem

Here, we define the feedback generation problem in an expert-novice perspective. We acknowledge that there may be more than 2 levels of expertise in some SBT applications. However, this can be easily addressed using the one-vs-rest approach.

In SBT, the feedback generation problem is to find the optimal action that can be taken to change a novice instance to an expert instance. Suppose the dataset D consists of m features, n instances defined by the feature vector x = (x1, ..., xm), and associated with a class label y ∈ {0, 1} (1: expert, 0: novice). H(x) is a prediction model learnt over D. The feedback generation problem can then be defined as follows.

Problem 1. Given a prediction model H(x) and a novice instance x, the problem is to find the optimal action A : x → xf that changes x to an instance xf under limited cost C such that xf has the highest probability of being in the expert class:

where, feedback A : x → xf involves one or multiple feature changes (increase/decrease). For example, A : (force = 0.2, duration = 0.3) → (force = 0.5, duration = 0.3) is "increase force to 0.5". C(x, xf) is the cost function measuring the potential cost of feedback A. In SBT, C (x, xf ) = ‖x - xf‖0, i.e., the number of changed features. The cost limit C is often a small integer such as 1 or 2 in SBT so as to meet the requirements discussed in Section 1.

4 ProposedMethod

To tackle the feedback generation problem, we propose the use of a neural network as the prediction model H(x) and introduce a method that directly generates feedback from the neural network. Let Hθ(x), with parameters/weights θ, be the neural network learnt with respect to the loss function J(θ, x, y), where x is the input or feature vector, y the class value associated with x, and y∗ the target class we want x to be in.

Recall that during the training process, the weights θ are updated so that the loss J(θ, x, y) is minimized. Therefore, if we keep θ fixed while the input x is updated so that J(θ, x, y) is maximized, we can get a new instance that has high confidence of being in the opposite class to its original class y [Szegedy et al., 2013]. To maximize J(θ, x, y), the input can be updated in the positive direction of the gradient following Equation (1), where ϵ is the learning rate.

This is the property that has been used to generate adversarial examples in image classification. Since adversarial examples require small perturbations in the input image, [Goodfellow et al., 2014] applied a sign function to linearize the loss function around the current value of θ , as shown in Equation (2). This method updates all the pixels of the input image once to get small perturbations.

Equation (1) works well for two-class tasks. However, for multi-class tasks, there are more than one opposite classes to y. This means using Equation (1) cannot guarantee the new instance has high confidence in the target class y∗ . The alternative is minimizing the loss J(θ, x, y∗ ) with respect to the specific target class y∗ as defined in Equation (3).

Although Equation (3) works for both two-class and multiclass tasks, it still has two potential problems that limit its use for feedback generation in SBT. First, it may change all input features, thus violating the constraint (e.g., ‖x-xf‖0 ≤ C) of Problem 1. Second, the update may explode the values of inputs to extremely small or large values, similar to the exploding gradient problem [Pascanu et al., 2012]. However, in practice, some features may have a certain value range outside of which the feature is meaningless.

To solve the first problem, we introduce a L1 regularization term to J(θ, x, y∗ ) to control the sparsity of the change so as to generate simple feedback. The new loss function is defined in Equation (4), where λ is the regularization parameter and x0 is the original input that needs to be changed.

To solve the second problem, we propose a bounded update approach (see Equations (5) and(6)) as an alternative to Equation (3). It incorporates the value range (defined by lower and upper bounds) of a feature into the update to ensure the updated feature value is still within range.

Sx is the sign of the partial derivative of J ′ (θ, x, y∗ ) with respect to x. The upper and lower bounds of x are a and b respectively, i.e. xi ∈ [ai, bi].

According to Equation (5), if the gradient ∇xi J ′ (θ, x, y∗ ) is positive (i.e., Sxi = 1), the update will become xi = xi - ϵ(xi - ai) which means xi moves a small step towards its lower bound ai. Similarly, a negative gradient gives xi = xi + ϵ(bi - xi), a move towards its upper bound b. No update will be applied if the gradient is zero, as in this case, Sxi = 0. This bounded update not only guarantees the correct update direction to minimize the loss, but also ensures that xi ∈ [ai, bi] always holds true (see Lemma 1 and proof).

Proof. The sign function Sxi only has 3 outputs: 1,0 or -1.

In this case, ai ≤ x′i = xi ≤ bi holds true.

In this case, x′i - ai = (1 - ϵ)(xi - ai) and x′i - bi = ϵ(ai - xi) + (xi - bi). Then, ai ≤ xi ≤ bi and 0 < ϵ ≪ 1 gives xi - ai ≥ 0, xi - bi ≤ 0 and 1 - ϵ > 0. Therefore, x′i - ai ≥ 0 and x′i - bi ≤ 0, that is, ai ≤ x′i ≤ bi.

And in this case, x′i - ai = ϵ(bi - xi) + (xi - ai) and x′i -bi = (1-ϵ)(xi-bi). Similarly, bi-xi ≥ 0, xi-ai ≥ 0 and 1 - ϵ > 0 gives x′i - ai ≥ 0 and x′i - bi ≤ 0, that is, ai ≤ x′i ≤ bi.

To conclude, in all cases, ai ≤ x′i ≤ bi holds true.

Equations (4), (5) and (6) give the definition of the proposed "neural network-based feedback (NNFB) method". NNFB takes a novice instance x as input, iteratively updates x (different from the one-time-update in generating adversarial examples) until it converges or meets the terminating criteria. Let the generated new instance be xf , the feedback is then the action A : x → xf (see the example in Problem 1).

When feedback is delivered, we need to ensure that it contains only C features. Although, the L1 regularization reduces the number of feature changes in general, in the absence of valid feedback with a low number of feature changes, it may still result in ones with higher numbers of feature changes. To overcome this issue, we suggest a post-selection process that iteratively tests all feature changes and select the ones with C or less changes that result in the best improvements.

The proposed method (NNFB) is easily generalizable to different SBT applications. First, the regularization term in J ′ (θ, x, y∗ ) can be adjusted accordingly for different applications (for example, L2 norm for applications that prefer small changes). Furthermore, NNFB offers flexible control over feature changes as the lower and upper bounds are adjustable for different features and even for different input instances. For example, we can set ai = bi = xi for a categorical feature that cannot be changed, such as prior simulation experience. This flexibility also benefits those applications that have discrete cost functions as some explicit cost limits can be easily incorporated into the bounds.

5 Experimental Validation

In this section, we first describe the two real-world datasets that were used in the experiments. Then, we briefly introduce the existing methods that the proposed method was compared against, followed by the experimental setup. Finally, we discuss the experiment results.

5.1 Datasets

We tested our method on two real-world SBT datasets. These datasets were collected from a temporal bone surgical simulator designed to train surgeons in ear-related surgeries. 7 expert and 12 novice surgeons performed two different surgeries that require very different surgical skills: cortical mastoidectomy - dataset 1 (D1) and posterior tympanotomy - dataset 2 (D2). Surgical skill is defined by 6 numeric skill metrics: stroke length, drill speed, acceleration, time elapsed, the straightness of the trajectory and drill force (see example in Section 3.2 for more details). The skill metrics were recorded by the simulator at a rate of approximately 15 Hz. Overall, D1 includes 60K skill instances (28K expert and 32K novice) while D2 includes 14K skill instances (9K expert and 5K novice). Both datasets were normalized to the range of [0, 1] using feature scaling as follows.

where, x and x′ are the original and scaled feature vectors respectively and xmin and xmax are the minimum and maximum feature values of x respectively.

5.2 ComparedMethods

Existing feedback generation methods compared with NNFB are as follows.

• Split Voting (SV): This is the random forest based stateof-the-art generation method for providing real-time feedback [Zhou et al., 2013a] as discussed in Section 2.
• Integer Linear Programming (ILP): This method solves the random forest feedback generation problem by transforming it to an integer linear programming problem [Cui et al., 2015] as discussed in Section 2.
• Random Iterative (RI): This method randomly selects a feature and iteratively selects the best value among the feature's value partitions in the random forest [Cui et al., 2015].
• Random Random (RR): This method randomly picks a feature from a novice instance and selects a random change to that feature as the suggested feedback.

5.3 Experimental Setup

For testing, we randomly chose one novice participant, then took all instances performed by this novice as the test set. The remaining instances were used for training. This simulates the real-world scenario of an unknown novice using the simulator. Parameter tuning was performed on the training data based on a 11-fold leave-one-novice-out cross-validation. In each fold, we took all instances from one randomly chosen novice as the validation set.

All methods were restricted to generate feedback with only one feature change, which is a typical requirement in SBT. This is a binary task as there are only 2 skill levels (expert and novice). All methods were then evaluated using 2 measures: 1) efficiency and 2) effectiveness. Overall, a good feedback generation method should have high effectiveness and high efficiency (low time-cost).

Efficiency was measured using the time-cost (in seconds) spent on average to generate feedback for one novice instance. The novice instance x will be changed to the target instance xf by the feedback A : x → xf (see Section 3.3). Thus, we use the quality of the target instances (i.e., {xf}) to measure the effectiveness of the feedback. As defined in Equation (8), effectiveness E is the percentage of expert instances in {xf}.

However, how instances are classified is dependent on the classifier used. To obtain more convincing results, we used 6 classifiers of different types for evaluation. The evaluation classifiers are: neural network (NN), random forest (RF), logistic regression (LR), SVM (RBF kernal), naive Bayes (NB) and KNN (K = 10). A generation method that scores consistently high levels of E across classifiers is deemed effective.

Experiments were carried out on a typical PC with 2.40GHz CPU. The ILP solver used for the ILP method was CPLEX 1 as suggested by the authors, and the neural network/random forest implementations we used were from scikit-learn. Default settings in scikit-learn were used for parameters not specifically mentioned here.

5.4 Parameter Tuning

Parameter tuning was performed on the training data with a 11-fold leave-one-novice-out cross-validation as mentioned above in Section 5.3. A two-layer neural network architecture was used for NNFB. For D1, a neural network with 250 hidden neurons was selected for NNFB while a random forest with 120 trees was selected for SV, RI and ILP. For D2, NNFB used a neural network with 120 hidden neurons while SV, RI and ILP used a random forest with 100 trees. These parameters were selected based on the turning point of the number of hidden neurons or the number of trees with respect to the mean squared error (MSE) of the neural network and random forest respectively.

In terms of the regularization parameter λ in NNFB, Figure 3 indicates that larger λ results in simple feedback with a fewer number of feature changes. When λ = 1, a feedback on average consists of only one feature change, but remains highly confident (H(xf) > 0.7) to change a novice instance to an expert instance. Therefore, we chose λ = 1 for NNFB. Since datasets have been normalised, the upper bounds for all features are 1 and the lower bounds are 0. Other settings for NNFB include Rectified Linear Unit (ReLU) activation function [Glorot et al., 2011], cross entropy loss and learning rate ϵ = 1 × 10-4 .

5.5 Results

We first demonstrate the overall performance considering both effectiveness and efficiency. Figure 4 illustrates the effectiveness of each method as evaluated using 6 different evaluation classifiers with respect to the time-cost (inverses to efficiency) for each dataset. As seen in the figure, the proposed method shows the desired performance of highest effectiveness at an acceptably low time-cost (within the realtime time-limit) when compared to the other methods. This proves that the adversarial technique can be used to generate effective and timely feedback for SBT. Note that the slightly higher variance of the NNFB method indicates the varying resistance of test classifiers to adversarial generation.

1 https://www-01.ibm.com/software/commerce/optimization/cplexoptimizer

Detailed results for effectiveness of the feedback generation methods across the 6 evaluation classifiers is shown in Table 1 for the datatsets D1 and D2. On both datasets, NNFB achieved comparable performance to ILP and outperformed all others methods across all classifiers. However, as shown in Table 2, ILP violates the real-time time-limit as discussed in Section 2, and as such, will not be suitable for most SBT applications. Although both RR and SV are more efficient than the proposed method in feedback generation, they show significantly lower levels of effectiveness when compared to NNFB. Thus, it can be concluded that in terms of both effectiveness and efficiency, the proposed method is the best suited for providing real-time feedback in SBT applications.

6 Conclusion and Discussion

In this paper, we introduced a technique for the adversarial generation of real-time feedback with neural networks for SBT. The proposed method (NNFB) applies a bounded adversarial update on the novice skill vector to generate an optimal expert skill vector in order to be used in the provision of feedback. To ensure that the suggested action is simple enough to practically undertake, we adopted L1 regularization to obtain feedback with a fewer number of feature changes. We explored theoretically the validity of NNFB, and showed empirically that it outperforms existing methods in providing effective real-time feedback.

Improving human performance in practice is a very challenging task. It involves many aspects of the learning process such as the learning environment, the task complexity, the knowledge level of the leaner, and the feedback intervention. In the future, we will deploy the proposed method to SBT environments and conduct user studies with human experts to further validate the method and investigate its effectiveness in teaching skills in practical applications.