EventDrop: Data Augmentation for Event-based Learning

1 Introduction

Event sensors, such as DVS event cameras [Patrick et al., 2008] and NeuTouch tactile sensor [Taunyazov et al., 2020], are bio-inspired devices that mimic the efficient event-driven communication mechanisms of the brain. Compared to conventional sensors (e.g., RGB cameras), which synchronously capture the scene at a fixed rate, event sensors asynchronously report the changes (called events) of the scene. For example, DVS cameras capture the changes in luminosity over time for each pixel independently rather than intensity images as RGB cameras do. Event sensors usually have the advantages of higher dynamic range, higher temporal resolution, lower time latency, and higher power efficiency [Gehrig et al., 2019]. These advantages have stimulated research into machine learning for event data. Unsurprisingly, deep learning, which performs extremely well on a variety of tasks, remains a competitive method for learning with event sensors.

∗Corresponding Author

A challenging problem in deep learning is over-fitting, which causes a model that exhibits excellent performance on training data to degrade dramatically when validated against new and unseen data. A simple solution to the overfitting problem is to significantly increase the amount of labeled data, which is theoretically feasible but may be costprohibitive in practice. The over-fitting problem is more severe in learning with event data since event datasets remain small relative to conventional datasets (e.g., ImageNet).

Data augmentation is a way to increase both the amount and diversity of data from existing data, which can improve the generalization ability of deep learning models. For images, common augmentation techniques include Translation, Rotating, Flipping, Cropping, Contrast, Sharpness, Shearing, etc [Cubuk et al., 2019]. Event data are fundamentally different from frame-like data (e.g., images), and hence we cannot directly use these augmentation techniques that are originally developed for frame-like data to augment asynchronous event data.

In this paper, we present EventDrop, a novel method to augment event data by dropping events. This is motivated by the observations that the number of events significantly changes over time even for the same scene and that occlusion often occurs in many vision tasks. To address these issues, we propose three strategies to select certain events to be dropped, including Random drop, Drop by time, and Drop by area. Figure 1 shows a specific example of augmented events with different operations of EventDrop. Through these augmentation operations, we can enlarge both the amount of training data as well as the data diversity, which will benefit deep learning models. To the best of our knowledge, EventDrop is the first work to augment asynchronous event data by dropping events.

The closely-related works to this study are Dropout [Srivastava et al., 2014], Cutout [DeVries and Taylor, 2017], RE [Zhong et al., 2020] and SpecAugment [Park et al., 2019], all of which introduce some noise to improve the generalization ability of deep learning models. However, Dropout drops the units and their connections in the models that can be in the intermediate layers, while our method only drops events in the input space. EventDrop can be considered as an extension of Dropout in the input space. Compared to Cutout and RE, both of which deal with images by considering occlusion, EventDrop works with event data and deals with both sensor noise and occlusion. SpecAugment works on audio, while our method deals with event-based data.

In summary, our main contributions are as follows:

• We propose EventDrop, a novel method for augmenting asynchronous event data, which is simple to implement, computationally low-cost, and can be applied to various event-based tasks.
• We evaluate the proposed method on two public event datasets with different event representations. Experimental results show that the proposed method significantly improves the generalization performance across a variety of deep networks.

2 RelatedWork

2.1 Event-based Learning

Event-based learning has been increasingly popular due to the advantages of event sensors (e.g, low time latency, low power consumption, and high dynamic range) [Gallego et al., 2020; Lee et al., 2020]. Event-based learning algorithms can be grouped into two major approaches. One approach is to first convert asynchronous events into frame-like data, such that frame-based learning methods can be applied directly (e.g., state-of-the-art DNNs). Some representative works include event frame [Rebecq et al., 2017], Event Count Image [Maqueda et al., 2018], Voxel Grid [Zhu et al., 2019], and Event Spike Tensor (EST) [Gehrig et al., 2019]. While such methods can make use of the powerful ability of modern DNNs through event conversion, they may discard some useful information about the events (e.g., polarity, temporal information, density).

The other approach is to directly use spiking neural networks (SNNs) on the asynchronous event data. The eventdriven property of SNNs makes them inherently suitable for dealing with event data. Compared to standard DNNs, SNNs are more biologically plausible and more energy efficient when implemented on neuromorphic processors. Eventbased learning with SNNs has been used for object recognition [Gu et al., 2020], visual-tactile perception [Taunyazov et al., 2020], etc. While SNNs are attractive for dealing with event data, the spike function is not differentiable and hence one cannot directly use backpropagation methods to train the SNNs. Several solutions have been proposed to address this issue, such as converting DNNs to SNNs and approximating the derivative of the spike function [Wu et al., 2019]. However, the overall performance of SNNs is often inferior to standard DNNs.

In this study, we focus on data augmentation for eventbased learning with DNNs, but the proposed method can be also applied for SNN-based methods.

2.2 Regularization

Regularization is a key technique for mitigating over-fitting in the training of deep learning models. Common regularization strategies include weight decay, and Dropout [Goodfellow et al., 2016]. The basic idea of weight decay is to penalize the model weights by adding a term to the loss function. Popular forms of weight decay are L1 and L2 regularization. Dropout is also a widely-used regularization technique, which simulates sparsity for the layer it is applied to. In the standard Dropout method [Srivastava et al., 2014], units and their connections are randomly dropped out from the model with a certain probability (e.g., 0.5) during training. Many variants have been proposed to further improve the speed or regularization effectiveness [Labach et al., 2019]. Compared to Dropout, this study drops events in the input space rather than drops the units and their connections in the models.

2.3 Data Augmentation

Data augmentation can be also regarded as a regularization method that improves the generalization ability of deep learning models. It is widely accepted that deep learning models over-fit, and benefit strongly from larger datasets. Data augmentation is a practical technique to increase the amount of training data as well as the data diversity. Many studies have demonstrated that deep learning models can significantly improve their generalization ability by applying some transforms on the input images [Krizhevsky et al., 2012], such as Translation, Rotation, Flipping, and Cropping. Recently, A popular augmentation technique called SamplePair [Inoue, 2018] is proposed for image classification, which creates a new sample from one image by overlaying another image that is randomly selected from training data.

Different from existing works that deal with images, this study works on the augmentation of event data, which remains unexplored to the best of our knowledge. We focus particularly on the object occlusion problem and noisy event data. The closely-related works that also deal with occlusion are Cutout [DeVries and Taylor, 2017], RE [Zhong et al., 2020], and SpecAugment [Park et al., 2019]. Specifically, Cutout applies a fixed-size zero mask to a random location of each input image, while RE erases the pixels in the randomly selected region with random values. Compared to Cutout and RE, which deal with images, our approach deals with event data and considers both object occlusion and sensor noise. SpecAugment is an augmentation method for audio, which operates on the log mel spectrogram of the input audio. Both SpecAugment and our method are inspired by Cutout. However, SpecAugment works on audio, while our method deals with event data (which are asynchronous events).

3 Event Representation

State-of-the-art DNNs usually deal with frame-like data (e.g., images, videos) and cannot be directly used for event data since event data are a stream of asynchronous events. An individual event alone, contains little information about the scene. To make use of event data, certain methods have been proposed to learn useful frame-like representations that can be exploited by DNNs. Popular event representations include Event Frame [Rebecq et al., 2017], Event Count Image [Maqueda et al., 2018], Voxel Grid [Zhu et al., 2019], and EST [Gehrig et al., 2019], which we will introduce in the following. Figure 2 shows the general framework of converting asynchronous event data into popular event representations.

Let ε be a sequence of events, which encode the location, time, polarity (sign) of the changes. It can be described as:

where (xi, yi) is the coordinate of the pixel triggering the event ei, ti is the timestamp when the event is generated, and pi is the polarity of the event. The polarity takes two values: 1 and -1, representing positive and negative events, respectively. I is the number of events.

Event Frame represents events using the histograms of events for each pixel, which can be written as (denoted by VEF ):

where δ(·) is an indicator function. (xl, ym) is the pixel coordinate in the Event Frame representation, and xl ∈ {0, 1, · · · ,W - 1}, ym ∈ {0, 1, · · · ,H - 1}. The Event Frame can be regarded as a 2D image with a resolution of H ×W.

Event Count Image is similar to Event Frame, but it uses separate histograms for positive events and negative events. Event Count Image VEC is described as:

where ε+ and ε- are event sequences with positive polarity and negative polarity, respectively. The Event Count Image can be regarded as a two-channel image with each channel corresponding to one polarity.

Voxel Grid VVG considers the temporal information of the events, which is not explicitly handled in Event Frame and Event Count Image. It is written as

where 1ti is an indicator function, which takes 1 when ti is in the interval (tn-1, tn] and 0 otherwise. cn is the temporal index of the Voxel Grid representation, and cn ∈ {0, 1, · · · , C - 1}. ∆T is temporal bin size.

Similar to Voxel Grid, EST is also a grid-based representation that is learned end-to-end directly from asynchronous event data through differentiable kernel convolution and quantization. EST considers both temporal information and polarity about events, which is described as:

where f±(x, y, t) is the normalized timestamp, and f±(x, y, t) = t-t1 ∆T where t1 is the first timestamp, ∆T is the bin size. k(x, y, t) is a trilinear kernel, which is written as

In addition to the above representations, there are other event representations such as HOTS [Lagorce et al., 2016], HATS [Sironi et al., 2018], and Matrix-LSTM [Cannici et al., 2020]. In this study, we take the four representations as representatives to analyze how EventDrop enhances the performance of DNNs.

4 ProposedMethod: Augmenting Event Data by Dropping Events

4.1 Motivation

This work is motivated by two observations. The first observation is that the output of event cameras for the same scene under the same lighting condition may vary significantly over time. This may be because event cameras are somehow noisy, and events are usually triggered when the change about the scene reaches or surpasses a threshold. By randomly dropping a proportion of events, it is possible to improve the diversity of event data and hence increase the performance of downstream applications.

The second observation is that occlusion often occurs in many tasks such as object recognition and tracking. The generalization ability of a machine learning model depends highly on the diversity of training data, including various levels of occlusion. However, the available training data usually suffer from limited variance in the occlusion level. A machine learning model trained on the data with limited or no (totally visible) occlusion variance may generalize poorly on new samples that are partially occluded. By generating new samples that simulate partially occluded cases, the model is able to better recognize objects with partial occlusion.

4.2 Strategies of Dropping Events

To address the above issues, we propose three strategies of dropping events to augment event data, namely Random drop, Drop by time, and Drop by area. The first strategy is to prepare the model for noisy event data. The other two strategies are for simulating the occlusion problem. Figure 3 illustrates the idea of different dropping strategies. We describe the three strategies in the following.

• Random drop. The basic idea of Random drop is to randomly drop a proportion of events in the sequence. This is to overcome the noise from the event sensors.
• Drop by time. Drop by time is to drop events triggered within a random period of time. It tries to increase the diversity of training data by stimulating the case that objects are partially occluded during certain time period.
• Drop by area. Drop by area is to drop events triggered within a randomly selected pixel area. It also aims to improve the data diversity by simulating various cases that some parts of the objects are partially occluded.

Algorithm 1: Procedures of augmenting event data with EventDrop

Input : A sequence of events ε = {ei}Ii=1 = {xi, yi, ti, pi}Ii=1 , pixel resolution (W, H). Output: Augmented event sequences ε∗ . 1 Initialize the ε∗ to an empty set, namely ε∗ = {}; 2 Operation←Random.choice(identity, drop by time, drop by area, random drop); 3 if Operation == identity then 4 ε∗ ← ε; 5 end 6 if Operation == drop by time then 7 ρ ←Rand(1, 10)/10; 8 Tmin ←Rand(t1, tI); 9 Tmax ←max(tI, Tmin +ρ ∗ (tI - t1)); 10 for ei ∈ ε do 11 if (ti < Tmin)‖(ti > Tmax) then 12 Add ei into ε∗ ; 13 end 14 end 15 end 16 if Operation == drop by area then 17 ρ ←Rand(1, 6)/20; 18 x0 ←Rand(0,W); 19 y0 ← Rand(0, H); 20 for ei ∈ ε do 21 if(xi ∈ [x0, x0+ρ∗W])&(yi ∈ [y0, y0+ρ∗H] then 22 Do nothing; 23 end 24 else 25 Add ei into ε∗ ; 26 end 27 end 28 end 29 if Operation == random drop then 30 ρ ←Rand(1, 10)/10; 31 ε∗ = Random.choices(ε, I ∗ (1 - ρ)) 32 end 33 return ε∗ .

4.3 Implementation

In this section, we describe the implementation of EventDrop. Algorithm 1 gives the procedures of augmenting event data with EventDrop. This algorithm takes as input a sequence of asynchronous events and corresponding image resolution (W,H). We first define four augmentation techniques, namely Identity, Random drop, Drop by time, and Drop by area, and conduct one augmentation technique that is randomly selected on the event sequence. The random policy exploration in [Cubuk et al., 2020] is adopted in this study due to its simplicity and excellent performance. The probability p of each of these augmentation operations being chosen is set to equal (namely, p = 0.25). The magnitude of Random drop and Drop by time is discretized into 9 different levels and that of Drop by area into 5 levels. Specifically, when conducting the Drop by time operation, a magnitude is first randomly selected and then a period of time is selected. The events triggered within the selected time period would be deleted from the event sequence, the remaining event sequence will be returned as the output of the algorithm. In the Drop by area operation, a pixel region is first selected by a random magnitude and a random location, and then the events within the selected region would be dropped. In the Random drop operation, a proportion of events are randomly selected to be dropped. Overall, EventDrop is simple to implement and computationally low-cost. We have implemented EventDrop in PyTorch and the source code is available at https://github.com/fuqianggu/EventDrop.

5 Experiments and Results

5.1 Datasets

We evaluate the proposed EventDrop augmentation technique using two public event datasets: N-Caltech101 [Orchard et al., 2015] and N-Cars [Sironi et al., 2018]. N-Caltech101 (Neuromorphic-Caltech101) is the event version of the popular Caltech101 dataset [Fei-Fei et al., 2004]. To convert the images to event sequences, an ATIS event camera was installed on a motorized pan-tilt unit, and automatically moved while pointing at images from the original dataset (Caltech101) that were shown on a LCD monitor. N-Cars (Neuromorphic-Cars) is a real-world event dataset for recognizing whether a car is present in a scene. It was recorded using an ATIS camera that was mounted behind the windshield of a car.

5.2 Experiment Setup

We evaluate the proposed method using four state-of-theart deep learning architectures, namely ResNet-34 architecture [He et al., 2016], VGG-19 [Simonyan and Zisserman, 2014], MobileNet-V2 [Sandler et al., 2018], and InceptionV3 [Szegedy et al., 2016]. All the networks are pretrained on ImageNet [Russakovsky et al., 2015]. Since the number of input channels and output classes for our case are different from these pre-trained models, we adopt the approach used in [Gehrig et al., 2019] and replace the first and last layer of the pre-trained models with random weights, and then fine-tune all the parameters on the task.

Since event data are a stream of asynchronous events and cannot be directly applied to deep nets, we consider and implement the four event representations introduced in Section 3. For the implementation of EST, we replace the neural network with a trilinear kernel to convolve with the normalized timestamps for computational efficiency. Note that the considered deep learning models take as input 2D images, while some event representations we considered (e.g., Voxel Grid and EST) are 3D or 4D tensor. To adapt to these pretrained model, we concatenate the event representation along the polarity and/or temporal dimension as channels.

The Adam optimizer is used to train the model by minimizing the cross-entropy loss. The initial learning rate is set to 1×10-4 until the iteration reaches up to 100, after which the learning rate is reduced by a factor of 0.5 every 10 iterations. The total number of iterations is set to 200. We use a batch size of 4 for both datasets. To conduct a robust evaluation, we run the model on each dataset for multiple rounds with different random seeds, and report the mean and standard deviation values. We perform early stopping on a validation set using the splits provided by the EST [Gehrig et al., 2019] on N-Caltech101 and 20% of the training data on N-Cars.

5.3 Results on N-Caltech101

We first analyze the results of EventDrop on the NCaltech101 dataset. The results from the same models without data augmentation are considered as the baselines. Table 1 compares the performance of EventDrop and the baselines. We can see that EventDrop improves the performance of all the models used with different event representations. The accuracy achieved with Voxel Grid and EST representations is much higher than that with Event Frame and Event Count representations. This is attributed to the fact that Voxel Grid and EST contain temporal information about the events that is discarded by Event Frame and Event Count. Since EST further considers the polarity information about the events, it behaves slightly better than Voxel Grid. The same trend can be found when comparing Event Frame (without polarity information) and Event Count (with polarity information). Among these deep nets, MobileNet-V2 seems to perform slightly better than ResNet-34 and Inception-V3, while VGG-19 performs the worst, probably because it is relatively old.

5.4 Results on N-Cars

We then compare the results of EventDrop with the baselines on N-Cars. As can be seen from Table 2, EventDrop outperforms the baselines with different deep learning architectures and event representations. The improvement on N-Cars dataset is generally greater than that on N-Caltech101 dataset. This might be because N-Cars is a real-world data dataset, and EventDrop works better with real-world cases where sensor noise and occlusion occur more likely than simulation

(i.e., where the event camera looks at a projected scene rather than a real-world scene). The improvement of EventDrop can reach up to about 4.5% (by ResNet-34 with EST representation). The Event Count and EST representations, which consider polarity information, perform better than the Event Frame and Voxel Grid that do not take polarity into account. The performance of the four deep nets is similar among the baselines, while ResNet-34 and MobileNet-V2 achieve better accuracy when the training data is augmented with EventDrop.

5.5 Comparison of Different Dropping Strategies

We also compare the performance of different dropping strategies on both datasets. In the implementation of Drop by time, Drop by area, and Random drop operations, the probability of each operation being conducted is set to 0.5, and the corresponding magnitude is randomly selected from the value set described in Section 4. For EventDrop, the three dropping strategies and Identity operations are randomly selected to conduct with equal probability. As demonstrated in Table 3, EventDrop that integrates different dropping strategies outperforms the baselines on both datasets, and the improvement of EventDrop over the baselines is bigger on N-Cars dataset than on N-Caltech101 dataset.

Specifically, for N-Caltech101 dataset, EventDrop and Drop by area operations have better performance than Drop by time and Random drop operations in general. The Drop by time operation seems not to improve the baselines when using Voxel Grid and EST representations but it improves the performance when using event frame and Event Count representations. This might be explained by N-Caltech101 being a simulated dataset in which the sensor noise and occlusion in time are negligible, and hence discarding some events that are selected randomly or triggered during a certain period of time does not increase the diversity of data. By contrast, for N-Cars dataset, all the dropping operations result in a better accuracy than the baselines. This might be because the realworld event dataset (N-Cars) suffers more from sensor noise and various occlusions, and dropping operations can better increase the data diversity.

5.6 Effect of the Amount of Training Data

In this section, we analyze the effect of using different amounts of training data. The ratio we considered ranges from 0.1 to 1 where 0.1 represents only 10% training data that are randomly selected are used to train the network. To reduce the search space, we fix the random seed that is shared by the baselines and EventDrop, and then compare their performance. Figure 4 shows that EventDrop consistently improves the baselines in general. It can achieve about 94% accuracy even with only 10% training data compared to about 91.5% by the baseline. Although the improvement of EventDrop over the baseline is marginal when it is trained with certain ratios of training data, such a problem would obviously be reduced by averaging the results over more runs. It is also clear that the improvement of EventDrop is stable when using the more relatively robust EST representation.

6 Conclusion

In this paper, we propose a new augmentation method for event-based learning, which we call EventDrop. It is easy to implement, computationally low-cost, and does not involve any parameter learning. We have demonstrated that by dropping events selected with certain strategies, we can significantly improve the object classification accuracy of different deep networks on two event datasets. While we show the application of our approach for event-based learning with deep nets, our approach can be also applied to learning with SNNs. For future work, we will apply our approach to other eventbased learning tasks, such as visual inertial odometry, place recognition, traffic flow estimation, and simultaneous localization and mapping.