Disguise Adversarial Networks for Click-through Rate Prediction

1 Introduction

Improving users' click-through rate prediction on Advertisement (Ads) is a long-term research topic in the AI community [Richardson et al., 2007; Chen and Yan, 2012]. In practical advertising industry, the "click pay cost per click" business model allows the recommender accumulating high amount of revenues if they could precisely serve the right Ads to the right user. In existing works, such CTR prediction problem is conventionally tackled by supervised learning [K¨onig et al., 2009], in which a classifier/discriminator is trained to map contents (feature) to the practical click/non-click behaviors (label) in historic Ads impressions.

The challenges in this prediction problem stem from the severe imbalances between clicks and non-clicks in the "traffic log" of Ads impressions. For instance, in a mobile Ads dataset [Ava, 2015], only 17% Ads impressions are clicked by users while the other 83% are all non-clicks, yielding to an extremely imbalanced label distribution with quite limited positive samples. There are in fact a number of algorithms designed to handle classification problems with imbalanced labels. According to [Japkowicz, 2000], these methods could be generally profiled into down-sampling and up-sampling categories.

Down-sampling approaches reduce the major class size and make a balanced subgroup for classification. While those methods could alleviate the label imbalance issue to some extent, they do not notice the essential pitfalls of the imbalanced classification-the information about minor class is still insufficient. In the context of Ads recommendation, we know that most Ads impressions are annoying and human may naturally dislike them. Therefore, it is more meaningful for a classifier to get more chances to know interesting Ads contents (minor class up-sampling), rather than losing opportunities to see disliked Ads (major class down-sampling).

Oversampling method tries to generate more samples in the minor group. The first prevalent over-sampling strategy is to parametrically fit a probabilistic distribution (a.k.a. generative model) from limited observed samples in the minor class. Then, new data points can be sampled out from such a fitted distribution [Bao et al., 2017b]. Such oversampling method is only suitable for well structured data, e.g. gene [Bao et al., 2017a]. Otherwise, it is really hard to find a reasonable parametric generative model to well depict the data generation mechanism. Unfortunately, like most real world data, the Ads feature does not exhibit obvious inherent structure. Alternatively, up-sampling methods may also synthesize new data points by combining a set of observed data points in the minor class [Chawla et al., 2002; Han et al., 2005]. Such strategy was built on the assumption that existing data are sufficient to span the complete space of the minor class, which is not true in practice. The Ads contents and other real world data could be highly heterogeneous. There are great chances that neither the data nor its related items has ever been observed within these limited samples. Accordingly, the assumption that new data could all be produced by combing existing observations does not make too much sense here.

While we admit the rationality of minor class augmentation, existing strategies almost suffer restrictions in handling practical problems. At least, they are not suitable for the CTR prediction problem. We noted that an ideal minor class augmentation algorithm should exhibit two desired properties 1) the data generation function of it should be general and robust 2) the seeds used in it for new data generation should not solely come from limited samples in the minor class.

To fully cover the aforementioned two properties, we proposed a disguise adversarial network (DAN) for minor class augmentation (up-sampling). Our DAN was inspired by the recent progress of generative adversarial learning [Goodfellow et al., 2014] and was especially designed to solve the CTR prediction problem with imbalanced Ads' labels. The proposed DAN incorporates a disguise neural network to generate more samples to enrich the minor class by disguising negative samples. The purpose of the disguise neural network is to cheat a discriminator to believe all these disguised samples are all positive. On the other hand, a discriminator neural network is also implemented to clearly assign these disguised samples into their inherent classes via an information theoretic discriminative clustering strategy.

The DAN framework was applied and tested on CTR prediction problems including both mobile and display Ads. Compared with traditional imbalanced classification algorithms and other generative adversarial networks, the proposed DAN improves in both recommendation frequency and accuracy. More impressively, our approach is also very effective when less training data are available. The DAN approach can maintain reasonable good performances even though reducing the size of training samples to 10%, which shows a promising direction for algorithm speeding up.

2 Preliminaries

DAN is inspired by the generative adversarial network (GAN) [Goodfellow et al., 2014] that showed great promises in the computer vision society [Denton et al., 2015]. While there are different innovations on GAN, the central concept of it can be well interpreted as a gambling process involving a generator (G) and a discriminator (D). They could be both implemented by deep neural networks (DNN). The generator maps a randomly sampled vector zi ∼ Pz(z) as an image yi = G(zi). The discriminator is designed to identify fake image zi from real world image xi ∼ Pdata(x). In a nutshell, the gambling process could be formulated with the following min-max optimization:

Conditional GAN [Gauthier, 2014] is an extension of the basic GAN that takes the label information into consideration. In detail, the generator in conditional GAN takes two streams of information as the input yi = G(zi, li), where li is the assigned categorical label of the ith generated sample. There were different implementations of this conditional GAN by either concatenating l and z as a long vector or taking them as two inputs of a multi-modal DNN [Ren et al., 2016]. InfoGan is another prevalent approach that [Chen et al., 2016] generalizes the categorical information into the generator by adding a mutual information term. It views the categorical information and other control information (e.g. the angle of the image) as side information and quantizes them as a long vector l. InfoGAN then maximizes the mutual information between the generated image y and its corresponding side information vector l.

There were also improvements on the discriminator side, e.g. SGAN [Odena, 2016]. The discriminator in SGAN assigns data into N classes of real data and one extra class of fake data. In the adversarial gambling game of SGAN, the generator tries to put all fake data into N real data classes but the discriminator is optimized to allocate fake data into the fake category, i.e. the (N + 1)th category. As SGAN, the main purpose of DAN is also set to enhance classification performances of the discriminator rather than distinguishing fake data from real ones.

3 Disguise Adversarial Networks

3.1 Motivation

We motivate the algorithm of Disguise Adversarial Networks (DAN) in the context of CTR prediction. As stated above, one significant problem in Ads data is the imbalanced label distribution, i.e. limited positive samples (clicks) v.s. abundant negative samples (non-clicks). A natural solution to this problem is to enrich the information of the minor positive group. However, it is perhaps impossible to directly seek for more positive samples from users in real world. Inspired by GAN, we consider an alternative approach to generate more positive samples via a generative deep neural network.

We consider a way called "Ads Makeup". The general assumption is that if we could slightly change some properties of the disliked Ads, it may have a chance to become an interesting one. We discuss the intuition behind this Ads makeup approach by taking a mobile Ads "super bowl game" as an example. We consider that the Ads feature vector contains one entry denoting the show time of the Ads. If the "super bowl game" Ads impression was pushed to the user at 10:00 AM (with the Ads' time feature denoted as 'morning'), this Ads may not be clicked because morning time is always the business hour. Accordingly, a non-click record about "super bowl Ads" is accumulated in the training data. But it does not necessarily mean the "super bowl Ads" itself is bad and is not liked by the user. Alternatively, if we fix all other features in this Ads the same but only change the Ad's show time feature from 'morning' to 'night', this disliked Ads may become a popular one and gain a click.

Following the rationality discussed above, we believe there should be a huge amount of non-clicks in the historic data that could be converted to interesting ones with slight modifications. We design a "Disguise Neural Network" (Fig 1) to transform and makeup non-clicks. However, there is still a lack of the metric to evaluate the quality of such an Ads disguise approach. We hence define a "Discriminator Neural Network" to mimic real user's behavior on Ads clicking. From the aspect of the Disguise Neural Network, its objective is to ultimately disguise the Ads and encourage the discriminator to classify these disguised Ads as positive. In an adversarial view, the discriminator should avoid being cheated by the disguise network and come up with a "smart" objective to identify these disguised Ads.

In detail, we discuss two possible objectives for the discriminator. The first option is the "tough objective" which is set up to assign all disguised Ads to the negative class (nonclicks). However, such objective is too strict to be reasonable. There should be a number of successfully disguised Ads, such as the super bowl Ads, that could be assigned to the positive class after makeup. If treating all disguised Ads as non-clicks, these successfully disguised Ads and their corresponding favorite feature content may not be correctly fed to the positive side of the classifier. Therefore, we consider a more reasonable "mild objective" for the discriminator. It should assign the disguised data into two classes (both clicks and nonclicks) according to their inherent properties. To achieve this, we design a clustering objective for the discriminator to encourage a large margin between two classes. We will explain how to mathematically design the discriminative objective in Section 3.3. But please keep in mind here that the "mild objective" could smartly treat both successfully and unsuccessfully disguised samples without any biased pre-judgment.

Although we have motivated the Disguise Adversarial Networks (DAN) in the context of Ads CTR prediction, we still remind readers that all subsequent discussions about DAN model are also seamlessly adapted to general classification problems. In the next two subsections, we will elaborately illustrate formulations of these two adversarial objectives in DAN.

3.2 Disguise Objective

The disguise learning part was linked by solid arrows in Fig.1. There are M positive samples (clicks, red bar in Fig.1) x+ ∼ PΩ+(x) and N negative samples (non-clicks, blue bar in Fig.1) x- ∼ PΩ-(x), M < N for imbalanced cases. Ω+ (resp. Ω-) represents the positive samples' (resp. negative samples') space. We introduce a Disguise Neural Network T(·) to map the negative samples to z = T(x-) ∼ PˆΩ(z), where ˆΩ is space spanned by those disguised samples z

(green bar in Fig.1). z shares the same dimension as the real world data x+ and x-.

The learning purpose of disguise neural network is to encourage the equivalence of two distributions PΩ+(x) and PˆΩ(x). From the "disguise objective panel" in Fig.1, it is apparent that the disguise objective intends to assign all disguised data to the positive class with the discriminator neural network D(·). The last layer in neural network D is connected with a sigmoid output to indicate the probability that the input sample is positive. Therefore, we could mathematically write the disguise learning objective:

The first term in Eq.2 is the KL divergence between the output probability and the positive distribution. It is a part of the cross-entropy term. The second term uses a ℓ1 distance to restrict the disguise process could only 'slightly' change the original data.

We would like to highlight the conceptual differences between this disguise neural network and the generator neural network in GAN [Goodfellow et al., 2014]. GAN is designed to generate a brand new sample that never exists in real world. It thus requires sampling a random seed (vector) to feed in the generator and to produce a fabricate output. In our DAN, everything is from real world data and the inputs to the disguise neural network are practical negative samples. It just adjusts negative data to make them comply with the distribution in the positive class. Therefore, we prefer to use the word "disguise" rather than "generator" in our approach. Such characteristic is apparent in Fig.1 that no random vector sampling function was observed in DAN's infrastructure.

3.3 Discriminator Objective

As indicated by dotted arrows in Fig.1, the discriminator neural network takes both real and disguised data as inputs. Therefore, both of these two types of data contribute to the final loss in the discriminator neural network. The real data come with the user's clicking labels, so there is no difficulty to define a supervised loss for this part of data via the crossentropy loss. The second part of the loss is the attitude of the discriminator neural network about disguised data. We will elaborately discuss it in the following part.

As stated in Section 3.1, there were two possible ways for the discriminator neural network to handle disguised data via either the "tough loss" or the "mild loss". Here, we adopt the more reasonable "mild loss" that allows some negative data to be converted as positive ones. However, the difficulty is that we have no idea about which part of the disguised data behave like positive samples and which part are still negative. To address such a problem, we follow an existing work to maximize the information theoretic margin between positively and negatively disguised samples [Krause et al., 2010]. Unlike other margins in supervised learning, such information theoretic margin is absolutely unsupervised. This approach is also termed as discriminative clustering in multiple early works [Deng et al., 2016a][Shi and Sha, 2012].

We assume there are N unlabeled points. When assigning these N points to 2 classes (l = 1 or l = 0) by a discriminator neural network D(·), the assignment confidence of the discriminator could be well characterized by the following additive conditional entropyMD(x),

As indicated in [Krause et al., 2010], the conditional entropy captures the discriminative clustering margin between two classes and hence we call MD(x) as the information theoretic margin in our approach. This term should be minimized to encourage a large margin between the clustering results.

We combine the aforementioned two parts of losses altogether and form the final training objective for the discriminator neural network:

where the first two terms come from the cross-entropy of the real world labeled data and the last term penalizes the margin of disguised data.

To note, our approach is quite different from traditional GAN in which the discriminator is only set up to classify whether a sample is real or fake. Our discriminator objective function shares a very similar framework as semi-supervised classification. In traditional semi-supervised learning, the unlabeled data used for training are pre-fixed. But unsupervised samples in DAN are produced by a disguise neural network. Therefore, in our approach, the discriminator neural network may get a chance to access to different versions of disguised samples in multiple iterations. More importantly, the disguise neural network also evolves and could produce more difficult samples for the discriminator along with training iterations going on. Therefore, compared with traditional semi-supervised learning, DAN could be optimized with more diverse and difficult unsupervised samples in the training phase. This is the exact reason why we believe DAN can achieve much better performance than traditional semisupervised methods that only adopt pre-fixed unlabeled samples as assistance.

The training of DAN is involved in a bilevel optimization [Bard, 2013] that requires minimizing the disguise and the discriminator losses in turn. Bilevel optimization was used in a number of practical learning problems including GAN [Radford et al., 2015], sparse learning [Deng et al., 2013] and reinforcement learning [Lillicrap et al., 2015; Deng et al., 2017a]. We divide all training samples into multiple minibatches and iteratively feed these mini-batches to train DAN. Algorithm 1 summarized our detailed training steps.

4 Experiments

4.1 Experimental Setup

We evaluate the performance of GAN in CTR prediction on two datasets including both display and mobile Ads. Display Ads dataset record the Ads impressions from Criteo in

Algorithm 1: Training DAN

Input : Ads Features X and their labels Y Initialization: Randomly Initialize all parameters ΘT of disguise neural network T and ΘD of discriminator Neural Network D; Find all negative samples X- in X

1 for epoch=1...K do

2

3

4

5

6

7

for all mini-batches in Training Data do

Minimize the the disguise loss in Eq.2 with

X- and update ΘT in T

Feed all negative samples X- through disguise neural network with current ΘT and get the the disguised samples Z

Feed the discriminator with both supervised data (X) and unlabeled disguised data Z = T (X-); treat Z as unsupervised samples Minimize the discriminator loss in Eq.4 and update ΘD in D

end

8 end

Output

: Discriminator Neural Network D with parameter ΘD

an one-week time period that include various undisclosed features along with the click labels [Cri, 2015]. The feature for each display Ads is composed of 13 integer and 26 categorical features. The mobile Ads dataset come from [Ava, 2015] that cover mobile Ads impressions in 10-days period. All provided attributes in this dataset are anonymous categorical features such as "device type", "app id" etc. Because both of these two datasets contain categorical features, we tried to convert each categorical feature as a binary vector indicating which category the certain item belongs to. A brief summary about these two datasets were provided in Table 1. The last column "Dim" in the table reports feature dimensions of each Ads dataset after converting the categorical feature to numerical values. These numerical features are used in our DAN and other competing methods.

It is also noted that Ads impressions in both datasets were accumulated in an ascending order over time. We obey the time order and uniformly divide each dataset as 100 bulks. Each bulk contains 1% Ads impressions in a certain period and different bulks are consecutive in time. It is important to note the time-varying effects on Ads impressions that one specific Ads could be quite hot in a short period but quickly losing its popularity afterwards. To fully take this timevarying effect into consideration, we train our model with the last 20 bulks of Ads impressions in the history and predict the CTR in the next 5 bulks. The whole process was incrementally moved forward following the time order of bulks.

Many previous works have indicated that conventional classification accuracy indicators such as precision and recall are not suitable for evaluating the CTR performance because the data themselves are highly imbalanced [Yan et al., 2014]. Therefore, they suggest the use of area-under-curves (AUC) as a robust indicator to validate the performances of different CTR algorithms. AUC should be reported in the range of (0.5, 1), with AUC=0.5 meaning random guessing. A higher AUC implies a better classification result. We also prefer to use AUC as a main indicator to report CTR prediction performances.

While the neural networks in DAN could be implemented with arbitrary structures, such as the fancy hierarchical structure in [Deng et al., 2016b], we still prefer to implement them with the simple multi-layer perception (MLP). In detail, both the disguise and discriminator neural networks are configured with 4 layers and each layer contains 32 nodes. The output layer of the disguise neural network shares the same nodes' number as its input layer. The output of the discriminator neural network is a sigmoid function indicating the clicking probability. We do not carefully tune the network structure because this general setting is already good enough to produce reasonable results as reported below.

4.2 Overall Evaluation

To visualize the effectiveness of adversarial training in DAN, we first compare the DAN's performance with a discriminator neural network. For fair comparisons, both the DAN and the discriminator neural network share the same network structure as stated above. Mathematically, the discriminator was trained with the supervised loss while DAN was optimized with the adversarial loss defined in Eq.4. We train both the DAN and the discriminator neural network for 25 epochs. At the end of every 5th epoch, the corresponding CTR performances on testing data were calculated and reported in Fig.2.

In the figure, we used dotted and solid lines to represent the supervised discriminator and DAN, respectively. Different colors correspond to results on different datasets. At the beginning of training, both supervised and adversarial discriminators achieved similar performances. But after the 5th epoch, the adversarial training strategy leads to an obvious improvement in AUC on both datasets. Besides, the trend of curves also empirically imply the evolutions of DAN in the gambling process. It achieved relatively higher performance than ever with more iterations going on.

We also compare our DAN with other competing methods. In general, these competing methods could be divided as four types as listed in the first column of Table 2. In detail, Deep Neural network (DNN) and Support Vector Machine (SVM) were chosen as the supervised algorithms. We further chose cluster centroid algorithm (Centroid) [Ganganwar, 2012], adaptive synthetic sampling (ADASYN) [He et al., 2008], synthetic minority over-sampling technique (SMOTE) [Chawla et al., 2002] as representative imbalanced classification techniques, in which the first one is a down-sampling strategy and the later two are up-sampling approaches.

We also consider other GAN methods as competitors. However, except for SGAN [Odena, 2016], most other GANs are not designed for classification purpose. Therefore, we slightly modified existing GAN (Modified GAN) to make it applied to the CTR problem. In the implementation, we first ran a traditional GAN to generate many fake Ads. After the GAN training, the generated fake Ads that could successfully pass the authenticity verification step (i.e. by the discriminator in traditional GAN) were used as unsupervised samples altogether with supervised CTR records for semi-supervised training as in [Deng et al., 2016a]. Finally, different DAN implementations were also discussed. In addition to the DAN model presented in the paper, we alternatively consider two other variations. The first variation is the tough-loss DAN (TL-DAN) as stated in Section 3.1. The other variation is the Noise seeded DAN (Noise GAN) that is designed to use a random generator to generate more samples for the discriminator. In Noise DAN, the discriminator neural network is the same as in DAN but fake samples are all generated from random noise.

We report the performances of these methods in Table.2. In addition to the AUC indicator, we also report two other indicators that may intuitively explain how different methods perform in the practical recommendation scenario. The first indicator is the recommendation frequency (Rd-frequency) that reports how frequently a recommendation decision is rendered. In our setting, a recommendation is only made when the CTR prediction model indicates to the positive class. Therefore, the recommendation frequency means the percentage of positive labels predicted by the recommendation method. Further, we investigate the CTR based on the recommended items (Rd-CTR) that calculates the CTR among all recommendations. Such evaluation strategy is similar as the off-line evaluation policy introduced in [Li et al., 2010].

We observed that all machine learning models render pretty similar but relatively low recommendation frequency. Even though based on the best performed DAN model, its recommendation frequency is only 11% and 5% on two datasets. But, the DAN improves Rd-CTR for a large amount. From the AUC score, it is observed that our DAN significantly outperforms other learning methods. Only the DAN model could achieve the AUC score higher than 0.7 on two datasets. By comparing two datasets, performances on Mobile dataset is obviously worse than on Display dataset. This is partially because the Criteo display data were already up-sampled by the data provider to maintain some balance level and the mobile data is much closer to real-world data. In addition, we have not observed obvious advantages of existing imbalanced classification algorithms when compared with deep learning approaches. Such claim is apparent by comparing all methods in the "Imbalanced" category with the DNN result in the "Supervised" category in Table 2. It implies the deep learning model perhaps already owns some inherent properties to robustly treat samples with imbalanced label distributions. But the adversarial deep learning (such as our DAN) models further improve traditional supervised deep learning.

4.3 Training Complexity

Deep learning framework always gains the reputation of "heavy to train". Among all reported competing methods, the DAN requires the heaviest training complexity. It is conceivable because DAN involves two (deep) neural networks. Even worse, the learning objectives of these two neural networks are designed to go against each other. The gambling essence of adversarial training inevitably adds complexity to the optimization. Our practical training always requires 3 hours to finish 25 epochs on 9 million historic data with 4 GPUs parallelized. When using the same data to train a supervised neural network, the consumed time is only about 1 hour. Rather than hardware improvement, we also consider alternative approach to improve the training speed by reducing total training samples as in Table 3. The AUC results for DAN are calculated on out-of-sample data.

The experimental results here comply with most findings in deep learning that a larger training size may lead to a higher classification performance. But when taking training complexity into consideration, decreasing the sample size in DAN may be a good trade-off. It is observed that AUCs on testing data do not drop significantly even though decreasing the training sample size to 10%. The AUC of DAN is still better than most approaches in Table 2 with 10% training data.

Such plausible property may be partially due to the self-dataaugmentation mechanism encoded in the adversarial learning framework. Therefore, it is reasonable to reduce the total training size while maintaining a good performance.

5 Discussions

While our DAN was motivated and innovated in Ads CTR prediction, we should remind readers that it is meanwhile flexibly applied to other classification problems involving imbalanced data. While the algorithm and experiment in this paper were mainly designed on the two-class classification problem, it is still possible to extend this framework to handling multi-class tasks. One intuitive extension is just to implement DAN in an one vs others manner [Deng et al., 2017b]. But it requires training at least C (the class number) different DANs which is impractical when C is large. We will consider more efficient approaches in our future works. One downside that have been observed from experiment is the extremely low recommendation frequency as listed in Table 2. We may consider other approaches, e.g. training multiple machines with different initializations or configurations, to improve the recommendation frequency in an ensemble manner.