Streamed Learning: One-Pass SVMs

1 Introduction

Learning in a streaming model poses the restriction that we are constrained both in terms of time, as well as storage. Such scenarios are quite common, for example, in cases such as analyzing network traffic data, when the data arrives in a streamed fashion at a very high rate. Streaming model also applies to cases such as disk-resident large datasets which cannot be stored in memory. Unfortunately, standard learning algorithms do not scale well for such cases. To address such scenarios, we propose applying the stream model of computation [Muthukrishnan, 2005] to supervised learning problems. In the stream model, we are allowed only one pass (or a small number of passes) over an ordered data set, and polylogarithmic storage and polylogarithmic computation per element.

In spite of the severe limitations imposed by the streaming framework, streaming algorithms have been successfully employed in many different domains [Guha et al., 2003]. Many of the problems in geometry can be adapted to the streaming setting and since many learning problems have equivalent geometric formulations, streaming algorithms naturally motivate the development of efficient techniques for solving (or approximating) large-scale batch learning problems.

In this paper, we study the application of the stream model to the problem of maximum-margin classification, in the context of ℓ2-SVMs [Vapnik, 1998; Cristianini and ShaweTaylor, 2000]. Since the support vector machine is a widely used classification framework, we believe success here will encourage further research into other frameworks. SVMs are known to have a natural formulation in terms of the minimum enclosing ball problem in a high dimensional space [Tsang et al., 2005; 2007]. This latter problem has been extensively studied in the computational geometry literature and admits natural streaming algorithms [Zarrabi-Zadeh and Chan, 2006; Agarwal et al., 2004]. We adapt these algorithms to the classification setting, provide some extensions, and outline some open issues. Our experiments show that we can learn efficiently in just one pass and get competetive classification accuracies on synthetic and real datasets.

2 Scaling up SVMTraining

Support Vector Machines (SVM) are maximum-margin kernel-based linear classifiers [Cristianini and Shawe-Taylor, 2000] that are known to provide provably good generalization bounds [Vapnik, 1998]. Traditional SVM training is formulated in terms of a quadratic program (QP) which is typically optimized by a numerical solver. For a training size of N points, the typical time complexity is O(N 3 ) and storage required is O(N 2 ) and such requirements make SVMs prohibitively expensive for large scale applications. Typical approaches to large scale SVMs, such as chunking [Vapnik, 1998], decomposition methods [Chang and Lin, 2001] and SMO [Platt, 1999] work by dividing the original problem into smaller subtasks or by scaling down the training data in some manner [Yu et al., 2003; Lee and Mangasarian, 2001]. However, these approaches are typically heuristic in nature: they may converge very slowly and do not provide rigorous guarantees on training complexity [Tsang et al., 2005]. There has been a recent surge in interest in the online learning literature for SVMs due to the success of various gradient descent approaches such as stochastic gradient based methods [Zhang, 2004] and stochastic sub-gradient based approaches[ShalevShwartz et al., 2007]. These methods solve the SVM optimization problem iteratively in steps, are quite efficient, and have very small computational requirements. Another recent online algorithm LASVM [Bordes et al., 2005] combines online learning with active sampling and yields considerably good performance doing single pass (or more passes) over the data. However, although fast and easy to train, for most of the stochastic gradient based approaches, doing a single pass over the data does not suffice and they usually require running for several iterations before converging to a reasonable solution.

3 Two-Class Soft Margin SVMas the MEB Problem

A minimum enclosing ball (MEB) instance is defined by a set of points x1, ..., xN ∈ RD and a metric d : RD ×RD → R≥0 . The goal is to find a point (the center) c ∈ RD that minimizes the radius R = maxn d(xn, c).

The 2-class ℓ2-SVM [Tsang et al., 2005] is defined by a hypothesis f(x) = wT ϕ(x), and a training set consisting of N points {zn = (xn, yn)}Nn=1 with yn ∈ {-1, 1} and xn ∈ RD . The primal of the two-classs ℓ2-SVM (we consider the unbiased case one-the extension is straightforward) can be written as

The only difference between the ℓ2-SVM and the standard SVM is that the penalty term has the form (C ∑n ξn 2 ) rather than (C ∑n ξn).

We assume a kernel K with associated nonlinear feature map ϕ. We further assume that K has the property K(x, x) = κ, where κ is a fixed constant [Tsang et al., 2005]. Most standard kernels such as the isotropic, dot product (normalized inputs), and normalized kernels satisfy this criterion.

Suppose we replace the mapping ϕ(xn) on xn by another nonlinear mapping ˜ ϕ(zn) on zn such that (for unbiased case)

The mapping is done in a way that that the label information yn is subsumed in the new feature map ˜ ϕ (essentially, converting a supervised learning problem into an unsupervised one). The first term in the mapping corresponds to the feature term and the second term accounts for a regularization effect, where C is the misclassification cost. en is a vector of dimension N, having all entries as zero, except the nth entry which is equal to one.

It was shown in [Tsang et al., 2005] that the MEB instance ( ˜ ϕ(z1), ˜ ϕ(z2), . . . ˜ ϕ(zN)), with the metric defined by the induced inner product, is dual to the corresponding ℓ2-SVM instance (1). The weight vector w of the maximum margin hypothesis can then be obtained from the center c of the MEB using the constraints induced by the Lagrangian [Tsang et al., 2007].

4 Approximate and Streaming MEBs

The minimum enclosing ball problem has been extensively studied in the computational geometry literature. An instance of MEB, with a metric defined by an inner product, can be solved using quadratic programming[Boyd and Vandenberghe, 2004]. However, this becomes prohibitively expensive as the dimensionality and cardinality of the data increases; for an N-point SVM instance in D dimensions, the resulting MEB instance consists of N points in N + D dimensions.

Thus, attention has turned to efficient approximate solutions for the MEB. A δ-approximate solution to the MEB (δ > 1) is a point c such that maxn d(xn, c) ≤ δR∗ , where R∗ is the radius of the true MEB solution. For example, A (1 + ϵ)-approximation for the MEB can be obtained by extracting a very small subset (of size O(1/ϵ)) of the input called a core-set [Agarwal et al., 2005], and running an exact MEB algorithm on this set [B˘adoiu and Clarkson, 2002]. This is the method originally employed in the CVM [Tsang et al., 2005]. [Har-Peled et al., 2007] take a more direct approach, constructing an explicit core set for the (approximate) maximum-margin hyperplane, without relying on the MEB formulation. Both these algorithms take linear training time and require very small storage. Note that a δ-approximation for the MEB directly yields a δ-approximation for the regularized cost function associated with the SVM problem.

Unfortunately, the core-set approach cannot be adapted to a streaming setting, since it requires O(1/ϵ) passes over the training data. Two one-pass streaming algorithms for the MEB problem are known. The first [Agarwal et al., 2004] finds a (1 + ϵ) approximation using O((1/ε)⌊D/2⌋ ) storage and O((1/ε)⌊D/2⌋N) time. Unfortunately, the exponential dependence on D makes this algorithm impractical. At the other end of the space-approximation tradeoff, the second algorithm [Zarrabi-Zadeh and Chan, 2006] stores only the center and the radius of the current ball, requiring O(D) space. This algorithm yields a 3/2-approximation to the optimal enclosing ball radius.

4.1 The StreamSVMAlgorithm

We adapt the algorithm of [Zarrabi-Zadeh and Chan, 2006] for computing an approximate maximum margin classifier. The algorithm initializes with a single point (and therefore an MEB of radius zero). When a new point is read in off the stream, the algorithm checks whether or not the current MEB can enclose this point. If so, the point is discarded. If not, the point is used to suitably update the center and radius of the current MEB. All such selected points define a core set of the original point set.

Let pi be the input point causing an update to the MEB and Bi be the resulting ball after the update. From figure 1, it is easy to verify that the new center ci lies on the line joining the old center ci-1 and the new point pi. The radius ri and the center ci of the resulting MEB can be defined by simple update equations.

Here 2δi = (||pi - ci-1|| - ri-1) is the closest distance of the new point pi from the old ball Bi-1. Using these, we can define a closed-form analytical update equation for the new ball Bi:

It can be shown that, for adversarially constructed data, the radius of the MEB computed by the algorithm has a lowerbound of (1 + √2)/2 and a worst-case upper-bound of 3/2 [Zarrabi-Zadeh and Chan, 2006].

We adapt these updates in a natural way in the augmented feature space ˜ ϕ (see Algorithm 1). Each selected point belongs to the core set for the MEB. The support vectors of the corresponding SVM instance come from this set. It is easy to verify that the update equations for weight vector (w) and the margin (R) in StreamSVM correspond to the center and radius updates for the ball in equation 7 and 4 respectively. The ξ 2 term is the distance calculation is included to account for the fact that the distance computations are being done in the D + N dimensional augmented feature space ˜ ϕ which, for the linear kernel case, is given by:

Also note that, because we perform only a single pass over the data and the en components are all mutually orthogonal, we never need to explicitly store them. The number of updates to the weight vector is limited by the number of core vectors of the MEB, which we have experimentally found to be much smaller as compared to other algorithms (such as Perceptron). The space complexity of StreamSVM is small since only the weight vector and the radius need be stored.

4.2 Kernelized StreamSVM

Although our main exposition and experiments are with linear kernels, it is straightforward to extend the algorithm for nonlinear kernels. In that case, algorithm 1, instead of storing the weight vector w, stores an Ndimensional vector of Lagrange coefficients α initialized as [y1, . . . , 0]. The distance computation is line 5 are replaced by d2 = ∑n,m αnαmk(xn, xm) + k(xn, xn) - 2yn ∑m αmk(xn, xm) + ξ 2 + 1/C, and the weight vector updates in line 7 can be replaced by Lagrange coefficients updates α1:n-1 = α1:n-1(1 - 1 2 (1-R/d)), αn = 12 (1 - R/d) yn.

Algorithm 1 StreamSVM

• 1: Input: examples (xn, yn)n∈1...N, slack parameter C 2: Output: weights (w), radius (R), number of support vectors (M) 3: Initialize: M = 1;R = 0; ξ 2 = 1,w= y1x1 4: for n = 2 to N do 5: Compute distance to center: d = √‖w - ynxn‖2 + ξ2 + 1/C 6: if d ≥ R then 7: w =w+ 12 (1-R/d) (ynxn - w) 8: R = R+ 12 (d-R) 9: ξ2 = ξ2 [1 - 12 (1-R/d)]2 + [ 12 (1 - R/d)]2 10: M =M +1 11: end if 12: end for

Algorithm 2 StreamSVM with lookahead L

Input: examples (xn, yn)n∈1...N, slack parameter C, lookahead parameter L ≥ 1

Output: weights (w), radius (R), upper bound on number of support vectors (M)

• 1: Initialize: M = 1;R = 0; ξ 2 = 1; S = ∅;w = y1x1 2: for n = 2 to N do 3: Compute distance to center: d = √‖w - ynxn‖2 + ξ2 + 1/C 4: if d ≥ R then 5: Add example n to the active set: S = S ∪ {ynxn} 6: if |S| = L then 7: Update w,R, ξ2 to enclose the ball (w,R, ξ2 ) and all points in S 8: M =M +L ; S = ∅ 9: end if 10: end if 11: end for 12: if |S| > 0 then 13: Update w,R, ξ2 to enclose the ball (w,R, ξ2 ) and all points in S 14: M =M + |S| 15: end if

4.3 StreamSVMapproximation bounds and extension to multiple balls

It was shown in [Zarrabi-Zadeh and Chan, 2006] that any streaming MEB algorithm that uses only O(D) storage obtains a lower-bound of (1 + √2)/2 and an upper-bound of 3/2 on the quality of solution (i.e., the radius of final MEB). Clearly, this is a conservative approximation and would affect the obtained margin of the resulting SVM classifier (and hence the classification performance). In order to do better in just a single pass, one possible conjecture could be that the algorithm must remember more. To this end, we therefore extended algorithm-1 to simultaneously store L weight vectors (or "balls"). The space complexity of this algorithm is L(D + 1) floats and it still makes only a single pass over the data. In the MEB setting, our algorithm chooses with each arriving datapoint (that is not already enclosed in any of the balls) how the current L + 1 balls (the L balls plus the new data point) should be merged, resulting again into a set of L balls. At the end, the final set of L balls are merged together to give the final MEB. A special variant of the L balls case is when all but one of the L balls are of zero radius. This amounts to storing a ball of non-zero radius and to keeping a buffer of L many data-points (we call this the lookahead algorithm - Algorithm 2). Any incoming point, if not already enclosed in the current ball, is stored in the buffer. We solve the MEB problem (using a quadratic program of size L) whenever the buffer is full. Note that algorithm 1 is a special case of algorithm 2 with L=1, with the MEB updates available in a closed analytical form (rather than having to solve a QP).

Algorithm 1 takes linear time in terms of the input size. Algorithm 2 which uses a lookahead of L solves a quadratic program of size L whenever the buffer gets full. This step takes O(L3 ) times. The number of such updates is O(N/L) (in practice, it is considerably less than N/L) and thus the over all complexity for the lookahead case is O(NL2 ). For small lookaheads, this is roughly O(N).

5 Experiments

We evaluate our algorithm on several synthetic and real datasets and compare it against several state-of-the-art SVM solvers. We use 3 crieria for evaluations: a) Single-pass classification accuracies compared against single-pass of online SVM solvers such as iterative sub-gradient solver Pegasos [Shalev-Shwartz et al., 2007], LASVM [Bordes et al., 2005], and Perceptron [Rosenblatt, 1988]. b) Comparison with CVM [Tsang et al., 2005] which is a batch SVM algorithm based on the MEB formulation. c) Effect of using lookahead in StreamSVM. For fairness, all the algorithms used a linear kernel.

5.1 Single-Pass Classification Accuracies

The single-pass classification accuracies of StreamSVM and other online SVM solvers are shown in table-1. Details of the datasets used are shown in table-1. To get a sense of how good the single-pass approximation of our algorithm is, we also report the classification accuracies of batch-mode (i.e., all data in memory, and multiple passes) libSVM solver with linear kernel on all the datasets. The results suggest that our single-pass algorithm StreamSVM, using a small reasonable lookahead, performs comparably to the batch-mode libSVM, and does significantly better than a single pass of other online SVM solvers.

5.2 Comparison with CVM

We compared our algorithm with CVM which, like our algorithm, is based on a MEB formulation. CVM is highly efficient for large datasets but it operates in batch mode, making one pass through the data for each core vector. We are interested in knowing how many passes the CVM must make over the data before it achieves an accuracy comparable to our streaming algorithm. For that purpose, we compared the accuracy of our single-pass StreamSVM against two and more passes of CVM to see how long does it take for CVM to beat StreamSVM (we note here that CVM requires at least two passes over the data to return a solution). We used a linear kernel for both. Shown in Figure 2 are the results on MNIST 8vs9 data and it turns out that it takes several hundreds of passes of CVM to beat the single pass accuracy of StreamSVM. Similar results were obtained for other datasets but we do not report them here due to space limitations.

5.3 Effect of Lookahead

We also investigated the effect of doing higher-order lookaheads on the data. For this, we varied L (the lookahead parameter) and, for each L, tested Algorithm 2 on 100 random permutations of the data stream order, also recording the standard deviation of the classification accuracies with respect to the data-order permutations. Note that the algorithm still performs a single pass over the data. Figure 3 shows the results on the MNIST 8vs9 data (similar results were obtained for other datasets but not shown due to space limitations). In this figure, we see two effects. Firstly, as the lookahead increase, performance goes up. This is to be expected since in the limit, as the lookahead approaches the data set size, we will solve the exact MEB problem (albeit at a high computational cost). The important thing to note here is that even with a small lookahead of 10, the performance converges. Secondly, we see that the standard deviation of the result decreases as the lookahead increases. This shows experimentally that higher lookaheads make the algorithm less susceptible to badly ordered data. This is interesting from an empirical perspective, given that we can show that in theory, any value of L < N cannot improve upon the 3/2-approximation guaranteed for L = 1.

6 Analysis, Open Problems, and Extensions

There are several open problems that this work brings up:

1. Are the (1 + √2)/2 lower-bound and the 3/2 upperbound on MEB radius indeed the best achievable in a single pass over the data?
2. Is it possible to use a richer geometric structure instead of a ball and come up with streaming variants with provably good approximation bounds?

We discuss these in some more detail here.

6.1 Improving the Theoretical Bounds

One might conjecture that storing more information (i.e., more points) would give better approximation guarantees in the streaming setting. Although the empirical results showed that such approaches do result in better classification accuracies, this is not theoretically true in many cases.

For instance, in the adversarial stream setting, one can show that neither the lookahead algorithm nor its more general case (the multiple balls algorithm) improves the bounds given by the simple no-lookahead case (Algorithm-1). In particular, one can prove an identical upper- and lower-bound for the lookahead algorithm as for the no-lookahead algorithm. To obtain the 3/2-upper bound result, one can show a nearly identical construction as to [Zarrabi-Zadeh and Chan, 2006] where L - 1 points are packed in a small, carefully constructed cloud the boundary of the true MEB.

Alternatively, one can analyze these algorithms in the random stream setting. Here, the input points are chosen adversarially, but their order is permuted randomly. The lookahead model is not strengthened in this setting either: we can show both that the lower bound for no-lookahead algorithms, as well as the 3/2-upper bound for the specific no-lookahead algorithm described, generalize. For the former, see Figure 4. We place (N - 1)/2 points around (0, 1) and (N - 1)/2 points around (0,-1) and one point at (1 + √2, 0). The algorithm will only beat the (1 + √2)/2 lower bound if the singleton appears in the first L points, where L is the lookahead used. Assuming the lookahead is polylogarithmic in N (which must be true for a streaming algorithm), this means that as N -→∞, the probability of a better bound tends toward zero. Note, however, that this applies only to the lookahead model, not to the more general multiple balls model, where it may be possible to obtain a tighter bounds in the random stream setting.

6.2 Ellipsoidal Balls

Instead of using a minimum enclosing ball of points, an alternative could be to use a minimum volume ellipsoid (MVE)

[Kumar et al., 2005]. An ellipsoid in RD is defined as follows: {x : (x - c)′A(x - c) <= 1} where c ∈ RD , A ∈ RDxD , and A ⪰ 0 (positive semi-definite).

Note that a ball, upon inclusion of a new point, expands equally in all dimensions which may be unnecessary. On the other hand, an ellipsoid can have several axes and scales of variations (modulated by the covariance matrix A). This allows the ellipsoid to expand only along those directions where needed. In addition, such an approach can also be seen along the lines of confidence weighted linear classifiers [Dredze et al., 2008]. The confidence weighted (CW) method assumes a Gaussian distribution over the space of weight vectors and updates the mean and covariance parameters upon witnessing each incoming example. Just as CW maintains the models uncertainty using a Gaussian, an ellipsoid generaization can model the uncertainty using the covariance matrix A. Recent work has shown that there exist streaming possibilities for MVE [Mukhopadhyay and Greene, 2008]. The approximation gaurantees, however, are very conservative. It would be interesting to come up with improved streaming algorithms for the MVE case and adapt them for classification settings.

7 Conclusion

Within the streaming framework for learning, we have presented an efficient, single-pass ℓ2-SVM learning algorithm using a streaming algorithm for the minimum enclosing ball problem. We have also extended this algorithm to use a lookahead to increase robustness against poorly ordered data. Our algorithm, StreamSVM, satisfies a proven theoretical bound: it provides a ( 3 2 )-approximation to the optimal solution. Despite this conservative bound, our algorithm is experimentally competitive with alternative techniques in terms of accuracy, and learns much simpler solutions. We believe that a careful study of stream-based learning would lead to high quality scalable solutions for other classification problems, possibly with alternative losses and with tighter approximation bounds.