Feature Mining and Neuro-Fuzzy Inference System for Steganalysis of LSB Matching Stegangoraphy in Grayscale Images

1 Introduction

Steganalysis is the science and art of detecting the presence of hidden data in digital images, audios, videos and other media. In steganography or the hiding of secret data in digital media, the most common cover is digital images. To this date, many steganographical or embedding methods, such as LSB embedding, spread spectrum steganography, F5 algorithm and some other JPEG steganography, have been very successfully steganalyzed [Fridrich et al., 2003; Ker, 2005a; Fridrich et al. 2002; Harmsen and Pearlman 2003; Choubassi and Moulin 2005; Liu et al., 2006a]. Several other embedding paradigms, include stochastic modulation [Fridrich and Goljan, 2003; Moulin and Briassouli, 2004] and LSB matching [Sharp 2001], however, are much more difficult to detect.

.notdef Support for this research was received from ICASA (of New Mexico Tech) and a DoD IASP Capacity Building grant

The literature does provide a few detectors for LSB matching steganography. One of the first papers on detection of embedding by noise adding is the paper by Harmsen and Pearlman [Harmsen and Pearlman, 2003], wherein the measure of histogram characteristic function center of mass (HCFCOM), is extracted and a Bayesian multivariate classifier is applied. Based on the contribution of Harmsen and Pearlman [2003], Ker [2005b] proposes two novel ways of applying the HCF: calibrating the output using a down-sampled image and computing the adjacency histogram instead of the usual histogram. The best discriminators are Adjacency HCFCOM (A.HCFCOM) and Calibrated Adjacency HCFCOM (C.A.HCFCOM) to improve the probability of detection for LSB matching in grayscale images. Farid and Lyu describe an approach to detecting hidden messages in images by using a wavelet-like decomposition to build high-order statistical models of natural images [Lyu and Farid, 2004 and 2005]. Fridrich et al. [2005] propose a Maximum Likelihood (ML) estimator for estimating the number of embedding changes for non-adaptive ±K embedding in images. Based on the stego-signal estimation, Holotyak et al. [2005] present a blind steganalysis classifying on high order statistics of the estimation signal.

Unfortunately, the publications mentioned above did not fully address the issue of image complexity that is very important in evaluating the detection performance (though Fridrich et al. [2005] report that the ML estimator starts to fail to reliably estimate the message length once the variance of sample exceeds 9, indicating that the detection performance decreases with the increase in the image complexity).

Recently, the shape parameter of Generalized Gaussian Distribution (GGD) in the wavelet domain is introduced by the authors to measure the image complexity in steganalysis [Liu et al., 2006b]; although the method proposed therein is successful in detecting LSB matching steganography in color images and outperforms other well-known methods, its performance is not so good in grayscale images, which is generally more difficult.

On the other side, many steganalysis methods are based on feature mining and machine learning. In feature mining, besides feature extraction, another general problem is how to choose the good measures from the extracted features. Avcibas et al. [2003] propose a steganalysis using image quality metrics. In their method, they apply analysis of variance (ANOVA) to feature selection, the higher the F statistic, the lower the p value, and the better the feature will be. Essentially, the ANOVA applied by Avcibas et al. [2003] is significancebased feature selection like other statistics such as T-test, etc. But these statistics just consider the significance of individual feature, not the interaction of features. There has been little research that addresses in depth the feature selection problem with specific respect to steganalysis.

In this paper, we propose four types of features and a Dynamic Evolving Neural Fuzzy Inference System [Kasabov and Song, 2002; Kasabov, 2002] (DENFIS)- based feature selection to the steganalysis of LSB matching steganography in grayscale images. The four types of features consist of the shape parameter of GGD in the wavelet domain to measure the image complexity, the entropy and the high order statistics in the histogram of the nearest neighbors, and correlation features. We also adopt the well-known gene selection method of Support Vector Machine Recursive Feature Elimination (SVMRFE) [Guyon et al., 2002] for choosing good measures in steganalysis.

Comparing against other well-known methods in terms of steganalysis performance, our feature set, overall, performs the best. DENFIS outperforms some traditional learning classifiers. SVM-RFE and DENFIS-based feature selection outperform statistical significance based feature selection such as T-test in steganalysis.

2 Feature Mining

2.1 Image Complexity

Several papers [Srivastava et al., 2003; Winkler, 1996; Sharifi and Leon-Garcia, 1995] describe the statistical models of images such as probability models for images based on Markov Random Field models (MRFs), Gaussian Mixture Model (GMM) and GGD model in transform domains, such as DCT, DFT, or DWT.

Experiments show that a good Probability Density Function (PDF) approximation for the marginal density of coefficients at a particular subband produced by various types of wavelet transforms may be achieved by adaptively varying two parameters of the GGD [Sharifi and Leon-Garcia, 1995; Moulin and Liu, 1999], which is defined as

Where 1 0 ( ) , 0 t z z e t dt z .notdef .notdef .notdef .notdef = > .notdef is the Gamma function.

Here .notdef models the width of the PDF peak, while .notdef is inversely proportional to the decreasing rate of the peak; .notdef is referred to as the scale parameter and .notdef is called the shape parameter. The GGD model contains the Gaussian and Laplacian PDFs as special cases, using .notdef = 2 and .notdef = 1, respectively.

2.2 Entropy and High order Statistics of the Histogram of the Nearest Neighbors

There is evidence that adjacent pixels in ordinary images are highly correlated [Huang and Mumford, 1999; Liu et al., 2006a]. Consider the histogram of the nearest neighbors, denote the grayscale value at the point (i, j) as x, the grayscale value at the nearest point (i+1, j) in the horizontal direction as y, and the grayscale value at the nearest point (i, j+1) in the vertical direction as z. The variable H(x, y, z) denotes the occurrence of the pair (x, y, z), or the histogram of the nearest neighbors (NNH).

The entropy of NNH (NNH_E) is calculated as follows:

Where .notdef denotes the distribution density of the NNH.

The symbol H .notdef denotes the standard deviation of H. The r th high order statistics of NNH is given as:

Where N is the number of possible gray scales of the image, e.g., for 8-bit grayscale image, N=256.

2.3 Correlation Features

The following three correlation features are extracted.

1. The correlation between the Least Significant Bit Plane (LSBP) and the second Least Significant Bit Plane (LSBP2) and the autocorrelation in the LSBP: M1(1:m, 1:n) denotes the binary bits of the LSBP and M2(1:m, 1:n) denotes the binary bits of the LSBP2.

where 1 2 2 2 1 2 ( ) and ( ) . M M Var M Var M .notdef .notdef = =

C(k, l), the autocorrelation of LSBP is defined as:

2. The autocorrelation in the image histogram: The histogram probability density is denoted as (.notdef0, .notdef1, .notdef2….notdefN-1). The histogram probability densities, He, Ho, Hl1, and Hl2 are denoted as follows:

The autocorrelation coefficients C2 and CH(l), where l is the lag distance, are defined as follows:

3. The correlation in the difference between the image and the denoised version: The original cover is denoted as F, the stego-image is denoted as F´, D (·) denotes some denoising function, the differences between the image and the denoised are:

Generally, the statistics of EF and EF´ are different. The correlation features in the difference domain are extracted as follows. Firstly, the test image is decomposed by wavelet transform. Find the coefficients in HL, LH and HH subbands with the absolute value smaller than the threshold value, t, set these coefficients to zero, and reconstruct the image using the inverse wavelet transform on the updated wavelet coefficients. The reconstructed image is treated as denoised image. The difference between test image and reconstructed version is Et, where t is the threshold value.

where , , (1: ,1: ); ( 1: , 1: ). t k t t l t E E m k n l E E k m l n = .notdef .notdef = + + The variables k and l denote the lag distance.

3 Neuro-Fuzzy Inference System Based Feature Selection

Neuro-fuzzy inference systems and evolving neuro-fuzzy inference systems are introduced in [Kasabov, 2002]. The dynamic evolving neuro-fuzzy system (DENFIS) proposed by [Kasabov and Song, 2002] uses the TakagiSugeno type of fuzzy inference method [Takagi and Sugeno, 1985].

To improve the detection performance, based on our previous work [Liu and Sung, 2006], we propose a feature selection method based on the DENFIS supervised learning, described as follows:

1. Each individual feature is ranked in the order from the highest train accuracy to the lowest train accuracy with the use of DENFIS.
2. The feature with the highest train accuracy is chosen as the first feature. After this step, the chosen feature set, F1, consists of the best feature, e1, corresponding to feature dimension one.
3. The (N+1) st feature set, FN+1 = {e1, e2 ,…, eN , eN+1} is produced by adding eN+1 into the present N-dimensional feature set, FN = {e1, e2, …, eN} according to the following method: Each feature ei (i .notdef 1, 2, …, N) outside of FN is added into FN; the classification accuracy of each feature set FN + {ei} is compared, the ec with the highest train accuracy is put into the set of candidates, C. The candidate set C generally includes multiple features, but only one feature will be chosen. The strategy is to measure the similarity of chosen features and each of the candidates. Pearson's correlation between the candidate ec, ec .notdef C and the element ei, ei .notdef FN (i = 1, 2… N) is calculated. To measure the similarity, the sum of the square of the correlation (SC) is defined as follows:

Where, ec .notdef C, ei .notdef FN (i = 1, 2… N). The ec with the minimal value of SC(ec) is chosen as eN+1. We call this feature selection DENFIS-MSC (for Minimum of SC).

4 Experiments

4.1 Experimental Setup

The original images in our experiments are 5000 TIFF raw format digital pictures from Olympus C740. These images are 24-bit, 640.notdef480 pixels, lossless true color and never compressed. According to the method in [Lyu and Farid, 2004 and 2005], we cropped the original images into 256.notdef256 pixels in order to get rid of the low complexity parts of the images. The cropped color images are converted into grayscales and we hid data in these grayscales with different hiding ratio. LSB matching stego-images are produced. The hidden data in different covers are different. The hiding ratio is 12.5%.

4.2 Feature Extraction and Comparison

The following features are extracted:

1. Shape parameter .notdef of GGD of HH wavelet subband to measure image complexity, described in (1).
2. Entropy of the histogram of the nearest neighbors, NNH_E, defined in (2)
3. The high order statistics of the histogram of the nearest neighbors, NNH_HOS(r) in (3). r is set from 3 to 22, total 20 high order statistics.
4. Correlations features consist of C1 in (4), C(k,l) in (5), C2 in (6), CH(l) in (7), and CE(t; k,l).

We set the following lag distance to k and l in C(k,l) and get 14 features:

1. k = 0, l = 1, 2, 3, and 4; l = 0, k = 1, 2, 3, and 4.
2. k = 1, l = 1; k = 2, l =2; k=3, l = 3; k = 4 and l = 4.
3. k = 1, l =2; k=2, l =1.
4. In (7), l is set to 1, 2, 3, and 4. In (8), we set the following lag distance to k and l in CE(t; k,l) and get following pairs: CE(t; 0,1), CE(t; 0,2), CE(t;1,0), CE(t; 2,0), CE(t; 1,1), CE(t; 1,2), and CE(t; 2,1). t is set 1, 1.5, 2, 2.5, 3, 3.5, 4, 4.5, and 5.

Henceforth, we use CF to denote the fourth type of correlation features; and use EHCC (for Entropy, High order statistics, Complexity, and Correlation features) to denote types 1 to 4 features.

To compare EHCC with other well-known features, the Histogram Characteristic Function Center of Mass (HCFCOM) features [Harmsen and Pearlman, 2003] are extracted because the hiding process of LSB matching steganography can be modeled in the context of additive noise. We extend HCFCOM feature set to the high order moments. HCFHOM stands for HCF center of mass High Order Moments; HCFHOM(r) denotes the r th order statistics. In our experiments, the HCFHOM feature set consists of HCFCOM and HCFHOM(r) (r = 2, 3, and 4). Based on Harmsen and Pearlman's work, Ker [2005b] proposed A.HCFCOM and C.A.HCFCOM. Additionally, Farid and Lyu [2004, 2005] described an approach to detecting hidden messages in images by building HighOrder Moment statistics in Multi-Scale decomposition domain (we call the features HOMMS), which consists of 72-dimension features in grayscale images.

4.3 Detection Performance on Feature Sets

To compare the detection performances on these feature sets with different classifiers, Besides DENFIS, we apply the following classifiers to each feature sets. These classifiers are Naive Bayes Classifier (NBC), Support Vector Machines (SVM), Quadratic Bayes Normal Classifier (QDC), Nearest Mean Scaled Classifier (NMSC), K-Nearest Neighbor classifier (KNN) and Adaboost that produces a classifier composed from a set of weak rules [Vapnik, 1998; Schlesinger and Hlavac, 2002; Heijden et al., 2004; Webb, 2002; Schapire and Singer, 1999; Friedman et al., 2000].

Thirty experiments are done on each feature set using each classifier. Training samples are chosen at random and the remaining samples are for test. The ratio of training samples to test samples is 2:3. The average classification accuracy is compared.

Table 1 compares the detection performances (mean values and standard deviations) on each feature set with the use of different classifiers. In each category of image complexity, the highest test accuracy is in bold. Table 1 indicates that, regarding the classification accuracy of feature sets, on the average, EHCC performs best, followed by CF, HOMMS performs worst; regarding the classification performance of classifiers, SVM is the best; regarding image complexity, the test accuracy decreases while the image complexity increases. In the low complexity of .notdef < 0.4, the highest test accuracy is 86.9%; in the high complexity of .notdef > 1, the highest test accuracy is 62.9%. It shows that image complexity is an important factor to the detection performance.

To obtain a higher detection performance, we combine EHCC, HCFCOM, HOMMS, C.A. HCFCOM, and A. HCFCOM features, and apply three feature section methods, SVM-RFE [Guyon et al., 2002], DENFIS-MSC, and T-test (here we apply T-test instead of ANOVA because cover samples and steganography samples are unpaired in each category of image complexity) to the features, and compare different classifiers and the three feature selections in the feature dimensions 1 to 40.

Fig. 1 compares the detection performance on the SVM-RFE feature set with the use of DENFIS, SVM, NBC, NMSC, and KNN. It indicates that in the low image complexity, DENFIS and SVM are the best; in the mediate and high image complexity, DENFIS is the best.

Fig. 1 also shows that the detection performance decreases as the image complexity increases.

Fig. 2 compares the three feature selections in the image complexity of .notdef < 0.4 with the use of SVM, DENFIS, and NMSC. It indicates that the feature selections SVM-RFE and DENFIS-MSC are better than T-test. Furthermore, applying SVM and DENFIS to SVM-RFE and DENFIS-MSC feature sets, the test accuracies are better than the highest value of table 1. Due to the page limit, we don't list comparison of the feature selections in the high image complexity. Our experiments also indicate that SVM-RFE and DENFISMSC outperform T-test.

5 Conclusions

In this paper, we present a scheme of steganalysis of LSB matching steganography in grayscale images based on feature mining and neuro-fuzzy inference system. Four types of features are extracted, a DENFIS-based feature selection is used, and SVM-RFE is used as well to obtain better detection accuracy. In comparison with other features of HCFHOM, HOMMS, A.HCFCOM, and C.A.HCFCOM, overall, our features perform the best. DENFIS outperforms some traditional learning classifiers. SVM-RFE and DENFIS based feature selection outperform statistical significance based feature selection such as T-test. Experimental results also indicate that it is still very challenging for the steganalysis of LSB matching steganography in grayscale images with high complexity.