{"type":"ScholarlyArticle","title":"Cuckoo Feature Hashing: Dynamic Weight Sharing for Sparse Analytics","authors":[{"name":"Jinyang Gao","slug":"jinyang-gao","given":"Jinyang","family":"Gao","affiliation":"National University of Singapore"},{"name":"Beng Chin Ooi","slug":"beng-chin-ooi","given":"Beng Chin","family":"Ooi","affiliation":"National University of Singapore"},{"name":"Yanyan Shen","slug":"yanyan-shen","given":"Yanyan","family":"Shen","affiliation":"Shanghai Jiao Tong University"},{"name":"Wang-Chien Lee","slug":"wang-chien-lee","given":"Wang-Chien","family":"Lee","affiliation":"The Pennsylvania State University"}],"abstract":"Feature hashing is widely used to process large scale sparse features for learning of predictive models. Collisions inherently happen in the hashing process and hurt the model performance. In this paper, we develop a feature hashing scheme called Cuckoo Feature Hashing(CCFH) based on the principle behind Cuckoo hashing, a hashing scheme designed to resolve collisions. By providing multiple possible hash locations for each feature, CCFH prevents the collisions between predictive features by dynamically hashing them into alternative locations during model training. Experimental results on prediction tasks with hundred-millions of features demonstrate that CCFH can achieve the same level of performance by using only 15%-25% parameters compared with conventional feature hashing.","doi":"10.24963/ijcai.2018/295","url":"https://ijcai.org/proceedings/2018/295","pdf":"https://ijcai.org/proceedings/2018/0295.pdf","markdown":"https://ijcai.org/proceedings/2018/295.md","bibtex":"https://ijcai.org/proceedings/2018/bibtex/295","volume":{"slug":"2018","title":"Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2018","url":"https://ijcai.org/proceedings/2018/","year":2018,"location":"Stockholm","dates":"13-19 July 2018","editors":"Jérôme Lang","isbn":"978-0-9992411-2-7","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2018"},"section":"Machine Learning","track":"Main track","number":295,"pages":{"first":2135,"last":2141},"keywords":["Machine Learning: Feature Selection ; Learning Sparse Models","Machine Learning: Dimensionality Reduction and Manifold Learning","Computer Vision: Big Data and Large Scale Methods"],"videos":[],"references":[{"position":1,"raw":"[Attenberg et al., 2009] Josh Attenberg, Kilian Weinberger, Anirban Dasgupta, Alex Smola, and Martin Zinkevich. Collaborative email-spam filtering with the hashing trick. In CEAS, 2009.","title":"Collaborative email-spam filtering with the hashing trick","authors":"Josh Attenberg, Kilian Weinberger, Anirban Dasgupta, Alex Smola, Martin Zinkevich","year":"2009","venue":"CEAS","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Caragea et al., 2012] Cornelia Caragea, Adrian Silvescu, and Prasenjit Mitra. Protein sequence classification using feature hashing. Proteome science, 2012.","title":"Protein sequence classification using feature hashing","authors":"Cornelia Caragea, Adrian Silvescu, Prasenjit Mitra","year":"2012","venue":"Proteome science","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Chen et al., 2015] Wenlin Chen, James T Wilson, Stephen Tyree, Kilian Q Weinberger, and Yixin Chen. Compress- ing neural networks with the hashing trick. In ICML, 2015.","title":"Compressing neural networks with the hashing trick","authors":"Wenlin Chen, James T Wilson, Stephen Tyree, Kilian Q Weinberger, Yixin Chen","year":"2015","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Cunningham and Ghahramani, 2015] John P Cunningham and Zoubin Ghahramani. Linear dimensionality reduction: survey, insights, and generalizations. JMLR, 2015.","title":"Linear dimensionality reduction: survey, insights, and generalizations","authors":"P John, Zoubin Cunningham, Ghahramani","year":"2015","venue":"JMLR","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[He and Chua, 2017] Xiangnan He and Tat-Seng Chua. Neu- ral factorization machines for sparse predictive analytics. In SIGIR, 2017.","title":"Xiangnan He and Tat-Seng Chua. Neural factorization machines for sparse predictive analytics","authors":"Chua He","year":"2017","venue":"SIGIR","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[He and Niyogi, 2004] Xiaofei He and Partha Niyogi. Lo- cality preserving projections. In NIPS, 2004.","title":"Locality preserving projections","authors":"Xiaofei He, Partha Niyogi","year":"2004","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Ihaka and Gentleman, 1996] Ross Ihaka and Robert Gentle- man. R: a language for data analysis and graphics. JCGS, 1996.","title":"R: a language for data analysis and graphics","authors":"Ross Ihaka, Robert Gentleman","year":"1996","venue":"JCGS","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Jang et al., 2011] Jiyong Jang, David Brumley, and Shobha Venkataraman. Bitshred: feature hashing malware for scalable triage and semantic analysis. In CCS, 2011.","title":"Bitshred: feature hashing malware for scalable triage and semantic analysis","authors":"Jang","year":"2011","venue":"CCS","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Juan et al., 2016] Yuchin Juan, Yong Zhuang, Wei-Sheng Chin, and Chih-Jen Lin. Field-aware factorization ma- chines for ctr prediction. In RecSys, 2016.","title":"Field-aware factorization machines for ctr prediction","authors":"Juan","year":"2016","venue":"RecSys","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Kingma and Ba, 2014] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.","title":"Adam: A method for stochastic optimization","authors":"P Diederik, Jimmy Kingma, Ba","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Kutzelnigg, 2006] Reinhard Kutzelnigg. Bipartite random graphs and cuckoo hashing. In Fourth Colloquium on Mathematics and Computer Science Algorithms, Trees, Combinatorics and Probabilities. Discrete Mathematics and Theoretical Computer Science, 2006.","title":"Bipartite random graphs and cuckoo hashing","authors":"Reinhard Kutzelnigg","year":"2006","venue":"Fourth Colloquium on Mathematics and Computer Science Algorithms, Trees, Combinatorics and Probabilities. Discrete Mathematics and Theoretical Computer Science","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Ma et al., 2009] Justin Ma, Lawrence K Saul, Stefan Sav- age, and Geoffrey M Voelker. Identifying suspicious urls: an application of large-scale online learning. In ICML, 2009.","title":"Identifying suspicious urls: an application of large-scale online learning","authors":"Justin Ma, Lawrence K Saul, Stefan Savage, Geoffrey M Voelker","year":"2009","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Mahout, 2012] Apache Mahout. Scalable machine learning and data mining, 2012.","title":"Apache Mahout. Scalable machine learning and data mining","authors":"Mahout","year":"2012","venue":null,"doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Meng et al., 2016] Xiangrui Meng, Joseph Bradley, Burak Yavuz, Evan Sparks, Shivaram Venkataraman, Davies Liu, Jeremy Freeman, DB Tsai, Manish Amde, and Sean Owen. Mllib: Machine learning in apache spark. JMLR, 2016.","title":"Mllib: Machine learning in apache spark","authors":"Meng","year":"2016","venue":"JMLR","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Mika et al., 1999] Sebastian Mika, Gunnar Ratsch, Jason Weston, Bernhard Scholkopf, and Klaus-Robert Mullers. Fisher discriminant analysis with kernels. In Neural Net- works for Signal Processing, 1999.","title":"Fisher discriminant analysis with kernels","authors":"Sebastian Mika, Gunnar Ratsch, Jason Weston, Bernhard Scholkopf, Klaus-Robert Mullers","year":"1999","venue":"Neural Networks for Signal Processing","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Pagh and Rodler, 2001] Rasmus Pagh and Flemming Friche Rodler. Cuckoo hashing. Springer, 2001.","title":"Rasmus Pagh and Flemming Friche Rodler. Cuckoo hashing","authors":"Rodler Pagh","year":"2001","venue":null,"doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Pedregosa et al., 2011] Fabian Pedregosa, Gaël Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg, et al. Scikit-learn: Machine learning in python. JMLR, 2011.","title":"Scikit-learn: Machine learning in python","authors":"Fabian Pedregosa, Gaël Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg","year":"2011","venue":"JMLR","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Richardson et al., 2007] Matthew Richardson, Ewa Domi- nowska, and Robert Ragno. Predicting clicks: estimating the click-through rate for new ads. In WWW, 2007.","title":"Predicting clicks: estimating the click-through rate for new ads","authors":"Matthew Richardson, Ewa Dominowska, Robert Ragno","year":"2007","venue":"WWW","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Shan et al., 2016] Ying Shan, T Ryan Hoens, Jian Jiao, Hai- jing Wang, Dong Yu, and JC Mao. Deep crossing: Web- scale modeling without manually crafted combinatorial features. In SIGKDD, 2016.","title":"Deep crossing: Webscale modeling without manually crafted combinatorial features","authors":"Ying Shan, Ryan Hoens, Jian Jiao, Haijing Wang, Dong Yu, J C Mao","year":"2016","venue":"SIGKDD","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Shepitsen et al., 2008] Andriy Shepitsen, Jonathan Gem- mell, Bamshad Mobasher, and Robin Burke. Personalized recommendation in social tagging systems using hierarchi- cal clustering. In RecSys, 2008.","title":"Personalized recommendation in social tagging systems using hierarchical clustering","authors":"Andriy Shepitsen, Jonathan Gemmell, Bamshad Mobasher, Robin Burke","year":"2008","venue":"RecSys","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Shi et al., 2009] Qinfeng Shi, James Petterson, Gideon Dror, John Langford, Alex Smola, and SVN Vish- wanathan. Hash kernels for structured data. JMLR, 2009.","title":"Hash kernels for structured data","authors":"Shi","year":"2009","venue":"JMLR","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Song et al., 2011] Jingkuan Song, Yi Yang, Zi Huang, Heng Tao Shen, and Richang Hong. Multiple feature hash- ing for real-time large scale near-duplicate video retrieval. In ACM MM, 2011.","title":"Multiple feature hashing for real-time large scale near-duplicate video retrieval","authors":"Song","year":"2011","venue":"ACM MM","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Wall et al., 2003] Michael E Wall, Andreas Rechtsteiner, and Luis M Rocha. Singular value decomposition and principal component analysis. In A practical approach to microarray data analysis. 2003.","title":"Singular value decomposition and principal component analysis. In A practical approach to microarray data analysis","authors":"Andreas Michael E Wall, Luis M Rechtsteiner, Rocha","year":"2003","venue":null,"doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Weinberger et al., 2009] Kilian Weinberger, Anirban Das- gupta, John Langford, Alex Smola, and Josh Attenberg. Feature hashing for large scale multitask learning. In ICML, 2009.","title":"Feature hashing for large scale multitask learning","authors":"Kilian Weinberger, Anirban Dasgupta, John Langford, Alex Smola, Josh Attenberg","year":"2009","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Yan et al., 2007] Shuicheng Yan, Dong Xu, Benyu Zhang, Hong-Jiang Zhang, Qiang Yang, and Stephen Lin. Graph embedding and extensions: A general framework for di- mensionality reduction. TPAMI, 2007.","title":"Graph embedding and extensions: A general framework for dimensionality reduction","authors":"Shuicheng Yan, Dong Xu, Benyu Zhang, Hong-Jiang Zhang, Qiang Yang, Stephen Lin","year":"2007","venue":"TPAMI","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Zaharia et al., 2010] Matei Zaharia, Mosharaf Chowdhury, Michael J Franklin, Scott Shenker, and Ion Stoica. Spark: Cluster computing with working sets. HotCloud, 2010.","title":"Spark: Cluster computing with working sets","authors":"Matei Zaharia, Mosharaf Chowdhury, J Michael, Scott Franklin, Ion Shenker, Stoica","year":"2010","venue":null,"doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Zhou et al., 2015] Li Zhou, David G Andersen, Mu Li, and Alexander J Smola. Cuckoo linear algebra. In SIGKDD, 2015.","title":"Cuckoo linear algebra","authors":"Li Zhou, Mu David G Andersen, Alexander J Li, Smola","year":"2015","venue":"SIGKDD","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}