{"type":"ScholarlyArticle","title":"Deterministic Binary Filters for Convolutional Neural Networks","authors":[{"name":"Vincent W.-S. Tseng","slug":"vincent-w-s-tseng","given":"Vincent W.-S.","family":"Tseng","affiliation":"University of Oxford"},{"name":"Sourav Bhattacharya","slug":"sourav-bhattacharya","given":"Sourav","family":"Bhattacharya","affiliation":"† Cornell University Nokia Bell Labs *"},{"name":"Javier Fernández Marqués","slug":"javier-fernandez-marques","given":"Javier Fernández","family":"Marqués","affiliation":"University of Oxford"},{"name":"Milad Alizadeh","slug":"milad-alizadeh","given":"Milad","family":"Alizadeh","affiliation":"University of Oxford"},{"name":"Catherine Tong","slug":"catherine-tong","given":"Catherine","family":"Tong","affiliation":"University of Oxford"},{"name":"Nicholas D. Lane","slug":"nicholas-d-lane","given":"Nicholas D.","family":"Lane","affiliation":"† Cornell University Nokia Bell Labs *"}],"abstract":"We propose Deterministic Binary Filters, an approach to Convolutional Neural Networks that learns weighting coefficients of predefined orthogonal binary basis instead of the conventional approach of learning directly the convolutional filters. This approach results in model architectures with significantly fewer parameters (4x to 16x) and smaller model sizes (32x due to the use of binary rather than floating point precision). We show our deterministic filter design can be integrated into well-known network architectures (such as ResNet and SqueezeNet) with as little as 2% loss of accuracy (under datasets like CIFAR-10). Under ImageNet, they result in 3x model size reduction compared to sub-megabyte binary networks while reaching comparable accuracy levels.","doi":"10.24963/ijcai.2018/380","url":"https://ijcai.org/proceedings/2018/380","pdf":"https://ijcai.org/proceedings/2018/0380.pdf","markdown":"https://ijcai.org/proceedings/2018/380.md","bibtex":"https://ijcai.org/proceedings/2018/bibtex/380","volume":{"slug":"2018","title":"Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2018","url":"https://ijcai.org/proceedings/2018/","year":2018,"location":"Stockholm","dates":"13-19 July 2018","editors":"Jérôme Lang","isbn":"978-0-9992411-2-7","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2018"},"section":"Machine Learning","track":"Main track","number":380,"pages":{"first":2739,"last":2747},"keywords":["Machine Learning: Deep Learning","Computer Vision: Computer Vision"],"videos":[],"references":[{"position":1,"raw":"[Adachi et al., 1997] F. Adachi, M. Sawahashi, and K. Okawa. Tree-structured generation of orthogonal spreading codes with different lengths for forward link of ds-cdma mobile radio. Electronics Letters, 33(1):27-28, Jan 1997.","title":"Tree-structured generation of orthogonal spreading codes with different lengths for forward link of ds-cdma mobile radio","authors":"F Adachi, M Sawahashi, K Okawa","year":"1997","venue":"Electronics Letters","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Adachi et al., 1998] Fumiyuki Adachi, Mamoru Sawahashi, and Hirohito Suda. Wideband ds-cdma for next-generation mobile communications systems. IEEE communications Magazine, 36(9):56-69, 1998.","title":"Wideband ds-cdma for next-generation mobile communications systems","authors":"Fumiyuki Adachi, Mamoru Sawahashi, Hirohito Suda","year":"1998","venue":"IEEE communications Magazine","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Andreev et al., 2003] Boris D. Andreev, Edward L. Title- baum, and Eby G. Friedman. Orthogonal code generator for 3g wireless transceivers. In Proceedings of the 13th ACM Great Lakes Symposium on VLSI, GLSVLSI '03, pages 229-232, New York, NY, USA, 2003. ACM.","title":"Orthogonal code generator for 3g wireless transceivers","authors":"Boris D Andreev, Edward L Titlebaum, G Eby, Friedman","year":"2003","venue":"Proceedings of the 13th ACM Great Lakes Symposium on VLSI, GLSVLSI '03","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Bhattacharya and Lane, 2016a] S. Bhattacharya and N. D. Lane. From smart to deep: Robust activity recognition on smartwatches using deep learning. In 2016 IEEE Interna- tional Conference on Pervasive Computing and Communi- cation Workshops (PerCom Workshops), pages 1-6, March 2016.","title":"From smart to deep: Robust activity recognition on smartwatches using deep learning","authors":"Lane ; Bhattacharya, S Bhattacharya, N D Lane","year":"2016","venue":"IEEE International Conference on Pervasive Computing and Communication Workshops (PerCom Workshops)","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Bhattacharya and Lane, 2016b] Sourav Bhattacharya and Nicholas D. Lane. Sparsification and separation of deep learning layers for constrained resource inference on wear- ables. In Proceedings of the 14th ACM Conference on Em- bedded Network Sensor Systems CD-ROM, SenSys '16, pages 176-189, New York, NY, USA, 2016. ACM.","title":"Sparsification and separation of deep learning layers for constrained resource inference on wearables","authors":"Lane ; Bhattacharya, Sourav Bhattacharya, Nicholas D Lane","year":"2016","venue":"Proceedings of the 14th ACM Conference on Embedded Network Sensor Systems CD-ROM, SenSys '16","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Courbariaux and Bengio, 2016] Matthieu Courbariaux and Yoshua Bengio. Binarynet: Training deep neural net- works with weights and activations constrained to +1 or -1. CoRR, abs/1602.02830, 2016.","title":"Binarynet: Training deep neural networks with weights and activations constrained to +1 or -1","authors":"Bengio Courbariaux, Matthieu Courbariaux, Yoshua Bengio","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Esser et al., 2015] Steve K Esser, Rathinakumar Ap- puswamy, Paul Merolla, John V. Arthur, and Dharmen- dra S Modha. Backpropagation for energy-efficient neuromorphic computing. In Advances in Neural Infor- mation Processing Systems 28, pages 1117-1125.","title":"Backpropagation for energy-efficient neuromorphic computing","authors":"Steve K Esser, Rathinakumar Appuswamy, Paul Merolla, John V Arthur, Dharmendra S Modha","year":null,"venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"Curran Associates, Inc., 2015.","title":null,"authors":"Curran Associates, Inc","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Fernandez-Marques et al., 2018] Javier Fernandez- Marques, Vincent T.-S. Tseng, Sourav Bhattacharya, and Nicholas D. Lane. On-the-fly deterministic binary filters for memory efficient keyword spotting applications on embedded devices. In Proceedings of the 2nd Inter- national Workshop on Deep Learning for Mobile Systems and Applications, EMDL '18. ACM, 2018.","title":"On-the-fly deterministic binary filters for memory efficient keyword spotting applications on embedded devices","authors":"Javier Fernandez-Marques, T.-S Vincent, Sourav Tseng, Nicholas D Bhattacharya, Lane","year":"2018","venue":"Proceedings of the 2nd International Workshop on Deep Learning for Mobile Systems and Applications, EMDL '18","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Frosst and Hinton, 2017] Nicholas Frosst and Geoffrey E. Hinton. Distilling a neural network into a soft decision tree. CoRR, abs/1711.09784, 2017.","title":"Distilling a neural network into a soft decision tree","authors":"Nicholas Frosst, Geoffrey E Hinton","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Georgiev et al., 2017] Petko Georgiev, Nicholas D. Lane, Cecilia Mascolo, and David Chu. Accelerating mobile au- dio sensing algorithms through on-chip gpu offloading. In Proceedings of the 15th Annual International Conference on Mobile Systems, Applications, and Services, MobiSys '17, pages 306-318, New York, NY, USA, 2017. ACM.","title":"Accelerating mobile audio sensing algorithms through on-chip gpu offloading","authors":"Petko Georgiev, Nicholas D Lane, Cecilia Mascolo, David Chu","year":"2017","venue":"Proceedings of the 15th Annual International Conference on Mobile Systems, Applications, and Services, MobiSys '17","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Han et al., 2015] Song Han, Huizi Mao, and William J. Dally. Deep compression: Compressing deep neural net- work with pruning, trained quantization and huffman cod- ing. CoRR, abs/1510.00149, 2015.","title":"Deep compression: Compressing deep neural network with pruning, trained quantization and huffman coding","authors":"Han","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[He et al., 2015] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. CoRR, abs/1512.03385, 2015.","title":"Deep residual learning for image recognition","authors":"Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Howard et al., 2017] Andrew G. Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. Mo- bilenets: Efficient convolutional neural networks for mo- bile vision applications. CoRR, abs/1704.04861, 2017.","title":"Mobilenets: Efficient convolutional neural networks for mobile vision applications","authors":"Andrew G Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, Hartwig Adam","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Iandola et al., 2016] Forrest N. Iandola, Matthew W. Moskewicz, Khalid Ashraf, Song Han, William J. Dally, and Kurt Keutzer. Squeezenet: Alexnet-level accuracy with 50x fewer parameters and <1mb model size. CoRR, abs/1602.07360, 2016.","title":"Squeezenet: Alexnet-level accuracy with 50x fewer parameters and <1mb model size","authors":"Forrest N Iandola, Matthew W Moskewicz, Khalid Ashraf, Song Han, William J Dally, Kurt Keutzer","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Jarrett et al., 2009] Kevin Jarrett, Koray Kavukcuoglu, Marc'Aurelio Ranzato, and Yann LeCun. What is the best multi-stage architecture for object recognition? In IEEE 12th International Conference on Computer Vision, ICCV 2009, 2009.","title":"What is the best multi-stage architecture for object recognition?","authors":"Kevin Jarrett, Koray Kavukcuoglu, Marc'aurelio Ranzato, Yann Lecun","year":"2009","venue":"IEEE 12th International Conference on Computer Vision, ICCV 2009","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Juefei-Xu et al., 2017] Felix Juefei-Xu, Vishnu Naresh Boddeti, and Marios Savvides. Local binary convolutional neural networks. In Computer Vision and Pattern Recogni- tion (CVPR), 2017 IEEE Conference on, volume 1, 2017.","title":"Local binary convolutional neural networks","authors":"Juefei-Xu","year":"2017","venue":"Computer Vision and Pattern Recognition (CVPR)","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Kim et al., 2009] S. Kim, M. Kim, C. Shin, J. Lee, and Y. Kim. Efficient implementation of ovsf code genera- tor for umts systems. In 2009 IEEE Pacific Rim Confer- ence on Communications, Computers and Signal Process- ing, pages 483-486, Aug 2009.","title":"Efficient implementation of ovsf code generator for umts systems","authors":"S Kim, M Kim, C Shin, J Lee, Y Kim","year":"2009","venue":"IEEE Pacific Rim Conference on Communications, Computers and Signal Processing","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Krizhevsky et al., 2012] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. In Advances in Neural In- formation Processing Systems 25, pages 1097-1105.","title":"Imagenet classification with deep convolutional neural networks","authors":"Alex Krizhevsky, Ilya Sutskever, Geoffrey E Hinton","year":null,"venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Lane et al., 2015] Nicholas D. Lane, Sourav Bhattacharya, Petko Georgiev, Claudio Forlivesi, and Fahim Kawsar. An early resource characterization of deep learning on wear- ables, smartphones and internet-of-things devices. In Pro- ceedings of the 2015 International Workshop on Internet of Things Towards Applications, IoT-App '15, pages 7-12, New York, NY, USA, 2015. ACM.","title":"An early resource characterization of deep learning on wearables, smartphones and internet-of-things devices","authors":"Nicholas D Lane, Sourav Bhattacharya, Petko Georgiev, Claudio Forlivesi, Fahim Kawsar","year":"2015","venue":"Proceedings of the 2015 International Workshop on Internet of Things Towards Applications, IoT-App '15","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Lane et al., 2017] N. D. Lane, S. Bhattacharya, A. Mathur, P. Georgiev, C. Forlivesi, and F. Kawsar. Squeezing deep learning into mobile and embedded devices. IEEE Perva- sive Computing, 16(3):82-88, 2017.","title":"Squeezing deep learning into mobile and embedded devices","authors":"N D Lane, S Bhattacharya, A Mathur, P Georgiev, C Forlivesi, F Kawsar","year":"2017","venue":"IEEE Pervasive Computing","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[LeCun et al., 1989] Y. LeCun, B. Boser, J. S. Denker, D. Henderson, R. E. Howard, W. Hubbard, and L. D. Jackel. Backpropagation applied to handwritten zip code recognition. Neural Comput., 1(4):541-551, December 1989.","title":"Backpropagation applied to handwritten zip code recognition","authors":"Y Lecun, B Boser, J S Denker, D Henderson, R E Howard, W Hubbard, L D Jackel","year":"1989","venue":"Neural Comput","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Mathur et al., 2017] Akhil Mathur, Nicholas D. Lane, Sourav Bhattacharya, Aidan Boran, Claudio Forlivesi, and Fahim Kawsar. Deepeye: Resource efficient local execu- tion of multiple deep vision models using wearable com- modity hardware. In Proceedings of the 15th Annual In- ternational Conference on Mobile Systems, Applications, and Services, MobiSys '17, pages 68-81, New York, NY, USA, 2017. ACM.","title":"Deepeye: Resource efficient local execution of multiple deep vision models using wearable commodity hardware","authors":"Akhil Mathur, Nicholas D Lane, Sourav Bhattacharya, Aidan Boran, Claudio Forlivesi, Fahim Kawsar","year":"2017","venue":"Proceedings of the 15th Annual International Conference on Mobile Systems, Applications, and Services, MobiSys '17","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[McDanel et al., 2017] Bradley McDanel, Surat Teerapit- tayanon, and H. T. Kung. Embedded binarized neural net- works. In Proceedings of the 2017 International Confer- ence on Embedded Wireless Systems and Networks, EWSN 2017, Uppsala, Sweden, February 20-22, 2017, pages 168-173, 2017.","title":"Embedded binarized neural net-works","authors":"Bradley Mcdanel, Surat Teerapittayanon, H T Kung","year":"2017","venue":"Proceedings of the 2017 International Conference on Embedded Wireless Systems and Networks, EWSN 2017","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Pinto et al., 2009] Nicolas Pinto, David Doukhan, James J. DiCarlo, and David D. Cox. A high-throughput screen- ing approach to discovering good forms of biologically inspired visual representation. PLOS Computational Bi- ology, 5:1-12, 11 2009.","title":"A high-throughput screening approach to discovering good forms of biologically inspired visual representation","authors":"Nicolas Pinto, David Doukhan, James J Dicarlo, David D Cox","year":"2009","venue":"PLOS Computational Biology","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Purohit et al., 2013] G. Purohit, V. K. Chaubey, K. S. Raju, and P. V. Reddy. Fpga based implementation and testing of ovsf code. In 2013 International Conference on Advanced Electronic Systems (ICAES), pages 88-92, Sept 2013.","title":"Fpga based implementation and testing of ovsf code","authors":"G Purohit, V K Chaubey, K S Raju, P V Reddy","year":"2013","venue":"2013 International Conference on Advanced Electronic Systems (ICAES)","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Rastegari et al., 2016] Mohammad Rastegari, Vicente Or- donez, Joseph Redmon, and Ali Farhadi. Xnor-net: Ima- genet classification using binary convolutional neural net- works. CoRR, abs/1603.05279, 2016.","title":"Xnor-net: Imagenet classification using binary convolutional neural networks","authors":"Mohammad Rastegari, Vicente Ordonez, Joseph Redmon, Ali Farhadi","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Rintakoski et al., 2004] T. Rintakoski, M. Kuulusa, and J. Nurmi. Hardware unit for ovsf/walsh/hadamard code generation [3g mobile communication applications]. In 2004 International Symposium on System-on-Chip, 2004. Proceedings., pages 143-145, Nov 2004.","title":"Hardware unit for ovsf/walsh/hadamard code generation [3g mobile communication applications","authors":"T Rintakoski, M Kuulusa, J Nurmi","year":"2004","venue":"2004 International Symposium on System-on-Chip","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Saxe et al., 2011] Andrew Saxe, Pang W. Koh, Zhenghao Chen, Maneesh Bhand, Bipin Suresh, and Andrew Y. Ng. On random weights and unsupervised feature learning. In Proceedings of the 28th International Conference on Ma- chine Learning (ICML-11). ACM, 2011.","title":"On random weights and unsupervised feature learning","authors":"Andrew Saxe, Pang W Koh, Zhenghao Chen, Maneesh Bhand, Bipin Suresh, Andrew Y Ng","year":"2011","venue":"Proceedings of the 28th International Conference on Machine Learning (ICML-11)","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Soudry et al., 2014] Daniel Soudry, Itay Hubara, and Ron Meir. Expectation backpropagation: Parameter-free train- ing of multilayer neural networks with continuous or dis- crete weights. In Advances in Neural Information Process- ing Systems 27, pages 963-971.","title":"Expectation backpropagation: Parameter-free training of multilayer neural networks with continuous or discrete weights","authors":"Daniel Soudry, Itay Hubara, Ron Meir","year":null,"venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"Curran Associates, Inc., 2014.","title":null,"authors":"Curran Associates, Inc","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Suleiman et al., 2017] A. Suleiman, Y. H. Chen, J. Emer, and V. Sze. Towards closing the energy gap between hog and cnn features for embedded vision. In 2017 IEEE In- ternational Symposium on Circuits and Systems (ISCAS), pages 1-4, May 2017.","title":"Towards closing the energy gap between hog and cnn features for embedded vision","authors":"A Suleiman, Y H Chen, J Emer, V Sze","year":"2017","venue":"IEEE International Symposium on Circuits and Systems (ISCAS)","doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Sze et al., 2017] Vivienne Sze, Yu-Hsin Chen, Tien-Ju Yang, and Joel S. Emer. Efficient processing of deep neural networks: A tutorial and survey. CoRR, abs/1703.09039, 2017.","title":"Efficient processing of deep neural networks: A tutorial and survey","authors":"Vivienne Sze, Yu-Hsin Chen, Tien-Ju Yang, Joel S Emer","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[Tahavori et al., 2017] Fatemeh Tahavori, Emma Stack, Veena Agarwal, Malcolm Burnett, Ann Ashburn, Seyed Amir Hoseinitabatabaei, and William Harwin. Physical activity recognition of elderly people and people with parkinson's (pwp) during standard mobility tests using wearable sensors. In Smart Cities Conference (ISC2), 2017 International, pages 1-4. IEEE, 2017.","title":"Physical activity recognition of elderly people and people with parkinson's (pwp) during standard mobility tests using wearable sensors","authors":"Fatemeh Tahavori, Emma Stack, Veena Agarwal, Malcolm Burnett, Ann Ashburn, Seyed Amir Hoseinitabatabaei, William Harwin","year":"2017","venue":"Smart Cities Conference (ISC2), 2017 International","doi":null,"url":null,"internal_url":null},{"position":35,"raw":"[Vanhoucke et al., 2011] Vincent Vanhoucke, Andrew Se- nior, and Mark Z. Mao. Improving the speed of neural networks on cpus. In Deep Learning and Unsupervised Feature Learning Workshop, NIPS 2011, 2011.","title":"Improving the speed of neural networks on cpus","authors":"Vincent Vanhoucke, Andrew Senior, Mark Z Mao","year":"2011","venue":"Deep Learning and Unsupervised Feature Learning Workshop, NIPS 2011","doi":null,"url":null,"internal_url":null},{"position":36,"raw":"[Wang et al., 2016] Yunhe Wang, Chang Xu, Shan You, Dacheng Tao, and Chao Xu. Cnnpack: Packing convolu- tional neural networks in the frequency domain. In D. D. Lee, M. Sugiyama, U. V. Luxburg, I. Guyon, and R. Gar- nett, editors, Advances in Neural Information Processing Systems 29, pages 253-261.","title":"Cnnpack: Packing convolutional neural networks in the frequency domain","authors":"Yunhe Wang, Chang Xu, Shan You, Dacheng Tao, Chao Xu","year":null,"venue":"Advances in Neural Information Processing Systems 29","doi":null,"url":null,"internal_url":null},{"position":37,"raw":"Curran Associates, Inc., 2016.","title":null,"authors":"Curran Associates, Inc","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":38,"raw":"[Yao et al., 2017] Shuochao Yao, Yiran Zhao, Aston Zhang, Lu Su, and Tarek F. Abdelzaher. Compressing deep neural network structures for sensing systems with a compressor- critic framework. CoRR, abs/1706.01215, 2017.","title":"Compressing deep neural network structures for sensing systems with a compressorcritic framework","authors":"Shuochao Yao, Yiran Zhao, Aston Zhang, Lu Su, Tarek F Abdelzaher","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}