{"type":"ScholarlyArticle","title":"Rubik's Optical Neural Networks: Multi-task Learning with Physics-aware Rotation Architecture","authors":[{"name":"Yingjie Li","slug":"yingjie-li","given":"Yingjie","family":"Li","affiliation":"University of Utah"},{"name":"Weilu Gao","slug":"weilu-gao","given":"Weilu","family":"Gao","affiliation":"University of Utah"},{"name":"Cunxi Yu","slug":"cunxi-yu","given":"Cunxi","family":"Yu","affiliation":"University of Utah"}],"abstract":"Recently, there are increasing efforts on advancing optical neural networks (ONNs), which bring significant advantages for machine learning (ML) in terms of power efficiency, parallelism, and computational speed. With the considerable benefits in computation speed and energy efficiency, there are significant interests in leveraging ONNs into medical sensing, security screening, drug detection, and autonomous driving. However, due to the challenge of implementing reconfigurability, deploying multi-task learning (MTL) algorithms on ONNs requires re-building and duplicating the physical diffractive systems, which significantly degrades the energy and cost efficiency in practical application scenarios. This work presents a novel ONNs architecture, namely, RubikONNs, which utilizes the physical properties of optical systems to encode multiple feed-forward functions by physically rotating the hardware similarly to rotating a Rubik's Cube. To optimize MTL performance on RubikONNs, two domain-specific physics-aware training algorithms RotAgg and RotSeq are proposed. Our experimental results demonstrate more than 4x improvements in energy and cost efficiency with marginal accuracy degradation compared to the state-of-the-art approaches.","doi":"10.24963/ijcai.2023/847","url":"https://ijcai.org/proceedings/2023/847","pdf":"https://ijcai.org/proceedings/2023/0847.pdf","markdown":"https://ijcai.org/proceedings/2023/847.md","bibtex":"https://ijcai.org/proceedings/2023/bibtex/847","volume":{"slug":"2023","title":"Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2023","url":"https://ijcai.org/proceedings/2023/","year":2023,"location":"Macao, SAR","dates":"19-25 August 2023","editors":"Edith Elkind","isbn":"978-1-956792-03-4","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2023"},"section":"Late Papers","track":"Main Track","number":847,"pages":{"first":7197,"last":7206},"keywords":["Machine Learning: ML: Classification","Multidisciplinary Topics and Applications: MDA: AI hardware","Multidisciplinary Topics and Applications: MDA: Physical sciences"],"videos":[],"references":[{"position":1,"raw":"[Chen et al., 2022a] Ruiyang Chen, Yingjie Li, Minhan Lou, Jichao Fan, Yingheng Tang, Berardi Sensale-Rodriguez, Cunxi Yu, and Weilu Gao. Physics-aware machine learn- ing and adversarial attack in complex-valued reconfig- urable diffractive all-optical neural network. Laser & Pho- tonics Reviews, page 2200348, 2022.","title":"Physics-aware machine learning and adversarial attack in complex-valued reconfigurable diffractive all-optical neural network","authors":"Chen","year":"2022","venue":"Laser & Photonics Reviews","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Chen et al., 2022b] Ruiyang Chen, Yingjie Li, Minhan Lou, Cunxi Yu, and Weilu Gao. Complex-valued reconfigurable diffractive optical neural networks using cost-effective spatial light modulators. In CLEO: Applications and Tech- nology, pages JTh3B-56. Optica Publishing Group, 2022.","title":"Complex-valued reconfigurable diffractive optical neural networks using cost-effective spatial light modulators","authors":"Chen","year":"2022","venue":"CLEO: Applications and Technology","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Clanuwat et al., 2018] Tarin Clanuwat, Mikel Bober-Irizar, Asanobu Kitamoto, Alex Lamb, Kazuaki Yamamoto, and David Ha. Deep learning for classical japanese literature. 2018.","title":"Deep learning for classical japanese literature","authors":"Tarin Clanuwat, Mikel Bober-Irizar, Asanobu Kitamoto, Alex Lamb, Kazuaki Yamamoto, David Ha","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Cohen et al., 2017] Gregory Cohen, Saeed Afshar, Jonathan Tapson, and Andre Van Schaik. Emnist: Extending mnist to handwritten letters. In 2017 International Joint Con- ference on Neural Networks (IJCNN), pages 2921-2926. IEEE, 2017.","title":"Emnist: Extending mnist to handwritten letters","authors":"Gregory Cohen, Saeed Afshar, Jonathan Tapson, Andre Van Schaik","year":"2017","venue":"2017 International Joint Conference on Neural Networks (IJCNN)","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Duan et al., 2023] Zhengyang Duan, Hang Chen, and Xing Lin. Optical multi-task learning using multi-wavelength diffractive deep neural networks. Nanophotonics, 2023.","title":"Optical multi-task learning using multi-wavelength diffractive deep neural networks","authors":"Duan","year":"2023","venue":"Nanophotonics","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Feldmann et al., 2019] J Feldmann, N Youngblood, C David Wright, H Bhaskaran, and WHP Pernice. All- optical spiking neurosynaptic networks with self-learning capabilities. Nature, 569(7755):208-214, 2019.","title":"Alloptical spiking neurosynaptic networks with self-learning capabilities","authors":"Feldmann","year":"2019","venue":"Nature","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Gao et al., 2021] Weilu Gao, Cunxi Yu, and Ruiyang Chen. Artificial intelligence accelerators based on graphene op- toelectronic devices. Advanced Photonics Research, 2(6):2100048, 2021.","title":"Artificial intelligence accelerators based on graphene optoelectronic devices","authors":"Gao","year":"2021","venue":"Advanced Photonics Research","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Gu et al., 2020] Jiaqi Gu, Zheng Zhao, Chenghao Feng, Mingjie Liu, Ray T Chen, and David Z Pan. Towards area- efficient optical neural networks: an fft-based architecture. In 2020 25th Asia and South Pacific Design Automation Conference (ASP-DAC), pages 476-481. IEEE, 2020.","title":"Towards areaefficient optical neural networks: an fft-based architecture","authors":"Jiaqi Gu, Zheng Zhao, Chenghao Feng, Mingjie Liu, Ray T Chen, David Z Pan","year":"2020","venue":"2020 25th Asia and South Pacific Design Automation Conference (ASP-DAC)","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Gu et al., 2022] Jiaqi Gu, Hanqing Zhu, Chenghao Feng, Zixuan Jiang, Mingjie Liu, Shuhan Zhang, Ray T Chen, and David Z Pan. Adept: Automatic differentiable de- sign of photonic tensor cores. In Proceedings of the 59th ACM/IEEE Design Automation Conference, pages 937- 942, 2022.","title":"Adept: Automatic differentiable design of photonic tensor cores","authors":"Jiaqi Gu, Hanqing Zhu, Chenghao Feng, Zixuan Jiang, Mingjie Liu, Shuhan Zhang, Ray T Chen, David Z Pan","year":"2022","venue":"Proceedings of the 59th ACM/IEEE Design Automation Conference","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Jouppi et al., 2017] Norman P Jouppi, Cliff Young, Nishant Patil, David Patterson, Gaurav Agrawal, Raminder Bajwa, Sarah Bates, Suresh Bhatia, Nan Boden, Al Borchers, et al. In-datacenter Performance Analysis of a Tensor Process- ing Unit. Int'l Symp. on Computer Architecture (ISCA), pages 1-12, 2017.","title":"In-datacenter Performance Analysis of a Tensor Processing Unit","authors":"Cliff Norman P Jouppi, Nishant Young, David Patil, Gaurav Patterson, Raminder Agrawal, Sarah Bajwa, Suresh Bates, Nan Bhatia, Al Boden, Borchers","year":"2017","venue":"Int'l Symp. on Computer Architecture (ISCA)","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[LeCun, 1998] Yann LeCun. The mnist database of hand- written digits. http://yann.lecun.com/exdb/mnist/, 1998. Accessed: 2022-05-21.","title":"The mnist database of handwritten digits","authors":"Yann Lecun","year":"2022","venue":null,"doi":null,"url":"http://yann.lecun.com/exdb/mnist/,1998","internal_url":null},{"position":12,"raw":"[Li and Yu, 2021] Yingjie Li and Cunxi Yu. Late breaking results: physical adversarial attacks of diffractive deep neural networks. In DAC, 2021.","title":"Late breaking results: physical adversarial attacks of diffractive deep neural networks","authors":"Yu ; ] Yingjie Li, Cunxi Li, Yu","year":"2021","venue":"DAC","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Li et al., 2021] Yingjie Li, Ruiyang Chen, Berardi Sensale Rodriguez, Weilu Gao, and Cunxi Yu. Multi-task learning in diffractive deep neural networks via hardware-software co-design. Scientific Reports, pages 1-9, 2021.","title":"Multi-task learning in diffractive deep neural networks via hardware-software co-design","authors":"Yingjie Li, Ruiyang Chen, Sensale Berardi, Weilu Rodriguez, Cunxi Gao, Yu","year":"2021","venue":"Scientific Reports","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Li et al., 2022] Yingjie Li, Ruiyang Chen, Weilu Gao, and Cunxi Yu. Physics-aware differentiable discrete code- sign for diffractive optical neural networks. In Proceed- ings of the 41st IEEE/ACM International Conference on Computer-Aided Design, pages 1-9, 2022.","title":"Physics-aware differentiable discrete codesign for diffractive optical neural networks","authors":"Yingjie Li, Ruiyang Chen, Weilu Gao, Cunxi Yu","year":"2022","venue":"Proceedings of the 41st IEEE/ACM International Conference on Computer-Aided Design","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Lin et al., 2018] Xing Lin, Yair Rivenson, Nezih T Yardimci, Muhammed Veli, Yi Luo, Mona Jarrahi, and Ay- dogan Ozcan. All-optical machine learning using diffrac- tive deep neural networks. Science, 361(6406):1004- 1008, 2018.","title":"All-optical machine learning using diffractive deep neural networks","authors":"Xing Lin, Yair Rivenson, T Nezih, Muhammed Yardimci, Yi Veli, Mona Luo, Aydogan Jarrahi, Ozcan","year":"2018","venue":"Science","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Lou and et al., 2023] Minhan Lou and et al. Effects of in- terlayer reflection and interpixel interaction in diffractive optical neural networks. Optics Letters, 2023.","title":"Minhan Lou and et al. Effects of interlayer reflection and interpixel interaction in diffractive optical neural networks","authors":"Lou","year":"2023","venue":"Optics Letters","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Mengu et al., 2020] Deniz Mengu, Yair Rivenson, and Ay- dogan Ozcan. Scale-, shift-, and rotation-invariant diffrac- tive optical networks. ACS photonics, 8(1):324-334, 2020.","title":"Scale-, shift-, and rotation-invariant diffractive optical networks","authors":"Deniz Mengu, Yair Rivenson, Aydogan Ozcan","year":"2020","venue":"ACS photonics","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Mengu et al., 2023] Deniz Mengu, Anika Tabassum, Mona Jarrahi, and Aydogan Ozcan. Snapshot multispectral imag- ing using a diffractive optical network. Light: Science & Applications, 12(1):86, 2023.","title":"Snapshot multispectral imaging using a diffractive optical network","authors":"Deniz Mengu, Anika Tabassum, Mona Jarrahi, Aydogan Ozcan","year":"2023","venue":"Light: Science & Applications","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Seshadri et al., 2022] Kiran Seshadri, Berkin Akin, James Laudon, Ravi Narayanaswami, and Amir Yazdanbakhsh. An evaluation of edge tpu accelerators for convolutional neural networks. In 2022 IEEE International Sympo- sium on Workload Characterization (IISWC), pages 79- 91. IEEE, 2022.","title":"An evaluation of edge tpu accelerators for convolutional neural networks","authors":"Kiran Seshadri, Berkin Akin, James Laudon, Ravi Narayanaswami, Amir Yazdanbakhsh","year":"2022","venue":"IEEE International Symposium on Workload Characterization (IISWC)","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Shen et al., 2017] Yichen Shen, Nicholas C Harris, Scott Skirlo, Mihika Prabhu, Tom Baehr-Jones, Michael Hochberg, Xin Sun, Shijie Zhao, Hugo Larochelle, Dirk Englund, et al. Deep learning with coherent nanophotonic circuits. Nature Photonics, 11(7):441, 2017.","title":"Deep learning with coherent nanophotonic circuits","authors":"Yichen Shen, Nicholas C Harris, Scott Skirlo, Mihika Prabhu, Tom Baehr-Jones, Michael Hochberg, Xin Sun, Shijie Zhao, Hugo Larochelle, Dirk Englund","year":"2017","venue":"Nature Photonics","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Strubell et al., 2019] Emma Strubell, Ananya Ganesh, and Andrew McCallum. Energy and policy considerations for deep learning in nlp. arXiv preprint arXiv:1906.02243, 2019.","title":"Energy and policy considerations for deep learning in nlp","authors":"Emma Strubell, Ananya Ganesh, Andrew Mccallum","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Tait et al., 2017] Alexander N Tait, Thomas Ferreira De Lima, Ellen Zhou, Allie X Wu, Mitchell A Nahmias, Bhavin J Shastri, and Paul R Prucnal. Neuromorphic photonic networks using silicon photonic weight banks. Scientific reports, 7(1):1-10, 2017.","title":"Neuromorphic photonic networks using silicon photonic weight banks","authors":"Thomas Alexander N Tait, Ferreira De, Ellen Lima, Allie X Zhou, Mitchell A Wu, Bhavin J Nahmias, Paul R Shastri, Prucnal","year":"2017","venue":"Scientific reports","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Tang et al., 2023] Yingheng Tang, Princess Tara Zamani, Ruiyang Chen, Jianzhu Ma, Minghao Qi, Cunxi Yu, and Weilu Gao. Device-system end-to-end design of pho- tonic neuromorphic processor using reinforcement learn- ing. Laser & Photonics Reviews, 17(2):2200381, 2023.","title":"Device-system end-to-end design of photonic neuromorphic processor using reinforcement learning","authors":"Tang","year":"2023","venue":"Laser & Photonics Reviews","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Xiao et al., 2017] Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist: a novel image dataset for bench- marking machine learning algorithms. arXiv preprint arXiv:1708.07747, 2017.","title":"Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms","authors":"Han Xiao, Kashif Rasul, Roland Vollgraf","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Yan et al., 2022] Tao Yan, Rui Yang, Ziyang Zheng, Xing Lin, Hongkai Xiong, and Qionghai Dai. All-optical graph representation learning using integrated diffractive pho- tonic computing units. Science Advances, 8(24):eabn7630, 2022.","title":"All-optical graph representation learning using integrated diffractive photonic computing units","authors":"Tao Yan, Rui Yang, Ziyang Zheng, Xing Lin, Hongkai Xiong, Qionghai Dai","year":"2022","venue":"Science Advances","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Yin et al., 2022] Jiaqi Yin, Zhiru Zhang, and Cunxi Yu. Ex- act memory-and communication-aware scheduling of dnns on pipelined edge tpus. In 2022 IEEE/ACM 7th Sympo- sium on Edge Computing (SEC), pages 203-215. IEEE, 2022.","title":"Exact memory-and communication-aware scheduling of dnns on pipelined edge tpus","authors":"Jiaqi Yin, Zhiru Zhang, Cunxi Yu","year":"2022","venue":"IEEE/ACM 7th Symposium on Edge Computing (SEC)","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Yin et al., 2023] Jiaqi Yin, Yingjie Li, Daniel Robinson, and Cunxi Yu. Respect: Reinforcement learning based edge scheduling on pipelined coral edge tpus. Design Au- tomation Conference (DAC'23), 2023.","title":"Respect: Reinforcement learning based edge scheduling on pipelined coral edge tpus","authors":"Jiaqi Yin, Yingjie Li, Daniel Robinson, Cunxi Yu","year":"2023","venue":"Design Automation Conference (DAC'23)","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Ying et al., 2020] Zhoufeng Ying, Chenghao Feng, Zheng Zhao, Shounak Dhar, Hamed Dalir, Jiaqi Gu, Yue Cheng, Richard Soref, David Z Pan, and Ray T Chen. Electronic- photonic arithmetic logic unit for high-speed computing. Nature communications, 2020.","title":"Electronicphotonic arithmetic logic unit for high-speed computing","authors":"Zhoufeng Ying, Chenghao Feng, Zheng Zhao, Shounak Dhar, Hamed Dalir, Jiaqi Gu, Yue Cheng, Richard Soref, Ray T David Z Pan, Chen","year":"2020","venue":"Nature communications","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}