{"type":"ScholarlyArticle","title":"Learning to Binarize Continuous Features for Neuro-Rule Networks","authors":[{"name":"Wei Zhang","slug":"wei-zhang","given":"Wei","family":"Zhang","affiliation":"School of Computer Science and Technology , East China Normal University; Shanghai Institute for AI Education"},{"name":"Yongxiang Liu","slug":"yongxiang-liu","given":"Yongxiang","family":"Liu","affiliation":"School of Computer Science and Technology , East China Normal University"},{"name":"Zhuo Wang","slug":"zhuo-wang","given":"Zhuo","family":"Wang","affiliation":"Department of Computer Science and Technology , Tsinghua University"},{"name":"Jianyong Wang","slug":"jianyong-wang","given":"Jianyong","family":"Wang","affiliation":"Department of Computer Science and Technology , Tsinghua University"}],"abstract":"Neuro-Rule Networks (NRNs) emerge as a promising neuro-symbolic method, enjoyed by the ability to equate fully-connected neural networks with logic rules. To support learning logic rules consisting of boolean variables, converting input features into binary representations is required. Different from discrete features that could be directly transformed by one-hot encodings, continuous features need to be binarized based on some numerical intervals. Existing studies usually select the bound values of intervals based on empirical strategies (e.g., equal-width interval). However, it is not optimal since the bounds are fixed and cannot be optimized to accommodate the ultimate training target. In this paper, we propose AutoInt, an approach that automatically binarizes continuous features and enables the intervals to be optimized with NRNs in an end-to-end fashion. Specifically, AutoInt automatically selects an interval for a given continuous feature in a soft manner to enable a differentiable learning procedure of interval-related parameters. Moreover, it introduces an additional soft K-means clustering loss to make the interval centres approach the original feature value distribution, thus reducing the risk of overfitting intervals. We conduct comprehensive experiments on public datasets and demonstrate the effectiveness of AutoInt in boosting the performance of NRNs.","doi":"10.24963/ijcai.2023/510","url":"https://ijcai.org/proceedings/2023/510","pdf":"https://ijcai.org/proceedings/2023/0510.pdf","markdown":"https://ijcai.org/proceedings/2023/510.md","bibtex":"https://ijcai.org/proceedings/2023/bibtex/510","volume":{"slug":"2023","title":"Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2023","url":"https://ijcai.org/proceedings/2023/","year":2023,"location":"Macao, SAR","dates":"19-25 August 2023","editors":"Edith Elkind","isbn":"978-1-956792-03-4","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2023"},"section":"Machine Learning","track":"Main Track","number":510,"pages":{"first":4584,"last":4592},"keywords":["Machine Learning: ML: Neuro-symbolic methods"],"videos":[],"references":[{"position":1,"raw":"[Alexander and Mozer, 1994] Jay A. Alexander and Michael Mozer. Template-based algorithms for connectionist rule extraction. In NeurIPS, pages 609-616, 1994.","title":"Template-based algorithms for connectionist rule extraction","authors":"Jay A Alexander, Michael Mozer","year":"1994","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bengio et al., 2013] Yoshua Bengio, Nicholas Léonard, and Aaron Courville. Estimating or propagating gradients through stochastic neurons for conditional computation. arXiv preprint arXiv:1308.3432, 2013.","title":"Estimating or propagating gradients through stochastic neurons for conditional computation","authors":"Yoshua Bengio, Nicholas Léonard, Aaron Courville","year":"2013","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Bezdek, 2013] James C Bezdek. Pattern recognition with fuzzy objective function algorithms. Springer Science & Business Media, 2013.","title":"Pattern recognition with fuzzy objective function algorithms","authors":"C James, Bezdek","year":"2013","venue":null,"doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Castelvecchi, 2016] Davide Castelvecchi. Can we open the black box of ai? Nature News, 538(7623):20, 2016.","title":"Can we open the black box of ai?","authors":"Davide Castelvecchi","year":"2016","venue":"Nature News","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Cheng et al., 2022] Kewei Cheng, Jiahao Liu, Wei Wang, and Yizhou Sun. Rlogic: Recursive logical rule learning from knowledge graphs. In KDD, pages 179-189, 2022.","title":"Rlogic: Recursive logical rule learning from knowledge graphs","authors":"Kewei Cheng, Jiahao Liu, Wei Wang, Yizhou Sun","year":"2022","venue":"KDD","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Dehaspe and Toivonen, 1999] Luc Dehaspe and Hannu Toivonen. Discovery of frequent DATALOG patterns. Data Min. Knowl. Discov., 3(1):7-36, 1999.","title":"Discovery of frequent DATALOG patterns","authors":"Toivonen Dehaspe, Luc Dehaspe, Hannu Toivonen","year":"1999","venue":"Data Min. Knowl. Discov","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Dong et al., 2019] Honghua Dong, Jiayuan Mao, Tian Lin, Chong Wang, Lihong Li, and Denny Zhou. Neural logic machines. In ICLR, 2019.","title":"Neural logic machines","authors":"Honghua Dong, Jiayuan Mao, Tian Lin, Chong Wang, Lihong Li, Denny Zhou","year":"2019","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Dougherty et al., 1995] James Dougherty, Ron Kohavi, and Mehran Sahami. Supervised and unsupervised discretiza- tion of continuous features. In ICML, pages 194-202, 1995.","title":"Supervised and unsupervised discretization of continuous features","authors":"James Dougherty, Ron Kohavi, Mehran Sahami","year":"1995","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Fard et al., 2020] Maziar Moradi Fard, Thibaut Thonet, and Éric Gaussier. Deep k-means: Jointly clustering with k- means and learning representations. Pattern Recognit. Lett., 138:185-192, 2020.","title":"Deep k-means: Jointly clustering with kmeans and learning representations","authors":"Maziar Moradi Fard, Thibaut Thonet, Éric Gaussier","year":"2020","venue":"Pattern Recognit. Lett","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Garcez et al., 2022] Artur d'Avila Garcez, Sebastian Bader, Howard Bowman, Luis C Lamb, Leo de Penning, BV Il- luminoo, Hoifung Poon, and COPPE Gerson Zaverucha. Neural-symbolic learning and reasoning: A survey and in- terpretation. Neuro-Symbolic Artificial Intelligence: The State of the Art, 342:1, 2022.","title":"Neural-symbolic learning and reasoning: A survey and interpretation","authors":"Garcez","year":"2022","venue":"Neuro-Symbolic Artificial Intelligence: The State of the Art","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Glanois et al., 2022] Claire Glanois, Zhaohui Jiang, Xuen- ing Feng, Paul Weng, Matthieu Zimmer, Dong Li, Wulong Liu, and Jianye Hao. Neuro-symbolic hierarchical rule in- duction. In ICML, pages 7583-7615, 2022.","title":"Neuro-symbolic hierarchical rule induction","authors":"Claire Glanois, Zhaohui Jiang, Xuening Feng, Paul Weng, Matthieu Zimmer, Dong Li, Wulong Liu, Jianye Hao","year":"2022","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Hu et al., 2016] Zhiting Hu, Xuezhe Ma, Zhengzhong Liu, Eduard H. Hovy, and Eric P. Xing. Harnessing deep neural networks with logic rules. In ACL, 2016.","title":"Harnessing deep neural networks with logic rules","authors":"Hu","year":"2016","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Jin et al., 2022] Yang Jin, Linchao Zhu, and Yadong Mu. Complex video action reasoning via learnable markov logic network. In CVPR, pages 3232-3241, 2022.","title":"Complex video action reasoning via learnable markov logic network","authors":"Yang Jin, Linchao Zhu, Yadong Mu","year":"2022","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Liu and Setiono, 1997] Huan Liu and Rudy Setiono. Fea- ture selection via discretization. IEEE Trans. Knowl. Data Eng., 9(4):642-645, 1997.","title":"Feature selection via discretization","authors":"Setiono ; ] Huan Liu, Rudy Liu, Setiono","year":"1997","venue":"IEEE Trans. Knowl. Data Eng","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Manhaeve et al., 2018] Robin Manhaeve, Sebastijan Du- mancic, Angelika Kimmig, Thomas Demeester, and Luc De Raedt. Deepproblog: Neural probabilistic logic programming. In NeurIPS, pages 3753-3763, 2018.","title":"Robin Manhaeve, Sebastijan Dumancic","authors":"Manhaeve","year":"2018","venue":"Deepproblog: Neural probabilistic logic programming","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Marra and Kuzelka, 2021] Giuseppe Marra and Ondrej Kuzelka. Neural markov logic networks. In UAI, pages 908-917, 2021.","title":"Neural markov logic networks","authors":"Kuzelka Marra, Giuseppe Marra, Ondrej Kuzelka","year":"2021","venue":"UAI","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Marra et al., 2020] Giuseppe Marra, Sebastijan Dumancic, Robin Manhaeve, and Luc De Raedt. From statistical re- lational to neural symbolic artificial intelligence: a survey. In IJCAI, pages 4943-4950, 2020.","title":"From statistical relational to neural symbolic artificial intelligence: a survey","authors":"Giuseppe Marra, Sebastijan Dumancic, Robin Manhaeve, Luc De, Raedt","year":"2020","venue":"IJCAI","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Meilicke et al., 2019] Christian Meilicke, Melisachew Wudage Chekol, Daniel Ruffinelli, and Heiner Stuckenschmidt. Anytime bottom-up rule learn- ing for knowledge graph completion. In IJCAI, pages 3137-3143, 2019.","title":"Anytime bottom-up rule learning for knowledge graph completion","authors":"Meilicke","year":"2019","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2019/435"},{"position":19,"raw":"[Qiao et al., 2021] Litao Qiao, Weijia Wang, and Bill Lin. Learning accurate and interpretable decision rule sets from neural networks. In AAAI, pages 4303-4311, 2021.","title":"Learning accurate and interpretable decision rule sets from neural networks","authors":"Qiao","year":"2021","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Qu and Tang, 2019] Meng Qu and Jian Tang. Probabilistic logic neural networks for reasoning. In NeurIPS, pages 7710-7720, 2019.","title":"Probabilistic logic neural networks for reasoning","authors":"Meng Qu, Jian Tang","year":"2019","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Qu et al., 2021] Meng Qu, Junkun Chen, Louis-Pascal A. C. Xhonneux, Yoshua Bengio, and Jian Tang. Rnn- logic: Learning logic rules for reasoning on knowledge graphs. In ICLR, 2021.","title":"Rnnlogic: Learning logic rules for reasoning on knowledge graphs","authors":"Meng Qu, Junkun Chen, A C Louis-Pascal, Yoshua Xhonneux, Jian Bengio, Tang","year":"2021","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Quinlan, 2014] J Ross Quinlan. C4. 5: programs for ma- chine learning. Elsevier, 2014.","title":"C4. 5: programs for machine learning","authors":"Quinlan Ross","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Rudin, 2019] Cynthia Rudin. Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead. Nature Machine Intelligence, 1(5):206-215, 2019.","title":"Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead","authors":"Cynthia Rudin","year":"2019","venue":"Nature Machine Intelligence","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Sarker et al., 2021] Md Kamruzzaman Sarker, Lu Zhou, Aaron Eberhart, and Pascal Hitzler. Neuro-symbolic artificial intelligence: Current trends. arXiv preprint arXiv:2105.05330, 2021.","title":"Neuro-symbolic artificial intelligence: Current trends","authors":"Lu Md Kamruzzaman Sarker, Aaron Zhou, Pascal Eberhart, Hitzler","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Sharmin et al., 2019] Sadia Sharmin, Mohammad Shoyaib, Amin Ahsan Ali, Muhammad Asif Hossain Khan, and Oksam Chae. Simultaneous feature selection and dis- cretization based on mutual information. Pattern Recog- nit., 91:162-174, 2019.","title":"Simultaneous feature selection and discretization based on mutual information","authors":"Sadia Sharmin, Mohammad Shoyaib, Ahsan Amin, Muhammad Ali, Asif Hossain, Oksam Khan, Chae","year":"2019","venue":"Pattern Recognit","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Wang et al., 2020] Zhuo Wang, Wei Zhang, Ning Liu, and Jianyong Wang. Transparent classification with multilayer logical perceptrons and random binarization. In AAAI, pages 6331-6339, 2020.","title":"Transparent classification with multilayer logical perceptrons and random binarization","authors":"Zhuo Wang, Wei Zhang, Ning Liu, Jianyong Wang","year":"2020","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Wang et al., 2021] Zhuo Wang, Wei Zhang, Ning Liu, and Jianyong Wang. Scalable rule-based representation learn- ing for interpretable classification. In NeurIPS, 2021.","title":"Scalable rule-based representation learning for interpretable classification","authors":"Zhuo Wang, Wei Zhang, Ning Liu, Jianyong Wang","year":"2021","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Xie et al., 2019] Yaqi Xie, Ziwei Xu, Kuldeep S. Meel, Mo- han S. Kankanhalli, and Harold Soh. Embedding symbolic knowledge into deep networks. In NeurIPS, pages 4235- 4245, 2019.","title":"Embedding symbolic knowledge into deep networks","authors":"Yaqi Xie, Ziwei Xu, Kuldeep S Meel, Mohan S Kankanhalli, Harold Soh","year":"2019","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Yadav et al., 2021] Rohan Kumar Yadav, Lei Jiao, Ole- Christoffer Granmo, and Morten Goodwin. Human-level interpretable learning for aspect-based sentiment analysis. In AAAI, pages 14203-14212, 2021.","title":"Human-level interpretable learning for aspect-based sentiment analysis","authors":"Rohan Kumar Yadav, Lei Jiao, Ole-Christoffer Granmo, Morten Goodwin","year":"2021","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Yang et al., 2017] Fan Yang, Zhilin Yang, and William W. Cohen. Differentiable learning of logical rules for knowl- edge base reasoning. In NeurIPS, pages 2319-2328, 2017.","title":"Differentiable learning of logical rules for knowledge base reasoning","authors":"Fan Yang, Zhilin Yang, William W Cohen","year":"2017","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Yang et al., 2022] Zhun Yang, Joohyung Lee, and Chiyoun Park. Injecting logical constraints into neural networks via straight-through estimators. In ICML, pages 25096- 25122, 2022.","title":"Injecting logical constraints into neural networks via straight-through estimators","authors":"Zhun Yang, Joohyung Lee, Chiyoun Park","year":"2022","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Zhang et al., 2022a] Wei Zhang, Junbing Yan, Zhuo Wang, and Jianyong Wang. Neuro-symbolic interpretable collab- orative filtering for attribute-based recommendation. In WWW, pages 3229-3238, 2022.","title":"Neuro-symbolic interpretable collaborative filtering for attribute-based recommendation","authors":"Zhang","year":"2022","venue":"WWW","doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Zhang et al., 2022b] Yao Zhang, Yun Xiong, Yiheng Sun, Caihua Shan, Tian Lu, Hui Song, and Yangyong Zhu. Rudi: Explaining behavior sequence models by automatic statistics generation and rule distillation. In CIKM, pages 2651-2660, 2022.","title":"Rudi: Explaining behavior sequence models by automatic statistics generation and rule distillation","authors":"Zhang","year":"2022","venue":"CIKM","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}