{"type":"ScholarlyArticle","title":"Bridging the Gap between Reality and Ideality of Entity Matching: A Revisiting and Benchmark Re-Construction","authors":[{"name":"Tianshu Wang","slug":"tianshu-wang","given":"Tianshu","family":"Wang","affiliation":"Chinese Information Processing Laboratory , Institute of Software , Chinese Academy of Sciences; Hangzhou Institute for Advanced Study , University of Chinese Academy of Sciences"},{"name":"Hongyu Lin","slug":"hongyu-lin","given":"Hongyu","family":"Lin","affiliation":"Chinese Information Processing Laboratory , Institute of Software , Chinese Academy of Sciences"},{"name":"Cheng Fu","slug":"cheng-fu","given":"Cheng","family":"Fu","affiliation":"Chinese Information Processing Laboratory , Institute of Software , Chinese Academy of Sciences"},{"name":"Xianpei Han","slug":"xianpei-han","given":"Xianpei","family":"Han","affiliation":"Chinese Information Processing Laboratory , Institute of Software , Chinese Academy of Sciences; State Key Laboratory of Computer Science , Institute of Software , Chinese Academy of Sciences; Beijing Academy of Artificial Intelligence"},{"name":"Le Sun","slug":"le-sun","given":"Le","family":"Sun","affiliation":"Chinese Information Processing Laboratory , Institute of Software , Chinese Academy of Sciences; State Key Laboratory of Computer Science , Institute of Software , Chinese Academy of Sciences"},{"name":"Feiyu Xiong","slug":"feiyu-xiong","given":"Feiyu","family":"Xiong","affiliation":"Alibaba Group , China"},{"name":"Hui Chen","slug":"hui-chen","given":"Hui","family":"Chen","affiliation":"Alibaba Group , China"},{"name":"Minlong Lu","slug":"minlong-lu","given":"Minlong","family":"Lu","affiliation":"Alibaba Group , China"},{"name":"Xiuwen Zhu","slug":"xiuwen-zhu","given":"Xiuwen","family":"Zhu","affiliation":"Alibaba Group , China"}],"abstract":"Entity matching (EM) is the most critical step for entity resolution (ER). While current deep learning-based methods achieve very impressive performance on standard EM benchmarks, their real-world application performance is much frustrating. In this paper, we highlight that such the gap between reality and ideality stems from the unreasonable benchmark construction process, which is inconsistent with the nature of entity matching and therefore leads to biased evaluations of current EM approaches. To this end, we build a new EM corpus and re-construct EM benchmarks to challenge critical assumptions implicit in the previous benchmark construction process by step-wisely changing the restricted entities, balanced labels, and single-modal records in previous benchmarks into open entities, imbalanced labels, and multi-modal records in an open environment. Experimental results demonstrate that the assumptions made in the previous benchmark construction process are not coincidental with the open environment, which conceal the main challenges of the task and therefore significantly overestimate the current progress of entity matching. The constructed benchmarks and code are publicly released at https://github.com/tshu-w/ember.","doi":"10.24963/ijcai.2022/552","url":"https://ijcai.org/proceedings/2022/552","pdf":"https://ijcai.org/proceedings/2022/0552.pdf","markdown":"https://ijcai.org/proceedings/2022/552.md","bibtex":"https://ijcai.org/proceedings/2022/bibtex/552","volume":{"slug":"2022","title":"Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2022","url":"https://ijcai.org/proceedings/2022/","year":2022,"location":"Vienna","dates":"23-29 July 2022","editors":"Luc De Raedt","isbn":"978-1-956792-00-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2022"},"section":"Multidisciplinary Topics and Applications","track":"Main Track","number":552,"pages":{"first":3978,"last":3984},"keywords":["Data Mining: Intelligent Database Systems","Multidisciplinary Topics and Applications: Databases"],"videos":[{"label":"Video #1","url":"https://slideslive.com/38986594","length_sec":81,"provider":"slideslive"},{"label":"Video #2","url":"https://slideslive.com/38984737","length_sec":346,"provider":"slideslive"}],"references":[{"position":1,"raw":"[Christophides et al., 2021] Vassilis Christophides, Vasilis Efthymiou, Themis Palpanas, George Papadakis, and Kostas Stefanidis. An overview of end-to-end entity res- olution for big data. ACM Comput. Surv., 53(6):127:1- 127:42, 2021.","title":"An overview of end-to-end entity resolution for big data","authors":"Vassilis Christophides, Vasilis Efthymiou, Themis Palpanas, George Papadakis, Kostas Stefanidis","year":"2021","venue":"ACM Comput. Surv","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Dosovitskiy et al., 2021] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Min- derer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Trans- formers for image recognition at scale. In 9th Inter- national Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenRe- view.net, 2021.","title":"An image is worth 16x16 words: Transformers for image recognition at scale","authors":"Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, Neil Houlsby","year":"2021","venue":"9th International Conference on Learning Representations, ICLR 2021, Virtual Event","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Fellegi and Sunter, 1969] Ivan P. Fellegi and Alan B. Sunter. A theory for record linkage. Journal of the Ameri- can Statistical Association, 64(328):1183-1210, dec 1969.","title":"A theory for record linkage","authors":"Ivan P Fellegi, Alan B Sunter","year":"1969","venue":"Journal of the American Statistical Association","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Fu et al., 2020] Cheng Fu, Xianpei Han, Jiaming He, and Le Sun. Hierarchical matching network for heterogeneous entity resolution. In Proceedings of the Twenty-Ninth In- ternational Joint Conference on Artificial Intelligence, IJ- CAI 2020, pages 3665-3671. ijcai.org, 2020.","title":"Hierarchical matching network for heterogeneous entity resolution","authors":"Cheng Fu, Xianpei Han, Jiaming He, Le Sun","year":"2020","venue":"Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence, IJ-CAI 2020","doi":null,"url":null,"internal_url":"/proceedings/2020/507"},{"position":5,"raw":"[Ge et al., 2021] Congcong Ge, Pengfei Wang, Lu Chen, Xi- aoze Liu, Baihua Zheng, and Yunjun Gao. CollaborEM: A self-supervised entity matching framework using multi- features collaboration. IEEE Transactions on Knowledge and Data Engineering, pages 1-1, 2021.","title":"CollaborEM: A self-supervised entity matching framework using multifeatures collaboration","authors":"Congcong Ge, Pengfei Wang, Lu Chen, Xiaoze Liu, Baihua Zheng, Yunjun Gao","year":"2021","venue":"IEEE Transactions on Knowledge and Data Engineering","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Jiménez-Ruiz et al., 2020] Ernesto Jiménez-Ruiz, Oktie Hassanzadeh, Vasilis Efthymiou, Jiaoyan Chen, and Kavitha Srinivas. Semtab 2019: Resources to benchmark tabular data to knowledge graph matching systems. In The Semantic Web -17th International Conference, ESWC 2020, Heraklion, Crete, Greece, May 31-June 4, 2020, Proceedings, volume 12123 of Lecture Notes in Computer Science, pages 514-530. Springer, 2020.","title":"Semtab 2019: Resources to benchmark tabular data to knowledge graph matching systems","authors":"Ernesto Jiménez-Ruiz, Oktie Hassanzadeh, Vasilis Efthymiou, Jiaoyan Chen, Kavitha Srinivas","year":"2020","venue":"The Semantic Web -17th International Conference, ESWC 2020","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Li et al., 2020] Yuliang Li, Jinfeng Li, Yoshihiko Suhara, AnHai Doan, and Wang-Chiew Tan. Deep entity matching with pre-trained language models. Proc. VLDB Endow., 14(1):50-60, 2020.","title":"Deep entity matching with pre-trained language models","authors":"Yuliang Li, Jinfeng Li, Yoshihiko Suhara, Anhai Doan, Wang-Chiew Tan","year":"2020","venue":"Proc. VLDB Endow","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Meyer and Bolosky, 2012] Dutch T. Meyer and William J. Bolosky. A study of practical deduplication. ACM Trans. Storage, 7(4):14:1-14:20, 2012.","title":"A study of practical deduplication","authors":"Meyer, T Bolosky ; Dutch, William J Meyer, Bolosky","year":"2012","venue":"ACM Trans. Storage","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Mudgal et al., 2018] Sidharth Mudgal, Han Li, Theodoros Rekatsinas, AnHai Doan, Youngchoon Park, Ganesh Krishnan, Rohit Deep, Esteban Arcaute, and Vijay Raghavendra. Deep learning for entity matching: A de- sign space exploration. In Proceedings of the 2018 Inter- national Conference on Management of Data, SIGMOD Conference 2018, Houston, TX, USA, June 10-15, 2018, pages 19-34. ACM, 2018.","title":"Deep learning for entity matching: A design space exploration","authors":"Sidharth Mudgal, Han Li, Theodoros Rekatsinas, Anhai Doan, Youngchoon Park, Ganesh Krishnan, Rohit Deep, Esteban Arcaute, Vijay Raghavendra","year":"2018","venue":"Proceedings of the 2018 International Conference on Management of Data, SIGMOD Conference 2018","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Nie et al., 2019] Hao Nie, Xianpei Han, Ben He, Le Sun, Bo Chen, Wei Zhang, Suhui Wu, and Hao Kong. Deep sequence-to-sequence entity matching for heterogeneous entity resolution. In Proceedings of the 28th ACM In- ternational Conference on Information and Knowledge Management, CIKM 2019, Beijing, China, November 3- 7, 2019, pages 629-638. ACM, 2019.","title":"Deep sequence-to-sequence entity matching for heterogeneous entity resolution","authors":"Xianpei Hao Nie, Ben Han, Le He, Bo Sun, Wei Chen, Suhui Zhang, Hao Wu, Kong","year":"2019","venue":"Proceedings of the 28th ACM International Conference on Information and Knowledge Management, CIKM 2019","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Papadakis et al., 2021] George Papadakis, Ekaterini Ioan- nou, Emanouil Thanos, and Themis Palpanas. The Four Generations of Entity Resolution. Synthesis Lectures on Data Management. Morgan & Claypool Publishers, 2021.","title":"The Four Generations of Entity Resolution","authors":"George Papadakis, Ekaterini Ioannou, Emanouil Thanos, Themis Palpanas","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Primpeli and Bizer, 2020] Anna Primpeli and Christian Bizer. Profiling entity matching benchmark tasks. In CIKM '20: The 29th ACM International Conference on Information and Knowledge Management, Virtual Event, Ireland, October 19-23, 2020, pages 3101-3108. ACM, 2020.","title":"Anna Primpeli and Christian Bizer. Profiling entity matching benchmark tasks","authors":"Bizer Primpeli","year":"2020","venue":"CIKM '20: The 29th ACM International Conference on Information and Knowledge Management","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Primpeli et al., 2019] Anna Primpeli, Ralph Peeters, and Christian Bizer. The WDC training dataset and gold stan- dard for large-scale product matching. In Companion of The 2019 World Wide Web Conference, WWW 2019, San Francisco, CA, USA, May 13-17, 2019, pages 381-386. ACM, 2019.","title":"The WDC training dataset and gold standard for large-scale product matching","authors":"Anna Primpeli, Ralph Peeters, Christian Bizer","year":"2019","venue":"Companion of The 2019 World Wide Web Conference, WWW 2019","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Shao et al., 2020] Jingyu Shao, Qing Wang, Asiri Wi- jesinghe, and Erhard Rahm. Ergan: Generative adversarial networks for entity resolution. In 20th IEEE International Conference on Data Mining, ICDM 2020, Sorrento, Italy, November 17-20, 2020, pages 1250-1255. IEEE, 2020.","title":"Ergan: Generative adversarial networks for entity resolution","authors":"Jingyu Shao, Qing Wang, Asiri Wijesinghe, Erhard Rahm","year":"2020","venue":"20th IEEE International Conference on Data Mining, ICDM 2020","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Thirumuruganathan et al., 2018] Saravanan Thirumuru- ganathan, Shameem Ahamed Puthiya Parambath, Mourad Ouzzani, Nan Tang, and Shafiq R. Joty. Reuse and adaptation for entity resolution through transfer learning. CoRR, abs/1809.11084, 2018.","title":"Reuse and adaptation for entity resolution through transfer learning","authors":"Saravanan Thirumuruganathan, Shameem Ahamed Puthiya, Mourad Parambath, Nan Ouzzani, Shafiq R Tang, Joty","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Thirumuruganathan et al., 2021] Saravanan Thirumuru- ganathan, Han Li, Nan Tang, Mourad Ouzzani, Yash Govind, Derek Paulsen, Glenn Fung, and AnHai Doan. Deep learning for blocking in entity matching: A design space exploration. Proc. VLDB Endow., 14(11):2459- 2472, 2021.","title":"Deep learning for blocking in entity matching: A design space exploration","authors":"Saravanan Thirumuruganathan, Han Li, Nan Tang, Mourad Ouzzani, Yash Govind, Derek Paulsen, Glenn Fung, Anhai Doan","year":"2021","venue":"Proc. VLDB Endow","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Wang et al., 2011] Jiannan Wang, Guoliang Li, Jeffrey Xu Yu, and Jianhua Feng. Entity matching: How similar is similar. Proc. VLDB Endow., 4(10):622-633, 2011.","title":"Entity matching: How similar is similar","authors":"Wang","year":"2011","venue":"Proc. VLDB Endow","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Wang et al., 2021] Jin Wang, Yuliang Li, and Wataru Hi- rota. Machamp: A generalized entity matching bench- mark. In CIKM '21: The 30th ACM International Con- ference on Information and Knowledge Management, Vir- tual Event, Queensland, Australia, November 1 -5, 2021, pages 4633-4642. ACM, 2021.","title":"Machamp: A generalized entity matching benchmark","authors":"Jin Wang, Yuliang Li, Wataru Hirota","year":"2021","venue":"CIKM '21: The 30th ACM International Conference on Information and Knowledge Management, Virtual Event","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Wilke and Rahm, 2021] Moritz Wilke and Erhard Rahm. Towards multi-modal entity resolution for product match- ing. In Proceedings of the 32nd GI-Workshop Grundlagen von Datenbanken, Online Event, Germany, September 01- 03, 2021, volume 3075 of CEUR Workshop Proceedings. CEUR-WS.org, 2021.","title":"Towards multi-modal entity resolution for product matching","authors":"Rahm ; ] Moritz Wilke, Erhard Wilke, Rahm","year":"2021","venue":"Proceedings of the 32nd GI-Workshop Grundlagen von Datenbanken","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Zhao and He, 2019] Chen Zhao and Yeye He. Auto-em: End-to-end fuzzy entity-matching using pre-trained deep models and transfer learning. In The World Wide Web Con- ference, WWW 2019, San Francisco, CA, USA, May 13-17, 2019, pages 2413-2424. ACM, 2019.","title":"Auto-em: End-to-end fuzzy entity-matching using pre-trained deep models and transfer learning","authors":"Chen Zhao, Yeye He","year":"2019","venue":"The World Wide Web Conference, WWW 2019","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}