{"type":"ScholarlyArticle","title":"Semantic Smoothing of Document Models for Agglomerative Clustering","authors":[{"name":"Xiaohua Zhou","slug":"xiaohua-zhou","given":"Xiaohua","family":"Zhou","affiliation":""},{"name":"Xiaodan Zhang","slug":"xiaodan-zhang","given":"Xiaodan","family":"Zhang","affiliation":""},{"name":"Xiaohua Hu","slug":"xiaohua-hu","given":"Xiaohua","family":"Hu","affiliation":""}],"abstract":"In this paper, we argue that the agglomerative clustering with vector cosine similarity measure performs poorly due to two reasons. First, the nearest neighbors of a document belong to different classes in many cases since any pair of documents shares lots of “general” words. Second, the sparsity of class-specific “core&dquo; words leads to grouping documents with the same class labels into different clusters. Both problems can be resolved by suitable smoothing of document model and using Kullback-Leibler divergence of two smoothed models as pairwise document distances. Inspired by the recent work in information retrieval, we propose a novel context-sensitive semantic smoothing method that can automatically identifies multiword phrases in a document and then statistically map phrases to individual document terms. We evaluate the new model-based similarity measure on three datasets using complete linkage criterion for agglomerative clustering and find out it significantly improves the clustering quality over the traditional vector cosine measure.","doi":null,"url":"https://ijcai.org/proceedings/2007/471","pdf":"https://ijcai.org/proceedings/2007/0471.pdf","markdown":"https://ijcai.org/proceedings/2007/471.md","bibtex":"https://ijcai.org/proceedings/2007/bibtex/471","volume":{"slug":"2007","title":"Proceedings of the Twentieth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2007","url":"https://ijcai.org/proceedings/2007/","year":2007,"location":"Hyderabad, India","dates":"","editors":"Ramon López de Mántaras","isbn":"","publisher":"International Joint Conferences on Artificial Intelligence","doi_prefix":""},"section":null,"track":"","number":471,"pages":{"first":2928,"last":2932},"keywords":[],"videos":[],"references":[],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}