---
title: "Semantic Smoothing of Document Models for Agglomerative Clustering"
authors:
  - "Xiaohua Zhou"
  - "Xiaodan Zhang"
  - "Xiaohua Hu"
doi: ""
url: "https://ijcai.org/proceedings/2007/471"
pdf: "https://ijcai.org/proceedings/2007/0471.pdf"
venue: "Proceedings of the Twentieth International Joint Conference on Artificial Intelligence"
year: 2007
pages: "2928-2932"
track: ""
keywords:
---

# Semantic Smoothing of Document Models for Agglomerative Clustering

Xiaohua Zhou, Xiaodan Zhang, Xiaohua Hu

*Proceedings of the Twentieth International Joint Conference on Artificial Intelligence*, pages 2928-2932.

## Abstract

In this paper, we argue that the agglomerative clustering with vector cosine similarity measure performs poorly due to two reasons. First, the nearest neighbors of a document belong to different classes in many cases since any pair of documents shares lots of “general” words. Second, the sparsity of class-specific “core&dquo; words leads to grouping documents with the same class labels into different clusters. Both problems can be resolved by suitable smoothing of document model and using Kullback-Leibler divergence of two smoothed models as pairwise document distances. Inspired by the recent work in information retrieval, we propose a novel context-sensitive semantic smoothing method that can automatically identifies multiword phrases in a document and then statistically map phrases to individual document terms. We evaluate the new model-based similarity measure on three datasets using complete linkage criterion for agglomerative clustering and find out it significantly improves the clustering quality over the traditional vector cosine measure.