Dynamically Weighted Hidden Markov Model for Spam Deobfuscation

Seunghak Lee, Iryoung Jeong, Seungjin Choi

Spam deobfuscation is a processing to detect obfuscated words appeared in spam emails and to convert them back to the original words for correct recognition. Lexicon tree hidden Markov model (LT–HMM) was recently shown to be useful in spam deobfuscation. However, LT–HMM suffers from a huge number of states, which is not desirable for practical applications. In this paper we present a complexity–reduced HMM, referred to as dynamically weighted HMM (DW–HMM) where the states involving the same emission probability are grouped into super–states, while preserving state transition probabilities of the original HMM. DW–HMM dramatically reduces the number of states and its state transition probabilities are determined in the decoding phase. We illustrate how we convert a LT–HMM to its associated DW–HMM. We confirm the useful behavior of DW–HMM in the task of spam deobfuscation, showing that it significantly reduces the number of states while maintaining the high accuracy.