{"type":"ScholarlyArticle","title":"Max Markov Chain","authors":[{"name":"Yu Zhang","slug":"yu-zhang","given":"Yu","family":"Zhang","affiliation":"School of Computing and Augmented Intelligence , Arizona State University"},{"name":"Mitchell Bucklew","slug":"mitchell-bucklew","given":"Mitchell","family":"Bucklew","affiliation":"School of Computing and Augmented Intelligence , Arizona State University"}],"abstract":"In this paper, we introduce Max Markov Chain (MMC), a novel model for sequential data with sparse correlations among the state variables.\r\nIt may also be viewed as a special class of approximate models for High-order Markov Chains (HMCs). \r\nMMC is desirable for domains where the sparse correlations are long-term and vary in their temporal stretches. \r\nAlthough generally intractable, parameter optimization for MMC can be solved analytically. \r\nHowever, based on this result,\r\nwe derive an approximate solution that is highly efficient empirically.\r\nWhen compared with HMC and approximate HMC models, MMC \r\ncombines  better sample efficiency, model parsimony, and an outstanding computational advantage. \r\nSuch a quality allows MMC to scale to large domains \r\nwhere the competing models would struggle to perform. \r\nWe compare MMC with several baselines with synthetic and real-world datasets to demonstrate MMC as a valuable alternative for  stochastic modeling.","doi":"10.24963/ijcai.2023/639","url":"https://ijcai.org/proceedings/2023/639","pdf":"https://ijcai.org/proceedings/2023/0639.pdf","markdown":"https://ijcai.org/proceedings/2023/639.md","bibtex":"https://ijcai.org/proceedings/2023/bibtex/639","volume":{"slug":"2023","title":"Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2023","url":"https://ijcai.org/proceedings/2023/","year":2023,"location":"Macao, SAR","dates":"19-25 August 2023","editors":"Edith Elkind","isbn":"978-1-956792-03-4","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2023"},"section":"Uncertainty in AI","track":"Main Track","number":639,"pages":{"first":5758,"last":5767},"keywords":["Uncertainty in AI: UAI: Bayesian networks","Uncertainty in AI: UAI: Causality, structural causal models and causal inference","Uncertainty in AI: UAI: Tractable probabilistic models"],"videos":[],"references":[{"position":1,"raw":"[Baum and Petrie, 1966] Leonard E Baum and Ted Petrie. Statistical inference for probabilistic functions of finite state markov chains. The annals of mathematical statis- tics, 37(6):1554-1563, 1966.","title":"Statistical inference for probabilistic functions of finite state markov chains","authors":"E Leonard, Ted Baum, Petrie","year":"1966","venue":"The annals of mathematical statistics","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Begleiter et al., 2004] Ron Begleiter, Ran El-Yaniv, and Golan Yona. On prediction using variable order markov models. Journal of Artificial Intelligence Research, 22:385-421, 2004.","title":"On prediction using variable order markov models","authors":"Ron Begleiter, Ran El-Yaniv, Golan Yona","year":"2004","venue":"Journal of Artificial Intelligence Research","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Benjamin, 1979] Lorna Smith Benjamin. Use of structural analysis of social behavior (sasb) and markov chains to study dyadic interactions. Journal of Abnormal Psychol- ogy, 88(3):303, 1979.","title":"Use of structural analysis of social behavior (sasb) and markov chains to study dyadic interactions","authors":"Lorna Smith, Benjamin","year":"1979","venue":"Journal of Abnormal Psychology","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Berchtold and Raftery, 2002] André Berchtold and Adrian Raftery. The mixture transition distribution model for high-order markov chains and non-gaussian time series. Statistical Science, 17(3):328-356, 2002.","title":"The mixture transition distribution model for high-order markov chains and non-gaussian time series","authors":"Berchtold, Raftery, André Berchtold, Adrian Raftery","year":"2002","venue":"Statistical Science","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Berchtold, 1995] André Berchtold. Autoregressive mod- elling of markov chains. In Statistical Modelling, pages 19-26. Springer, 1995.","title":"Autoregressive modelling of markov chains","authors":"Berchtold ;, André Berchtold","year":"1995","venue":"Statistical Modelling","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Bhattacharjya et al., 2022] Debarun Bhattacharjya, Saurabh Sihag, Oktie Hassanzadeh, and Liza Bialik. Summary markov models for event sequences. arXiv preprint arXiv:2205.03375, 2022.","title":"Summary markov models for event sequences","authors":"Bhattacharjya","year":"2022","venue":null,"doi":null,"url":null,"internal_url":"/proceedings/2022/670"},{"position":7,"raw":"[Chin, 1977] Edwin H Chin. Modeling daily precipitation occurrence process with markov chain. Water resources research, 13(6):949-956, 1977.","title":"Modeling daily precipitation occurrence process with markov chain","authors":"Edwin H Chin","year":"1977","venue":"Water resources research","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Cowles and Carlin, 1996] Mary Kathryn Cowles and Bradley P Carlin. Markov chain monte carlo conver- gence diagnostics: a comparative review. Journal of the American Statistical Association, 91(434):883-904, 1996.","title":"Markov chain monte carlo convergence diagnostics: a comparative review","authors":"Carlin ; Cowles, Mary, Kathryn Cowles, Bradley P Carlin","year":"1996","venue":"Journal of the American Statistical Association","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Elfeki and Dekking, 2001] Amro Elfeki and Michel Dekking. A markov chain model for subsurface character- ization: theory and applications. Mathematical geology, 33(5):569-589, 2001.","title":"Amro Elfeki and Michel Dekking. A markov chain model for subsurface characterization: theory and applications","authors":"Dekking Elfeki","year":"2001","venue":"Mathematical geology","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Fine et al., 1998] Shai Fine, Yoram Singer, and Naftali Tishby. The hierarchical hidden markov model: Analysis and applications. Machine learning, 32(1):41-62, 1998.","title":"The hierarchical hidden markov model: Analysis and applications","authors":"Shai Fine, Yoram Singer, Naftali Tishby","year":"1998","venue":"Machine learning","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Galley, 2006] Michel Galley. A skip-chain conditional ran- dom field for ranking meeting utterances by importance. In Proceedings of the 2006 Conference on Empirical Meth- ods in Natural Language Processing, pages 364-372, 2006.","title":"A skip-chain conditional random field for ranking meeting utterances by importance","authors":"Michel Galley","year":"2006","venue":"Proceedings of the 2006 Conference on Empirical Methods in Natural Language Processing","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Gyftodimos and Flach, 2002] Elias Gyftodimos and Peter A Flach. Hierarchical bayesian networks: A probabilistic reasoning model for structured domains. In Proceedings of the ICML-2002 Workshop on Development of Represen- tations, pages 23-30. Citeseer, 2002.","title":"Hierarchical bayesian networks: A probabilistic reasoning model for structured domains","authors":"Gyftodimos, Flach, Elias Gyftodimos, Peter A Flach","year":"2002","venue":"Proceedings of the ICML-2002 Workshop on Development of Representations","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Hochreiter and Schmidhuber, 1997] Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8):1735-1780, 1997.","title":"Long short-term memory","authors":"Schmidhuber Hochreiter, Sepp Hochreiter, Jürgen Schmidhuber","year":"1997","venue":"Neural computation","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Jacobs and Lewis, 1978] Patricia A Jacobs and Peter AW Lewis. Discrete time series generated by mixtures. i: Cor- relational and runs properties. Journal of the Royal Sta- tistical Society: Series B (Methodological), 40(1):94-105, 1978.","title":"Discrete time series generated by mixtures. i: Correlational and runs properties","authors":"Patricia A Jacobs, Peter Aw Lewis","year":"1978","venue":"Journal of the Royal Statistical Society: Series B (Methodological)","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Kearns and Koller, 1999] Michael Kearns and Daphne Koller. Efficient reinforcement learning in factored mdps. In IJCAI, volume 16, pages 740-747, 1999.","title":"Efficient reinforcement learning in factored mdps","authors":"Koller Kearns, Michael Kearns, Daphne Koller","year":"1999","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/1999/106"},{"position":16,"raw":"[Metzner et al., 2009] Philipp Metzner, Christof Schütte, and Eric Vanden-Eijnden. Transition path theory for markov jump processes. Multiscale Modeling & Simula- tion, 7(3):1192-1219, 2009.","title":"Transition path theory for markov jump processes","authors":"Philipp Metzner, Christof Schütte, Eric Vanden-Eijnden","year":"2009","venue":"Multiscale Modeling & Simulation","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Milch et al., 2005] Brian Milch, Bhaskara Marthi, David Sontag, Stuart Russell, Daniel L Ong, and Andrey Kolobov. Approximate inference for infinite contingent bayesian networks. In International Workshop on Arti- ficial Intelligence and Statistics, pages 238-245. PMLR, 2005.","title":"Approximate inference for infinite contingent bayesian networks","authors":"Brian Milch, Bhaskara Marthi, David Sontag, Stuart Russell, Daniel L Ong, Andrey Kolobov","year":"2005","venue":"International Workshop on Artificial Intelligence and Statistics","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Parzen, 1999] Emanuel Parzen. Stochastic processes. SIAM, 1999.","title":"Stochastic processes","authors":"Emanuel Parzen","year":"1999","venue":null,"doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Pearl, 2003] Judea Pearl. Statistics and causal inference: A review. Test, 12(2):281-345, 2003.","title":"Judea Pearl. Statistics and causal inference: A review","authors":"Pearl","year":"2003","venue":"Test","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Pentland and Liu, 1999] Alex Pentland and Andrew Liu. Modeling and prediction of human behavior. Neural com- putation, 11(1):229-242, 1999.","title":"Modeling and prediction of human behavior","authors":"Liu Pentland, Alex Pentland, Andrew Liu","year":"1999","venue":"Neural computation","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Petropoulos et al., 2017] Anastasios Petropoulos, Sotirios P Chatzis, and Stelios Xanthopoulos. A hidden markov model with dependence jumps for predictive modeling of multidimensional time-series. Information Sciences, 412:50-66, 2017.","title":"A hidden markov model with dependence jumps for predictive modeling of multidimensional time-series","authors":"Anastasios Petropoulos, Sotirios P Chatzis, Stelios Xanthopoulos","year":"2017","venue":"Information Sciences","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Raftery and Tavaré, 1994] Adrian Raftery and Simon Tavaré. Estimation and modelling repeated patterns in high order markov chains with the mixture transition distribution model. Journal of the Royal Statistical Society: Series C (Applied Statistics), 43(1):179-199, 1994.","title":"Estimation and modelling repeated patterns in high order markov chains with the mixture transition distribution model","authors":"Adrian Raftery, Simon Tavaré","year":"1994","venue":"Journal of the Royal Statistical Society: Series C (Applied Statistics)","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Raftery, 1985] Adrian E Raftery. A model for high-order markov chains. Journal of the Royal Statistical Society: Series B (Methodological), 47(3):528-539, 1985.","title":"A model for high-order markov chains","authors":"Adrian E Raftery","year":"1985","venue":"Journal of the Royal Statistical Society: Series B (Methodological)","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Randall, 2006] Dana Randall. Rapidly mixing markov chains with applications in computer science and physics. Computing in Science & Engineering, 8(2):30-41, 2006.","title":"Rapidly mixing markov chains with applications in computer science and physics","authors":"Dana Randall","year":"2006","venue":"Computing in Science & Engineering","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Roucos et al., 1982] S Roucos, J Makhoul, and R Schwartz. A variable-order markov chain for coding of speech spec- tra. In ICASSP'82. IEEE International Conference on Acoustics, Speech, and Signal Processing, volume 7, pages 582-585. IEEE, 1982.","title":"A variable-order markov chain for coding of speech spectra","authors":"Roucos, Makhoul, Schwartz","year":"1982","venue":"IEEE International Conference on Acoustics, Speech, and Signal Processing","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Russell, 2010] Stuart J Russell. Artificial intelligence a modern approach. Pearson Education, Inc., 2010.","title":"Artificial intelligence a modern approach","authors":"Russell Stuart","year":"2010","venue":null,"doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Sallans and Hinton, 2004] Brian Sallans and Geoffrey E Hinton. Reinforcement learning with factored states and actions. The Journal of Machine Learning Research, 5:1063-1088, 2004.","title":"Reinforcement learning with factored states and actions","authors":"Brian Sallans, Geoffrey E Hinton","year":"2004","venue":"The Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Schmidt, 2000] Bernard Schmidt. The modelling of human behaviour, volume 132. Society for Computer Simulation International, 2000.","title":"The modelling of human behaviour","authors":"Bernard Schmidt","year":"2000","venue":null,"doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Stewart, 1994] William J Stewart. Introduction to the nu- merical solution of Markov chains. Princeton University Press, 1994.","title":"Introduction to the numerical solution of Markov chains","authors":"J William, Stewart","year":"1994","venue":null,"doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Sukthankar et al., 2014] Gita Sukthankar, Christopher Geib, Hung Bui, David Pynadath, and Robert P Gold- man. Plan, activity, and intent recognition: Theory and practice. Newnes, 2014.","title":"Plan, activity, and intent recognition: Theory and practice","authors":"Gita Sukthankar, Christopher Geib, Hung Bui, David Pynadath, Robert P Goldman","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Sutton and McCallum, 2004] Charles Sutton and Andrew McCallum. Collective segmentation and labeling of dis- tant entities in information extraction. In ICML Workshop on Statistical Relational Learning and Its Connections to Other Fields, 2004.","title":"Collective segmentation and labeling of distant entities in information extraction","authors":"Sutton, Mccallum, Charles Sutton, Andrew Mccallum","year":"2004","venue":"ICML Workshop on Statistical Relational Learning and Its Connections to Other Fields","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Sutton et al., 1999] Richard S Sutton, Doina Precup, and Satinder Singh. Between mdps and semi-mdps: A frame- work for temporal abstraction in reinforcement learning. Artificial intelligence, 112(1-2):181-211, 1999.","title":"Between mdps and semi-mdps: A framework for temporal abstraction in reinforcement learning","authors":"Doina Richard S Sutton, Satinder Precup, Singh","year":"1999","venue":"Artificial intelligence","doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Ye et al., 2004] Nong Ye, Yebin Zhang, and Connie M Borror. Robustness of the markov-chain model for cyber-attack detection. IEEE transactions on reliability, 53(1):116-123, 2004.","title":"Robustness of the markov-chain model for cyber-attack detection","authors":"Nong Ye, Yebin Zhang, Connie M Borror","year":"2004","venue":"IEEE transactions on reliability","doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[Zhang and Bucklew, 2022] Yu Zhang and Mitchell Buck- lew. Max markov chain. arXiv preprint arXiv:2211.01496, 2022.","title":"Max markov chain","authors":"Yu Zhang, Mitchell Bucklew","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":35,"raw":"[Zhang et al., 2015] Yu Zhang, Sarath Sreedharan, and Sub- barao Kambhampati. Capability models and their applica- tions in planning. In AAMAS, pages 1151-1159, 2015.","title":"Capability models and their applications in planning","authors":"Yu Zhang, Sarath Sreedharan, Subbarao Kambhampati","year":"2015","venue":"AAMAS","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}