{"type":"ScholarlyArticle","title":"A Distributed and Scalable Machine Learning Approach for Big Data","authors":[{"name":"Hongliang Guo","slug":"hongliang-guo","given":"Hongliang","family":"Guo","affiliation":""},{"name":"Jie Zhang","slug":"jie-zhang","given":"Jie","family":"Zhang","affiliation":""}],"abstract":"With the rapid development of data sensing and collection technologies, we can easily obtain large volumes of data (big data). However, big data poses huge challenges to many popular machine learning techniques which take all the data at the same time for processing. To address the big data related challenges, we first partition the data along its feature space, and apply the parallel block coordinate descent algorithm for distributed computation; then, we continue to partition the data along the sample space, and propose a novel matrix decomposition and combination approach for distributed processing. The final results from all the entities are guaranteed to be the same as the centralized solution. Extensive experiments performed on Hadoop confirm that our proposed approach is superior in terms of both testing errors and convergence rate (computation time) over the canonical distributed machine learning techniques that deal with big data.","doi":null,"url":"https://ijcai.org/proceedings/2016/217","pdf":"https://ijcai.org/proceedings/2016/0217.pdf","markdown":"https://ijcai.org/proceedings/2016/217.md","bibtex":"https://ijcai.org/proceedings/2016/bibtex/217","volume":{"slug":"2016","title":"Proceedings of the Twenty-Fifth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2016","url":"https://ijcai.org/proceedings/2016/","year":2016,"location":"New York, NY, USA","dates":"9–15 July 2016","editors":"Subbarao Kambhampati","isbn":"978-1-57735-770-4; 978-1-57735-771-1","publisher":"AAAI Press / International Joint Conferences on Artificial Intelligence","doi_prefix":""},"section":null,"track":"","number":217,"pages":{"first":1512,"last":1517},"keywords":[],"videos":[],"references":[],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}