{"type":"ScholarlyArticle","title":"Oasis: Data Curation and Assessment System for Pretraining of Large Language Models","authors":[{"name":"Tong Zhou","slug":"tong-zhou","given":"Tong","family":"Zhou","affiliation":"The Laboratory of Cognition and Decision Intelligence for Complex Systems , Institute of Automation , Chinese Academy of Sciences"},{"name":"Yubo Chen","slug":"yubo-chen","given":"Yubo","family":"Chen","affiliation":"The Laboratory of Cognition and Decision Intelligence for Complex Systems , Institute of Automation , Chinese Academy of Sciences; School of Artificial Intelligence , University of Chinese Academy of Sciences"},{"name":"Pengfei Cao","slug":"pengfei-cao","given":"Pengfei","family":"Cao","affiliation":"The Laboratory of Cognition and Decision Intelligence for Complex Systems , Institute of Automation , Chinese Academy of Sciences; School of Artificial Intelligence , University of Chinese Academy of Sciences"},{"name":"Kang Liu","slug":"kang-liu","given":"Kang","family":"Liu","affiliation":"The Laboratory of Cognition and Decision Intelligence for Complex Systems , Institute of Automation , Chinese Academy of Sciences; School of Artificial Intelligence , University of Chinese Academy of Sciences; Shanghai Artificial Intelligence Laboratory"},{"name":"Shengping Liu","slug":"shengping-liu","given":"Shengping","family":"Liu","affiliation":"Beijing Unisound Information Technology Co. , Ltd"},{"name":"Jun Zhao","slug":"jun-zhao","given":"Jun","family":"Zhao","affiliation":"The Laboratory of Cognition and Decision Intelligence for Complex Systems , Institute of Automation , Chinese Academy of Sciences; School of Artificial Intelligence , University of Chinese Academy of Sciences"}],"abstract":"Data is one of the most critical elements in building a large language model. However, existing systems either fail to customize a corpus curation pipeline or neglect to leverage comprehensive corpus assessment for iterative optimization of the curation. To this end, we present a pretraining corpus curation and assessment platform called Oasis — a one-stop system for data quality improvement and quantification with user-friendly interactive interfaces.  Specifically, the interactive modular rule filter module can devise customized rules according to explicit feedback. The debiased neural filter module builds the quality classification dataset in a negative-centric manner to remove the undesired bias. The adaptive document deduplication module could execute large-scale deduplication with limited memory resources. These three parts constitute the customized data curation module. And in the holistic data assessment module, a corpus can be assessed in local and global views, with three evaluation means including human, GPT-4, and heuristic metrics. We exhibit a complete process to use Oasis for the curation and assessment of pretraining data. In addition, an 800GB bilingual corpus curated by Oasis is publicly released.","doi":"10.24963/ijcai.2024/1048","url":"https://ijcai.org/proceedings/2024/1048","pdf":"https://ijcai.org/proceedings/2024/1048.pdf","markdown":"https://ijcai.org/proceedings/2024/1048.md","bibtex":"https://ijcai.org/proceedings/2024/bibtex/1048","volume":{"slug":"2024","title":"Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2024","url":"https://ijcai.org/proceedings/2024/","year":2024,"location":"Jeju, Korea","dates":"3-9 August 2024","editors":"Kate Larson","isbn":"978-1-956792-04-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2024"},"section":"Demo Track","track":"Demo Track","number":1048,"pages":{"first":8855,"last":8859},"keywords":["Natural Language Processing: NLP: Language models","Natural Language Processing: NLP: Applications","Natural Language Processing: NLP: Tools"],"videos":[],"references":[{"position":1,"raw":"[Abadji et al., 2022] Julien Abadji, Pedro Ortiz Suarez, Lau- rent Romary, and Benoît Sagot. Towards a cleaner document-oriented multilingual crawled corpus. arXiv preprint arXiv:2201.06642, 2022.","title":"Towards a cleaner document-oriented multilingual crawled corpus","authors":"Julien Abadji, Pedro Ortiz Suarez, Laurent Romary, Benoît Sagot","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Anil et al., 2023] Rohan Anil, Andrew M Dai, Orhan Fi- rat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, et al. Palm 2 technical report. arXiv preprint arXiv:2305.10403, 2023.","title":"Palm 2 technical report","authors":"Rohan Anil, Andrew M Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Chen et al., 2023] Lichang Chen, Shiyang Li, Jun Yan, Hai Wang, Kalpa Gunaratna, Vikas Yadav, Zheng Tang, Vijay Srinivasan, Tianyi Zhou, Heng Huang, et al. Alpagasus: Training a better alpaca with fewer data. arXiv preprint arXiv:2307.08701, 2023.","title":"Alpagasus: Training a better alpaca with fewer data","authors":"Chen","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Dodge et al., 2021] Jesse Dodge, Maarten Sap, Ana Marasović, William Agnew, Gabriel Ilharco, Dirk Groen- eveld, Margaret Mitchell, and Matt Gardner. Documenting large webtext corpora: A case study on the colossal clean crawled corpus. arXiv preprint arXiv:2104.08758, 2021.","title":"Documenting large webtext corpora: A case study on the colossal clean crawled corpus","authors":"Jesse Dodge, Maarten Sap, Ana Marasović, William Agnew, Gabriel Ilharco, Dirk Groeneveld, Margaret Mitchell, Matt Gardner","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Fries et al., 2022] Jason Fries, Leon Weber, Natasha Seelam, Gabriel Altay, Debajyoti Datta, Samuele Garda, Sunny Kang, Rosaline Su, Wojciech Kusa, Samuel Cahyawijaya, et al. Bigbio: a framework for data-centric biomedical nat- ural language processing. Advances in Neural Information Processing Systems, 35:25792-25806, 2022.","title":"Bigbio: a framework for data-centric biomedical natural language processing","authors":"Jason Fries, Leon Weber, Natasha Seelam, Gabriel Altay, Debajyoti Datta, Samuele Garda, Sunny Kang, Rosaline Su, Wojciech Kusa, Samuel Cahyawijaya","year":"2022","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Gao et al., 2020] Leo Gao, Stella Biderman, Sid Black, Lau- rence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, et al. The pile: An 800gb dataset of diverse text for language modeling. arXiv preprint arXiv:2101.00027, 2020.","title":"The pile: An 800gb dataset of diverse text for language modeling","authors":"Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Gunasekar et al., 2023] Suriya Gunasekar, Yi Zhang, Jyoti Aneja, Caio César Teodoro Mendes, Allie Del Giorno, Sivakanth Gopi, Mojan Javaheripi, Piero Kauffmann, Gus- tavo de Rosa, Olli Saarikivi, et al. Textbooks are all you need. arXiv preprint arXiv:2306.11644, 2023.","title":"Textbooks are all you need","authors":"Suriya Gunasekar, Yi Zhang, Jyoti Aneja, Caio César, Teodoro Mendes, Allie Del Giorno, Sivakanth Gopi, Mojan Javaheripi, Piero Kauffmann, Gustavo De Rosa, Olli Saarikivi","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Jansen et al., 2022] Tim Jansen, Yangling Tong, Victoria Ze- vallos, and Pedro Ortiz Suarez. Perplexed by quality: A perplexity-based method for adult and harmful content de- tection in multilingual heterogeneous web data. arXiv preprint arXiv:2212.10440, 2022.","title":"Perplexed by quality: A perplexity-based method for adult and harmful content detection in multilingual heterogeneous web data","authors":"Tim Jansen, Yangling Tong, Victoria Zevallos, Pedro Ortiz Suarez","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Kaplan et al., 2020] Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361, 2020.","title":"Scaling laws for neural language models","authors":"Jared Kaplan, Sam Mccandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, Dario Amodei","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Kreutzer et al., 2022] Julia Kreutzer, Isaac Caswell, Lisa Wang, Ahsan Wahab, Daan van Esch, Nasanbayar Ulzii- Orshikh, Allahsera Tapo, Nishant Subramani, Artem Sokolov, Claytone Sikasote, et al. Quality at a glance: An audit of web-crawled multilingual datasets. Transactions of the Association for Computational Linguistics, 10:50-72, 2022.","title":"Quality at a glance: An audit of web-crawled multilingual datasets","authors":"Julia Kreutzer, Isaac Caswell, Lisa Wang, Ahsan Wahab, Daan Van Esch, Nasanbayar Ulzii-Orshikh, Allahsera Tapo, Nishant Subramani, Artem Sokolov, Claytone Sikasote","year":"2022","venue":"Transactions of the Association for Computational Linguistics","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Laurenc ¸on et al., 2022] Hugo Laurenc ¸on, Lucile Saulnier, Thomas Wang, Christopher Akiki, Albert Villanova del Moral, Teven Le Scao, Leandro Von Werra, Chenghao Mou, Eduardo González Ponferrada, Huu Nguyen, et al. The bigscience roots corpus: A 1.6 tb composite multilin- gual dataset. Advances in Neural Information Processing Systems, 35:31809-31826, 2022.","title":"The bigscience roots corpus: A 1.6 tb composite multilingual dataset","authors":"Laurenc ¸on","year":"2022","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Lee et al., 2023] Alycia Lee, Brando Miranda, and Sanmi Koyejo. Beyond scale: the diversity coefficient as a data quality metric demonstrates llms are pre-trained on for- mally diverse data. arXiv preprint arXiv:2306.13840, 2023.","title":"Beyond scale: the diversity coefficient as a data quality metric demonstrates llms are pre-trained on formally diverse data","authors":"Alycia Lee, Brando Miranda, Sanmi Koyejo","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Longpre et al., 2023] Shayne Longpre, Gregory Yauney, Emily Reif, Katherine Lee, Adam Roberts, Barret Zoph, Denny Zhou, Jason Wei, Kevin Robinson, David Mimno, et al. A pretrainer's guide to training data: Measuring the effects of data age, domain coverage, quality, & toxicity. arXiv preprint arXiv:2305.13169, 2023.","title":"A pretrainer's guide to training data: Measuring the effects of data age, domain coverage, quality, & toxicity","authors":"Shayne Longpre, Gregory Yauney, Emily Reif, Katherine Lee, Adam Roberts, Barret Zoph, Denny Zhou, Jason Wei, Kevin Robinson, David Mimno","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[McCarthy and Jarvis, 2010] Philip M McCarthy and Scott Jarvis. Mtld, vocd-d, and hd-d: A validation study of sophisticated approaches to lexical diversity assessment. Behavior research methods, 42(2):381-392, 2010.","title":"Mtld, vocd-d, and hd-d: A validation study of sophisticated approaches to lexical diversity assessment","authors":"M Philip, Scott Mccarthy, Jarvis","year":"2010","venue":"Behavior research methods","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[OpenAI, 2023] R OpenAI. Gpt-4 technical report. arXiv, pages 2303-08774, 2023.","title":"Gpt-4 technical report","authors":"Openai","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Penedo et al., 2023] Guilherme Penedo, Quentin Malartic, Daniel Hesslow, Ruxandra Cojocaru, Alessandro Cappelli, Hamza Alobeidli, Baptiste Pannier, Ebtesam Almazrouei, and Julien Launay. The refinedweb dataset for falcon llm: outperforming curated corpora with web data, and web data only. arXiv preprint arXiv:2306.01116, 2023.","title":"The refinedweb dataset for falcon llm: outperforming curated corpora with web data, and web data only","authors":"Guilherme Penedo, Quentin Malartic, Daniel Hesslow, Ruxandra Cojocaru, Alessandro Cappelli, Hamza Alobeidli, Baptiste Pannier, Ebtesam Almazrouei, Julien Launay","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Rae et al., 2021] Jack W Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susan- nah Young, et al. Scaling language models: Methods, analysis & insights from training gopher. arXiv preprint arXiv:2112.11446, 2021.","title":"Scaling language models: Methods, analysis & insights from training gopher","authors":"Sebastian Jack W Rae, Trevor Borgeaud, Katie Cai, Jordan Millican, Francis Hoffmann, John Song, Sarah Aslanides, Roman Henderson, Susannah Ring, Young","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":18,"raw":"Corby Rosset. Turing-nlg: A 17-billion- parameter language model by microsoft. Microsoft Blog, 1(2), 2020.","title":"Turing-nlg: A 17-billionparameter language model by microsoft","authors":"Corby Rosset","year":"2020","venue":"Microsoft Blog","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Touvron et al., 2023] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023.","title":"Llama: Open and efficient foundation language models","authors":"Touvron","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Wei et al., 2022] Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, et al. Emer- gent abilities of large language models. arXiv preprint arXiv:2206.07682, 2022.","title":"Emergent abilities of large language models","authors":"Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Wenzek et al., 2019] Guillaume Wenzek, Marie-Anne Lachaux, Alexis Conneau, Vishrav Chaudhary, Francisco Guzmán, Armand Joulin, and Edouard Grave. Ccnet: Extracting high quality monolingual datasets from web crawl data. arXiv preprint arXiv:1911.00359, 2019.","title":"Ccnet: Extracting high quality monolingual datasets from web crawl data","authors":"Guillaume Wenzek, Marie-Anne Lachaux, Alexis Conneau, Vishrav Chaudhary, Francisco Guzmán, Armand Joulin, Edouard Grave","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Yuan et al., 2021] Sha Yuan, Hanyu Zhao, Zhengxiao Du, Ming Ding, Xiao Liu, Yukuo Cen, Xu Zou, Zhilin Yang, and Jie Tang. Wudaocorpora: A super large-scale chinese corpora for pre-training language models. AI Open, 2:65- 68, 2021.","title":"Wudaocorpora: A super large-scale chinese corpora for pre-training language models","authors":"Sha Yuan, Hanyu Zhao, Zhengxiao Du, Ming Ding, Xiao Liu, Yukuo Cen, Xu Zou, Zhilin Yang, Jie Tang","year":"2021","venue":"AI Open","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}