{"type":"ScholarlyArticle","title":"GeoLLM: Comparative Study and Enhancement of Large Language Models for Geospatial Knowledge Using Retrieval Augmented Generation(RAG)","authors":[{"name":"Serah Akojenu","slug":"serah-akojenu","given":"Serah","family":"Akojenu","affiliation":"Data Science Nigeria"},{"name":"Olubayo Adekanmbi","slug":"olubayo-adekanmbi","given":"Olubayo","family":"Adekanmbi","affiliation":"Data Science Nigeria"},{"name":"Anthony Soronnadi","slug":"anthony-soronnadi","given":"Anthony","family":"Soronnadi","affiliation":"Data Science Nigeria"}],"abstract":"In recent years, large language models (LLMs) have demonstrated exceptional ability in natural language processing tasks, yet their potential to enhance geospatial intelligence remains largely untapped. Geospatial data, which is found not only in maps but also in regular text, presents unique opportunities for the application of LLMs. This research seeks to investigate if Large Language Models can serve as accurate sources of geospatial information. \r\nTo achieve this, we assessed the geospatial capability of 6 open LLMs: Mistral, GPT 3.5 and 4.o, LLaMA 2 and 3, and Gemini, using 30 curated geospatial prompts grouped into three categories: geographic coordinates, basic spatial calculations, and descriptions of places. These prompts were translated into Yoruba to ensure contextual relevance. Subsequently, we employed retrieval-augmented generation (RAG) to enhance the models’ geospatial knowledge base. Our findings revealed that GPT 4.o and Mistral consistently outperformed the other models across all categories and languages, while the RAG drastically improved the LLMs’ knowledge. This study demonstrates the potential of LLMs in being an accurate source of geospatial information through the use of retrieval augmented generation (RAG).","doi":"10.24963/ijcai.aai4g.2024/2","url":"https://ijcai.org/proceedings_aai4g/2024/2","pdf":"https://ijcai.org/proceedings_aai4g/2024/0002.pdf","markdown":"https://ijcai.org/proceedings_aai4g/2024/2.md","bibtex":"https://ijcai.org/proceedings_aai4g/2024/bibtex/2","volume":{"slug":"aai4g-2024","title":"Proceedings of the Second IJCAI AI for Good Symposium in Africa hosted by Deep Learning Indaba","short_title":"AI for Good Africa 2024","url":"https://ijcai.org/proceedings_aai4g/2024/","year":2024,"location":"Dakar, Senegal","dates":"1-7 September 2024","editors":"Avishkar Bhoopchand, Girmaw Abebe Tadesse, Sibusisiwe Makhanya, Frank Dignum, Georgina Curto","isbn":"978-1-956792-07-2","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.aai4g.2024"},"section":"Indaba","track":"Indaba","number":2,"pages":{"first":15,"last":18},"keywords":["Indaba: Indaba"],"videos":[],"references":[{"position":1,"raw":"[AI, 2023] Meta AI. Llama: Open and efficient foundation language models. https://www.llama.com/, 2023.","title":"Meta AI. Llama: Open and efficient foundation language models","authors":"Ai","year":"2023","venue":null,"doi":null,"url":"https://www.llama.com/,2023","internal_url":null},{"position":2,"raw":"[AI, 2024] Mistral AI. Mistral ai model. https://mistral.ai, 2024.","title":"Mistral ai model","authors":"A I Mistral","year":null,"venue":null,"doi":null,"url":"https://mistral.ai,2024","internal_url":null},{"position":3,"raw":"[Akojenu et al., 2023] Serah Sessi Akojenu, Olubayo Adekanmbi, and Anthony Soronnadi. Geo-visualization of hotspots of citizens dissatisfaction on social services using media print: A case study of fuel and cash scarcity in nigeria. In Submitted to Deep Learning Indaba 2023, 2023. Under review.","title":"Serah Sessi Akojenu, Olubayo Adekanmbi, and Anthony Soronnadi. Geo-visualization of hotspots of citizens dissatisfaction on social services using media print: A case study of fuel and cash scarcity in nigeria","authors":"Akojenu","year":"2023","venue":"Submitted to Deep Learning Indaba 2023","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Bhandari et al., 2023] Prabin Bhandari, Antonios Anasta- sopoulos, and Dieter Pfoser. Are large language models geospatially knowledgeable?, 2023.","title":"Are large language models geospatially knowledgeable?","authors":"Prabin Bhandari, Antonios Anastasopoulos, Dieter Pfoser","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Brown et al., 2020] Tom Brown, Benjamin Mann, Nick Ry- der, Melanie Subbiah, Jared D Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in Neural Information Processing Sys- tems, 33:1877-1901, 2020.","title":"Language models are few-shot learners","authors":"Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell","year":"2020","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[DeepMind, 2023] Google DeepMind. Gemini ai model. https://deepmind.google/technologies/gemini/pro/, 2023.","title":"Google DeepMind. Gemini ai model","authors":"Deepmind","year":"2023","venue":null,"doi":null,"url":"https://deepmind.google/technologies/gemini/pro/,2023","internal_url":null},{"position":7,"raw":"[Li et al., 2024] Jiarui Li, Ye Yuan, and Zehua Zhang. En- hancing llm factual accuracy with rag to counter halluci- nations: A case study on domain-specific queries in private knowledge-bases, 2024.","title":"Enhancing llm factual accuracy with rag to counter hallucinations: A case study on domain-specific queries in private knowledge-bases","authors":"Jiarui Li, Ye Yuan, Zehua Zhang","year":"2024","venue":null,"doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Mansourian and Oucheikh, 2024] Ali Mansourian and Rachid Oucheikh. Chatgeoai: Enabling geospatial analysis for public through natural language, with large language models. ISPRS International Journal of Geo-Information, 13(10):348, 2024.","title":"Chatgeoai: Enabling geospatial analysis for public through natural language, with large language models","authors":"Oucheikh Mansourian, Ali Mansourian, Rachid Oucheikh","year":"2024","venue":"ISPRS International Journal of Geo-Information","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Manvi et al., 2024] Rohin Manvi, Samar Khanna, Gengchen Mai, Marshall Burke, David Lobell, and Stefano Ermon. Geollm: Extracting geospatial knowl- edge from large language models. arXiv preprint arXiv:2310.06213v2, 2024. [OpenAI, 2024] OpenAI. Gpt-4 technical report. https: //openai.com/research/gpt-4, 2024.","title":"Geollm: Extracting geospatial knowledge from large language models","authors":"Rohin Manvi, Samar Khanna, Gengchen Mai, Marshall Burke, David Lobell, Stefano Ermon","year":"2024","venue":null,"doi":null,"url":"https://openai.com/research/gpt-4,2024","internal_url":null},{"position":10,"raw":"[Safavi and Koutra, 2021] Tara Safavi and Danai Koutra. Relational world knowledge representation in contextual language models: A review. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 1053-1067. Association for Computa- tional Linguistics, 2021.","title":"Relational world knowledge representation in contextual language models: A review","authors":"Tara Safavi, Danai Koutra","year":"2021","venue":"Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Sikiru et al., 2024] Rashidat Sikiru, Olubayo Adekanmbi, and Anthony Soronnadi. Comparative study of llms for personal financial decision in low resource language. In AfricaNLP Workshop at ICLR 2024, 2024.","title":"Comparative study of llms for personal financial decision in low resource language","authors":"Rashidat Sikiru, Olubayo Adekanmbi, Anthony Soronnadi","year":"2024","venue":"AfricaNLP Workshop at ICLR 2024","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}