Can we trust large language models to summarize food policy research papers and generate research briefs?
Generative large language models (LLMs), while widely accessible and capable of simulating policy recommendations, pose challenges in the assessment of their accuracy. Users, including policy analysts and decision-makers, bear the responsibility of evaluating the outcomes from these models. A signif...
| Autores principales: | , , |
|---|---|
| Formato: | Artículo preliminar |
| Lenguaje: | Inglés |
| Publicado: |
International Food Policy Research Institute
2023
|
| Materias: | |
| Acceso en línea: | https://hdl.handle.net/10568/137600 |
| _version_ | 1855514380036210688 |
|---|---|
| author | Kim, MinAh Koo, Jawoo Jung, Yunchul |
| author_browse | Jung, Yunchul Kim, MinAh Koo, Jawoo |
| author_facet | Kim, MinAh Koo, Jawoo Jung, Yunchul |
| author_sort | Kim, MinAh |
| collection | Repository of Agricultural Research Outputs (CGSpace) |
| description | Generative large language models (LLMs), while widely accessible and capable of simulating policy recommendations, pose challenges in the assessment of their accuracy. Users, including policy analysts and decision-makers, bear the responsibility of evaluating the outcomes from these models. A significant limitation of LLMs is their potential to overlook critical, context-specific factors. For example, in formulating food policies, it is vital to consider regional climate and environmental variables that influence water and resource availability. Nonetheless, due to their reliance on word sequencing probabilities from training datasets, LLMs might propose similar policies for distinct regions. Despite these limitations, LLMs offer considerable advantages for rapid policy analysis, particularly when resources are constrained. They serve as quick, accessible, and cost-effective tools for policy research and development, requiring minimal training and infrastructure. In our study, we assessed the efficacy of LLMs in generating policy briefs by inputting an IFPRI discussion paper into three different LLM-based approaches: a standard chatbot without extra data, a Retrieval Augmented Generation model integrating semantic search with LLM, and a custom-developed Brief Generator designed to create policy summaries from AI-analyzed paper structures. Our findings revealed that none of the LLM-generated briefs fully captured the original paper's intent, underscoring the need for further research. Future investigations should focus on gathering more empirical data with diverse text types and volumes to better understand these outcomes. |
| format | Artículo preliminar |
| id | CGSpace137600 |
| institution | CGIAR Consortium |
| language | Inglés |
| publishDate | 2023 |
| publishDateRange | 2023 |
| publishDateSort | 2023 |
| publisher | International Food Policy Research Institute |
| publisherStr | International Food Policy Research Institute |
| record_format | dspace |
| spelling | CGSpace1376002025-11-06T05:08:30Z Can we trust large language models to summarize food policy research papers and generate research briefs? Kim, MinAh Koo, Jawoo Jung, Yunchul policies artificial intelligence technological changes data Generative large language models (LLMs), while widely accessible and capable of simulating policy recommendations, pose challenges in the assessment of their accuracy. Users, including policy analysts and decision-makers, bear the responsibility of evaluating the outcomes from these models. A significant limitation of LLMs is their potential to overlook critical, context-specific factors. For example, in formulating food policies, it is vital to consider regional climate and environmental variables that influence water and resource availability. Nonetheless, due to their reliance on word sequencing probabilities from training datasets, LLMs might propose similar policies for distinct regions. Despite these limitations, LLMs offer considerable advantages for rapid policy analysis, particularly when resources are constrained. They serve as quick, accessible, and cost-effective tools for policy research and development, requiring minimal training and infrastructure. In our study, we assessed the efficacy of LLMs in generating policy briefs by inputting an IFPRI discussion paper into three different LLM-based approaches: a standard chatbot without extra data, a Retrieval Augmented Generation model integrating semantic search with LLM, and a custom-developed Brief Generator designed to create policy summaries from AI-analyzed paper structures. Our findings revealed that none of the LLM-generated briefs fully captured the original paper's intent, underscoring the need for further research. Future investigations should focus on gathering more empirical data with diverse text types and volumes to better understand these outcomes. 2023-12-31 2024-01-11T19:30:47Z 2024-01-11T19:30:47Z Working Paper https://hdl.handle.net/10568/137600 en Open Access application/pdf International Food Policy Research Institute Kim, MinAh; Koo, Jawoo; and Jung, Yunchul. 2023. Can we trust large language models to summarize food policy research papers and generate research briefs? Digital Innovation Working Paper December 2023. Washington, DC: International Food Policy Research Institute (IFPRI). https://hdl.handle.net/10568/137600 |
| spellingShingle | policies artificial intelligence technological changes data Kim, MinAh Koo, Jawoo Jung, Yunchul Can we trust large language models to summarize food policy research papers and generate research briefs? |
| title | Can we trust large language models to summarize food policy research papers and generate research briefs? |
| title_full | Can we trust large language models to summarize food policy research papers and generate research briefs? |
| title_fullStr | Can we trust large language models to summarize food policy research papers and generate research briefs? |
| title_full_unstemmed | Can we trust large language models to summarize food policy research papers and generate research briefs? |
| title_short | Can we trust large language models to summarize food policy research papers and generate research briefs? |
| title_sort | can we trust large language models to summarize food policy research papers and generate research briefs |
| topic | policies artificial intelligence technological changes data |
| url | https://hdl.handle.net/10568/137600 |
| work_keys_str_mv | AT kimminah canwetrustlargelanguagemodelstosummarizefoodpolicyresearchpapersandgenerateresearchbriefs AT koojawoo canwetrustlargelanguagemodelstosummarizefoodpolicyresearchpapersandgenerateresearchbriefs AT jungyunchul canwetrustlargelanguagemodelstosummarizefoodpolicyresearchpapersandgenerateresearchbriefs |