Man vs. machine: Experimental evidence on the quality and perceptions of AI-generated research content
Academic researchers want their research to be understood and used by non-technical audiences, but that requires communication that is more accessible in the form of non-technical and shorter summaries. The researcher must both signal the quality of the research and ensure that the content is salien...
| Main Authors: | , , , , , |
|---|---|
| Format: | Artículo preliminar |
| Language: | Inglés |
| Published: |
International Food Policy Research Institute
2024
|
| Subjects: | |
| Online Access: | https://hdl.handle.net/10568/169363 |
| _version_ | 1855523219071565824 |
|---|---|
| author | Keenan, Michael Koo, Jawoo Mwangi, Christine Wamuyu Karachiwalla, Naureen Breisinger, Clemens Kim, MinAh |
| author_browse | Breisinger, Clemens Karachiwalla, Naureen Keenan, Michael Kim, MinAh Koo, Jawoo Mwangi, Christine Wamuyu |
| author_facet | Keenan, Michael Koo, Jawoo Mwangi, Christine Wamuyu Karachiwalla, Naureen Breisinger, Clemens Kim, MinAh |
| author_sort | Keenan, Michael |
| collection | Repository of Agricultural Research Outputs (CGSpace) |
| description | Academic researchers want their research to be understood and used by non-technical audiences, but that requires communication that is more accessible in the form of non-technical and shorter summaries. The researcher must both signal the quality of the research and ensure that the content is salient by making it more readable. AI tools can improve salience; however, they can also lead to ambiguity in the signal since true effort is then difficult to observe. We implement an online factorial experiment providing non-technical audiences with a blog on an academic paper and vary the actual author of the blog from the same paper (human or ChatGPT) and whether respondents are told the blog is written by a human or AI tool. Even though AI-generated blogs are objectively of higher quality, they are rated lower, but not if the author is disclosed as AI, indicating that signaling is important and can be distorted by AI. Use of the blog does not vary by experimental arm. The findings suggest that, provided disclosure statements are included, researchers can potentially use AI to reduce effort costs without compromising signaling or salience. Academic researchers want their research to be understood and used by non-technical audiences, but that requires communication that is more accessible in the form of non-technical and shorter summaries. The researcher must both signal the quality of the research and ensure that the content is salient by making it more readable. AI tools can improve salience; however, they can also lead to ambiguity in the signal since true effort is then difficult to observe. We implement an online factorial experiment providing non-technical audiences with a blog on an academic paper and vary the actual author of the blog from the same paper (human or ChatGPT) and whether respondents are told the blog is written by a human or AI tool. Even though AI-generated blogs are objectively of higher quality, they are rated lower, but not if the author is disclosed as AI, indicating that signaling is important and can be distorted by AI. Use of the blog does not vary by experimental arm. The findings suggest that, provided disclosure statements are included, researchers can potentially use AI to reduce effort costs without compromising signaling or salience. |
| format | Artículo preliminar |
| id | CGSpace169363 |
| institution | CGIAR Consortium |
| language | Inglés |
| publishDate | 2024 |
| publishDateRange | 2024 |
| publishDateSort | 2024 |
| publisher | International Food Policy Research Institute |
| publisherStr | International Food Policy Research Institute |
| record_format | dspace |
| spelling | CGSpace1693632025-12-02T21:02:52Z Man vs. machine: Experimental evidence on the quality and perceptions of AI-generated research content Keenan, Michael Koo, Jawoo Mwangi, Christine Wamuyu Karachiwalla, Naureen Breisinger, Clemens Kim, MinAh artificial intelligence communication research Academic researchers want their research to be understood and used by non-technical audiences, but that requires communication that is more accessible in the form of non-technical and shorter summaries. The researcher must both signal the quality of the research and ensure that the content is salient by making it more readable. AI tools can improve salience; however, they can also lead to ambiguity in the signal since true effort is then difficult to observe. We implement an online factorial experiment providing non-technical audiences with a blog on an academic paper and vary the actual author of the blog from the same paper (human or ChatGPT) and whether respondents are told the blog is written by a human or AI tool. Even though AI-generated blogs are objectively of higher quality, they are rated lower, but not if the author is disclosed as AI, indicating that signaling is important and can be distorted by AI. Use of the blog does not vary by experimental arm. The findings suggest that, provided disclosure statements are included, researchers can potentially use AI to reduce effort costs without compromising signaling or salience. Academic researchers want their research to be understood and used by non-technical audiences, but that requires communication that is more accessible in the form of non-technical and shorter summaries. The researcher must both signal the quality of the research and ensure that the content is salient by making it more readable. AI tools can improve salience; however, they can also lead to ambiguity in the signal since true effort is then difficult to observe. We implement an online factorial experiment providing non-technical audiences with a blog on an academic paper and vary the actual author of the blog from the same paper (human or ChatGPT) and whether respondents are told the blog is written by a human or AI tool. Even though AI-generated blogs are objectively of higher quality, they are rated lower, but not if the author is disclosed as AI, indicating that signaling is important and can be distorted by AI. Use of the blog does not vary by experimental arm. The findings suggest that, provided disclosure statements are included, researchers can potentially use AI to reduce effort costs without compromising signaling or salience. 2024-12-31 2025-01-17T17:28:21Z 2025-01-17T17:28:21Z Working Paper https://hdl.handle.net/10568/169363 en https://hdl.handle.net/10568/127434 Open Access application/pdf International Food Policy Research Institute Keenan, Michael; Koo, Jawoo; Mwangi, Christine; Karachiwalla, Naureen; Breisinger, Clemens; and Kim, MinAh. 2024. Man vs. machine: Experimental evidence on the quality and perceptions of AI-generated research content. IFPRI Discussion Paper 2321. Washington, DC: International Food Policy Research Institute. https://hdl.handle.net/10568/169363 |
| spellingShingle | artificial intelligence communication research Keenan, Michael Koo, Jawoo Mwangi, Christine Wamuyu Karachiwalla, Naureen Breisinger, Clemens Kim, MinAh Man vs. machine: Experimental evidence on the quality and perceptions of AI-generated research content |
| title | Man vs. machine: Experimental evidence on the quality and perceptions of AI-generated research content |
| title_full | Man vs. machine: Experimental evidence on the quality and perceptions of AI-generated research content |
| title_fullStr | Man vs. machine: Experimental evidence on the quality and perceptions of AI-generated research content |
| title_full_unstemmed | Man vs. machine: Experimental evidence on the quality and perceptions of AI-generated research content |
| title_short | Man vs. machine: Experimental evidence on the quality and perceptions of AI-generated research content |
| title_sort | man vs machine experimental evidence on the quality and perceptions of ai generated research content |
| topic | artificial intelligence communication research |
| url | https://hdl.handle.net/10568/169363 |
| work_keys_str_mv | AT keenanmichael manvsmachineexperimentalevidenceonthequalityandperceptionsofaigeneratedresearchcontent AT koojawoo manvsmachineexperimentalevidenceonthequalityandperceptionsofaigeneratedresearchcontent AT mwangichristinewamuyu manvsmachineexperimentalevidenceonthequalityandperceptionsofaigeneratedresearchcontent AT karachiwallanaureen manvsmachineexperimentalevidenceonthequalityandperceptionsofaigeneratedresearchcontent AT breisingerclemens manvsmachineexperimentalevidenceonthequalityandperceptionsofaigeneratedresearchcontent AT kimminah manvsmachineexperimentalevidenceonthequalityandperceptionsofaigeneratedresearchcontent |