Capabilities and Limitations in Scientific Application of LLMs: Preliminary Data Analysis, Bias, and Validation

Abstract

Large language models (LLMs) are becoming increasingly popular within the academic community, and their use is now more frequently observed in social science. This article summarizes the emerging opportunities for integrating LLMs into textual data analysis and systematizes the limitations that scholars encounter along the way. The authors identify the most effective «points of entry» for incorporating LLMs into research, while devoting particular attention to issues of model bias, validation, and the replication of results produced with the AI-assistance. The paper proposes several possible strategies for enhancing the quality of work with generative models in accordance with the triangulation procedure. These include examining alternative prompts, testing various data samples, and employing a combination of models. Existing studies show that, when properly configured, LLMs can reduce time costs, expand researchers’ analytical capacities, and help uncover hidden patterns within large textual corpora. However, the effectiveness of scientific applications of LLMs directly depends on scholarly diligence, including a clear understanding of the tool’s scope, careful problem formulation, high-quality input data, and the ability to normalize unstructured data. Without these conditions, the use of such models risks devolving into a simulation of scientific inquiry. The article is intended to serve as a starting point for political scientists and international relations researchers interested in integrating of LLMs into their analytical work.