A Generative AI approach for creating and validating simplified versions of government documents

Abstract

Abstract. In the context of Brazil’s re-democratization and the need for greater transparency in public administration, the 1988 Constitution established the right to access public information. However, the complexity of legal language, particularly in court documents, poses a significant barrier to understanding for the general public, especially given that only about 25% of Brazilians aged 25 or older have completed or are pursuing higher education. This study addresses this issue by leveraging generative AI models to simplify legal texts from the Court of Accounts of Pernambuco into plain language, making them more accessible to individuals with a high school education level. The research evaluates the effectiveness of two Large Language Models (GPT and Gemini) and five prompt techniques (Tree of Thought, COSTAR, Zero Shot, One Shot, and Meta Prompting) in producing simplified versions of 14 preliminary decisions. A total of 140 simplified texts were generated and evaluated using an 18-question questionnaire based on plain language principles, with scores generated by AI models and validated through human review. The results show that Gemini with the Tree of Thought technique achieved the highest average score (67.64), based on responses to the plain language questionnaire, while GPT with the COSTAR technique performed best in preserving essential information and achieving the highest readability scores (Flesch Reading Ease: 55.26). However, omissions of critical information were a common issue across all models, highlighting the need for human oversight. The study also found that GPT outperformed Gemini in evaluation accuracy, with lower Mean Absolute Error (MAE) and Mean Absolute Percentage Error (MAPE) compared to human evaluations. Despite this, AI models tended to overestimate readability and comprehension, underscoring the importance of a hybrid approach that combines AI-generated assessments with human review. The findings demonstrate the potential of generative AI to reduce costs and improve accessibility to legal and governmental documents, while also emphasizing the need for further research to address limitations such as omissions, biases, and ethical considerations. This study contributes to the growing body of literature on AI-assisted text simplification and provides a foundation for future work in this area.