Performance Analysis of LLMs for Abstractive Summarization of Brazilian Legislative Documents

Abstract

Legislative documents present substantial obstacles to summarization due to their complex argument structures and specialized terminology. This research investigates the application of Large Language Models (LLMs) in summarizing Brazilian legislative proposals from the Chamber of Deputies, examining a dataset of over 56 thousand texts from 2013 to 2023. The paper explores three main summarization methodologies: extractive, abstractive, and hybrid, with an emphasis on abstractive summarization using LLMs. The performance of the LLM LLAMA2-13b is assessed using metrics such as ROUGE, BLEU, METEOR, BERTScore, and BERTopic, compared against reference summaries. The results show that LLMs can generate coherent and informative summaries, with positive evaluation metric results. Notably, the study reveals that traditional summary evaluation metrics may not be adequate for evaluating LLMs in summarization tasks. On the other hand, metrics based on pre-trained models like BERT provide a more effective evaluation of this innovative automatic summarization approach.