SEMINAR:Large Generative Models & Autophagy

Guest: Gizem Gezici, Scuola Normale Superiore of Pisa, Italy

Title:  Large Generative Models & Autophagy

Date/Time: 8 May 2024, 13.40

Location: FENS G032

Abstract: In recent years, large generative models (LGMs) have become increasingly powerful, particularly in conversational applications like ChatGPT. As a result, they are now widely embraced by the public and generate a significant amount of content across various online platforms. Since LGMs are typically trained on datasets gathered from the internet, the content generated by these models may be used to train and fine-tune the next wave of LGMs. It is estimated that the total amount of high-quality text in the world is up to 17 trillion tokens, with a growth rate of around 4% every year. One of the leading models, Llama 2 by Meta, has been trained on approximately two trillion tokens. Due to the potential risk of the exhaustion of human-generated data, recent research in LGMs delves into the long-term impact of this self-consuming (or autophagous) loop on text and image data. In this work, we aim to establish a pipeline by using Llama2 to analyse the long term effects of the self- consuming loop for the summarisation task. The main objective is to investigate the potential loss of diversity through measuring the impact on minority topics/groups, i.e. popularity bias, and on language, i.e. language impoverishment or standardisation.

Bio: Gizem Gezici is an Assistant Professor of Artificial Intelligence at Scuola Normale Superiore of Pisa, Italy. Her research focuses on the ethical dimensions of AI, with a particular interest in investigating the long-term impacts on sociotechnical systems. She has expertise in information retrieval (IR) and natural language processing (NLP), with a proven track record of analysing and improving search platforms, leveraging state-of-the-art IR and NLP techniques. Her research also delves into applying deep learning-based architectures, such as Large Language Models (LLMs), to downstream tasks in NLP. Prior to joining academia, she worked in an R&D Centre where she applied cutting-edge IR and NLP approaches from academia to real-world search products on a large scale.