AI models collapse when trained on recursively generated data.

ABSTRACT: Stable diffusion revolutionized image creation from descriptive text. GPT-2 (ref. ¹), GPT-3(.5) (ref. ²) and GPT-4 (ref. ³) demonstrated high performance across a variety of language tasks. ChatGPT introduced such language models to the public. It is now clear that generative artificial intelligence (AI) such as large language models (LLMs) is here to stay and will substantially change the ecosystem of online text and images. Here we consider what may happen to GPT-{n} once LLMs contribute much of the text found online. We find that indiscriminate use of model-generated content in training causes irreversible defects in the resulting models, in which tails of the original content distribution disappear. We refer to this effect as 'model collapse' and show t

SUBMITTER: Shumailov I 

PROVIDER: S-EPMC11269175 | biostudies-literature | 2024 Jul

REPOSITORIES: biostudies-literature