A large quantitative analysis of written language challenges the idea that all languages are equally complex.

ABSTRACT: One of the fundamental questions about human language is whether all languages are equally complex. Here, we approach this question from an information-theoretic perspective. We present a large scale quantitative cross-linguistic analysis of written language by training a language model on more than 6500 different documents as represented in 41 multilingual text collections consisting of ~ 3.5 billion words or ~ 9.0 billion characters and covering 2069 different languages that are spoken as a native language by more than 90% of the world population. We statistically infer the entropy of each language model as an index of what we call average prediction complexity. We compare complexity rankings across corpora and show that a language that tends to be more complex than another language in o

SUBMITTER: Koplenig A 

PROVIDER: S-EPMC10505229 | biostudies-literature | 2023 Sep

REPOSITORIES: biostudies-literature