Studying Lexical Dynamics and Language Change via Generalized Entropies: The Problem of Sample Size.

ABSTRACT: Recently, it was demonstrated that generalized entropies of order α offer novel and important opportunities to quantify the similarity of symbol sequences where α is a free parameter. Varying this parameter makes it possible to magnify differences between different texts at specific scales of the corresponding word frequency spectrum. For the analysis of the statistical properties of natural languages, this is especially interesting, because textual data are characterized by Zipf's law, i.e., there are very few word types that occur very often (e.g., function words expressing grammatical relationships) and many word types with a very low frequency (e.g., content words carrying most of the meaning of a sentence). Here, this approach is systematically and empirically studied by analyzing the

SUBMITTER: Koplenig A 

PROVIDER: S-EPMC7514953 | biostudies-literature | 2019 May

REPOSITORIES: biostudies-literature