An adaptive method for determining the optimal number of topics in topic modeling.

ABSTRACT: Topic models have been successfully applied to information classification and retrieval. The difficulty in successfully applying these technologies is to select the appropriate number of topics for a given corpus. Selecting too few topics can result in information loss and topic omission, known as underfitting. Conversely, an excess of topics can introduce noise and complexity, resulting in overfitting. Therefore, this article considers the inter-class distance and proposes a new method to determine the number of topics based on clustering results, named average inter-class distance change rate (AICDR). AICDR employs the Ward's method to calculate inter-class distances, then calculates the average inter-class distance for different numbers of topics, and determines the optimal numbe

SUBMITTER: Xu Y 

PROVIDER: S-EPMC11888909 | biostudies-literature | 2025

REPOSITORIES: biostudies-literature