Escaping The Curse of Dimensionality in Bayesian Model-Based Clustering.

ABSTRACT: Bayesian mixture models are widely used for clustering of high-dimensional data with appropriate uncertainty quantification. However, as the dimension of the observations increases, posterior inference often tends to favor too many or too few clusters. This article explains this behavior by studying the random partition posterior in a non-standard setting with a fixed sample size and increasing data dimensionality. We provide conditions under which the finite sample posterior tends to either assign every observation to a different cluster or all observations to the same cluster as the dimension grows. Interestingly, the conditions do not depend on the choice of clustering prior, as long as all possible partitions of observations into clusters have positive prior probabilities, and hold irr

SUBMITTER: Chandra NK 

PROVIDER: S-EPMC11999651 | biostudies-literature | 2023 Apr

REPOSITORIES: biostudies-literature