Unreasonable effectiveness of learning neural networks: From accessible states and robust ensembles to basic algorithmic schemes.

ABSTRACT: In artificial neural networks, learning from data is a computationally demanding task in which a large number of connection weights are iteratively tuned through stochastic-gradient-based heuristic processes over a cost function. It is not well understood how learning occurs in these systems, in particular how they avoid getting trapped in configurations with poor computational performance. Here, we study the difficult case of networks with discrete weights, where the optimization landscape is very rough even for simple architectures, and provide theoretical and numerical evidence of the existence of rare-but extremely dense and accessible-regions of configurations in the network weight space. We define a measure, the robust ensemble (RE), which suppresses trapping by isolated configuratio

SUBMITTER: Baldassi C 

PROVIDER: S-EPMC5137727 | biostudies-literature | 2016 Nov

REPOSITORIES: biostudies-literature