1. Главная
  2. Ресурсы
  3. Библиотечный поиск
A random matrixapproach to neuralnetworks Cosme LouartZhenyu LiaoRomain Couillet2018 год

A random matrix approach to neural networks
статья из журнала

Полный текстСтраница публикацииПубликация в OpenAlex
Аннотация: This article studies the Gram random matrix model $G=\frac{1}{T}\Sigma^{{\mathsf{T}}}\Sigma$, $\Sigma=\sigma(WX)$, classically found in the analysis of random feature maps and random neural networks, where $X=[x_{1},\ldots,x_{T}]\in\mathbb{R}^{p\times T}$ is a (data) matrix of bounded norm, $W\in\mathbb{R}^{n\times p}$ is a matrix of independent zero-mean unit variance entries and $\sigma:\mathbb{R}\to\mathbb{R}$ is a Lipschitz continuous (activation) function—$\sigma(WX)$ being understood entry-wise. By means of a key concentration of measure lemma arising from nonasymptotic random matrix arguments, we prove that, as $n,p,T$ grow large at the same rate, the resolvent $Q=(G+\gamma I_{T})^{-1}$, for $\gamma>0$, has a similar behavior as that met in sample covariance matrix models, involving notably the moment $\Phi=\frac{T}{n}{\mathrm{E}}[G]$, which provides in passing a deterministic equivalent for the empirical spectral measure of $G$. Application-wise, this result enables the estimation of the asymptotic performance of single-layer random neural networks. This in turn provides practical insights into the underlying mechanisms into play in random neural networks, entailing several unexpected consequences, as well as a fast practical means to tune the network hyperparameters.
Год издания: 2018
Авторы: Cosme Louart, Zhenyu Liao, Romain Couillet
Издательство: Institute of Mathematical Statistics
Источник: The Annals of Applied Probability
Ключевые слова: Random Matrices and Applications, Matrix Theory and Algorithms, Markov Chains and Monte Carlo Methods
Другие ссылки: The Annals of Applied Probability (HTML)
hal.science (PDF)
hal.science (HTML)
HAL (Le Centre pour la Communication Scientifique Directe) (HTML)
arXiv (Cornell University) (PDF)
arXiv (Cornell University) (HTML)
HAL (Le Centre pour la Communication Scientifique Directe) (HTML)
HAL (Le Centre pour la Communication Scientifique Directe) (HTML)
HAL (Le Centre pour la Communication Scientifique Directe) (PDF)
HAL (Le Centre pour la Communication Scientifique Directe) (HTML)
HAL (Le Centre pour la Communication Scientifique Directe) (HTML)
Project Euclid (Cornell University) (PDF)
Project Euclid (Cornell University) (HTML)