Online Spatio-Temporal Learning in Deep Neural Networksстатья из журнала
Аннотация: Biological neural networks are equipped with an inherent capability to continuously adapt through online learning. This aspect remains in stark contrast to learning with error backpropagation through time (BPTT) that involves offline computation of the gradients due to the need to unroll the network through time. Here, we present an alternative online learning algorithm ic framework for deep recurrent neural networks (RNNs) and spiking neural networks (SNNs), called online spatio-temporal learning (OSTL). It is based on insights from biology and proposes the clear separation of spatial and temporal gradient components. For shallow SNNs, OSTL is gradient equivalent to BPTT enabling for the first time online training of SNNs with BPTT-equivalent gradients. In addition, the proposed formulation unveils a class of SNN architectures trainable online at low time complexity. Moreover, we extend OSTL to a generic form, applicable to a wide range of network architectures, including networks comprising long short-term memory (LSTM) and gated recurrent units (GRUs). We demonstrate the operation of our algorithm ic framework on various tasks from language modeling to speech recognition and obtain results on par with the BPTT baselines.
Год издания: 2022
Издательство: Institute of Electrical and Electronics Engineers
Источник: IEEE Transactions on Neural Networks and Learning Systems
Ключевые слова: Advanced Memory and Neural Computing, Neural Networks and Applications, Neural Networks and Reservoir Computing
Другие ссылки: IEEE Transactions on Neural Networks and Learning Systems (PDF)
IEEE Transactions on Neural Networks and Learning Systems (HTML)
arXiv (Cornell University) (PDF)
arXiv (Cornell University) (HTML)
PubMed (HTML)
IEEE Transactions on Neural Networks and Learning Systems (HTML)
arXiv (Cornell University) (PDF)
arXiv (Cornell University) (HTML)
PubMed (HTML)
Открытый доступ: hybrid
Том: 34
Выпуск: 11
Страницы: 8894–8908