Regularized One-Layer Neural Networks for Distributed and Incremental Environments

Abstract

Deploying machine learning models at scale is still a major challenge; one reason is that performance degrades when they are put into production. It is therefore very important to ensure the maximum possible generalization capacity of the models and regularization plays a key role in avoiding overfitting. We describe Regularized One-Layer Artificial Neural Network (ROLANN), a novel regularized training method for one-layer neural networks. Despite its simplicity, this network model has several advantages: it is noniterative, has low complexity, and is capable of incremental and privacy-preserving distributed learning, while maintaining or improving accuracy over other state- of-the-art methods as demonstrated by the experimental study in which it has been compared with ridge regression, lasso and elastic net over several data sets.

This work has been supported by grant Machine Learning on the Edge (Ayudas Fundación BBVA a Equipos de Investigación Científica 2019), also by the National Plan for Scientific and Technical R&I of the Spanish Government (Grant PID2019-109238GB-C2), and by the Xunta de Galicia (Grant ED431C 2018/34) with the European Union ERDF funds. CITIC is partially funded by “Consellería de Cultura, Educación e Universidades from Xunta de Galicia” (Grant ED431G 2019/01).