The optimal regularization and its application in extreme learning machine for regression analysis and multi-class classification

Abstract

Extreme Learning Machine (ELM) proposed by Huang et al. (2006) is a newly developed single layer feed-forward neural network (SLFN). It is attractive for its high training efficiency and satisfactory performance, especially when dealing with a large amount of data, which are often in high-dimensional space. However, current ELM cannot solve the over-fitting problem among other several problems. While minimizing residuals of output errors for the training data, it tends to generate an over-fitting model, whose generalization ability is relatively weak. Even if the model fits the training data perfectly, it performs unsatisfactory for the testing data. In training process, we aim to minimize residuals of output errors of training data. It tends to generate an over-fitting model, which has poor generalization ability. The model maybe fit the training data perfectly, but performs badly in testing data. Furthermore, in order to improve accuracy, the traditional way is increasing the number of hidden-layer neurons, but excessive hidden-layer neurons result in an ill-posed normal matrix and a model which is over sensitive to the change of the training data. In such case, the performance of ELM is significantly affected by the outliers in the training data. In order to overcome these problems, we apply the regularization to the original ELM. In this study, the A-optimal design regularization is performed to improve the generalization ability and stability of ELM. The performance of ELM with the A-optimal design regularization will be evaluated through two main applications, respectively, regression analysis and satellite image multi-class classification.