2.7.3.   Training of Neural Networks

Many different learning algorithms exist for the training (calibration) of neural networks, which can be considered as an optimization problem. The first algorithm for the training of multilayer networks was proposed by Rumelhart and McClelland in 1986 [61]. This algorithm belongs to the gradient descent algorithms following the steepest descent of the error surface in the hyperspace of the adjustable parameters to find an acceptable minimum. To improve the excessively slow convergence of the algorithm for low gradients and to prevent trapping and oscillations in local minima a momentum term was introduced, which remembers the last change of the weights [62]. This algorithm, which smoothes the error surface, is also known as conjugate gradient descent. The introduction of individual learning rates and momentum terms for each weight significantly speeds up the algorithm and is known as delta bar algorithm [63] and SuperSAB algorithm [64]. One of the most modern algorithms of the gradient algorithms is the resilient propagation (Rprop) [65],[66]. This algorithm combines the ideas of the algorithms described before. Yet, the weights are not adapted depending on the magnitude of the derivative but depending only on the sequence of the signs of the derivative. Consequently, the learning is spread equally over the complete network in contrast to all the other methods described before. Another advantage is the insensitivity of the algorithm to the different parameters [59], which can be all set to theses values proposed in [65],[66] and implemented in [28],[67]. The algorithm has been successfully used for several chemometric applications due to its speed [68],[69].

Many modern learning algorithms with an amazing convergence speed belong to the second-order optimization methods, which utilize the Hessian matrix of partial second derivatives of the cost function, which describes how the error depends on the weights. The popular Levenberg-Marquardt algorithm [70] combines the gradient descent direction with the direction calculated from the estimated inverse Hessian matrix. The conjugate gradient algorithms also utilize the second-order information, but the estimation of the Hessian matrix is avoided. The scaled conjugate gradient algorithm (SCG) [71] makes use of the pseudo-second derivative, is insensitive to its parameters, which can be used as suggested in [71] and implemented in [65],[66], and has proven to be a very efficient algorithm with respect to convergence speed and optimization quality [101]. Recently, genetic algorithms have been suggested for training NN [72],[73]. As genetic algorithms are global optimization algorithms, they are more likely to find the global minimum of the error surface than gradient algorithms (see section 2.8.5 for more information about genetic algorithms). Yet, genetic learning algorithms are faced with several problems like long computing times, several parameters to be adjusted and troubles in fine-tuning the weights and biases. Combining genetic algorithms for the rough optimization and gradient algorithms for the fine-tuning has created a hybrid algorithm to overcome the last problem. Yet, the extremely high computing times render genetic algorithms for the training of neural networks unusable in practice, especially when several neural nets have to be trained for some kind of optimization (see chapters 7 and 8).

In this work, two learning algorithms are used. The Rprop algorithm is applied to the neural networks used in chapter 3. For all other networks SCG was used, as this algorithm shows a very fast initial convergence allowing the reduction in the number of training cycles during network optimization processes. All networks were trained with a maximum number of 2000 learning steps, whereby a method called early stopping was applied. This technique helps to anticipate the so-called overtraining effect [74]. An overtrained neural network learns a small calibration data set by heart. Thereby the noise in the data is learnt instead of generalizing the functional relationship of the data. For more details of overtraining, see section 2.8. Early stopping was implemented by monitoring the calibration data by a crossvalidation procedure (see section 2.4). The training is stopped when the error of crossvalidation of the calibration data starts going up, as the net may start loosing its generalization ability at this moment. Early stopping is not an ultimate solution for preventing overtraining as a premature stopping of the training also stops the calibration of the functional relationship behind the data. Early stopping is only a tool, which should be used in combination with the network optimization procedures (see section 2.8.2) and becomes less important with more optimized networks.