Training Recurrent Networks by Evolino

Analysis

Why This Paper Matters

This paper addresses a fundamental limitation of gradient-based training for recurrent neural networks (RNNs): the prevalence of local minima that can render gradient information useless. By proposing Evolino, the authors offer a practical alternative that leverages evolutionary algorithms for the nonlinear hidden weights while retaining efficient linear computation for output weights. This hybrid approach is particularly significant because it combines the global search capabilities of evolution with the analytical optimality of linear regression, avoiding the need for backpropagation through time.

The work is also notable for introducing evolutionary recurrent support vector machines, which extend the margin-maximization principle to RNNs. This opens up new possibilities for combining evolutionary optimization with kernel methods, potentially leading to more robust sequence learning models.

Technical Contributions

• Evolino algorithm: Separates the training of RNNs into two parts: evolution of hidden-to-hidden and input-to-hidden weights, and analytical computation of hidden-to-output weights using pseudo-inverse or quadratic programming.
• Evolutionary recurrent SVM: By replacing linear regression with quadratic programming to maximize the margin, the method produces the first evolutionary support vector machine for recurrent architectures.
• Application to LSTM: Shows that Evolino can be applied to LSTM networks, enabling them to solve tasks that gradient-based LSTM struggles with, such as certain continuous function generation problems.
• Comparison with Echo State Networks: Demonstrates that Evolino-based LSTM can solve tasks that Echo State Networks cannot, highlighting the advantage of evolving reservoir weights rather than using random fixed ones.

Results

The paper reports that Evolino-based LSTM achieves higher accuracy in continuous function generation tasks compared to conventional gradient descent RNNs, including gradient-based LSTM. Specific metrics are not detailed in the abstract, but the claim is supported by experimental comparisons. Additionally, Evolino-based LSTM successfully solves tasks that Echo State Networks fail on, indicating superior representational capacity or optimization capability.

Significance

Evolino contributes a novel training paradigm for RNNs that is particularly valuable when gradient information is unreliable. It broadens the applicability of RNNs to problems with rugged loss landscapes and provides a foundation for further research into hybrid evolutionary-analytical methods. The introduction of evolutionary recurrent SVMs also suggests potential cross-pollination between evolutionary computation and kernel methods, which could inspire new algorithms for sequence classification and regression. While the paper is from 2007, its ideas remain relevant for modern deep learning, especially in contexts where gradient-based training is challenging or where interpretability of linear output weights is desired.