Constant Error Flow!
Best 90s idea Hochreiter (back then an undergrad student on Schmidhuber’s long time lag recurrent net project, since 2002 assistant professor in Berlin)
Led to Long Short-Term Memory (LSTM):
No loss of short time lag capability
O(1) update complexity per time step and weight
Back to J. Schmidhuber's Recurrent neural network page