#theano-rnn
Demonstration of recurrent neural network implemented with Theano
- Theano
- Scikit-learn This relies on scikits-learn simply because I subclass their BaseEstimator class, but this dependency could easily be removed.
- A reasonably good Python distribution with numpy and scipy. I recommend Enthought because it is heavily optimized and it has a free academic license.
- If you want to use the Hessian-Free optimizer then you will also need: theano-hf
- rnn.py : this is the most basic implementation of a "vanilla" RNN. It is designed for readability. It processes each sequence at a time. There are three test functions which show how to train an RNN with real-valued, binary or softmax outputs using stochastic gradient descent.
- rnn_minibatch.py : this is similar to rnn.py but it is slightly more complicated code and processes multiple sequences at once for speed (i.e. in "mini-batches"). It also hooks into scipy.optimize to use more sophisticated optimization methods. It again includes three test functions based on output type.
- hf_example.py: this uses the class defined by rnn.py but instead of training it with stochastic gradient descent, it trains it with Martens and Sutskever's variant of Hessian-Free optimization.
There are other Theano rnn implementations publicly available, for example:
- Razvan Pascanu's implementation
- Razvan also provides a simple sketch. This was the starting point for this code. on the Theano-users list.
- Matthias Zoehrer's implementation