Szakdoga

My goal is to build an RNN that creates music lyrics based on the dataset.

Runnable installation

pip install -r requirements.txt

Train

To train use for example

python lyrics.py --artist=30y --epochs=250 --patience_limit=25 --lstm_layers=3 \
--lstm_units=64 --embedding=32 --size_x=100 --model_name=foobar

You can change any params, or you can just delete them (except for artist) or:

python lyrics.py --help

for info. For tensorboard use

tensorboard --logdir=target/train_log

from base folder (only works if you have at least 1 running/finished training).

Predict

python demo.py --model_name=${name_of_the_model}

or if you keep it blank you get the list of models you can predict from.

Latex

Installation

sudo apt-get update
sudo apt-get install texlive-full

Compiling

./doc/gen_pdf.sh

use -s flag if you want it to be silent.

Sources:

• RNN effectiveness (karpathy)
• Difficulty of training RNNs
• Visualizing and understanding RNNs
• AI written movie
• LSTM
  • paper about LSTM
  • colah
  • LSTM & diagrams
  • LSTM hard math
  • Deep sparse rectifier
  • Thesis on LSTM hyphenation (TDK, BME)
  • LSTM dropout
• Multitask learning
  • Multi task learning
  • Bidirectional Long Short-Term Memory Models and Auxiliary Loss
  • Semi-supervised Multitask Learning for Sequence Labeling

Frameworks

• Dynet
• Pytorch
• Keras
• Tensorflow

Word based encoding

• fastText
• fastText genism wrapper
• tokenization
• word embeddings (and motivation behind it)

Heuristics

• Bias-variance tradeoff (?)
• perplexity

Environment

• Anaconda

Deep learning A-Z

Deep learning Goodfellow

Software 2.0 - (karpathy)

Intro to ANN in tensorflow