Scary_Stories_with_GPT_2.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "Pzxl1vYX-1kk"
   },
   "source": [
    "Setup:\n",
    "\n",
    "1) Make sure GPU is enabled, go to edit->notebook settings->Hardware Accelerator GPU\n",
    "\n",
    "2) Make a copy to your google drive, click on copy to drive in panel"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "iW0abT07ZkhZ"
   },
   "source": [
    "Note: Colab will reset after 12 hours make sure to save your model checkpoints to google drive around 10-11 hours mark or before, then go to runtime->reset all runtimes. Now copy your train model back into colab and start training again from the previous checkpoint."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "iLXW02eIYpcB"
   },
   "source": [
    "clone and cd into repo, nshepperd's fork https://github.com/nshepperd/gpt-2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 102
    },
    "colab_type": "code",
    "id": "ICYu3w9hIJkC",
    "outputId": "391e36a2-733c-40eb-94b9-d498d6bedc3c"
   },
   "outputs": [],
   "source": [
    "!git clone https://github.com/nshepperd/gpt-2.git"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "id": "6eEIs3ApZUVO",
    "outputId": "74d0384e-ca78-4173-fc98-610f80691722"
   },
   "outputs": [],
   "source": [
    "cd gpt-2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "Qtn1qZPgZLb0"
   },
   "source": [
    "Install requirements"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 681
    },
    "colab_type": "code",
    "id": "434oOx0bZH6J",
    "outputId": "f4dff668-a208-47f7-bdf0-3117051e8dcd"
   },
   "outputs": [],
   "source": [
    "!pip3 install -r requirements.txt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "WvUQhgK3PQ4L"
   },
   "source": [
    "Mount drive to access google drive for saving and accessing checkpoints later. Have to log in to your google account"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "o1hrgeKFYsuE"
   },
   "source": [
    "Download the model data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 136
    },
    "colab_type": "code",
    "id": "A498TySgHYyF",
    "outputId": "9e844511-eb52-4127-883d-a1570d688935"
   },
   "outputs": [],
   "source": [
    "!python3 download_model.py 117M"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 136
    },
    "colab_type": "code",
    "id": "5UDpEGjfO8Q2",
    "outputId": "b7548751-514b-410a-fb62-3ceddf233997"
   },
   "outputs": [],
   "source": [
    "!python3 download_model.py 345M"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "Zq-YwRnNOBYO"
   },
   "source": [
    "encoding"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "7oJPQtdLbbeK"
   },
   "outputs": [],
   "source": [
    "!export PYTHONIOENCODING=UTF-8"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 204
    },
    "colab_type": "code",
    "id": "QOCvrs-DHvxa",
    "outputId": "e8de43f2-e64d-4730-9b24-c7e8b62e346f"
   },
   "outputs": [],
   "source": [
    "!wget https://tmp-scary-story.s3.us-east-2.amazonaws.com/scary.txt #https://archive.org/stream/B-001-014-456/B-001-014-456_djvu.txt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "yPfJ5b3CQXqr"
   },
   "source": [
    "\n",
    "Start training, add --model_name '345M' to use 345 model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 1000
    },
    "colab_type": "code",
    "id": "pEn_ihcGI00T",
    "outputId": "86e020ea-1468-4685-8143-c8083e1f5a1e"
   },
   "outputs": [],
   "source": [
    "!PYTHONPATH=src ./train.py --dataset ./scary.txt --model_name '345M'"
   ]
  }
 ],
 "metadata": {
  "accelerator": "GPU",
  "colab": {
   "name": "Scary Stories with GPT-2.ipynb",
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}