examples/getting-started-movielens/03-Training-with-HugeCTR.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "afe54ce9",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Copyright 2021 NVIDIA Corporation. All Rights Reserved.\n",
    "#\n",
    "# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
    "# you may not use this file except in compliance with the License.\n",
    "# You may obtain a copy of the License at\n",
    "#\n",
    "#     http://www.apache.org/licenses/LICENSE-2.0\n",
    "#\n",
    "# Unless required by applicable law or agreed to in writing, software\n",
    "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
    "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
    "# See the License for the specific language governing permissions and\n",
    "# limitations under the License.\n",
    "# ==============================================================================\n",
    "\n",
    "# Each user is responsible for checking the content of datasets and the\n",
    "# applicable licenses and determining if suitable for the intended use."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "83fad6cb",
   "metadata": {},
   "source": [
    "<img src=\"https://developer.download.nvidia.com/notebooks/dlsw-notebooks/merlin_merlin_getting-started-movielens-03-training-with-hugectr/nvidia_logo.png\" style=\"width: 90px; float: right;\">\n",
    "\n",
    "# Getting Started MovieLens: Training with HugeCTR\n",
    "\n",
    "This notebook is created using the latest stable [merlin-hugectr](https://catalog.ngc.nvidia.com/orgs/nvidia/teams/merlin/containers/merlin-hugectr/tags) container.\n",
    "\n",
    "## Overview\n",
    "\n",
    "In this notebook, we want to provide an overview what HugeCTR framework is, its features and benefits. We will use HugeCTR to train a basic neural network architecture.\n",
    "\n",
    "<b>Learning Objectives</b>:\n",
    "* Adopt NVTabular workflow to provide input files to HugeCTR\n",
    "* Define HugeCTR neural network architecture\n",
    "* Train a deep learning model with HugeCTR"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "16956c69",
   "metadata": {},
   "source": [
    "### Why use HugeCTR?\n",
    "\n",
    "HugeCTR is a GPU-accelerated recommender framework designed to distribute training across multiple GPUs and nodes and estimate Click-Through Rates (CTRs).<br>\n",
    "\n",
    "HugeCTR offers multiple advantages to train deep learning recommender systems:\n",
    "1. **Speed**: HugeCTR is a highly efficient framework written in C++. We experienced up to 10x speed up. HugeCTR on a NVIDIA DGX A100 system proved to be the fastest commercially available solution for training the architecture Deep Learning Recommender Model (DLRM) developed by Facebook.\n",
    "2. **Scale**: HugeCTR supports model parallel scaling. It distributes the large embedding tables over multiple GPUs or multiple nodes. \n",
    "3. **Easy-to-use**: Easy-to-use Python API similar to Keras. Examples for popular deep learning recommender systems architectures (Wide&Deep, DLRM, DCN, DeepFM) are available."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a7e099b5",
   "metadata": {},
   "source": [
    "### Other Features of HugeCTR\n",
    "\n",
    "HugeCTR is designed to scale deep learning models for recommender systems. It provides a list of other important features:\n",
    "* Proficiency in oversubscribing models to train embedding tables with single nodes that don’t fit within the GPU or CPU memory (only required embeddings are prefetched from a parameter server per batch)\n",
    "* Asynchronous and multithreaded data pipelines\n",
    "* A highly optimized data loader.\n",
    "* Supported data formats such as parquet and binary\n",
    "* Integration with Triton Inference Server for deployment to production"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e8c0c88f",
   "metadata": {},
   "source": [
    "### Getting Started"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "065096f1",
   "metadata": {},
   "source": [
    "In this example, we will train a neural network with HugeCTR. We will use preprocessed datasets generated via NVTabular in `02-ETL-with-NVTabular` notebook."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "add3372c",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/usr/local/lib/python3.8/dist-packages/tqdm/auto.py:22: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html\n",
      "  from .autonotebook import tqdm as notebook_tqdm\n"
     ]
    }
   ],
   "source": [
    "# External dependencies\n",
    "import os\n",
    "import nvtabular as nvt"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ac22a3ba",
   "metadata": {},
   "source": [
    "We define our base directory, containing the data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "b81cacce",
   "metadata": {},
   "outputs": [],
   "source": [
    "# path to preprocessed data\n",
    "INPUT_DATA_DIR = os.environ.get(\n",
    "    \"INPUT_DATA_DIR\", os.path.expanduser(\"/workspace/nvt-examples/movielens/data/\")\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6750ce01",
   "metadata": {},
   "source": [
    "Let's load our saved workflow from the `02-ETL-with-NVTabular` notebook."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "5ab5923c",
   "metadata": {},
   "outputs": [],
   "source": [
    "workflow = nvt.Workflow.load(os.path.join(INPUT_DATA_DIR, \"workflow\"))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "aa5405c1",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'userId': dtype('int64'),\n",
       " 'movieId': dtype('int64'),\n",
       " 'genres': dtype('int64'),\n",
       " 'rating': dtype('int8')}"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "workflow.output_dtypes"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "94bef620",
   "metadata": {},
   "source": [
    "Note: We do not have numerical output columns"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9e8ad562",
   "metadata": {},
   "source": [
    "Let's clear existing directory should it exist from previous runs and create the output folders."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "0c678d48",
   "metadata": {},
   "outputs": [],
   "source": [
    "MODEL_DIR = os.path.join(INPUT_DATA_DIR, \"model/movielens_hugectr/\")\n",
    "!rm -rf {MODEL_DIR}\n",
    "!mkdir -p {MODEL_DIR}\"1\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "435c7e86",
   "metadata": {},
   "source": [
    "## Scaling Accelerated training with HugeCTR"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5b76f6ea",
   "metadata": {},
   "source": [
    "HugeCTR is a deep learning framework dedicated to recommendation systems. It is written in CUDA C++. As HugeCTR optimizes the training in CUDA++, we need to define the training pipeline and model architecture and execute it via the commandline. We will use the Python API, which is similar to Keras models.\n",
    "\n",
    "For more information on HugeCTR please consult the [HugeCTR repository](https://github.com/NVIDIA-Merlin/HugeCTR)."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0624ca30",
   "metadata": {},
   "source": [
    "## Let's define our model\n",
    "\n",
    "Let's define our model. We will write the model to `./train_hugeCTR.py` and execute it afterwards."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "baaf3563",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Overwriting train_hugeCTR.py\n"
     ]
    }
   ],
   "source": [
    "%%writefile train_hugeCTR.py\n",
    "\n",
    "# External dependencies\n",
    "import os\n",
    "import nvtabular as nvt\n",
    "from nvtabular.ops import get_embedding_sizes\n",
    "import hugectr\n",
    "from mpi4py import MPI  \n",
    "\n",
    "# path to preprocessed data\n",
    "INPUT_DATA_DIR = os.environ.get(\n",
    "    \"INPUT_DATA_DIR\", os.path.expanduser(\"/workspace/nvt-examples/movielens/data/\")\n",
    ")\n",
    "\n",
    "MODEL_DIR = os.path.join(INPUT_DATA_DIR, \"model/movielens_hugectr/\")\n",
    "\n",
    "workflow = nvt.Workflow.load(os.path.join(INPUT_DATA_DIR, \"workflow\"))\n",
    "\n",
    "embeddings = get_embedding_sizes(workflow)\n",
    "\n",
    "solver = hugectr.CreateSolver(\n",
    "    vvgpu=[[0]],\n",
    "    batchsize=2048,\n",
    "    batchsize_eval=2048,\n",
    "    max_eval_batches=160,\n",
    "    i64_input_key=True,\n",
    "    use_mixed_precision=False,\n",
    "    repeat_dataset=True,\n",
    ")\n",
    "optimizer = hugectr.CreateOptimizer(optimizer_type=hugectr.Optimizer_t.Adam)\n",
    "reader = hugectr.DataReaderParams(\n",
    "    data_reader_type=hugectr.DataReaderType_t.Parquet,\n",
    "    source=[INPUT_DATA_DIR + \"train/_file_list.txt\"],\n",
    "    eval_source=INPUT_DATA_DIR + \"valid/_file_list.txt\",\n",
    "    check_type=hugectr.Check_t.Non,\n",
    "    slot_size_array=[162542, 56586, 21],\n",
    ")\n",
    "\n",
    "\n",
    "model = hugectr.Model(solver, reader, optimizer)\n",
    "\n",
    "model.add(\n",
    "    hugectr.Input(\n",
    "        label_dim=1,\n",
    "        label_name=\"label\",\n",
    "        dense_dim=0,\n",
    "        dense_name=\"dense\",\n",
    "        data_reader_sparse_param_array=[\n",
    "            hugectr.DataReaderSparseParam(\"data1\", nnz_per_slot=10, is_fixed_length=False, slot_num=3)\n",
    "        ],\n",
    "    )\n",
    ")\n",
    "model.add(\n",
    "    hugectr.SparseEmbedding(\n",
    "        embedding_type=hugectr.Embedding_t.LocalizedSlotSparseEmbeddingHash,\n",
    "        workspace_size_per_gpu_in_mb=200,\n",
    "        embedding_vec_size=16,\n",
    "        combiner=\"sum\",\n",
    "        sparse_embedding_name=\"sparse_embedding1\",\n",
    "        bottom_name=\"data1\",\n",
    "        optimizer=optimizer,\n",
    "    )\n",
    ")\n",
    "model.add(\n",
    "    hugectr.DenseLayer(\n",
    "        layer_type=hugectr.Layer_t.Reshape,\n",
    "        bottom_names=[\"sparse_embedding1\"],\n",
    "        top_names=[\"reshape1\"],\n",
    "        leading_dim=48,\n",
    "    )\n",
    ")\n",
    "model.add(\n",
    "    hugectr.DenseLayer(\n",
    "        layer_type=hugectr.Layer_t.InnerProduct,\n",
    "        bottom_names=[\"reshape1\"],\n",
    "        top_names=[\"fc1\"],\n",
    "        num_output=128,\n",
    "    )\n",
    ")\n",
    "model.add(\n",
    "    hugectr.DenseLayer(\n",
    "        layer_type=hugectr.Layer_t.ReLU,\n",
    "        bottom_names=[\"fc1\"],\n",
    "        top_names=[\"relu1\"],\n",
    "    )\n",
    ")\n",
    "model.add(\n",
    "    hugectr.DenseLayer(\n",
    "        layer_type=hugectr.Layer_t.InnerProduct,\n",
    "        bottom_names=[\"relu1\"],\n",
    "        top_names=[\"fc2\"],\n",
    "        num_output=128,\n",
    "    )\n",
    ")\n",
    "model.add(\n",
    "    hugectr.DenseLayer(\n",
    "        layer_type=hugectr.Layer_t.ReLU,\n",
    "        bottom_names=[\"fc2\"],\n",
    "        top_names=[\"relu2\"],\n",
    "    )\n",
    ")\n",
    "model.add(\n",
    "    hugectr.DenseLayer(\n",
    "        layer_type=hugectr.Layer_t.InnerProduct,\n",
    "        bottom_names=[\"relu2\"],\n",
    "        top_names=[\"fc3\"],\n",
    "        num_output=1,\n",
    "    )\n",
    ")\n",
    "model.add(\n",
    "    hugectr.DenseLayer(\n",
    "        layer_type=hugectr.Layer_t.BinaryCrossEntropyLoss,\n",
    "        bottom_names=[\"fc3\", \"label\"],\n",
    "        top_names=[\"loss\"],\n",
    "    )\n",
    ")\n",
    "\n",
    "model.compile()\n",
    "model.summary()\n",
    "model.fit(max_iter=2000, display=100, eval_interval=200, snapshot=1900)\n",
    "model.graph_to_json(graph_config_file=MODEL_DIR + \"1/movielens.json\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e177dee5",
   "metadata": {},
   "source": [
    "Now please run the script we outputted above in the terminal using the following command:\n",
    "\n",
    "```python train_hugeCTR.py```"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2ae972f8",
   "metadata": {},
   "source": [
    "After training terminates, we can see that multiple `.model` files and folders are generated."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "9a6a869f",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0_opt_sparse_1900.model  _dense_1900.model  _opt_dense_1900.model\r\n",
      "\r\n",
      "0_sparse_1900.model:\r\n",
      "emb_vector  key  slot_id\r\n"
     ]
    }
   ],
   "source": [
    "ls *.model"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2d7dace8",
   "metadata": {},
   "source": [
    "Let's move these files into the `movielens_hugectr` folder. When we start the Triton Inference Server, we will be able to point it to that directory and ask it to load our model using the files we move there below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "b085ceb4",
   "metadata": {},
   "outputs": [],
   "source": [
    "!mv *.model {MODEL_DIR}"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.10"
  },
  "merlin": {
   "containers": [
    "nvcr.io/nvidia/merlin/merlin-hugectr:latest"
   ]
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}