dev/docker-volume/notebooks/04-create-trusted-data-order-items.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Truted Data Generation - Order Items\n",
    "\n",
    "## Scope of notebook\n",
    "\n",
    "> Create `Order Items` dataset following the requirements below.\n",
    "\n",
    "* Order Items dataset - easy to read dataset with one-to-many relationship with Order dataset. Must contain all data from order items column."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Import libraries."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The autoreload extension is already loaded. To reload it, use:\n",
      "  %reload_ext autoreload\n"
     ]
    }
   ],
   "source": [
    "%load_ext autoreload\n",
    "%autoreload 2\n",
    "from src.config import RAW_DATA_PATH\n",
    "from src.IOController import create_pyspark_session\n",
    "from src.DataProcessor import create_trusted_order_items, explore_dataframe"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Spins off PySpark session."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Starting PySpark session. Check your terminal for detailed logging...\n",
      "PySpark session sucessfully created.\n"
     ]
    }
   ],
   "source": [
    "spark = create_pyspark_session()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Generates and exports Order Items dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Starting processing to generate Order Items dataset...\n",
      "Exporting dataset file system...\n",
      "Dataset sucessfully exported to `/home/jovyan/data/trusted/order_items`!\n",
      "(#rows, #columns) = (7494696, 14)\n",
      "root\n",
      " |-- order_id: string (nullable = true)\n",
      " |-- name: string (nullable = true)\n",
      " |-- addition: string (nullable = true)\n",
      " |-- discount: string (nullable = true)\n",
      " |-- quantity: float (nullable = true)\n",
      " |-- sequence: float (nullable = true)\n",
      " |-- unitPrice: string (nullable = true)\n",
      " |-- externalId: string (nullable = true)\n",
      " |-- totalValue: string (nullable = true)\n",
      " |-- customerNote: string (nullable = true)\n",
      " |-- garnishItems: string (nullable = true)\n",
      " |-- integrationId: string (nullable = true)\n",
      " |-- totalAddition: string (nullable = true)\n",
      " |-- totalDiscount: string (nullable = true)\n",
      "\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>order_id</th>\n",
       "      <td>dd4f8f0a-c2cb-45c6-a002-c3be6b305e5f</td>\n",
       "      <td>dd4f8f0a-c2cb-45c6-a002-c3be6b305e5f</td>\n",
       "      <td>8dd80f0b-db00-4b88-b7e2-02ca706fc5a5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>name</th>\n",
       "      <td>Parmegiana de Filé de Frango (2 pessoas)</td>\n",
       "      <td>Lasanha Frango (2 pessoas)</td>\n",
       "      <td>Filé Mignon à Cubana</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>addition</th>\n",
       "      <td>{\"value\":\"0\",\"currency\":\"BRL\"}</td>\n",
       "      <td>{\"value\":\"0\",\"currency\":\"BRL\"}</td>\n",
       "      <td>{\"value\":\"0\",\"currency\":\"BRL\"}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>discount</th>\n",
       "      <td>{\"value\":\"0\",\"currency\":\"BRL\"}</td>\n",
       "      <td>{\"value\":\"0\",\"currency\":\"BRL\"}</td>\n",
       "      <td>{\"value\":\"0\",\"currency\":\"BRL\"}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>quantity</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>sequence</th>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>unitPrice</th>\n",
       "      <td>{\"value\":\"2800\",\"currency\":\"BRL\"}</td>\n",
       "      <td>{\"value\":\"1800\",\"currency\":\"BRL\"}</td>\n",
       "      <td>{\"value\":\"0\",\"currency\":\"BRL\"}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>externalId</th>\n",
       "      <td>0bcd6764fd5e466d9c04b18ac0eb69e6</td>\n",
       "      <td>a361f5eec6a44ac0817892e81bf22e80</td>\n",
       "      <td>e0e81b2027c241cca5bf60a4768800ec</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>totalValue</th>\n",
       "      <td>{\"value\":\"2800\",\"currency\":\"BRL\"}</td>\n",
       "      <td>{\"value\":\"1800\",\"currency\":\"BRL\"}</td>\n",
       "      <td>{\"value\":\"0\",\"currency\":\"BRL\"}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>customerNote</th>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>garnishItems</th>\n",
       "      <td>[{\"name\":\"COM Arroz branco\",\"addition\":{\"value...</td>\n",
       "      <td>[]</td>\n",
       "      <td>[{\"name\":\"334 - 1/2 porção \",\"addition\":{\"valu...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>integrationId</th>\n",
       "      <td>PMFR</td>\n",
       "      <td>LF</td>\n",
       "      <td>None</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>totalAddition</th>\n",
       "      <td>{\"value\":\"0\",\"currency\":\"BRL\"}</td>\n",
       "      <td>{\"value\":\"0\",\"currency\":\"BRL\"}</td>\n",
       "      <td>{\"value\":\"0\",\"currency\":\"BRL\"}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>totalDiscount</th>\n",
       "      <td>{\"value\":\"0\",\"currency\":\"BRL\"}</td>\n",
       "      <td>{\"value\":\"0\",\"currency\":\"BRL\"}</td>\n",
       "      <td>{\"value\":\"0\",\"currency\":\"BRL\"}</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                               0  \\\n",
       "order_id                    dd4f8f0a-c2cb-45c6-a002-c3be6b305e5f   \n",
       "name                    Parmegiana de Filé de Frango (2 pessoas)   \n",
       "addition                          {\"value\":\"0\",\"currency\":\"BRL\"}   \n",
       "discount                          {\"value\":\"0\",\"currency\":\"BRL\"}   \n",
       "quantity                                                       1   \n",
       "sequence                                                       1   \n",
       "unitPrice                      {\"value\":\"2800\",\"currency\":\"BRL\"}   \n",
       "externalId                      0bcd6764fd5e466d9c04b18ac0eb69e6   \n",
       "totalValue                     {\"value\":\"2800\",\"currency\":\"BRL\"}   \n",
       "customerNote                                                None   \n",
       "garnishItems   [{\"name\":\"COM Arroz branco\",\"addition\":{\"value...   \n",
       "integrationId                                               PMFR   \n",
       "totalAddition                     {\"value\":\"0\",\"currency\":\"BRL\"}   \n",
       "totalDiscount                     {\"value\":\"0\",\"currency\":\"BRL\"}   \n",
       "\n",
       "                                                  1  \\\n",
       "order_id       dd4f8f0a-c2cb-45c6-a002-c3be6b305e5f   \n",
       "name                     Lasanha Frango (2 pessoas)   \n",
       "addition             {\"value\":\"0\",\"currency\":\"BRL\"}   \n",
       "discount             {\"value\":\"0\",\"currency\":\"BRL\"}   \n",
       "quantity                                          1   \n",
       "sequence                                          3   \n",
       "unitPrice         {\"value\":\"1800\",\"currency\":\"BRL\"}   \n",
       "externalId         a361f5eec6a44ac0817892e81bf22e80   \n",
       "totalValue        {\"value\":\"1800\",\"currency\":\"BRL\"}   \n",
       "customerNote                                   None   \n",
       "garnishItems                                     []   \n",
       "integrationId                                    LF   \n",
       "totalAddition        {\"value\":\"0\",\"currency\":\"BRL\"}   \n",
       "totalDiscount        {\"value\":\"0\",\"currency\":\"BRL\"}   \n",
       "\n",
       "                                                               2  \n",
       "order_id                    8dd80f0b-db00-4b88-b7e2-02ca706fc5a5  \n",
       "name                                        Filé Mignon à Cubana  \n",
       "addition                          {\"value\":\"0\",\"currency\":\"BRL\"}  \n",
       "discount                          {\"value\":\"0\",\"currency\":\"BRL\"}  \n",
       "quantity                                                       1  \n",
       "sequence                                                       1  \n",
       "unitPrice                         {\"value\":\"0\",\"currency\":\"BRL\"}  \n",
       "externalId                      e0e81b2027c241cca5bf60a4768800ec  \n",
       "totalValue                        {\"value\":\"0\",\"currency\":\"BRL\"}  \n",
       "customerNote                                                None  \n",
       "garnishItems   [{\"name\":\"334 - 1/2 porção \",\"addition\":{\"valu...  \n",
       "integrationId                                               None  \n",
       "totalAddition                     {\"value\":\"0\",\"currency\":\"BRL\"}  \n",
       "totalDiscount                     {\"value\":\"0\",\"currency\":\"BRL\"}  "
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tmp = create_trusted_order_items(spark)\n",
    "explore_dataframe(tmp)\n",
    "tmp.limit(3).toPandas().T"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Conclusion:\n",
    "\n",
    "> Indeed we've achieves a resulting DataFrame as required. It has one-to-many relationship to `Orders` dataset (primary key is `order_id`). Checking it's shape and first three rows there's a better feeling that the processing went well. A quick remark is that it has 7,5 million rows, thus producing an average 2 items per order since the Order dataset hast 3,6 million items. Seems reasonable!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Order Items dataset successfully generated!"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}