examples/notebooks/rag/rag_1A_dpk_process_python.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "841e533d-ebb3-406d-9da7-b19e2c5f5866",
   "metadata": {},
   "source": [
    "<div style=\"background-color: #04D7FD; padding: 20px; text-align: left;\">\n",
    "    <h1 style=\"color: #000000; font-size: 36px; margin: 0;\">Data Processing for RAG with Data Prep Kit (Python)</h1>\n",
    "    \n",
    "</div>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b15976e3",
   "metadata": {},
   "source": [
    "## Before Running the notebook\n",
    "\n",
    "Please complete [setting up python dev environment](./setup-python-dev-env.md)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "053ecf08-5f62-4b99-9347-8a0955843d21",
   "metadata": {},
   "source": [
    "## Overview\n",
    "\n",
    "This notebook will process PDF documents as part of RAG pipeline\n",
    "\n",
    "![](media/rag-overview-2.png)\n",
    "\n",
    "This notebook will perform steps 1, 2 and 3 in RAG pipeline.\n",
    "\n",
    "Here are the processing steps:\n",
    "\n",
    "- **pdf2parquet** : Extract text from PDF and convert them into parquet files\n",
    "- **Chunk documents**: Split the PDFs into 'meaningful sections' (paragraphs, sentences ..etc)\n",
    "- **Doc_ID generation**: Each chunk is assigned a uniq id, based on content and hash\n",
    "- **Exact Dedup**: Chunks with exact same content are filtered out\n",
    "- **Text encoder**: Convert chunks into vectors using embedding models"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e8b10be1",
   "metadata": {},
   "source": [
    "## Step-1: Configuration"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "33345487",
   "metadata": {},
   "outputs": [],
   "source": [
    "from my_config import MY_CONFIG"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "facb3bbc",
   "metadata": {},
   "source": [
    "## Step-2:  Data\n",
    "\n",
    "We will use white papers  about LLMs.  \n",
    "\n",
    "- [Granite Code Models](https://arxiv.org/abs/2405.04324)\n",
    "- [Attention is all you need](https://arxiv.org/abs/1706.03762)\n",
    "\n",
    "You can of course substite your own data below"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f1fe7c0c",
   "metadata": {},
   "source": [
    "### 2.1 - Download data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "8739b7a2",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Local file 'input/attension.pdf' (2.22 MB) already exists. Skipping download.\n",
      "Local file 'input/granite.pdf' (1.27 MB) already exists. Skipping download.\n"
     ]
    }
   ],
   "source": [
    "import os, sys\n",
    "import shutil\n",
    "from utils import download_file\n",
    "\n",
    "## Download the data files\n",
    "shutil.os.makedirs(MY_CONFIG.INPUT_DATA_DIR, exist_ok=True)\n",
    "\n",
    "download_file (url = 'https://arxiv.org/pdf/1706.03762', local_file = os.path.join(MY_CONFIG.INPUT_DATA_DIR, 'attension.pdf' ))\n",
    "\n",
    "download_file (url = 'https://arxiv.org/pdf/2405.04324', local_file = os.path.join(MY_CONFIG.INPUT_DATA_DIR, 'granite.pdf' ))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "72510ae6-48b0-4b88-9e13-a623281c3a63",
   "metadata": {},
   "source": [
    "### 2.2 - Set input/output path variables for the pipeline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "60ac8bee-0960-4309-b225-d7a211b14262",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "✅ Cleared output directory\n"
     ]
    }
   ],
   "source": [
    "import os, sys\n",
    "import shutil\n",
    "\n",
    "if not os.path.exists(MY_CONFIG.INPUT_DATA_DIR ):\n",
    "    raise Exception (f\"❌ Input folder MY_CONFIG.INPUT_DATA_DIR = '{MY_CONFIG.INPUT_DATA_DIR}' not found\")\n",
    "\n",
    "output_parquet_dir = os.path.join (MY_CONFIG.OUTPUT_FOLDER, '01_parquet_out')\n",
    "output_chunk_dir = os.path.join (MY_CONFIG.OUTPUT_FOLDER, '02_chunk_out')\n",
    "output_docid_dir = os.path.join (MY_CONFIG.OUTPUT_FOLDER, '03_docid_out')\n",
    "output_exact_dedupe_dir = os.path.join (MY_CONFIG.OUTPUT_FOLDER, '04_exact_dedupe_out')\n",
    "output_embeddings_dir = os.path.join (MY_CONFIG.OUTPUT_FOLDER, '05_embeddings_out')\n",
    "\n",
    "## clear output folder\n",
    "shutil.rmtree(MY_CONFIG.OUTPUT_FOLDER, ignore_errors=True)\n",
    "shutil.os.makedirs(MY_CONFIG.OUTPUT_FOLDER, exist_ok=True)\n",
    "\n",
    "print (\"✅ Cleared output directory\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2449e5c7-078c-4ad6-a2f6-21d39d4da3fb",
   "metadata": {},
   "source": [
    "## Step-3: pdf2parquet -  Convert data from PDF to Parquet\n",
    "\n",
    "This step is reading the input folder containing all PDF files and ingest them in a parquet table using the [Docling package](https://github.com/DS4SD/docling).\n",
    "The documents are converted into a JSON format which allows to easily chunk it in the later steps.\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c0c574c4-9dc4-4dab-9ad6-b5338207e67a",
   "metadata": {},
   "source": [
    "### 3.1 -  Set Input/output Folder"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "482605b2-d814-456d-9195-49a2ec454ef0",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "🏃🏼 STAGE-1: Processing input='input' --> output='output/01_parquet_out'\n"
     ]
    }
   ],
   "source": [
    "STAGE = 1 \n",
    "\n",
    "input_folder = MY_CONFIG.INPUT_DATA_DIR\n",
    "output_folder =  output_parquet_dir\n",
    "\n",
    "print (f\"🏃🏼 STAGE-{STAGE}: Processing input='{input_folder}' --> output='{output_folder}'\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9bb15f02-ab5c-4525-a536-cfa1fd2ba70b",
   "metadata": {},
   "source": [
    "### 3.2 - Execute "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "b0cd8ebd-bf71-42d6-a397-8df0c7b66a26",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "00:23:40 INFO - pdf2parquet parameters are : {'artifacts_path': None, 'contents_type': <pdf2parquet_contents_types.JSON: 'application/json'>, 'do_table_structure': True, 'do_ocr': True, 'double_precision': 8}\n",
      "00:23:40 INFO - pipeline id pipeline_id\n",
      "00:23:40 INFO - code location None\n",
      "00:23:40 INFO - data factory data_ is using local data access: input_folder - input output_folder - output/01_parquet_out\n",
      "00:23:40 INFO - data factory data_ max_files -1, n_sample -1\n",
      "00:23:40 INFO - data factory data_ Not using data sets, checkpointing False, max files -1, random samples -1, files to use ['.pdf'], files to checkpoint ['.parquet']\n",
      "00:23:40 INFO - orchestrator pdf2parquet started at 2024-10-02 00:23:40\n",
      "00:23:40 INFO - Number of files is 2, source profile {'max_file_size': 2.112621307373047, 'min_file_size': 1.2146415710449219, 'total_file_size': 3.3272628784179688}\n",
      "00:23:40 INFO - Initializing models\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "bd58971a33d4410c91e742e735a6e6e3",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Fetching 10 files:   0%|          | 0/10 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "7f8e0a57cf5b485aa42d95f1afe1379b",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "model.pt:   0%|          | 0.00/201M [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "00:24:14 INFO - Completed 1 files (50.0%) in 0.3 min\n",
      "00:24:48 INFO - Completed 2 files (100.0%) in 0.871 min\n",
      "00:24:48 INFO - Done processing 2 files, waiting for flush() completion.\n",
      "00:24:48 INFO - done flushing in 0.0 sec\n",
      "00:24:48 INFO - Completed execution in 1.137 min, execution result 0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "✅ Stage:1 completed successfully\n",
      "CPU times: user 2min 9s, sys: 3.15 s, total: 2min 12s\n",
      "Wall time: 1min 11s\n"
     ]
    }
   ],
   "source": [
    "%%time \n",
    "\n",
    "import ast\n",
    "import os\n",
    "import sys\n",
    "\n",
    "from data_processing.runtime.pure_python import PythonTransformLauncher\n",
    "from data_processing.utils import ParamsUtils\n",
    "\n",
    "from pdf2parquet_transform import (\n",
    "    pdf2parquet_contents_type_cli_param,\n",
    "    pdf2parquet_contents_types,\n",
    ")\n",
    "from pdf2parquet_transform_python import Pdf2ParquetPythonTransformConfiguration\n",
    "\n",
    "\n",
    "# create parameters\n",
    "local_conf = {\n",
    "    \"input_folder\": input_folder,\n",
    "    \"output_folder\": output_folder,\n",
    "}\n",
    "ingest_config = {\n",
    "    pdf2parquet_contents_type_cli_param: pdf2parquet_contents_types.JSON,\n",
    "}\n",
    "\n",
    "params = {\n",
    "    # Data access. Only required parameters are specified\n",
    "    \"data_local_config\": ParamsUtils.convert_to_ast(local_conf),\n",
    "    \"data_files_to_use\": ast.literal_eval(\"['.pdf']\"),\n",
    "}\n",
    "\n",
    "\n",
    "sys.argv = ParamsUtils.dict_to_req(d=(params | ingest_config))\n",
    "# create launcher\n",
    "launcher = PythonTransformLauncher(Pdf2ParquetPythonTransformConfiguration())\n",
    "# launch\n",
    "return_code = launcher.launch()\n",
    "\n",
    "if return_code == 0:\n",
    "    print (f\"✅ Stage:{STAGE} completed successfully\")\n",
    "else:\n",
    "    raise Exception (\"❌ Job failed\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5ca790e0",
   "metadata": {},
   "source": [
    "### 3.3 -  Inspect Generated output\n",
    "\n",
    "Here we should see one entry per input file processed"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "fe59563d",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Output dimensions (rows x columns)=  (2, 12)\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>filename</th>\n",
       "      <th>contents</th>\n",
       "      <th>num_pages</th>\n",
       "      <th>num_tables</th>\n",
       "      <th>num_doc_elements</th>\n",
       "      <th>document_id</th>\n",
       "      <th>ext</th>\n",
       "      <th>hash</th>\n",
       "      <th>size</th>\n",
       "      <th>date_acquired</th>\n",
       "      <th>pdf_convert_time</th>\n",
       "      <th>source_filename</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>granite.pdf</td>\n",
       "      <td>{\"_name\":\"\",\"type\":\"pdf-document\",\"description...</td>\n",
       "      <td>28</td>\n",
       "      <td>17</td>\n",
       "      <td>348</td>\n",
       "      <td>4a32ba4c-8fdb-4eeb-a06b-d28493efe8e3</td>\n",
       "      <td>pdf</td>\n",
       "      <td>0650e590f33356ab8581c7eb0c23f1b928f0cfe1659587...</td>\n",
       "      <td>654989</td>\n",
       "      <td>2024-10-02T00:24:48.959612</td>\n",
       "      <td>34.223920</td>\n",
       "      <td>granite.pdf</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>attension.pdf</td>\n",
       "      <td>{\"_name\":\"\",\"type\":\"pdf-document\",\"description...</td>\n",
       "      <td>15</td>\n",
       "      <td>4</td>\n",
       "      <td>193</td>\n",
       "      <td>f275d75a-a072-4836-8a55-6a65f0d34577</td>\n",
       "      <td>pdf</td>\n",
       "      <td>6fe23d4f932c725077dfc8334f3f4da4e3aaf908d2aa23...</td>\n",
       "      <td>135814</td>\n",
       "      <td>2024-10-02T00:24:14.713654</td>\n",
       "      <td>18.004455</td>\n",
       "      <td>attension.pdf</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "        filename                                           contents  \\\n",
       "0    granite.pdf  {\"_name\":\"\",\"type\":\"pdf-document\",\"description...   \n",
       "1  attension.pdf  {\"_name\":\"\",\"type\":\"pdf-document\",\"description...   \n",
       "\n",
       "   num_pages  num_tables  num_doc_elements  \\\n",
       "0         28          17               348   \n",
       "1         15           4               193   \n",
       "\n",
       "                            document_id  ext  \\\n",
       "0  4a32ba4c-8fdb-4eeb-a06b-d28493efe8e3  pdf   \n",
       "1  f275d75a-a072-4836-8a55-6a65f0d34577  pdf   \n",
       "\n",
       "                                                hash    size  \\\n",
       "0  0650e590f33356ab8581c7eb0c23f1b928f0cfe1659587...  654989   \n",
       "1  6fe23d4f932c725077dfc8334f3f4da4e3aaf908d2aa23...  135814   \n",
       "\n",
       "                date_acquired  pdf_convert_time source_filename  \n",
       "0  2024-10-02T00:24:48.959612         34.223920     granite.pdf  \n",
       "1  2024-10-02T00:24:14.713654         18.004455   attension.pdf  "
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from utils import read_parquet_files_as_df\n",
    "\n",
    "output_df = read_parquet_files_as_df(output_folder)\n",
    "\n",
    "print (\"Output dimensions (rows x columns)= \", output_df.shape)\n",
    "\n",
    "output_df.head(5)\n",
    "\n",
    "## To display certain columns\n",
    "#parquet_df[['column1', 'column2', 'column3']].head(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "72274586",
   "metadata": {},
   "source": [
    "##  Step-4: Doc chunks\n",
    "\n",
    "Split the documents in chunks, according to their layout segmentation."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "96198fa6",
   "metadata": {},
   "source": [
    "### 4.1 - Set Input/output Folder"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "305f00a3",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "🏃🏼 STAGE-2: Processing input='output/01_parquet_out' --> output='output/02_chunk_out'\n"
     ]
    }
   ],
   "source": [
    "STAGE = 2\n",
    "\n",
    "input_folder = output_parquet_dir # previous output folder is the input folder for the current stage\n",
    "output_folder =  output_chunk_dir\n",
    "\n",
    "input_df = read_parquet_files_as_df(input_folder)  ## for debug purposes\n",
    "\n",
    "print (f\"🏃🏼 STAGE-{STAGE}: Processing input='{input_folder}' --> output='{output_folder}'\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "369f2cd1",
   "metadata": {},
   "source": [
    "### 4.2 - Execute "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "5b7b18d5",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "00:24:50 INFO - doc_chunk parameters are : {'chunking_type': <chunking_types.DL_JSON: 'dl_json'>, 'content_column_name': 'contents', 'doc_id_column_name': 'document_id', 'dl_min_chunk_len': None, 'output_chunk_column_name': 'contents', 'output_source_doc_id_column_name': 'source_document_id', 'output_jsonpath_column_name': 'doc_jsonpath', 'output_pageno_column_name': 'page_number', 'output_bbox_column_name': 'bbox'}\n",
      "00:24:50 INFO - pipeline id pipeline_id\n",
      "00:24:50 INFO - code location None\n",
      "00:24:50 INFO - data factory data_ is using local data access: input_folder - output/01_parquet_out output_folder - output/02_chunk_out\n",
      "00:24:50 INFO - data factory data_ max_files -1, n_sample -1\n",
      "00:24:50 INFO - data factory data_ Not using data sets, checkpointing False, max files -1, random samples -1, files to use ['.parquet'], files to checkpoint ['.parquet']\n",
      "00:24:50 INFO - orchestrator doc_chunk started at 2024-10-02 00:24:50\n",
      "00:24:50 INFO - Number of files is 2, source profile {'max_file_size': 0.12735748291015625, 'min_file_size': 0.035338401794433594, 'total_file_size': 0.16269588470458984}\n",
      "00:24:50 INFO - Completed 1 files (50.0%) in 0.0 min\n",
      "00:24:50 INFO - Completed 2 files (100.0%) in 0.004 min\n",
      "00:24:50 INFO - Done processing 2 files, waiting for flush() completion.\n",
      "00:24:50 INFO - done flushing in 0.0 sec\n",
      "00:24:50 INFO - Completed execution in 0.004 min, execution result 0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "✅ Stage:2 completed successfully\n",
      "CPU times: user 1.07 s, sys: 95.1 ms, total: 1.16 s\n",
      "Wall time: 1.19 s\n"
     ]
    }
   ],
   "source": [
    "%%time \n",
    "\n",
    "from data_processing.runtime.pure_python import PythonTransformLauncher\n",
    "from doc_chunk_transform_python import DocChunkPythonTransformConfiguration\n",
    "\n",
    "# Prepare the commandline params\n",
    "local_conf = {\n",
    "    \"input_folder\": input_folder,\n",
    "    \"output_folder\": output_folder,\n",
    "}\n",
    "worker_options = {\"num_cpus\" : MY_CONFIG.RAY_NUM_CPUS}\n",
    "params = {\n",
    "    # Data access. Only required parameters are specified\n",
    "    \"data_local_config\": ParamsUtils.convert_to_ast(local_conf),\n",
    "    # doc_chunk arguments\n",
    "    # ...\n",
    "}\n",
    "\n",
    "# Pass the commandline params\n",
    "sys.argv = ParamsUtils.dict_to_req(d=params)\n",
    "\n",
    "# create launcher\n",
    "launcher = PythonTransformLauncher(DocChunkPythonTransformConfiguration())\n",
    "# launch\n",
    "return_code = launcher.launch()\n",
    "\n",
    "if return_code == 0:\n",
    "    print (f\"✅ Stage:{STAGE} completed successfully\")\n",
    "else:\n",
    "    raise Exception (\"❌ Job failed\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "213afdf6",
   "metadata": {},
   "source": [
    "### 4.3 - Inspect Generated output\n",
    "\n",
    "We would see documents are split into many chunks"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "d8138d43",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Files processed : 2\n",
      "Chunks created : 211\n",
      "Input data dimensions (rows x columns)=  (2, 12)\n",
      "Output data dimensions (rows x columns)=  (211, 16)\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>filename</th>\n",
       "      <th>num_pages</th>\n",
       "      <th>num_tables</th>\n",
       "      <th>num_doc_elements</th>\n",
       "      <th>ext</th>\n",
       "      <th>hash</th>\n",
       "      <th>size</th>\n",
       "      <th>date_acquired</th>\n",
       "      <th>pdf_convert_time</th>\n",
       "      <th>source_filename</th>\n",
       "      <th>source_document_id</th>\n",
       "      <th>contents</th>\n",
       "      <th>doc_jsonpath</th>\n",
       "      <th>page_number</th>\n",
       "      <th>bbox</th>\n",
       "      <th>document_id</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>87</th>\n",
       "      <td>granite.pdf</td>\n",
       "      <td>28</td>\n",
       "      <td>17</td>\n",
       "      <td>348</td>\n",
       "      <td>pdf</td>\n",
       "      <td>0650e590f33356ab8581c7eb0c23f1b928f0cfe1659587...</td>\n",
       "      <td>654989</td>\n",
       "      <td>2024-10-02T00:24:48.959612</td>\n",
       "      <td>34.223920</td>\n",
       "      <td>granite.pdf</td>\n",
       "      <td>4a32ba4c-8fdb-4eeb-a06b-d28493efe8e3</td>\n",
       "      <td>6.3 Code Editing and Translation\\nTable 12: Pa...</td>\n",
       "      <td>$.main-text[189]</td>\n",
       "      <td>16</td>\n",
       "      <td>[106.69820404, 190.24554443, 504.00320435, 211...</td>\n",
       "      <td>f28d8c9a4fe81f0baf801daf9a95ddaf152a4ac5e8b8ac...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>154</th>\n",
       "      <td>attension.pdf</td>\n",
       "      <td>15</td>\n",
       "      <td>4</td>\n",
       "      <td>193</td>\n",
       "      <td>pdf</td>\n",
       "      <td>6fe23d4f932c725077dfc8334f3f4da4e3aaf908d2aa23...</td>\n",
       "      <td>135814</td>\n",
       "      <td>2024-10-02T00:24:14.713654</td>\n",
       "      <td>18.004455</td>\n",
       "      <td>attension.pdf</td>\n",
       "      <td>f275d75a-a072-4836-8a55-6a65f0d34577</td>\n",
       "      <td>3.2.2 Multi-Head Attention\\nMulti-head attenti...</td>\n",
       "      <td>$.main-text[55]</td>\n",
       "      <td>5</td>\n",
       "      <td>[107.46644592, 669.41210938, 503.99703979, 690...</td>\n",
       "      <td>da79f02a5f19c2f07de7a6f1da9df8db00f01a477582ac...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>67</th>\n",
       "      <td>granite.pdf</td>\n",
       "      <td>28</td>\n",
       "      <td>17</td>\n",
       "      <td>348</td>\n",
       "      <td>pdf</td>\n",
       "      <td>0650e590f33356ab8581c7eb0c23f1b928f0cfe1659587...</td>\n",
       "      <td>654989</td>\n",
       "      <td>2024-10-02T00:24:48.959612</td>\n",
       "      <td>34.223920</td>\n",
       "      <td>granite.pdf</td>\n",
       "      <td>4a32ba4c-8fdb-4eeb-a06b-d28493efe8e3</td>\n",
       "      <td>6.1.5 RepoBench, CrossCodeEval: Repository-Lev...</td>\n",
       "      <td>$.main-text[153]</td>\n",
       "      <td>12</td>\n",
       "      <td>[106.97065735, 224.31654358, 505.74191284, 290...</td>\n",
       "      <td>cd5bd4537bde007298a91de7fa2fb4b56516d2f1d31262...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "          filename  num_pages  num_tables  num_doc_elements  ext  \\\n",
       "87     granite.pdf         28          17               348  pdf   \n",
       "154  attension.pdf         15           4               193  pdf   \n",
       "67     granite.pdf         28          17               348  pdf   \n",
       "\n",
       "                                                  hash    size  \\\n",
       "87   0650e590f33356ab8581c7eb0c23f1b928f0cfe1659587...  654989   \n",
       "154  6fe23d4f932c725077dfc8334f3f4da4e3aaf908d2aa23...  135814   \n",
       "67   0650e590f33356ab8581c7eb0c23f1b928f0cfe1659587...  654989   \n",
       "\n",
       "                  date_acquired  pdf_convert_time source_filename  \\\n",
       "87   2024-10-02T00:24:48.959612         34.223920     granite.pdf   \n",
       "154  2024-10-02T00:24:14.713654         18.004455   attension.pdf   \n",
       "67   2024-10-02T00:24:48.959612         34.223920     granite.pdf   \n",
       "\n",
       "                       source_document_id  \\\n",
       "87   4a32ba4c-8fdb-4eeb-a06b-d28493efe8e3   \n",
       "154  f275d75a-a072-4836-8a55-6a65f0d34577   \n",
       "67   4a32ba4c-8fdb-4eeb-a06b-d28493efe8e3   \n",
       "\n",
       "                                              contents      doc_jsonpath  \\\n",
       "87   6.3 Code Editing and Translation\\nTable 12: Pa...  $.main-text[189]   \n",
       "154  3.2.2 Multi-Head Attention\\nMulti-head attenti...   $.main-text[55]   \n",
       "67   6.1.5 RepoBench, CrossCodeEval: Repository-Lev...  $.main-text[153]   \n",
       "\n",
       "     page_number                                               bbox  \\\n",
       "87            16  [106.69820404, 190.24554443, 504.00320435, 211...   \n",
       "154            5  [107.46644592, 669.41210938, 503.99703979, 690...   \n",
       "67            12  [106.97065735, 224.31654358, 505.74191284, 290...   \n",
       "\n",
       "                                           document_id  \n",
       "87   f28d8c9a4fe81f0baf801daf9a95ddaf152a4ac5e8b8ac...  \n",
       "154  da79f02a5f19c2f07de7a6f1da9df8db00f01a477582ac...  \n",
       "67   cd5bd4537bde007298a91de7fa2fb4b56516d2f1d31262...  "
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from utils import read_parquet_files_as_df\n",
    "\n",
    "output_df = read_parquet_files_as_df(output_folder)\n",
    "\n",
    "print (f\"Files processed : {input_df.shape[0]:,}\")\n",
    "print (f\"Chunks created : {output_df.shape[0]:,}\")\n",
    "\n",
    "print (\"Input data dimensions (rows x columns)= \", input_df.shape)\n",
    "print (\"Output data dimensions (rows x columns)= \", output_df.shape)\n",
    "\n",
    "output_df.sample(min(3, output_df.shape[0]))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ece021fd",
   "metadata": {},
   "source": [
    "## Step-5:  DOC ID generation\n",
    "\n",
    "This transform annotates documents with document \"ids\". It supports the following transformations of the original data:\n",
    "\n",
    " - Adding document hash: this enables the addition of a document hash-based id to the data. The hash is calculated with `hashlib.sha256(doc.encode(\"utf-8\")).hexdigest()`. To enable this annotation, set hash_column to the name of the column, where you want to store it.\n",
    " - Adding integer document id: this allows the addition of an integer document id to the data that is unique across all rows in all tables provided to the transform() method. To enable this annotation, set int_id_column to the name of the column, where you want to store it. **This is a pre-requisite for fuzzy dedup** in the pipeline."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e414c12c",
   "metadata": {},
   "source": [
    "### 5.1 - Set Input/output Folder"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "10251d3d",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "🏃🏼 STAGE-3: Processing input='output/02_chunk_out' --> output='output/03_docid_out'\n"
     ]
    }
   ],
   "source": [
    "\n",
    "STAGE  = 3\n",
    "\n",
    "input_folder = output_chunk_dir # previous output folder is the input folder for the current stage\n",
    "output_folder =  output_docid_dir\n",
    "\n",
    "input_df = read_parquet_files_as_df(input_folder)  ## for debug purposes\n",
    "\n",
    "print (f\"🏃🏼 STAGE-{STAGE}: Processing input='{input_folder}' --> output='{output_folder}'\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0f312347",
   "metadata": {},
   "source": [
    "### 5.2 - Execute "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "a8b76a71",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "00:24:50 INFO - Doc id parameters are : {'doc_column': 'contents', 'hash_column': 'chunk_hash', 'int_column': 'chunk_id', 'start_id': 0}\n",
      "00:24:50 INFO - pipeline id pipeline_id\n",
      "00:24:50 INFO - code location None\n",
      "00:24:50 INFO - data factory data_ is using local data access: input_folder - output/02_chunk_out output_folder - output/03_docid_out\n",
      "00:24:50 INFO - data factory data_ max_files -1, n_sample -1\n",
      "00:24:50 INFO - data factory data_ Not using data sets, checkpointing False, max files -1, random samples -1, files to use ['.parquet'], files to checkpoint ['.parquet']\n",
      "00:24:50 INFO - orchestrator doc_id started at 2024-10-02 00:24:50\n",
      "00:24:50 INFO - Number of files is 2, source profile {'max_file_size': 0.06398963928222656, 'min_file_size': 0.028062820434570312, 'total_file_size': 0.09205245971679688}\n",
      "00:24:50 INFO - Completed 1 files (50.0%) in 0.0 min\n",
      "00:24:50 INFO - Completed 2 files (100.0%) in 0.0 min\n",
      "00:24:50 INFO - Done processing 2 files, waiting for flush() completion.\n",
      "00:24:50 INFO - done flushing in 0.0 sec\n",
      "00:24:50 INFO - Completed execution in 0.0 min, execution result 0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "✅ Stage:3 completed successfully\n",
      "CPU times: user 13.4 ms, sys: 4.83 ms, total: 18.3 ms\n",
      "Wall time: 14.7 ms\n"
     ]
    }
   ],
   "source": [
    "%%time \n",
    "\n",
    "from data_processing.runtime.pure_python import PythonTransformLauncher\n",
    "from doc_id_transform_python import DocIDPythonTransformRuntimeConfiguration\n",
    "local_conf = {\n",
    "    \"input_folder\": input_folder,\n",
    "    \"output_folder\": output_folder,\n",
    "}\n",
    "params = {\n",
    "    # Data access. Only required parameters are specified\n",
    "    \"data_local_config\": ParamsUtils.convert_to_ast(local_conf),\n",
    "    # doc id configuration\n",
    "    \"doc_id_doc_column\": \"contents\",\n",
    "    \"doc_id_hash_column\": \"chunk_hash\",\n",
    "    \"doc_id_int_column\": \"chunk_id\",\n",
    "}\n",
    "sys.argv = ParamsUtils.dict_to_req(d=params)\n",
    "\n",
    "# launch\n",
    "\n",
    "launcher = PythonTransformLauncher(DocIDPythonTransformRuntimeConfiguration())\n",
    "\n",
    "return_code = launcher.launch()\n",
    "\n",
    "if return_code == 0:\n",
    "    print (f\"✅ Stage:{STAGE} completed successfully\")\n",
    "else:\n",
    "    raise Exception (\"❌ Ray job failed\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8c23338b",
   "metadata": {},
   "source": [
    "### 5.3 - Inspect Generated output"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "ec23aa3a",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Input data dimensions (rows x columns)=  (211, 16)\n",
      "Output data dimensions (rows x columns)=  (211, 18)\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>filename</th>\n",
       "      <th>num_pages</th>\n",
       "      <th>num_tables</th>\n",
       "      <th>num_doc_elements</th>\n",
       "      <th>ext</th>\n",
       "      <th>hash</th>\n",
       "      <th>size</th>\n",
       "      <th>date_acquired</th>\n",
       "      <th>pdf_convert_time</th>\n",
       "      <th>source_filename</th>\n",
       "      <th>source_document_id</th>\n",
       "      <th>contents</th>\n",
       "      <th>doc_jsonpath</th>\n",
       "      <th>page_number</th>\n",
       "      <th>bbox</th>\n",
       "      <th>document_id</th>\n",
       "      <th>chunk_hash</th>\n",
       "      <th>chunk_id</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>192</th>\n",
       "      <td>attension.pdf</td>\n",
       "      <td>15</td>\n",
       "      <td>4</td>\n",
       "      <td>193</td>\n",
       "      <td>pdf</td>\n",
       "      <td>6fe23d4f932c725077dfc8334f3f4da4e3aaf908d2aa23...</td>\n",
       "      <td>135814</td>\n",
       "      <td>2024-10-02T00:24:14.713654</td>\n",
       "      <td>18.004455</td>\n",
       "      <td>attension.pdf</td>\n",
       "      <td>f275d75a-a072-4836-8a55-6a65f0d34577</td>\n",
       "      <td>6.2 Model Variations\\nIn Table 3 rows (A), we ...</td>\n",
       "      <td>$.main-text[118]</td>\n",
       "      <td>9</td>\n",
       "      <td>[107.27760315, 318.93438721, 505.24127197, 350...</td>\n",
       "      <td>70948f748c6f275b39c70652e29d60dfd53c545e0d6d92...</td>\n",
       "      <td>70948f748c6f275b39c70652e29d60dfd53c545e0d6d92...</td>\n",
       "      <td>69</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>71</th>\n",
       "      <td>granite.pdf</td>\n",
       "      <td>28</td>\n",
       "      <td>17</td>\n",
       "      <td>348</td>\n",
       "      <td>pdf</td>\n",
       "      <td>0650e590f33356ab8581c7eb0c23f1b928f0cfe1659587...</td>\n",
       "      <td>654989</td>\n",
       "      <td>2024-10-02T00:24:48.959612</td>\n",
       "      <td>34.223920</td>\n",
       "      <td>granite.pdf</td>\n",
       "      <td>4a32ba4c-8fdb-4eeb-a06b-d28493efe8e3</td>\n",
       "      <td>6.1.5 RepoBench, CrossCodeEval: Repository-Lev...</td>\n",
       "      <td>$.tables[7]</td>\n",
       "      <td>13</td>\n",
       "      <td>[109.39778137, 486.89639282, 502.1010437, 679....</td>\n",
       "      <td>b7497dcda69d88caa6b7c3a462edb925ffa97ce5e42c52...</td>\n",
       "      <td>b7497dcda69d88caa6b7c3a462edb925ffa97ce5e42c52...</td>\n",
       "      <td>159</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>196</th>\n",
       "      <td>attension.pdf</td>\n",
       "      <td>15</td>\n",
       "      <td>4</td>\n",
       "      <td>193</td>\n",
       "      <td>pdf</td>\n",
       "      <td>6fe23d4f932c725077dfc8334f3f4da4e3aaf908d2aa23...</td>\n",
       "      <td>135814</td>\n",
       "      <td>2024-10-02T00:24:14.713654</td>\n",
       "      <td>18.004455</td>\n",
       "      <td>attension.pdf</td>\n",
       "      <td>f275d75a-a072-4836-8a55-6a65f0d34577</td>\n",
       "      <td>6.3 English Constituency Parsing\\nWe performed...</td>\n",
       "      <td>$.main-text[123]</td>\n",
       "      <td>9</td>\n",
       "      <td>[106.96768951, 69.592453, 504.24859619, 101.62...</td>\n",
       "      <td>93e01b0e6bafcfe5fcd113d1a3dfedad27d12f81038ff5...</td>\n",
       "      <td>93e01b0e6bafcfe5fcd113d1a3dfedad27d12f81038ff5...</td>\n",
       "      <td>73</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "          filename  num_pages  num_tables  num_doc_elements  ext  \\\n",
       "192  attension.pdf         15           4               193  pdf   \n",
       "71     granite.pdf         28          17               348  pdf   \n",
       "196  attension.pdf         15           4               193  pdf   \n",
       "\n",
       "                                                  hash    size  \\\n",
       "192  6fe23d4f932c725077dfc8334f3f4da4e3aaf908d2aa23...  135814   \n",
       "71   0650e590f33356ab8581c7eb0c23f1b928f0cfe1659587...  654989   \n",
       "196  6fe23d4f932c725077dfc8334f3f4da4e3aaf908d2aa23...  135814   \n",
       "\n",
       "                  date_acquired  pdf_convert_time source_filename  \\\n",
       "192  2024-10-02T00:24:14.713654         18.004455   attension.pdf   \n",
       "71   2024-10-02T00:24:48.959612         34.223920     granite.pdf   \n",
       "196  2024-10-02T00:24:14.713654         18.004455   attension.pdf   \n",
       "\n",
       "                       source_document_id  \\\n",
       "192  f275d75a-a072-4836-8a55-6a65f0d34577   \n",
       "71   4a32ba4c-8fdb-4eeb-a06b-d28493efe8e3   \n",
       "196  f275d75a-a072-4836-8a55-6a65f0d34577   \n",
       "\n",
       "                                              contents      doc_jsonpath  \\\n",
       "192  6.2 Model Variations\\nIn Table 3 rows (A), we ...  $.main-text[118]   \n",
       "71   6.1.5 RepoBench, CrossCodeEval: Repository-Lev...       $.tables[7]   \n",
       "196  6.3 English Constituency Parsing\\nWe performed...  $.main-text[123]   \n",
       "\n",
       "     page_number                                               bbox  \\\n",
       "192            9  [107.27760315, 318.93438721, 505.24127197, 350...   \n",
       "71            13  [109.39778137, 486.89639282, 502.1010437, 679....   \n",
       "196            9  [106.96768951, 69.592453, 504.24859619, 101.62...   \n",
       "\n",
       "                                           document_id  \\\n",
       "192  70948f748c6f275b39c70652e29d60dfd53c545e0d6d92...   \n",
       "71   b7497dcda69d88caa6b7c3a462edb925ffa97ce5e42c52...   \n",
       "196  93e01b0e6bafcfe5fcd113d1a3dfedad27d12f81038ff5...   \n",
       "\n",
       "                                            chunk_hash  chunk_id  \n",
       "192  70948f748c6f275b39c70652e29d60dfd53c545e0d6d92...        69  \n",
       "71   b7497dcda69d88caa6b7c3a462edb925ffa97ce5e42c52...       159  \n",
       "196  93e01b0e6bafcfe5fcd113d1a3dfedad27d12f81038ff5...        73  "
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from utils import read_parquet_files_as_df\n",
    "\n",
    "output_df = read_parquet_files_as_df(output_folder)\n",
    "\n",
    "print (\"Input data dimensions (rows x columns)= \", input_df.shape)\n",
    "print (\"Output data dimensions (rows x columns)= \", output_df.shape)\n",
    "\n",
    "output_df.sample(min(3, output_df.shape[0]))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4692975c-49ff-41ae-810e-0f5bc0bbdc53",
   "metadata": {},
   "source": [
    "## Step-6: Exact Dedup\n",
    "\n",
    "Remove documents having identical code to remove bias in the training data. On the content of each document, a SHA256 hash is computed,\n",
    "followed by de-duplication of record having identical hashes."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5acfd3a2-a236-4143-bcfc-15804f1da7fe",
   "metadata": {},
   "source": [
    "### 6.1 - Set Input/output Folder"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "4c7a1b94",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "🏃🏼 STAGE-4: Processing input='output/03_docid_out' --> output='output/04_exact_dedupe_out'\n"
     ]
    }
   ],
   "source": [
    "STAGE  = 4\n",
    "\n",
    "input_folder = output_docid_dir # previous output folder is the input folder for the current stage\n",
    "output_folder =  output_exact_dedupe_dir\n",
    "\n",
    "input_df = read_parquet_files_as_df(input_folder)  ## for debug purposes\n",
    "\n",
    "print (f\"🏃🏼 STAGE-{STAGE}: Processing input='{input_folder}' --> output='{output_folder}'\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3661cb37-39c7-4b09-a784-925bfa9eaf1e",
   "metadata": {},
   "source": [
    "### 6.2 - Execute "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "a624b2b2-faad-4325-ac7d-53a840f564ef",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "00:24:50 INFO - exact dedup params are {'doc_column': 'contents', 'doc_id_column': 'chunk_hash', 'use_snapshot': False, 'snapshot_directory': None}\n",
      "00:24:50 INFO - pipeline id pipeline_id\n",
      "00:24:50 INFO - code location None\n",
      "00:24:50 INFO - data factory data_ is using local data access: input_folder - output/03_docid_out output_folder - output/04_exact_dedupe_out\n",
      "00:24:50 INFO - data factory data_ max_files -1, n_sample -1\n",
      "00:24:50 INFO - data factory data_ Not using data sets, checkpointing False, max files -1, random samples -1, files to use ['.parquet'], files to checkpoint ['.parquet']\n",
      "00:24:50 INFO - orchestrator ededup started at 2024-10-02 00:24:50\n",
      "00:24:50 INFO - Number of files is 2, source profile {'max_file_size': 0.06945991516113281, 'min_file_size': 0.03227043151855469, 'total_file_size': 0.1017303466796875}\n",
      "00:24:50 INFO - Starting from the beginning\n",
      "00:24:50 INFO - Completed 1 files (50.0%) in 0.0 min\n",
      "00:24:50 INFO - Completed 2 files (100.0%) in 0.0 min\n",
      "00:24:50 INFO - Done processing 2 files, waiting for flush() completion.\n",
      "00:24:50 INFO - done flushing in 0.0 sec\n",
      "00:24:50 INFO - Completed execution in 0.0 min, execution result 0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "✅ Stage:4 completed successfully\n",
      "CPU times: user 22.1 ms, sys: 5.79 ms, total: 27.9 ms\n",
      "Wall time: 23.5 ms\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "\n",
    "from data_processing.runtime.pure_python import PythonTransformLauncher\n",
    "\n",
    "# Import ededup transform configuration\n",
    "from ededup_transform_python import EdedupPythonTransformRuntimeConfiguration\n",
    "\n",
    "\n",
    "# Prepare the commandline params\n",
    "local_conf = {\n",
    "    \"input_folder\": input_folder,\n",
    "    \"output_folder\": output_folder,\n",
    "}\n",
    "params = {\n",
    "    # Data access. Only required parameters are specified\n",
    "    \"data_local_config\": ParamsUtils.convert_to_ast(local_conf),\n",
    "    # ededup parameters\n",
    "    \"ededup_doc_column\": \"contents\",\n",
    "    \"ededup_doc_id_column\": \"chunk_hash\",\n",
    "    \n",
    "}\n",
    "\n",
    "# Pass the commandline params\n",
    "sys.argv = ParamsUtils.dict_to_req(d=params)\n",
    "\n",
    "# create launcher\n",
    "launcher = PythonTransformLauncher(EdedupPythonTransformRuntimeConfiguration())\n",
    "# launch\n",
    "return_code = launcher.launch()\n",
    "\n",
    "if return_code == 0:\n",
    "    print (f\"✅ Stage:{STAGE} completed successfully\")\n",
    "else:\n",
    "    raise Exception (\"❌ Ray job failed\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "eaf1c3c3",
   "metadata": {},
   "source": [
    "### 6.3 - Inspect Generated output"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "d824ebf6",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Input data dimensions (rows x columns)=  (211, 18)\n",
      "Output data dimensions (rows x columns)=  (211, 19)\n",
      "Input chunks before exact dedupe : 211\n",
      "Output chunks after exact dedupe : 211\n",
      "Duplicate chunks removed :   0\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>filename</th>\n",
       "      <th>num_pages</th>\n",
       "      <th>num_tables</th>\n",
       "      <th>num_doc_elements</th>\n",
       "      <th>ext</th>\n",
       "      <th>hash</th>\n",
       "      <th>size</th>\n",
       "      <th>date_acquired</th>\n",
       "      <th>pdf_convert_time</th>\n",
       "      <th>source_filename</th>\n",
       "      <th>source_document_id</th>\n",
       "      <th>contents</th>\n",
       "      <th>doc_jsonpath</th>\n",
       "      <th>page_number</th>\n",
       "      <th>bbox</th>\n",
       "      <th>document_id</th>\n",
       "      <th>chunk_hash</th>\n",
       "      <th>chunk_id</th>\n",
       "      <th>removed</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>194</th>\n",
       "      <td>attension.pdf</td>\n",
       "      <td>15</td>\n",
       "      <td>4</td>\n",
       "      <td>193</td>\n",
       "      <td>pdf</td>\n",
       "      <td>6fe23d4f932c725077dfc8334f3f4da4e3aaf908d2aa23...</td>\n",
       "      <td>135814</td>\n",
       "      <td>2024-10-02T00:24:14.713654</td>\n",
       "      <td>18.004455</td>\n",
       "      <td>attension.pdf</td>\n",
       "      <td>f275d75a-a072-4836-8a55-6a65f0d34577</td>\n",
       "      <td>6.3 English Constituency Parsing\\nTo evaluate ...</td>\n",
       "      <td>$.main-text[121]</td>\n",
       "      <td>9</td>\n",
       "      <td>[107.15766144, 167.93530273, 504.10968018, 210...</td>\n",
       "      <td>10c85ade191100c9586ffb4e5ded4944bc4fd865d0919f...</td>\n",
       "      <td>10c85ade191100c9586ffb4e5ded4944bc4fd865d0919f...</td>\n",
       "      <td>71</td>\n",
       "      <td>[]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>101</th>\n",
       "      <td>granite.pdf</td>\n",
       "      <td>28</td>\n",
       "      <td>17</td>\n",
       "      <td>348</td>\n",
       "      <td>pdf</td>\n",
       "      <td>0650e590f33356ab8581c7eb0c23f1b928f0cfe1659587...</td>\n",
       "      <td>654989</td>\n",
       "      <td>2024-10-02T00:24:48.959612</td>\n",
       "      <td>34.223920</td>\n",
       "      <td>granite.pdf</td>\n",
       "      <td>4a32ba4c-8fdb-4eeb-a06b-d28493efe8e3</td>\n",
       "      <td>6.5 Math Reasoning\\nTable 15: Performance on 4...</td>\n",
       "      <td>$.main-text[219]</td>\n",
       "      <td>19</td>\n",
       "      <td>[118.49487305, 699.65753174, 492.17700195, 710...</td>\n",
       "      <td>c39e0817c8d1edf1d322cef0535b5a63b80d2b2b4d1852...</td>\n",
       "      <td>c39e0817c8d1edf1d322cef0535b5a63b80d2b2b4d1852...</td>\n",
       "      <td>189</td>\n",
       "      <td>[]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>206</th>\n",
       "      <td>attension.pdf</td>\n",
       "      <td>15</td>\n",
       "      <td>4</td>\n",
       "      <td>193</td>\n",
       "      <td>pdf</td>\n",
       "      <td>6fe23d4f932c725077dfc8334f3f4da4e3aaf908d2aa23...</td>\n",
       "      <td>135814</td>\n",
       "      <td>2024-10-02T00:24:14.713654</td>\n",
       "      <td>18.004455</td>\n",
       "      <td>attension.pdf</td>\n",
       "      <td>f275d75a-a072-4836-8a55-6a65f0d34577</td>\n",
       "      <td>7 Conclusion\\nAcknowledgements We are grateful...</td>\n",
       "      <td>$.main-text[135]</td>\n",
       "      <td>10</td>\n",
       "      <td>[107.4437561, 212.26509094, 504.00241089, 232....</td>\n",
       "      <td>855fdc0d15cb042a43d799b9a38d4339ae1e25b2df99c4...</td>\n",
       "      <td>855fdc0d15cb042a43d799b9a38d4339ae1e25b2df99c4...</td>\n",
       "      <td>83</td>\n",
       "      <td>[]</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "          filename  num_pages  num_tables  num_doc_elements  ext  \\\n",
       "194  attension.pdf         15           4               193  pdf   \n",
       "101    granite.pdf         28          17               348  pdf   \n",
       "206  attension.pdf         15           4               193  pdf   \n",
       "\n",
       "                                                  hash    size  \\\n",
       "194  6fe23d4f932c725077dfc8334f3f4da4e3aaf908d2aa23...  135814   \n",
       "101  0650e590f33356ab8581c7eb0c23f1b928f0cfe1659587...  654989   \n",
       "206  6fe23d4f932c725077dfc8334f3f4da4e3aaf908d2aa23...  135814   \n",
       "\n",
       "                  date_acquired  pdf_convert_time source_filename  \\\n",
       "194  2024-10-02T00:24:14.713654         18.004455   attension.pdf   \n",
       "101  2024-10-02T00:24:48.959612         34.223920     granite.pdf   \n",
       "206  2024-10-02T00:24:14.713654         18.004455   attension.pdf   \n",
       "\n",
       "                       source_document_id  \\\n",
       "194  f275d75a-a072-4836-8a55-6a65f0d34577   \n",
       "101  4a32ba4c-8fdb-4eeb-a06b-d28493efe8e3   \n",
       "206  f275d75a-a072-4836-8a55-6a65f0d34577   \n",
       "\n",
       "                                              contents      doc_jsonpath  \\\n",
       "194  6.3 English Constituency Parsing\\nTo evaluate ...  $.main-text[121]   \n",
       "101  6.5 Math Reasoning\\nTable 15: Performance on 4...  $.main-text[219]   \n",
       "206  7 Conclusion\\nAcknowledgements We are grateful...  $.main-text[135]   \n",
       "\n",
       "     page_number                                               bbox  \\\n",
       "194            9  [107.15766144, 167.93530273, 504.10968018, 210...   \n",
       "101           19  [118.49487305, 699.65753174, 492.17700195, 710...   \n",
       "206           10  [107.4437561, 212.26509094, 504.00241089, 232....   \n",
       "\n",
       "                                           document_id  \\\n",
       "194  10c85ade191100c9586ffb4e5ded4944bc4fd865d0919f...   \n",
       "101  c39e0817c8d1edf1d322cef0535b5a63b80d2b2b4d1852...   \n",
       "206  855fdc0d15cb042a43d799b9a38d4339ae1e25b2df99c4...   \n",
       "\n",
       "                                            chunk_hash  chunk_id removed  \n",
       "194  10c85ade191100c9586ffb4e5ded4944bc4fd865d0919f...        71      []  \n",
       "101  c39e0817c8d1edf1d322cef0535b5a63b80d2b2b4d1852...       189      []  \n",
       "206  855fdc0d15cb042a43d799b9a38d4339ae1e25b2df99c4...        83      []  "
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from utils import read_parquet_files_as_df\n",
    "\n",
    "output_df = read_parquet_files_as_df(output_folder)\n",
    "\n",
    "print (\"Input data dimensions (rows x columns)= \", input_df.shape)\n",
    "print (\"Output data dimensions (rows x columns)= \", output_df.shape)\n",
    "print (f\"Input chunks before exact dedupe : {input_df.shape[0]:,}\")\n",
    "print (f\"Output chunks after exact dedupe : {output_df.shape[0]:,}\")\n",
    "print (\"Duplicate chunks removed :  \", (input_df.shape[0] - output_df.shape[0]))\n",
    "\n",
    "output_df.sample(min(3, output_df.shape[0]))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "85309751-8556-41c6-ac32-84acc941bc8d",
   "metadata": {},
   "source": [
    "## Fuzzy Dedup\n",
    "\n",
    "**Fuzzy dedupe is currently available in RAY version only**\n",
    "\n",
    "So we will skip this here"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5370950a-2a3a-4143-8218-f9b4808099ba",
   "metadata": {},
   "source": [
    "## Step-7:   Text encoding\n",
    "\n",
    "Encode text for the vector storage."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "74fd33b1",
   "metadata": {},
   "source": [
    "### 7.1 - Set Input/output Folder"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "20a153fa-fd56-401e-86be-4f7617affcc8",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "🏃🏼 STAGE-5: Processing input='output/04_exact_dedupe_out' --> output='output/05_embeddings_out'\n"
     ]
    }
   ],
   "source": [
    "STAGE  = 5\n",
    "\n",
    "input_folder = output_exact_dedupe_dir\n",
    "output_folder =  output_embeddings_dir\n",
    "\n",
    "input_df = read_parquet_files_as_df(input_folder)  ## for debug purposes\n",
    "\n",
    "print (f\"🏃🏼 STAGE-{STAGE}: Processing input='{input_folder}' --> output='{output_folder}'\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b9112479",
   "metadata": {},
   "source": [
    "### 7.2 - Execute"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "228df6b2-bc62-494b-9697-03ece98d7853",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "00:24:50 INFO - text_encoder parameters are : {'content_column_name': 'contents', 'output_embeddings_column_name': 'embeddings', 'model_name': 'sentence-transformers/all-MiniLM-L6-v2'}\n",
      "00:24:50 INFO - pipeline id pipeline_id\n",
      "00:24:50 INFO - code location None\n",
      "00:24:50 INFO - data factory data_ is using local data access: input_folder - output/04_exact_dedupe_out output_folder - output/05_embeddings_out\n",
      "00:24:50 INFO - data factory data_ max_files -1, n_sample -1\n",
      "00:24:50 INFO - data factory data_ Not using data sets, checkpointing False, max files -1, random samples -1, files to use ['.parquet'], files to checkpoint ['.parquet']\n",
      "00:24:50 INFO - orchestrator text_encoder started at 2024-10-02 00:24:50\n",
      "00:24:50 INFO - Number of files is 2, source profile {'max_file_size': 0.06981945037841797, 'min_file_size': 0.032629966735839844, 'total_file_size': 0.10244941711425781}\n",
      "00:24:52 INFO - Completed 1 files (50.0%) in 0.008 min\n",
      "00:24:53 INFO - Completed 2 files (100.0%) in 0.02 min\n",
      "00:24:53 INFO - Done processing 2 files, waiting for flush() completion.\n",
      "00:24:53 INFO - done flushing in 0.0 sec\n",
      "00:24:53 INFO - Completed execution in 0.046 min, execution result 0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "✅ Stage:5 completed successfully\n",
      "CPU times: user 1.78 s, sys: 103 ms, total: 1.88 s\n",
      "Wall time: 3.09 s\n"
     ]
    }
   ],
   "source": [
    "%%time \n",
    "\n",
    "from data_processing.runtime.pure_python import PythonTransformLauncher\n",
    "from text_encoder_transform_python import TextEncoderPythonTransformConfiguration\n",
    "\n",
    "local_conf = {\n",
    "    \"input_folder\": input_folder,\n",
    "    \"output_folder\": output_folder,\n",
    "}\n",
    "params = {\n",
    "    # Data access. Only required parameters are specified\n",
    "    \"data_local_config\": ParamsUtils.convert_to_ast(local_conf),\n",
    "    # text_encoder\n",
    "    \"text_encoder_model_name\": MY_CONFIG.EMBEDDING_MODEL,\n",
    "}\n",
    "\n",
    "sys.argv = ParamsUtils.dict_to_req(d=params)\n",
    "# create launcher\n",
    "launcher = PythonTransformLauncher(TextEncoderPythonTransformConfiguration())\n",
    "# Launch the ray actor(s) to process the input\n",
    "\n",
    "return_code = launcher.launch()\n",
    "\n",
    "if return_code == 0:\n",
    "    print (f\"✅ Stage:{STAGE} completed successfully\")\n",
    "else:\n",
    "    raise Exception (\"❌ Job failed\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b734852c",
   "metadata": {},
   "source": [
    "### 7.3 - Inspect Generated output"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "7b1c1d09",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Input data dimensions (rows x columns)=  (211, 19)\n",
      "Output data dimensions (rows x columns)=  (211, 20)\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>filename</th>\n",
       "      <th>num_pages</th>\n",
       "      <th>num_tables</th>\n",
       "      <th>num_doc_elements</th>\n",
       "      <th>ext</th>\n",
       "      <th>hash</th>\n",
       "      <th>size</th>\n",
       "      <th>date_acquired</th>\n",
       "      <th>pdf_convert_time</th>\n",
       "      <th>source_filename</th>\n",
       "      <th>source_document_id</th>\n",
       "      <th>contents</th>\n",
       "      <th>doc_jsonpath</th>\n",
       "      <th>page_number</th>\n",
       "      <th>bbox</th>\n",
       "      <th>document_id</th>\n",
       "      <th>chunk_hash</th>\n",
       "      <th>chunk_id</th>\n",
       "      <th>removed</th>\n",
       "      <th>embeddings</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>193</th>\n",
       "      <td>attension.pdf</td>\n",
       "      <td>15</td>\n",
       "      <td>4</td>\n",
       "      <td>193</td>\n",
       "      <td>pdf</td>\n",
       "      <td>6fe23d4f932c725077dfc8334f3f4da4e3aaf908d2aa23...</td>\n",
       "      <td>135814</td>\n",
       "      <td>2024-10-02T00:24:14.713654</td>\n",
       "      <td>18.004455</td>\n",
       "      <td>attension.pdf</td>\n",
       "      <td>f275d75a-a072-4836-8a55-6a65f0d34577</td>\n",
       "      <td>6.2 Model Variations\\nIn Table 3 rows (B), we ...</td>\n",
       "      <td>$.main-text[119]</td>\n",
       "      <td>9</td>\n",
       "      <td>[107.44257355, 248.49208069, 505.24127197, 312...</td>\n",
       "      <td>6b79d74f59d1218fa3cdff6d13b504c8bf80558f3e2522...</td>\n",
       "      <td>6b79d74f59d1218fa3cdff6d13b504c8bf80558f3e2522...</td>\n",
       "      <td>70</td>\n",
       "      <td>[]</td>\n",
       "      <td>[-0.0049973284, -0.10789071, 0.02143236, -0.02...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>210</th>\n",
       "      <td>attension.pdf</td>\n",
       "      <td>15</td>\n",
       "      <td>4</td>\n",
       "      <td>193</td>\n",
       "      <td>pdf</td>\n",
       "      <td>6fe23d4f932c725077dfc8334f3f4da4e3aaf908d2aa23...</td>\n",
       "      <td>135814</td>\n",
       "      <td>2024-10-02T00:24:14.713654</td>\n",
       "      <td>18.004455</td>\n",
       "      <td>attension.pdf</td>\n",
       "      <td>f275d75a-a072-4836-8a55-6a65f0d34577</td>\n",
       "      <td>Attention Visualizations Input-Input Layer5\\nF...</td>\n",
       "      <td>$.main-text[190]</td>\n",
       "      <td>15</td>\n",
       "      <td>[107.43354034, 157.36341858, 504.06988525, 189...</td>\n",
       "      <td>67626adb815bf2b27871df24d538ddc10ae68a3fbbd238...</td>\n",
       "      <td>67626adb815bf2b27871df24d538ddc10ae68a3fbbd238...</td>\n",
       "      <td>87</td>\n",
       "      <td>[]</td>\n",
       "      <td>[0.01508544, -0.015680796, 0.039181348, 0.0084...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>46</th>\n",
       "      <td>granite.pdf</td>\n",
       "      <td>28</td>\n",
       "      <td>17</td>\n",
       "      <td>348</td>\n",
       "      <td>pdf</td>\n",
       "      <td>0650e590f33356ab8581c7eb0c23f1b928f0cfe1659587...</td>\n",
       "      <td>654989</td>\n",
       "      <td>2024-10-02T00:24:48.959612</td>\n",
       "      <td>34.223920</td>\n",
       "      <td>granite.pdf</td>\n",
       "      <td>4a32ba4c-8fdb-4eeb-a06b-d28493efe8e3</td>\n",
       "      <td>6.1.1 HumanEvalSynthesize: Multilingual Code G...</td>\n",
       "      <td>$.main-text[117]</td>\n",
       "      <td>9</td>\n",
       "      <td>[107.46860504, 613.84277344, 456.97003174, 624...</td>\n",
       "      <td>3d5d963f59d4ecb05d1ec2d014747459e01cabe2944bba...</td>\n",
       "      <td>3d5d963f59d4ecb05d1ec2d014747459e01cabe2944bba...</td>\n",
       "      <td>134</td>\n",
       "      <td>[]</td>\n",
       "      <td>[-0.029933447, 0.031515192, -0.04598905, -0.01...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "          filename  num_pages  num_tables  num_doc_elements  ext  \\\n",
       "193  attension.pdf         15           4               193  pdf   \n",
       "210  attension.pdf         15           4               193  pdf   \n",
       "46     granite.pdf         28          17               348  pdf   \n",
       "\n",
       "                                                  hash    size  \\\n",
       "193  6fe23d4f932c725077dfc8334f3f4da4e3aaf908d2aa23...  135814   \n",
       "210  6fe23d4f932c725077dfc8334f3f4da4e3aaf908d2aa23...  135814   \n",
       "46   0650e590f33356ab8581c7eb0c23f1b928f0cfe1659587...  654989   \n",
       "\n",
       "                  date_acquired  pdf_convert_time source_filename  \\\n",
       "193  2024-10-02T00:24:14.713654         18.004455   attension.pdf   \n",
       "210  2024-10-02T00:24:14.713654         18.004455   attension.pdf   \n",
       "46   2024-10-02T00:24:48.959612         34.223920     granite.pdf   \n",
       "\n",
       "                       source_document_id  \\\n",
       "193  f275d75a-a072-4836-8a55-6a65f0d34577   \n",
       "210  f275d75a-a072-4836-8a55-6a65f0d34577   \n",
       "46   4a32ba4c-8fdb-4eeb-a06b-d28493efe8e3   \n",
       "\n",
       "                                              contents      doc_jsonpath  \\\n",
       "193  6.2 Model Variations\\nIn Table 3 rows (B), we ...  $.main-text[119]   \n",
       "210  Attention Visualizations Input-Input Layer5\\nF...  $.main-text[190]   \n",
       "46   6.1.1 HumanEvalSynthesize: Multilingual Code G...  $.main-text[117]   \n",
       "\n",
       "     page_number                                               bbox  \\\n",
       "193            9  [107.44257355, 248.49208069, 505.24127197, 312...   \n",
       "210           15  [107.43354034, 157.36341858, 504.06988525, 189...   \n",
       "46             9  [107.46860504, 613.84277344, 456.97003174, 624...   \n",
       "\n",
       "                                           document_id  \\\n",
       "193  6b79d74f59d1218fa3cdff6d13b504c8bf80558f3e2522...   \n",
       "210  67626adb815bf2b27871df24d538ddc10ae68a3fbbd238...   \n",
       "46   3d5d963f59d4ecb05d1ec2d014747459e01cabe2944bba...   \n",
       "\n",
       "                                            chunk_hash  chunk_id removed  \\\n",
       "193  6b79d74f59d1218fa3cdff6d13b504c8bf80558f3e2522...        70      []   \n",
       "210  67626adb815bf2b27871df24d538ddc10ae68a3fbbd238...        87      []   \n",
       "46   3d5d963f59d4ecb05d1ec2d014747459e01cabe2944bba...       134      []   \n",
       "\n",
       "                                            embeddings  \n",
       "193  [-0.0049973284, -0.10789071, 0.02143236, -0.02...  \n",
       "210  [0.01508544, -0.015680796, 0.039181348, 0.0084...  \n",
       "46   [-0.029933447, 0.031515192, -0.04598905, -0.01...  "
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from utils import read_parquet_files_as_df\n",
    "\n",
    "output_df = read_parquet_files_as_df(output_folder)\n",
    "\n",
    "print (\"Input data dimensions (rows x columns)= \", input_df.shape)\n",
    "print (\"Output data dimensions (rows x columns)= \", output_df.shape)\n",
    "\n",
    "output_df.sample(min(3, output_df.shape[0]))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f5e12630-be6b-4188-a925-77117155617b",
   "metadata": {},
   "source": [
    "## Step-8: Copy output to final output dir"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "16dee3b8-31dc-4168-8adb-f2a0a0b5e207",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "✅ Copied output from 'output/05_embeddings_out' --> 'output/output_final'\n"
     ]
    }
   ],
   "source": [
    "import shutil\n",
    "\n",
    "shutil.rmtree(MY_CONFIG.OUTPUT_FOLDER_FINAL, ignore_errors=True)\n",
    "shutil.copytree(src=output_folder, dst=MY_CONFIG.OUTPUT_FOLDER_FINAL)\n",
    "\n",
    "print (f\"✅ Copied output from '{output_folder}' --> '{MY_CONFIG.OUTPUT_FOLDER_FINAL}'\")"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}