totalseg_all_mwe.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/modelhub-ai/colab_totalsegmentator/blob/main/totalseg_all_mwe.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Pmb_HiJv7Sit"
      },
      "source": [
        "# **ModelHub - Whole Body CT Segmentation**\n",
        "\n",
        "This notebook provides a minimal working example of TotalSegmentator, a tool for the segmentation of 104 anatomical structures from CT images. The model was trained using a wide range of imaging CT data of different pathologies from several scanners, protocols and institutions.\n",
        "\n",
        "We test TotalSegmentator by implementing an end-to-end (cloud-based) pipeline on publicly available whole body CT scans hosted on the [Imaging Data Commons (IDC)](https://portal.imaging.datacommons.cancer.gov/), starting from raw DICOM CT data and ending with a DICOM SEG object storing the segmentation masks generated by the AI model. The testing dataset we use is external and independent from the data used in the development phase of the model (training and validation) and is composed by a wide variety of image types (from the area covered by the scan, to the presence of contrast and various types of artefacts).\n",
        "\n",
        "The way all the operations are executed - from pulling data, to data postprocessing, and the standardisation of the results - have the goal of promoting transparency and reproducibility."
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Please cite the following article if you use this code or pre-trained models:\n",
        "\n",
        "Wasserthal, J., Meyer, M., Breit, H.C., Cyriac, J., Yang, S. and Segeroth, M., 2022. TotalSegmentator: robust segmentation of 104 anatomical structures in CT images. arXiv preprint arXiv:2208.05868, [\n",
        "https://doi.org/10.48550/arXiv.2208.05868]( \t\n",
        "https://doi.org/10.48550/arXiv.2208.05868).\n",
        "\n",
        "The original code is published on\n",
        "[GitHub](https://github.com/wasserth/TotalSegmentator)  using the [Apache-2.0 license](https://github.com/wasserth/TotalSegmentator/blob/master/LICENSE)."
      ],
      "metadata": {
        "id": "KhTN-Q6jujt4"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "# **Disclaimer**\n",
        "\n",
        "The code and data of this repository are provided to promote reproducible research. They are not intended for clinical care or commercial use.\n",
        "\n",
        "The software is provided \"as is\", without warranty of any kind, express or implied, including but not limited to the warranties of merchantability, fitness for a particular purpose and noninfringement. In no event shall the authors or copyright holders be liable for any claim, damages or other liability, whether in an action of contract, tort or otherwise, arising from, out of or in connection with the software or the use or other dealings in the software."
      ],
      "metadata": {
        "id": "gS7cUQaJujzN"
      }
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "CZnoRi9Y7nEB"
      },
      "source": [
        "## **Environment Setup**\n",
        "\n",
        "This demo notebook is intended to be run using a GPU.\n",
        "\n",
        "To access a free GPU on Colab:\n",
        "`Edit > Notebooks Settings`.\n",
        "\n",
        "From the dropdown menu under `Hardware accelerator`, select `GPU`. Let's check the Colab instance is indeed equipped with a GPU."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "non5qVLIcG4M",
        "outputId": "deb10fea-ace4-4261-e65f-03409b4018df"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Wed Oct 19 22:21:00 2022\n",
            "\n",
            "Current directory : /content\n",
            "Hostname          : f9d86ec88e01\n",
            "Username          : root\n",
            "Python version    : 3.7.15 (default, Oct 12 2022, 19:14:55) \n"
          ]
        }
      ],
      "source": [
        "import os\n",
        "import sys\n",
        "import shutil\n",
        "\n",
        "import yaml\n",
        "\n",
        "import time\n",
        "import tqdm\n",
        "\n",
        "\n",
        "# useful information\n",
        "curr_dir = !pwd\n",
        "curr_droid = !hostname\n",
        "curr_pilot = !whoami\n",
        "\n",
        "print(time.asctime(time.localtime()))\n",
        "\n",
        "print(\"\\nCurrent directory :\", curr_dir[-1])\n",
        "print(\"Hostname          :\", curr_droid[-1])\n",
        "print(\"Username          :\", curr_pilot[-1])\n",
        "\n",
        "print(\"Python version    :\", sys.version.split('\\n')[0])"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "The authentication to Google is necessary to run BigQuery queries.\n",
        "\n",
        "Every operation throughout the whole notebook (BigQuery, fetching data from the IDC buckets) is completely free. The only thing that is needed in order to run the notebook is the set-up of a Google Cloud project. In order for the notebook to work as intended, you will need to specify the name of the project in the cell after the authentication one."
      ],
      "metadata": {
        "id": "0PKq2z2jkVJz"
      }
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "2DMUqTOVF5WP"
      },
      "outputs": [],
      "source": [
        "from google.colab import auth\n",
        "auth.authenticate_user()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "9Y43S1F35h8m"
      },
      "outputs": [],
      "source": [
        "from google.colab import files\n",
        "from google.cloud import storage\n",
        "from google.cloud import bigquery as bq\n",
        "\n",
        "# INSERT THE ID OF YOUR PROJECT HERE!\n",
        "project_id = \"idc-sandbox-000\""
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "oT7coYaGfd4t"
      },
      "source": [
        "Throughout this Colab notebook, for image pre-processing we will use [Plastimatch](https://plastimatch.org), a reliable and open source software for image computation. We will be running Plastimatch using the simple [PyPlastimatch](https://github.com/AIM-Harvard/pyplastimatch/tree/main/pyplastimatch) python wrapper. "
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "6NLhlFqyEgBf"
      },
      "outputs": [],
      "source": [
        "%%capture\n",
        "!apt install plastimatch"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "_ILwxgbcfcpr",
        "outputId": "fab8cb04-2ee4-46a5-9666-1264ffbda20f"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "plastimatch version 1.7.0\n"
          ]
        }
      ],
      "source": [
        "# check plastimatch was correctly installed\n",
        "!plastimatch --version"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "---\n",
        "\n",
        "Start by cloning the AIMI hub repository on the Colab instance.\n",
        "\n",
        "The AIMI hub repository stores all the code we will use for pulling, preprocessing, processing, and postprocessing the data for this use case - as long as the others shared through AIMI hub."
      ],
      "metadata": {
        "id": "MrNlgYZJIxkf"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "!git clone https://github.com/AIM-Harvard/aimi_alpha.git aimi"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "y5lBiCWMJF9d",
        "outputId": "2d18c92a-cf2b-416f-dd58-3be6a2bb2392"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Cloning into 'aimi'...\n",
            "remote: Enumerating objects: 528, done.\u001b[K\n",
            "remote: Counting objects: 100% (166/166), done.\u001b[K\n",
            "remote: Compressing objects: 100% (122/122), done.\u001b[K\n",
            "remote: Total 528 (delta 80), reused 91 (delta 38), pack-reused 362\u001b[K\n",
            "Receiving objects: 100% (528/528), 5.23 MiB | 30.24 MiB/s, done.\n",
            "Resolving deltas: 100% (274/274), done.\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "To organise the DICOM data in a more common (and human-understandable) fashion after downloading those from the buckets, we will make use of [DICOMSort](https://github.com/pieper/dicomsort). \n",
        "\n",
        "DICOMSort is an open source tool for custom sorting and renaming of dicom files based on their specific DICOM tags. In our case, we will exploit DICOMSort to organise the DICOM data by `PatientID` and `Modality` - so that the final directory will look like the following:\n",
        "\n",
        "```\n",
        "data/raw/nsclc-radiomics/dicom/$PatientID\n",
        " └─── CT\n",
        "       ├─── $SOPInstanceUID_slice0.dcm\n",
        "       ├─── $SOPInstanceUID_slice1.dcm\n",
        "       ├───  ...\n",
        "       │\n",
        "      RTSTRUCT \n",
        "       ├─── $SOPInstanceUID_RTSTRUCT.dcm\n",
        "      SEG\n",
        "       └─── $SOPInstanceUID_RTSEG.dcm\n",
        "\n",
        "```"
      ],
      "metadata": {
        "id": "Pcj1hYK7lzmA"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "!git clone https://github.com/pieper/dicomsort dicomsort"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Py53FxrlD9eK",
        "outputId": "a939eb40-be2b-4a93-fa31-b95d0af0d764"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Cloning into 'dicomsort'...\n",
            "remote: Enumerating objects: 130, done.\u001b[K\n",
            "remote: Counting objects:  25% (1/4)\u001b[K\rremote: Counting objects:  50% (2/4)\u001b[K\rremote: Counting objects:  75% (3/4)\u001b[K\rremote: Counting objects: 100% (4/4)\u001b[K\rremote: Counting objects: 100% (4/4), done.\u001b[K\n",
            "remote: Compressing objects: 100% (4/4), done.\u001b[K\n",
            "remote: Total 130 (delta 0), reused 1 (delta 0), pack-reused 126\u001b[K\n",
            "Receiving objects: 100% (130/130), 44.12 KiB | 2.45 MiB/s, done.\n",
            "Resolving deltas: 100% (63/63), done.\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "7ztrKrzSm1WT"
      },
      "source": [
        "We will also use DCMQI for converting the resulting segmentation into standard DICOM SEG objects."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "iYf72nA8m0wh"
      },
      "outputs": [],
      "source": [
        "%%capture\n",
        "dcmqi_release_url = \"https://github.com/QIICR/dcmqi/releases/download/v1.2.4/dcmqi-1.2.4-linux.tar.gz\"\n",
        "dcmqi_download_path = \"/content/dcmqi-1.2.4-linux.tar.gz\"\n",
        "dcmqi_path = \"/content/dcmqi-1.2.4-linux\"\n",
        "\n",
        "!wget -O $dcmqi_download_path $dcmqi_release_url\n",
        "\n",
        "!tar -xvf $dcmqi_download_path\n",
        "\n",
        "!mv $dcmqi_path/bin/* /bin"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "4JBJmF7rmz1S"
      },
      "source": [
        "---"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "CthT1kRuywEh"
      },
      "outputs": [],
      "source": [
        "%%capture\n",
        "!pip install pyplastimatch nnunet ipywidgets\n",
        "!pip install TotalSegmentator"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "import shutil\n",
        "import random\n",
        "\n",
        "import json\n",
        "import pprint\n",
        "import numpy as np\n",
        "import pandas as pd\n",
        "\n",
        "import pydicom\n",
        "import nibabel as nib\n",
        "import SimpleITK as sitk\n",
        "import pyplastimatch as pypla\n",
        "\n",
        "print(\"Python version               : \", sys.version.split('\\n')[0])\n",
        "print(\"Numpy version                : \", np.__version__)\n",
        "\n",
        "# ----------------------------------------\n",
        "\n",
        "#everything that has to do with plotting goes here below\n",
        "import matplotlib\n",
        "matplotlib.use(\"agg\")\n",
        "\n",
        "import matplotlib.pyplot as plt\n",
        "from matplotlib.colors import ListedColormap\n",
        "from matplotlib.patches import Patch\n",
        "\n",
        "%matplotlib inline\n",
        "%config InlineBackend.figure_format = \"png\"\n",
        "\n",
        "import ipywidgets as ipyw\n",
        "\n",
        "## ----------------------------------------\n",
        "\n",
        "# create new colormap appending the alpha channel to the selected one\n",
        "# (so that we don't get a \\\"color overlay\\\" when plotting the segmask superimposed to the CT)\n",
        "cmap = plt.cm.Reds\n",
        "my_reds = cmap(np.arange(cmap.N))\n",
        "my_reds[:, -1] = np.linspace(0, 1, cmap.N)\n",
        "my_reds = ListedColormap(my_reds)\n",
        "\n",
        "cmap = plt.cm.Greens\n",
        "my_greens = cmap(np.arange(cmap.N))\n",
        "my_greens[:, -1] = np.linspace(0, 1, cmap.N)\n",
        "my_greens = ListedColormap(my_greens)\n",
        "\n",
        "cmap = plt.cm.Blues\n",
        "my_blues = cmap(np.arange(cmap.N))\n",
        "my_blues[:, -1] = np.linspace(0, 1, cmap.N)\n",
        "my_blues = ListedColormap(my_blues)\n",
        "\n",
        "cmap = plt.cm.spring\n",
        "my_spring = cmap(np.arange(cmap.N))\n",
        "my_spring[:, -1] = np.linspace(0, 1, cmap.N)\n",
        "my_spring = ListedColormap(my_spring)\n",
        "## ----------------------------------------\n",
        "\n",
        "import seaborn as sns"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "3euRfHl7pe2u",
        "outputId": "391fa7c0-2315-40b5-b2c0-e26a6270df77"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Python version               :  3.7.15 (default, Oct 12 2022, 19:14:55) \n",
            "Numpy version                :  1.21.6\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Provided everything was set up correctly, we can run the BigQuery query and get all the information we need to download the testing data from the IDC platform.\n",
        "\n",
        "For this specific use case, we are going to be working with the \"CT lymph nodes\" collection hosted on IDC - which groups a collections of series that are close to whole body CT scans."
      ],
      "metadata": {
        "id": "aZOtlGtcmOq5"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "%%bigquery --project=$project_id cohort_df\n",
        "\n",
        "SELECT\n",
        "  dicom_pivot_v11.PatientID,\n",
        "  dicom_pivot_v11.collection_id,\n",
        "  dicom_pivot_v11.source_DOI,\n",
        "  dicom_pivot_v11.StudyInstanceUID,\n",
        "  dicom_pivot_v11.SeriesInstanceUID,\n",
        "  dicom_pivot_v11.SOPInstanceUID,\n",
        "  dicom_pivot_v11.gcs_url\n",
        "FROM\n",
        "  `bigquery-public-data.idc_v11.dicom_pivot_v11` dicom_pivot_v11\n",
        "WHERE\n",
        "  StudyInstanceUID IN (\n",
        "    SELECT\n",
        "      StudyInstanceUID\n",
        "    FROM\n",
        "      `bigquery-public-data.idc_v11.dicom_pivot_v11` dicom_pivot_v11\n",
        "    WHERE\n",
        "      (\n",
        "        dicom_pivot_v11.collection_id IN ('Community', 'ct_lymph_nodes')\n",
        "      )\n",
        "    GROUP BY\n",
        "      StudyInstanceUID\n",
        "  )\n",
        "GROUP BY\n",
        "  dicom_pivot_v11.PatientID,\n",
        "  dicom_pivot_v11.collection_id,\n",
        "  dicom_pivot_v11.source_DOI,\n",
        "  dicom_pivot_v11.StudyInstanceUID,\n",
        "  dicom_pivot_v11.SeriesInstanceUID,\n",
        "  dicom_pivot_v11.SOPInstanceUID,\n",
        "  dicom_pivot_v11.gcs_url\n",
        "ORDER BY\n",
        "  dicom_pivot_v11.PatientID ASC,\n",
        "  dicom_pivot_v11.collection_id ASC,\n",
        "  dicom_pivot_v11.source_DOI ASC,\n",
        "  dicom_pivot_v11.StudyInstanceUID ASC,\n",
        "  dicom_pivot_v11.SeriesInstanceUID ASC,\n",
        "  dicom_pivot_v11.SOPInstanceUID ASC,\n",
        "  dicom_pivot_v11.gcs_url ASC"
      ],
      "metadata": {
        "id": "_8hRj0E5FW0A"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# this works as intended only if the BQ query parses data from a single dataset\n",
        "# if not, feel free to set the name manually!\n",
        "dataset_name = cohort_df[\"collection_id\"].values[0]\n",
        "\n",
        "dataset_name"
      ],
      "metadata": {
        "id": "oZApCCVLkZyb",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        },
        "outputId": "19d17ad1-6a8f-4985-dc0d-db2bad3df148"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "'ct_lymph_nodes'"
            ],
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "string"
            }
          },
          "metadata": {},
          "execution_count": 12
        }
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "GNutD-SRN52T"
      },
      "outputs": [],
      "source": [
        "# create the directory tree\n",
        "!mkdir -p data models\n",
        "\n",
        "!mkdir -p data/raw \n",
        "!mkdir -p data/raw/tmp data/raw/$dataset_name\n",
        "!mkdir -p data/raw/$dataset_name/dicom\n",
        "\n",
        "!mkdir -p data/processed\n",
        "!mkdir -p data/processed/$dataset_name\n",
        "!mkdir -p data/processed/$dataset_name/nii\n",
        "!mkdir -p data/processed/$dataset_name/dicomseg\n",
        "\n",
        "!mkdir -p data/model_input/\n",
        "!mkdir -p data/totalsegmentator_output/"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0Qu6eXpHx6iV"
      },
      "source": [
        "## **Parsing Cohort Information from BigQuery Tables**\n",
        "\n",
        "We can check the various fields of the table we populated by running the BigQuery query.\n",
        "\n",
        "This table will store one entry for each DICOM file in the dataset (therefore, expect thousands of rows!)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 902
        },
        "id": "V27IJihCi9Kv",
        "outputId": "270bc2df-af94-4ac3-e86b-30fa02dd8707"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Total number of unique Patient IDs: 176\n",
            "<class 'pandas.core.frame.DataFrame'>\n",
            "RangeIndex: 110003 entries, 0 to 110002\n",
            "Data columns (total 7 columns):\n",
            " #   Column             Non-Null Count   Dtype \n",
            "---  ------             --------------   ----- \n",
            " 0   PatientID          110003 non-null  object\n",
            " 1   collection_id      110003 non-null  object\n",
            " 2   source_DOI         110003 non-null  object\n",
            " 3   StudyInstanceUID   110003 non-null  object\n",
            " 4   SeriesInstanceUID  110003 non-null  object\n",
            " 5   SOPInstanceUID     110003 non-null  object\n",
            " 6   gcs_url            110003 non-null  object\n",
            "dtypes: object(7)\n",
            "memory usage: 5.9+ MB\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "None"
            ]
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "       PatientID   collection_id                     source_DOI  \\\n",
              "0  ABD_LYMPH_001  ct_lymph_nodes  10.7937/K9/TCIA.2015.AQIIDCNM   \n",
              "1  ABD_LYMPH_001  ct_lymph_nodes  10.7937/K9/TCIA.2015.AQIIDCNM   \n",
              "2  ABD_LYMPH_001  ct_lymph_nodes  10.7937/K9/TCIA.2015.AQIIDCNM   \n",
              "3  ABD_LYMPH_001  ct_lymph_nodes  10.7937/K9/TCIA.2015.AQIIDCNM   \n",
              "4  ABD_LYMPH_001  ct_lymph_nodes  10.7937/K9/TCIA.2015.AQIIDCNM   \n",
              "\n",
              "                              StudyInstanceUID  \\\n",
              "0  61.7.22285965616260355338860879829667630274   \n",
              "1  61.7.22285965616260355338860879829667630274   \n",
              "2  61.7.22285965616260355338860879829667630274   \n",
              "3  61.7.22285965616260355338860879829667630274   \n",
              "4  61.7.22285965616260355338860879829667630274   \n",
              "\n",
              "                              SeriesInstanceUID  \\\n",
              "0  61.7.167248355135476067044532759811631626828   \n",
              "1  61.7.167248355135476067044532759811631626828   \n",
              "2  61.7.167248355135476067044532759811631626828   \n",
              "3  61.7.167248355135476067044532759811631626828   \n",
              "4  61.7.167248355135476067044532759811631626828   \n",
              "\n",
              "                                 SOPInstanceUID  \\\n",
              "0  61.7.100530760313930961000572615593503636820   \n",
              "1  61.7.100619337614589303607528629909134919710   \n",
              "2  61.7.100722470958405165423499101883203258976   \n",
              "3  61.7.100926126811826446149832025888003249166   \n",
              "4  61.7.102568601113976310733671672702929246062   \n",
              "\n",
              "                                             gcs_url  \n",
              "0  gs://public-datasets-idc/38101099-8fae-44b5-be...  \n",
              "1  gs://public-datasets-idc/90b51943-20e5-4ce0-b7...  \n",
              "2  gs://public-datasets-idc/949a8429-0b08-4120-ad...  \n",
              "3  gs://public-datasets-idc/9190ed3e-edf4-4771-9d...  \n",
              "4  gs://public-datasets-idc/e050baf5-59e9-4416-8a...  "
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-69ebe14e-a0fa-4fad-9c28-413296d71856\">\n",
              "    <div class=\"colab-df-container\">\n",
              "      <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>PatientID</th>\n",
              "      <th>collection_id</th>\n",
              "      <th>source_DOI</th>\n",
              "      <th>StudyInstanceUID</th>\n",
              "      <th>SeriesInstanceUID</th>\n",
              "      <th>SOPInstanceUID</th>\n",
              "      <th>gcs_url</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>ABD_LYMPH_001</td>\n",
              "      <td>ct_lymph_nodes</td>\n",
              "      <td>10.7937/K9/TCIA.2015.AQIIDCNM</td>\n",
              "      <td>61.7.22285965616260355338860879829667630274</td>\n",
              "      <td>61.7.167248355135476067044532759811631626828</td>\n",
              "      <td>61.7.100530760313930961000572615593503636820</td>\n",
              "      <td>gs://public-datasets-idc/38101099-8fae-44b5-be...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>ABD_LYMPH_001</td>\n",
              "      <td>ct_lymph_nodes</td>\n",
              "      <td>10.7937/K9/TCIA.2015.AQIIDCNM</td>\n",
              "      <td>61.7.22285965616260355338860879829667630274</td>\n",
              "      <td>61.7.167248355135476067044532759811631626828</td>\n",
              "      <td>61.7.100619337614589303607528629909134919710</td>\n",
              "      <td>gs://public-datasets-idc/90b51943-20e5-4ce0-b7...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>ABD_LYMPH_001</td>\n",
              "      <td>ct_lymph_nodes</td>\n",
              "      <td>10.7937/K9/TCIA.2015.AQIIDCNM</td>\n",
              "      <td>61.7.22285965616260355338860879829667630274</td>\n",
              "      <td>61.7.167248355135476067044532759811631626828</td>\n",
              "      <td>61.7.100722470958405165423499101883203258976</td>\n",
              "      <td>gs://public-datasets-idc/949a8429-0b08-4120-ad...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>ABD_LYMPH_001</td>\n",
              "      <td>ct_lymph_nodes</td>\n",
              "      <td>10.7937/K9/TCIA.2015.AQIIDCNM</td>\n",
              "      <td>61.7.22285965616260355338860879829667630274</td>\n",
              "      <td>61.7.167248355135476067044532759811631626828</td>\n",
              "      <td>61.7.100926126811826446149832025888003249166</td>\n",
              "      <td>gs://public-datasets-idc/9190ed3e-edf4-4771-9d...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>ABD_LYMPH_001</td>\n",
              "      <td>ct_lymph_nodes</td>\n",
              "      <td>10.7937/K9/TCIA.2015.AQIIDCNM</td>\n",
              "      <td>61.7.22285965616260355338860879829667630274</td>\n",
              "      <td>61.7.167248355135476067044532759811631626828</td>\n",
              "      <td>61.7.102568601113976310733671672702929246062</td>\n",
              "      <td>gs://public-datasets-idc/e050baf5-59e9-4416-8a...</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-69ebe14e-a0fa-4fad-9c28-413296d71856')\"\n",
              "              title=\"Convert this dataframe to an interactive table.\"\n",
              "              style=\"display:none;\">\n",
              "        \n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "       width=\"24px\">\n",
              "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
              "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
              "  </svg>\n",
              "      </button>\n",
              "      \n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      flex-wrap:wrap;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "      <script>\n",
              "        const buttonEl =\n",
              "          document.querySelector('#df-69ebe14e-a0fa-4fad-9c28-413296d71856 button.colab-df-convert');\n",
              "        buttonEl.style.display =\n",
              "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "        async function convertToInteractive(key) {\n",
              "          const element = document.querySelector('#df-69ebe14e-a0fa-4fad-9c28-413296d71856');\n",
              "          const dataTable =\n",
              "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                     [key], {});\n",
              "          if (!dataTable) return;\n",
              "\n",
              "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "            + ' to learn more about interactive tables.';\n",
              "          element.innerHTML = '';\n",
              "          dataTable['output_type'] = 'display_data';\n",
              "          await google.colab.output.renderOutput(dataTable, element);\n",
              "          const docLink = document.createElement('div');\n",
              "          docLink.innerHTML = docLinkHtml;\n",
              "          element.appendChild(docLink);\n",
              "        }\n",
              "      </script>\n",
              "    </div>\n",
              "  </div>\n",
              "  "
            ]
          },
          "metadata": {}
        }
      ],
      "source": [
        "pat_id_list = sorted(list(set(cohort_df[\"PatientID\"].values)))\n",
        "\n",
        "print(\"Total number of unique Patient IDs:\", len(pat_id_list))\n",
        "\n",
        "display(cohort_df.info())\n",
        "\n",
        "display(cohort_df.head())"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "AolLtXOLVt7D"
      },
      "source": [
        "---\n",
        "\n",
        "## **Set Run Parameters**\n",
        "\n",
        "From this cell, we can configure the nnU-Net inference step - specifying, for instance, the type of model we want to run (among the four different models the framework provides), whether we want to use test time augmentation, or whether we want to export the soft probability maps of the segmentation masks.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "TRmrt00589lv"
      },
      "outputs": [],
      "source": [
        "# FIXED PARAMETERS\n",
        "data_base_path = \"/content/data\"\n",
        "raw_base_path = \"/content/data/raw/tmp\"\n",
        "sorted_base_path = os.path.join(\"/content/data/raw/\", dataset_name, \"dicom\")\n",
        "\n",
        "processed_base_path = os.path.join(\"/content/data/processed/\", dataset_name)\n",
        "processed_nifti_path = os.path.join(processed_base_path, \"nii\")\n",
        "\n",
        "processed_dicomseg_path = os.path.join(processed_base_path, \"dicomseg\")\n",
        "\n",
        "model_input_folder = \"/content/data/model_input/\"\n",
        "model_output_folder = \"/content/data/totalsegmentator_output/\"\n",
        "\n",
        "dicomseg_json_path = \"/content/aimi/aimi/totalsegmentator/config/dicomseg_metadata_whole.json\""
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-n0qjPp1B_mv"
      },
      "source": [
        "## **Running the Analysis for a Single Patient**"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "import aimi.aimi as aimi\n",
        "\n",
        "from aimi import general_utils as aimi_utils\n",
        "from aimi import totalsegmentator as aimi_model"
      ],
      "metadata": {
        "id": "vPPZRieLM5K-"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "The following cells run all the processing pipeline, from pre-processing to post-processing."
      ],
      "metadata": {
        "id": "0pFSCy5HoOp_"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "pat_id = random.choice(cohort_df[\"PatientID\"].values)\n",
        "pat_df = cohort_df[cohort_df[\"PatientID\"] == pat_id].reset_index(drop = True)\n",
        "\n",
        "display(pat_df.info())\n",
        "display(pat_df.head())"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 884
        },
        "id": "BTURQtGPgeQF",
        "outputId": "c7a623d0-eaa5-4b0d-c241-93902ba20361"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "<class 'pandas.core.frame.DataFrame'>\n",
            "RangeIndex: 604 entries, 0 to 603\n",
            "Data columns (total 7 columns):\n",
            " #   Column             Non-Null Count  Dtype \n",
            "---  ------             --------------  ----- \n",
            " 0   PatientID          604 non-null    object\n",
            " 1   collection_id      604 non-null    object\n",
            " 2   source_DOI         604 non-null    object\n",
            " 3   StudyInstanceUID   604 non-null    object\n",
            " 4   SeriesInstanceUID  604 non-null    object\n",
            " 5   SOPInstanceUID     604 non-null    object\n",
            " 6   gcs_url            604 non-null    object\n",
            "dtypes: object(7)\n",
            "memory usage: 33.2+ KB\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "None"
            ]
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "       PatientID   collection_id                     source_DOI  \\\n",
              "0  ABD_LYMPH_051  ct_lymph_nodes  10.7937/K9/TCIA.2015.AQIIDCNM   \n",
              "1  ABD_LYMPH_051  ct_lymph_nodes  10.7937/K9/TCIA.2015.AQIIDCNM   \n",
              "2  ABD_LYMPH_051  ct_lymph_nodes  10.7937/K9/TCIA.2015.AQIIDCNM   \n",
              "3  ABD_LYMPH_051  ct_lymph_nodes  10.7937/K9/TCIA.2015.AQIIDCNM   \n",
              "4  ABD_LYMPH_051  ct_lymph_nodes  10.7937/K9/TCIA.2015.AQIIDCNM   \n",
              "\n",
              "                               StudyInstanceUID  \\\n",
              "0  61.7.186150680759207515465616855509993541745   \n",
              "1  61.7.186150680759207515465616855509993541745   \n",
              "2  61.7.186150680759207515465616855509993541745   \n",
              "3  61.7.186150680759207515465616855509993541745   \n",
              "4  61.7.186150680759207515465616855509993541745   \n",
              "\n",
              "                             SeriesInstanceUID  \\\n",
              "0  61.7.78354494451481445463641298847814601170   \n",
              "1  61.7.78354494451481445463641298847814601170   \n",
              "2  61.7.78354494451481445463641298847814601170   \n",
              "3  61.7.78354494451481445463641298847814601170   \n",
              "4  61.7.78354494451481445463641298847814601170   \n",
              "\n",
              "                                 SOPInstanceUID  \\\n",
              "0  61.7.100566048483125644867935332876562205946   \n",
              "1  61.7.100901945078719662881101095720774848351   \n",
              "2  61.7.101061308537768984067178774610468055521   \n",
              "3  61.7.101584183168773379529684508872480445240   \n",
              "4  61.7.101763697669609146583900831933266650178   \n",
              "\n",
              "                                             gcs_url  \n",
              "0  gs://public-datasets-idc/354163b0-1f3f-403c-81...  \n",
              "1  gs://public-datasets-idc/06d37009-f170-413f-ad...  \n",
              "2  gs://public-datasets-idc/3b8b2fef-4b71-4883-ae...  \n",
              "3  gs://public-datasets-idc/ea0eae2b-0fe4-440f-97...  \n",
              "4  gs://public-datasets-idc/5e0317e9-dbee-4044-a7...  "
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-8c9061dd-16be-4c71-8616-1ed48d4e674d\">\n",
              "    <div class=\"colab-df-container\">\n",
              "      <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>PatientID</th>\n",
              "      <th>collection_id</th>\n",
              "      <th>source_DOI</th>\n",
              "      <th>StudyInstanceUID</th>\n",
              "      <th>SeriesInstanceUID</th>\n",
              "      <th>SOPInstanceUID</th>\n",
              "      <th>gcs_url</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>ABD_LYMPH_051</td>\n",
              "      <td>ct_lymph_nodes</td>\n",
              "      <td>10.7937/K9/TCIA.2015.AQIIDCNM</td>\n",
              "      <td>61.7.186150680759207515465616855509993541745</td>\n",
              "      <td>61.7.78354494451481445463641298847814601170</td>\n",
              "      <td>61.7.100566048483125644867935332876562205946</td>\n",
              "      <td>gs://public-datasets-idc/354163b0-1f3f-403c-81...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>ABD_LYMPH_051</td>\n",
              "      <td>ct_lymph_nodes</td>\n",
              "      <td>10.7937/K9/TCIA.2015.AQIIDCNM</td>\n",
              "      <td>61.7.186150680759207515465616855509993541745</td>\n",
              "      <td>61.7.78354494451481445463641298847814601170</td>\n",
              "      <td>61.7.100901945078719662881101095720774848351</td>\n",
              "      <td>gs://public-datasets-idc/06d37009-f170-413f-ad...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>ABD_LYMPH_051</td>\n",
              "      <td>ct_lymph_nodes</td>\n",
              "      <td>10.7937/K9/TCIA.2015.AQIIDCNM</td>\n",
              "      <td>61.7.186150680759207515465616855509993541745</td>\n",
              "      <td>61.7.78354494451481445463641298847814601170</td>\n",
              "      <td>61.7.101061308537768984067178774610468055521</td>\n",
              "      <td>gs://public-datasets-idc/3b8b2fef-4b71-4883-ae...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>ABD_LYMPH_051</td>\n",
              "      <td>ct_lymph_nodes</td>\n",
              "      <td>10.7937/K9/TCIA.2015.AQIIDCNM</td>\n",
              "      <td>61.7.186150680759207515465616855509993541745</td>\n",
              "      <td>61.7.78354494451481445463641298847814601170</td>\n",
              "      <td>61.7.101584183168773379529684508872480445240</td>\n",
              "      <td>gs://public-datasets-idc/ea0eae2b-0fe4-440f-97...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>ABD_LYMPH_051</td>\n",
              "      <td>ct_lymph_nodes</td>\n",
              "      <td>10.7937/K9/TCIA.2015.AQIIDCNM</td>\n",
              "      <td>61.7.186150680759207515465616855509993541745</td>\n",
              "      <td>61.7.78354494451481445463641298847814601170</td>\n",
              "      <td>61.7.101763697669609146583900831933266650178</td>\n",
              "      <td>gs://public-datasets-idc/5e0317e9-dbee-4044-a7...</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-8c9061dd-16be-4c71-8616-1ed48d4e674d')\"\n",
              "              title=\"Convert this dataframe to an interactive table.\"\n",
              "              style=\"display:none;\">\n",
              "        \n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "       width=\"24px\">\n",
              "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
              "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
              "  </svg>\n",
              "      </button>\n",
              "      \n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      flex-wrap:wrap;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "      <script>\n",
              "        const buttonEl =\n",
              "          document.querySelector('#df-8c9061dd-16be-4c71-8616-1ed48d4e674d button.colab-df-convert');\n",
              "        buttonEl.style.display =\n",
              "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "        async function convertToInteractive(key) {\n",
              "          const element = document.querySelector('#df-8c9061dd-16be-4c71-8616-1ed48d4e674d');\n",
              "          const dataTable =\n",
              "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                     [key], {});\n",
              "          if (!dataTable) return;\n",
              "\n",
              "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "            + ' to learn more about interactive tables.';\n",
              "          element.innerHTML = '';\n",
              "          dataTable['output_type'] = 'display_data';\n",
              "          await google.colab.output.renderOutput(dataTable, element);\n",
              "          const docLink = document.createElement('div');\n",
              "          docLink.innerHTML = docLinkHtml;\n",
              "          element.appendChild(docLink);\n",
              "        }\n",
              "      </script>\n",
              "    </div>\n",
              "  </div>\n",
              "  "
            ]
          },
          "metadata": {}
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# init\n",
        "\n",
        "print(\"Processing patient: %s\"%(pat_id))\n",
        "\n",
        "patient_df = cohort_df[cohort_df[\"PatientID\"] == pat_id]\n",
        "\n",
        "dicomseg_fn = pat_id + \"_SEG.dcm\"\n",
        "\n",
        "input_nifti_fn = pat_id + \".nii.gz\"\n",
        "input_nifti_path = os.path.join(model_input_folder, input_nifti_fn)\n",
        "\n",
        "pred_nifti_fn = pat_id + \".nii.gz\"\n",
        "pred_nifti_path = os.path.join(model_output_folder, pred_nifti_fn)\n",
        "\n",
        "pred_softmax_folder_name = \"pred_softmax\"\n",
        "pred_softmax_folder_path = os.path.join(processed_nifti_path, pat_id, pred_softmax_folder_name)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "S7a6h_iINlvq",
        "outputId": "f265ef3d-256e-4028-a0c7-59cd33d405c3"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Processing patient: ABD_LYMPH_051\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# data cross-loading\n",
        "aimi_utils.gcs.download_patient_data(raw_base_path = raw_base_path,\n",
        "                                     sorted_base_path = sorted_base_path,\n",
        "                                     patient_df = patient_df,\n",
        "                                     remove_raw = True)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "NZDxWkvCNlxw",
        "outputId": "092dcb7b-e397-430b-d11c-200104f544d6"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Copying files from IDC buckets to /content/data/raw/tmp/ABD_LYMPH_051...\n",
            "Done in 26.7915 seconds.\n",
            "\n",
            "Sorting DICOM files...\n",
            "Done in 1.28231 seconds.\n",
            "Sorted DICOM data saved at: /content/data/raw/ct_lymph_nodes/dicom/ABD_LYMPH_051\n",
            "Removing un-sorted data at /content/data/raw/tmp/ABD_LYMPH_051...\n",
            "... Done.\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# DICOM CT to NIfTI - required for the processing\n",
        "aimi_utils.preprocessing.pypla_dicom_ct_to_nifti(sorted_base_path = sorted_base_path,\n",
        "                                                 processed_nifti_path = processed_nifti_path,\n",
        "                                                 pat_id = pat_id, verbose = True)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "FfsY-_EGNs19",
        "outputId": "5618e519-707a-445b-b360-acc086035165"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "\n",
            "Running 'plastimatch convert' with the specified arguments:\n",
            "  --input /content/data/raw/ct_lymph_nodes/dicom/ABD_LYMPH_051/CT\n",
            "  --output-img /content/data/processed/ct_lymph_nodes/nii/ABD_LYMPH_051/ABD_LYMPH_051_CT.nii.gz\n",
            "... Done.\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# prepare the `model_input` folder for the inference phase\n",
        "aimi_utils.preprocessing.prep_ct_input_data(processed_nifti_path = processed_nifti_path,\n",
        "                                            model_input_folder = model_input_folder,\n",
        "                                            pat_id = pat_id)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "0R547UiyNs4O",
        "outputId": "56f79edd-d7ad-42a6-94a6-d2d1b2d6b92d"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Copying /content/data/processed/ct_lymph_nodes/nii/ABD_LYMPH_051/ABD_LYMPH_051_CT.nii.gz\n",
            "to /content/data/model_input/ABD_LYMPH_051_0000.nii.gz...\n",
            "... Done.\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "The following cell runs the DL-based segmentation.\n"
      ],
      "metadata": {
        "id": "LYlxAj05gY6w"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "aimi_model.utils.processing.process_patient(pat_id = pat_id,\n",
        "                                            model_input_folder = model_input_folder,\n",
        "                                            model_output_folder = model_output_folder)"
      ],
      "metadata": {
        "id": "9Lw70nH9wlD9",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "9e64b84c-0bc5-482f-89c1-b3937d212b14"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Running TotalSegmentator in default mode (1.5mm)\n",
            "Done in 421.064 seconds.\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "---\n"
      ],
      "metadata": {
        "id": "yf08ZiR9LeOJ"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "processed_seg_folder = os.path.join(processed_nifti_path, pat_id, \"totalsegmentator\")\n",
        "\n",
        "shutil.copytree(model_output_folder, processed_seg_folder)"
      ],
      "metadata": {
        "id": "vsViIeibKLb4",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        },
        "outputId": "4e37c668-fad8-4eb3-c3f7-8c70dbdcaec7"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "'/content/data/processed/ct_lymph_nodes/nii/ABD_LYMPH_051/totalsegmentator'"
            ],
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "string"
            }
          },
          "metadata": {},
          "execution_count": 38
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Given that TotalSegmentator segments more than one hundred structures, the conversion from *.nii.gz to DICOM SEG might take a few minutes."
      ],
      "metadata": {
        "id": "G8H1es4f7S_k"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "aimi_model.utils.postprocessing.nifti_to_dicomseg(sorted_base_path = sorted_base_path,\n",
        "                                                  processed_base_path = processed_base_path,\n",
        "                                                  dicomseg_json_path = dicomseg_json_path,\n",
        "                                                  pat_id = pat_id)"
      ],
      "metadata": {
        "id": "-4vilcX1KLiI"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "---\n",
        "\n",
        "## **Data Download**"
      ],
      "metadata": {
        "id": "i2QqrdvXqQq0"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "%%capture\n",
        "\n",
        "archive_fn = \"%s.zip\"%(pat_id)\n",
        "\n",
        "try:\n",
        "  os.remove(archive_fn)\n",
        "except OSError:\n",
        "  pass\n",
        "\n",
        "seg_dicom_path = os.path.join(processed_dicomseg_path, pat_id, dicomseg_fn)\n",
        "ct_dicom_path = os.path.join(sorted_base_path, pat_id)\n",
        "\n",
        "!zip -j -r $archive_fn $ct_dicom_path $seg_dicom_path"
      ],
      "metadata": {
        "id": "ARmBD0lHdz_2"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "filesize = os.stat(archive_fn).st_size/1024e03\n",
        "print('Starting the download of \"%s\" (%2.1f MB)...\\n'%(archive_fn, filesize))\n",
        "\n",
        "files.download(archive_fn)"
      ],
      "metadata": {
        "id": "8pE0X9kid0D-",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 70
        },
        "outputId": "5ad963bd-c67e-4ad7-d61e-1fa0b6225b79"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Starting the download of \"ABD_LYMPH_051.zip\" (178.4 MB)...\n",
            "\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<IPython.core.display.Javascript object>"
            ],
            "application/javascript": [
              "\n",
              "    async function download(id, filename, size) {\n",
              "      if (!google.colab.kernel.accessAllowed) {\n",
              "        return;\n",
              "      }\n",
              "      const div = document.createElement('div');\n",
              "      const label = document.createElement('label');\n",
              "      label.textContent = `Downloading \"${filename}\": `;\n",
              "      div.appendChild(label);\n",
              "      const progress = document.createElement('progress');\n",
              "      progress.max = size;\n",
              "      div.appendChild(progress);\n",
              "      document.body.appendChild(div);\n",
              "\n",
              "      const buffers = [];\n",
              "      let downloaded = 0;\n",
              "\n",
              "      const channel = await google.colab.kernel.comms.open(id);\n",
              "      // Send a message to notify the kernel that we're ready.\n",
              "      channel.send({})\n",
              "\n",
              "      for await (const message of channel.messages) {\n",
              "        // Send a message to notify the kernel that we're ready.\n",
              "        channel.send({})\n",
              "        if (message.buffers) {\n",
              "          for (const buffer of message.buffers) {\n",
              "            buffers.push(buffer);\n",
              "            downloaded += buffer.byteLength;\n",
              "            progress.value = downloaded;\n",
              "          }\n",
              "        }\n",
              "      }\n",
              "      const blob = new Blob(buffers, {type: 'application/binary'});\n",
              "      const a = document.createElement('a');\n",
              "      a.href = window.URL.createObjectURL(blob);\n",
              "      a.download = filename;\n",
              "      div.appendChild(a);\n",
              "      a.click();\n",
              "      div.remove();\n",
              "    }\n",
              "  "
            ]
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<IPython.core.display.Javascript object>"
            ],
            "application/javascript": [
              "download(\"download_06183528-e0cb-4f6f-9d89-2b0bec0dd036\", \"ABD_LYMPH_051.zip\", 182724423)"
            ]
          },
          "metadata": {}
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [],
      "metadata": {
        "id": "pBWet9OLb_LB"
      },
      "execution_count": null,
      "outputs": []
    }
  ],
  "metadata": {
    "colab": {
      "collapsed_sections": [],
      "provenance": [],
      "machine_shape": "hm",
      "include_colab_link": true
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    },
    "language_info": {
      "name": "python"
    },
    "gpuClass": "premium",
    "accelerator": "GPU"
  },
  "nbformat": 4,
  "nbformat_minor": 0
}