notebooks/300_benchmarking/302_hpo_wandb.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "owTxSDKzB9oO"
   },
   "source": [
    "# Walkthrough on Hyperparameter Optimization using Weights and Biases"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Setting up the Working Directory\n",
    "This cell is to ensure we change the directory to anomalib source code to have access to the datasets and config files. We assume that you already went through `001_getting_started.ipynb` and install the required packages."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "from pathlib import Path\n",
    "\n",
    "from git.repo import Repo\n",
    "\n",
    "current_directory = Path.cwd()\n",
    "if current_directory.name == \"300_benchmarking\":\n",
    "    # On the assumption that, the notebook is located in\n",
    "    #   ~/anomalib/notebooks/300_benchmarking/\n",
    "    root_directory = current_directory.parent.parent\n",
    "elif current_directory.name == \"anomalib\":\n",
    "    # This means that the notebook is run from the main anomalib directory.\n",
    "    root_directory = current_directory\n",
    "else:\n",
    "    # Otherwise, we'll need to clone the anomalib repo to the `current_directory`\n",
    "    repo = Repo.clone_from(url=\"https://github.com/openvinotoolkit/anomalib.git\", to_path=current_directory)\n",
    "    root_directory = current_directory / \"anomalib\"\n",
    "\n",
    "os.chdir(root_directory)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "w72Q-XRmCKau"
   },
   "outputs": [],
   "source": [
    "%pip install ."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "BkrI2pI5C6Z-"
   },
   "source": [
    "> Note: Restart Runtime if promted by clicking the button at the end of the install logs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "gsWKy1FUCXoQ"
   },
   "source": [
    "## Download and Setup Dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "7E8nMjLVCXXg"
   },
   "outputs": [],
   "source": [
    "!wget https://openvinotoolkit.github.io/anomalib/_downloads/3f2af1d7748194b18c2177a34c03a2c4/hazelnut_toy.zip\n",
    "!unzip hazelnut_toy.zip -d datasets/ > /dev/null\n",
    "!rm hazelnut_toy.zip"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "id": "Zhc8ai8VGKGZ"
   },
   "source": [
    "## Creating training configuration\n",
    "\n",
    "Since we are using our [custom dataset](https://openvinotoolkit.github.io/anomalib/how_to_guides/train_custom_data.html) we need to modify the default configuration file. The following configuration is based on the one available here `anomalib/anomalib/models/stfpm/config.yaml`. We will first read the stfpm configs and replace the dataset configuration with our custom dataset configuration."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "from omegaconf import OmegaConf\n",
    "\n",
    "dataset_configuration_str = f\"\"\"\n",
    "  name: hazelnut\n",
    "  format: folder\n",
    "  root: {str(root_directory / \"datasets\" / \"hazelnut_toy\")}\n",
    "  normal_dir: good # name of the folder containing normal images.\n",
    "  abnormal_dir: colour # name of the folder containing abnormal images.\n",
    "  normal_test_dir: null # name of the folder containing normal test images.\n",
    "  mask_dir: mask/colour # optional\n",
    "  task: segmentation # classification or segmentation\n",
    "  normalization: imagenet\n",
    "  extensions: null\n",
    "  split_ratio: 0.2  # ratio of the normal images that will be used to create a test split\n",
    "  image_size: 256\n",
    "  center_crop: 256\n",
    "  train_batch_size: 32\n",
    "  eval_batch_size: 32\n",
    "  num_workers: 8\n",
    "  transform_config:\n",
    "    train: null\n",
    "    eval: null\n",
    "  test_split_mode: from_dir # options: [from_dir, synthetic]\n",
    "  test_split_ratio: 0.2 # fraction of train images held out testing (usage depends on test_split_mode)\n",
    "  val_split_mode: same_as_test # options: [same_as_test, from_test, synthetic]\n",
    "  val_split_ratio: 0.5 # fraction of train/test images held out for validation (usage depends on val_split_mode)\n",
    "  tiling:\n",
    "    apply: false\n",
    "    tile_size: null\n",
    "    stride: null\n",
    "    remove_border_count: 0\n",
    "    use_random_tiling: False\n",
    "    random_tile_count: 16\n",
    "\"\"\"\n",
    "\n",
    "config = OmegaConf.load(root_directory / \"anomalib\" / \"models\" / \"stfpm\" / \"config.yaml\")\n",
    "dataset_configurations = OmegaConf.create(dataset_configuration_str)\n",
    "config.dataset = dataset_configurations"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "id": "R7broXB-EYnx"
   },
   "source": [
    "## Create sweep configuration\n",
    "\n",
    "The following configuration file is based on the one available at `anomalib/tools/hpo/configs/stfpm.yaml`. For this example we will use the STFPM model. It is possible to define multiple parameters for each section. For instance, we can define multiple image_sizes, or backbones. The sweep will then run for each combination of parameters. We will also define the number of runs for each combination of parameters with `observation_budget`. The sweep will run for a total of 10 runs for this config file. You could increase the `observation_budget` to run more runs. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "sweep_config_str = \"\"\"\n",
    "observation_budget: 10\n",
    "method: bayes\n",
    "metric:\n",
    "  name: pixel_AUROC\n",
    "  goal: maximize\n",
    "parameters:\n",
    "  dataset:\n",
    "    category: capsule\n",
    "    image_size:\n",
    "      values:\n",
    "      - 256    # Add as many values as you want.\n",
    "  model:\n",
    "    backbone:\n",
    "      values:\n",
    "      - resnet18    # Add as many backbones as you want.\n",
    "    lr:\n",
    "      min: 0.1\n",
    "      max: 0.9\n",
    "    momentum:\n",
    "      min: 0.1\n",
    "      max: 0.9\n",
    "\"\"\"\n",
    "sweep_config = OmegaConf.create(sweep_config_str)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> NOTE: To speed up the runs, we will use a small number of epochs and observation_budget. You could increase these parametere to get better results."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "# NOTE: This is to speed up the runs. Increase these two get better results!\n",
    "config.trainer.max_epochs = 5\n",
    "sweep_config.observation_budget = 1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Save the config files to the current directory.\n",
    "OmegaConf.save(config, \"model_config.yaml\")\n",
    "OmegaConf.save(sweep_config, \"sweep_config.yaml\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "id": "ZVM8GvxUDCfV"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/home/sakcay/.pyenv/versions/3.8.13/envs/anomalib/lib/python3.8/site-packages/requests/__init__.py:102: RequestsDependencyWarning: urllib3 (1.26.14) or chardet (5.1.0)/charset_normalizer (2.0.12) doesn't match a supported version!\n",
      "  warnings.warn(\"urllib3 ({}) or chardet ({})/charset_normalizer ({}) doesn't match a supported \"\n",
      "/home/sakcay/.pyenv/versions/3.8.13/envs/anomalib/lib/python3.8/site-packages/anomalib/config/config.py:238: UserWarning: The seed value is now fixed to 0. Up to v0.3.7, the seed was not fixed when the seed value was set to 0. If you want to use the random seed, please select `None` for the seed value (`null` in the YAML file) or remove the `seed` key from the YAML file.\n",
      "  warn(\n",
      "/home/sakcay/.pyenv/versions/3.8.13/envs/anomalib/lib/python3.8/site-packages/anomalib/config/config.py:179: DeprecationWarning: The 'split_ratio' parameter is deprecated and will be removed in a future release. Please use 'test_split_ratio' instead.\n",
      "  warn(\n",
      "/home/sakcay/.pyenv/versions/3.8.13/envs/anomalib/lib/python3.8/site-packages/anomalib/config/config.py:275: UserWarning: config.project.unique_dir is set to False. This does not ensure that your results will be written in an empty directory and you may overwrite files.\n",
      "  warn(\n",
      "Global seed set to 0\n",
      "Create sweep with ID: li9ferch\n",
      "Sweep URL: https://wandb.ai/anomalib/stfpm_hazelnut/sweeps/li9ferch\n",
      "\u001b[34m\u001b[1mwandb\u001b[0m: Agent Starting Run: sht9w2kj with config:\n",
      "\u001b[34m\u001b[1mwandb\u001b[0m: \tdataset.image_size: 256\n",
      "\u001b[34m\u001b[1mwandb\u001b[0m: \tmodel.backbone: wide_resnet50_2\n",
      "\u001b[34m\u001b[1mwandb\u001b[0m: \tmodel.lr: 0.3510811344480953\n",
      "\u001b[34m\u001b[1mwandb\u001b[0m: \tmodel.momentum: 0.5209698279681179\n",
      "/home/sakcay/.pyenv/versions/3.8.13/envs/anomalib/lib/python3.8/site-packages/requests/__init__.py:102: RequestsDependencyWarning: urllib3 (1.26.14) or chardet (5.1.0)/charset_normalizer (2.0.12) doesn't match a supported version!\n",
      "  warnings.warn(\"urllib3 ({}) or chardet ({})/charset_normalizer ({}) doesn't match a supported \"\n",
      "\u001b[34m\u001b[1mwandb\u001b[0m: Currently logged in as: \u001b[33msamet-akcay\u001b[0m (\u001b[33manomalib\u001b[0m). Use \u001b[1m`wandb login --relogin`\u001b[0m to force relogin\n",
      "\u001b[34m\u001b[1mwandb\u001b[0m: wandb version 0.13.9 is available!  To upgrade, please run:\n",
      "\u001b[34m\u001b[1mwandb\u001b[0m:  $ pip install wandb --upgrade\n",
      "\u001b[34m\u001b[1mwandb\u001b[0m: Tracking run with wandb version 0.12.17\n",
      "\u001b[34m\u001b[1mwandb\u001b[0m: Run data is saved locally in \u001b[35m\u001b[1m/home/sakcay/projects/anomalib/wandb/run-20230202_061839-sht9w2kj\u001b[0m\n",
      "\u001b[34m\u001b[1mwandb\u001b[0m: Run \u001b[1m`wandb offline`\u001b[0m to turn off syncing.\n",
      "\u001b[34m\u001b[1mwandb\u001b[0m: Resuming run \u001b[33mleafy-sweep-1\u001b[0m\n",
      "\u001b[34m\u001b[1mwandb\u001b[0m: ⭐️ View project at \u001b[34m\u001b[4mhttps://wandb.ai/anomalib/stfpm_hazelnut\u001b[0m\n",
      "\u001b[34m\u001b[1mwandb\u001b[0m: 🧹 View sweep at \u001b[34m\u001b[4mhttps://wandb.ai/anomalib/stfpm_hazelnut/sweeps/li9ferch\u001b[0m\n",
      "\u001b[34m\u001b[1mwandb\u001b[0m: 🚀 View run at \u001b[34m\u001b[4mhttps://wandb.ai/anomalib/stfpm_hazelnut/runs/sht9w2kj\u001b[0m\n",
      "/home/sakcay/.pyenv/versions/3.8.13/envs/anomalib/lib/python3.8/site-packages/torchmetrics/utilities/prints.py:36: DeprecationWarning: From v0.10 an `'Binary*'`, `'Multiclass*', `'Multilabel*'` version now exist of each classification metric. Moving forward we recommend using these versions. This base metric will still work as it did prior to v0.10 until v0.11. From v0.11 the `task` argument introduced in this metric will be required and the general order of arguments may change, such that this metric will just function as an single entrypoint to calling the three specialized versions.\n",
      "  warnings.warn(*args, **kwargs)\n",
      "/home/sakcay/.pyenv/versions/3.8.13/envs/anomalib/lib/python3.8/site-packages/torchmetrics/utilities/prints.py:36: UserWarning: Metric `PrecisionRecallCurve` will save all targets and predictions in buffer. For large datasets this may lead to large memory footprint.\n",
      "  warnings.warn(*args, **kwargs)\n",
      "FeatureExtractor is deprecated. Use TimmFeatureExtractor instead. Both FeatureExtractor and TimmFeatureExtractor will be removed in version 2023.1\n",
      "FeatureExtractor is deprecated. Use TimmFeatureExtractor instead. Both FeatureExtractor and TimmFeatureExtractor will be removed in version 2023.1\n",
      "/home/sakcay/.pyenv/versions/3.8.13/envs/anomalib/lib/python3.8/site-packages/pytorch_lightning/trainer/connectors/callback_connector.py:151: LightningDeprecationWarning: Setting `Trainer(checkpoint_callback=False)` is deprecated in v1.5 and will be removed in v1.7. Please consider using `Trainer(enable_checkpointing=False)`.\n",
      "  rank_zero_deprecation(\n",
      "GPU available: True, used: True\n",
      "TPU available: False, using: 0 TPU cores\n",
      "IPU available: False, using: 0 IPUs\n",
      "HPU available: False, using: 0 HPUs\n",
      "`Trainer(limit_train_batches=1.0)` was configured so 100% of the batches per epoch will be used..\n",
      "`Trainer(limit_val_batches=1.0)` was configured so 100% of the batches will be used..\n",
      "`Trainer(limit_test_batches=1.0)` was configured so 100% of the batches will be used..\n",
      "`Trainer(limit_predict_batches=1.0)` was configured so 100% of the batches will be used..\n",
      "`Trainer(val_check_interval=1.0)` was configured so validation will run at the end of the training epoch..\n",
      "/home/sakcay/.pyenv/versions/3.8.13/envs/anomalib/lib/python3.8/site-packages/torchmetrics/utilities/prints.py:36: UserWarning: Metric `ROC` will save all targets and predictions in buffer. For large datasets this may lead to large memory footprint.\n",
      "  warnings.warn(*args, **kwargs)\n",
      "LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0,1,2,3]\n",
      "\n",
      "  | Name            | Type                     | Params\n",
      "-------------------------------------------------------------\n",
      "0 | image_threshold | AnomalyScoreThreshold    | 0     \n",
      "1 | pixel_threshold | AnomalyScoreThreshold    | 0     \n",
      "2 | model           | STFPMModel               | 49.7 M\n",
      "3 | loss            | STFPMLoss                | 0     \n",
      "4 | image_metrics   | AnomalibMetricCollection | 0     \n",
      "5 | pixel_metrics   | AnomalibMetricCollection | 0     \n",
      "-------------------------------------------------------------\n",
      "24.9 M    Trainable params\n",
      "24.9 M    Non-trainable params\n",
      "49.7 M    Total params\n",
      "198.900   Total estimated model params size (MB)\n",
      "/home/sakcay/.pyenv/versions/3.8.13/envs/anomalib/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py:1933: PossibleUserWarning: The number of training batches (1) is smaller than the logging interval Trainer(log_every_n_steps=50). Set a lower value for log_every_n_steps if you want to see logs for the training epoch.\n",
      "  rank_zero_warn(\n",
      "Epoch 0:   0%|                                            | 0/2 [00:00<?, ?it/s]/home/sakcay/.pyenv/versions/3.8.13/envs/anomalib/lib/python3.8/site-packages/pytorch_lightning/utilities/data.py:72: UserWarning: Trying to infer the `batch_size` from an ambiguous collection. The batch size we found is 28. To avoid any miscalculations, use `self.log(..., batch_size=batch_size)`.\n",
      "  warning_cache.warn(\n",
      "Epoch 0:  50%|▌| 1/2 [00:01<00:01,  1.32s/it, loss=56.4, v_num=w2kj, train_loss_\n",
      "Validation: 0it [00:00, ?it/s]\u001b[A\n",
      "Validation:   0%|                                         | 0/1 [00:00<?, ?it/s]\u001b[A\n",
      "Validation DataLoader 0:   0%|                            | 0/1 [00:00<?, ?it/s]\u001b[A\n",
      "Validation DataLoader 0: 100%|████████████████████| 1/1 [00:00<00:00,  8.26it/s]\u001b[A\n",
      "Epoch 0: 100%|█| 2/2 [00:02<00:00,  1.10s/it, loss=56.4, v_num=w2kj, train_loss_\u001b[A/home/sakcay/.pyenv/versions/3.8.13/envs/anomalib/lib/python3.8/site-packages/torchmetrics/utilities/prints.py:36: DeprecationWarning: `torchmetrics.functional.auc` has been move to `torchmetrics.utilities.compute` in v0.10 and will be removed in v0.11.\n",
      "  warnings.warn(*args, **kwargs)\n",
      "Epoch 0: 100%|█| 2/2 [00:02<00:00,  1.22s/it, loss=56.4, v_num=w2kj, train_loss_\n",
      "Epoch 1:  50%|▌| 1/2 [00:01<00:01,  1.03s/it, loss=50.7, v_num=w2kj, train_loss_\u001b[A\n",
      "Validation: 0it [00:00, ?it/s]\u001b[A\n",
      "Validation:   0%|                                         | 0/1 [00:00<?, ?it/s]\u001b[A\n",
      "Validation DataLoader 0:   0%|                            | 0/1 [00:00<?, ?it/s]\u001b[A\n",
      "Validation DataLoader 0: 100%|████████████████████| 1/1 [00:00<00:00,  8.09it/s]\u001b[A\n",
      "Epoch 1: 100%|█| 2/2 [00:02<00:00,  1.13s/it, loss=50.7, v_num=w2kj, train_loss_\u001b[A\n",
      "Epoch 2:  50%|▌| 1/2 [00:01<00:01,  1.12s/it, loss=45, v_num=w2kj, train_loss_st\u001b[A\n",
      "Validation: 0it [00:00, ?it/s]\u001b[A\n",
      "Validation:   0%|                                         | 0/1 [00:00<?, ?it/s]\u001b[A\n",
      "Validation DataLoader 0:   0%|                            | 0/1 [00:00<?, ?it/s]\u001b[A\n",
      "Validation DataLoader 0: 100%|████████████████████| 1/1 [00:00<00:00,  8.22it/s]\u001b[A\n",
      "Epoch 2: 100%|█| 2/2 [00:02<00:00,  1.21s/it, loss=45, v_num=w2kj, train_loss_st\u001b[A\n",
      "Epoch 3:  50%|▌| 1/2 [00:01<00:01,  1.14s/it, loss=40.6, v_num=w2kj, train_loss_\u001b[A\n",
      "Validation: 0it [00:00, ?it/s]\u001b[A\n",
      "Validation:   0%|                                         | 0/1 [00:00<?, ?it/s]\u001b[A\n",
      "Validation DataLoader 0:   0%|                            | 0/1 [00:00<?, ?it/s]\u001b[A\n",
      "Validation DataLoader 0: 100%|████████████████████| 1/1 [00:00<00:00,  8.10it/s]\u001b[A\n",
      "Epoch 3: 100%|█| 2/2 [00:02<00:00,  1.21s/it, loss=40.6, v_num=w2kj, train_loss_\u001b[A\n",
      "Epoch 3: 100%|█| 2/2 [00:02<00:00,  1.21s/it, loss=40.6, v_num=w2kj, train_loss_\u001b[A\n",
      "\u001b[34m\u001b[1mwandb\u001b[0m: Waiting for W&B process to finish... \u001b[32m(success).\u001b[0m\n",
      "\u001b[34m\u001b[1mwandb\u001b[0m:                                                                                \n",
      "\u001b[34m\u001b[1mwandb\u001b[0m: \n",
      "\u001b[34m\u001b[1mwandb\u001b[0m: Run history:\n",
      "\u001b[34m\u001b[1mwandb\u001b[0m:               epoch ▁▁▃▃▆▆██\n",
      "\u001b[34m\u001b[1mwandb\u001b[0m:         image_AUROC ▁███\n",
      "\u001b[34m\u001b[1mwandb\u001b[0m:       image_F1Score ▁▆██\n",
      "\u001b[34m\u001b[1mwandb\u001b[0m:         pixel_AUROC █▁▃▆\n",
      "\u001b[34m\u001b[1mwandb\u001b[0m:       pixel_F1Score ▁▂▆█\n",
      "\u001b[34m\u001b[1mwandb\u001b[0m:    train_loss_epoch █▅▂▁\n",
      "\u001b[34m\u001b[1mwandb\u001b[0m: trainer/global_step ▁▁▃▃▆▆██\n",
      "\u001b[34m\u001b[1mwandb\u001b[0m: \n",
      "\u001b[34m\u001b[1mwandb\u001b[0m: Run summary:\n",
      "\u001b[34m\u001b[1mwandb\u001b[0m:               epoch 3\n",
      "\u001b[34m\u001b[1mwandb\u001b[0m:         image_AUROC 1.0\n",
      "\u001b[34m\u001b[1mwandb\u001b[0m:       image_F1Score 1.0\n",
      "\u001b[34m\u001b[1mwandb\u001b[0m:         pixel_AUROC 0.40607\n",
      "\u001b[34m\u001b[1mwandb\u001b[0m:       pixel_F1Score 0.04239\n",
      "\u001b[34m\u001b[1mwandb\u001b[0m:    train_loss_epoch 27.43848\n",
      "\u001b[34m\u001b[1mwandb\u001b[0m: trainer/global_step 3\n",
      "\u001b[34m\u001b[1mwandb\u001b[0m: \n",
      "\u001b[34m\u001b[1mwandb\u001b[0m: Synced \u001b[33mleafy-sweep-1\u001b[0m: \u001b[34m\u001b[4mhttps://wandb.ai/anomalib/stfpm_hazelnut/runs/sht9w2kj\u001b[0m\n",
      "\u001b[34m\u001b[1mwandb\u001b[0m: Synced 3 W&B file(s), 0 media file(s), 0 artifact file(s) and 0 other file(s)\n",
      "\u001b[34m\u001b[1mwandb\u001b[0m: Find logs at: \u001b[35m\u001b[1m./wandb/run-20230202_061839-sht9w2kj/logs\u001b[0m\n"
     ]
    }
   ],
   "source": [
    "!python ./tools/hpo/sweep.py --model stfpm --model_config model_config.yaml --sweep_config sweep_config.yaml"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "OhS88SuxFFzi"
   },
   "source": [
    "While only a few parameters were shown in this example, you can call HPO on any of the parameters defined in the `model` and `dataset` section of the model configuration file."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "FX57TV8gFirU"
   },
   "source": [
    "Congratulations 🎉 You have now successfully optimized a model on your dataset.\n",
    "\n",
    "To go into more detail, refer our documentation on [hyperparameter optimization](https://openvinotoolkit.github.io/anomalib/tutorials/hyperparameter_optimization.html)."
   ]
  }
 ],
 "metadata": {
  "colab": {
   "provenance": []
  },
  "kernelspec": {
   "display_name": "anomalib",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.13"
  },
  "vscode": {
   "interpreter": {
    "hash": "ae223df28f60859a2f400fae8b3a1034248e0a469f5599fd9a89c32908ed7a84"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}