notebooks/700_metrics/701a_aupimo.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# AUPIMO\n",
    "\n",
    "Basic usage of the metric AUPIMO (pronounced \"a-u-pee-mo\")."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "# What is AUPIMO?\n",
    "\n",
    "The `Area Under the Per-Image Overlap [curve]` (AUPIMO) is a metric of recall (higher is better) designed for visual anomaly detection.\n",
    "\n",
    "Inspired by the [ROC](https://en.wikipedia.org/wiki/Receiver_operating_characteristic) and [PRO](https://link.springer.com/article/10.1007/s11263-020-01400-4) curves, \n",
    "\n",
    "> AUPIMO is the area under a curve of True Positive Rate (TPR or _recall_) as a function of False Positive Rate (FPR) restricted to a fixed range. \n",
    "\n",
    "But:\n",
    "- the TPR (Y-axis) is *per-image* (1 image = 1 curve/score);\n",
    "- the FPR (X-axis) considers the (average of) **normal** images only; \n",
    "- the FPR (X-axis) is in log scale and its range is [1e-5, 1e-4]\\* (harder detection task!).\n",
    "\n",
    "\\* The score (the area under the curve) is normalized to be in [0, 1].\n",
    "\n",
    "AUPIMO can be interpreted as\n",
    "\n",
    "> average segmentation recall in an image given that the model (nearly) does not yield false positives in normal images.\n",
    "\n",
    "References in the last cell.\n",
    "\n",
    "![AUROC vs. AUPRO vs. AUPIMO](./roc_pro_pimo.svg)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Setup"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Install `anomalib` using `pip`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# TODO(jpcbertoldo): replace by `pip install anomalib` when AUPIMO is released  # noqa: TD003\n",
    "%pip install ../.."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Change the directory to have access to the datasets."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "from pathlib import Path\n",
    "\n",
    "# NOTE: Provide the path to the dataset root directory.\n",
    "#   If the datasets is not downloaded, it will be downloaded\n",
    "#   to this directory.\n",
    "dataset_root = Path.cwd().parent.parent / \"datasets\" / \"MVTec\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Imports"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import torch\n",
    "from matplotlib import pyplot as plt\n",
    "from matplotlib.ticker import MaxNLocator, PercentFormatter\n",
    "from scipy import stats\n",
    "\n",
    "from anomalib import TaskType\n",
    "from anomalib.data import MVTec\n",
    "from anomalib.engine import Engine\n",
    "from anomalib.metrics import AUPIMO\n",
    "from anomalib.models import Padim"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Data Module\n",
    "\n",
    "We will use dataset Leather from MVTec AD. \n",
    "\n",
    "> See the notebooks below for more details on datamodules. \n",
    "> [github.com/openvinotoolkit/anomalib/tree/main/notebooks/100_datamodules](https://github.com/openvinotoolkit/anomalib/tree/main/notebooks/100_datamodules)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "task = TaskType.SEGMENTATION\n",
    "datamodule = MVTec(\n",
    "    root=dataset_root,\n",
    "    category=\"leather\",\n",
    "    image_size=256,\n",
    "    train_batch_size=32,\n",
    "    eval_batch_size=32,\n",
    "    num_workers=8,\n",
    "    task=task,\n",
    ")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Model\n",
    "\n",
    "We will use `PaDiM` (performance is not the best, but it is fast to train).\n",
    "\n",
    "> See the notebooks below for more details on models. \n",
    "> [github.com/openvinotoolkit/anomalib/tree/main/notebooks/200_models](https://github.com/openvinotoolkit/anomalib/tree/main/notebooks/200_models)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Instantiate the model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = Padim(\n",
    "    # only use one layer to speed it up\n",
    "    layers=[\"layer1\"],\n",
    "    n_features=64,\n",
    "    backbone=\"resnet18\",\n",
    "    pre_trained=True,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Average AUPIMO (Basic)\n",
    "\n",
    "The easiest way to use AUPIMO is via the collection of pixel metrics in the engine.\n",
    "\n",
    "By default, the average AUPIMO is calculated."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "engine = Engine(\n",
    "    pixel_metrics=\"AUPIMO\",  # others can be added\n",
    "    accelerator=\"auto\",  # \\<\"cpu\", \"gpu\", \"tpu\", \"ipu\", \"hpu\", \"auto\">,\n",
    "    devices=1,\n",
    "    logger=False,\n",
    ")\n",
    "engine.fit(datamodule=datamodule, model=model)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "F1Score class exists for backwards compatibility. It will be removed in v1.1. Please use BinaryF1Score from torchmetrics instead\n",
      "Metric `AUPIMO` will save all targets and predictions in buffer. For large datasets this may lead to large memory footprint.\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "880e325e4e4842b2b679340ca8007849",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Testing: |          | 0/? [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">┏━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┓\n",
       "┃<span style=\"font-weight: bold\">        Test metric        </span>┃<span style=\"font-weight: bold\">       DataLoader 0        </span>┃\n",
       "┡━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━┩\n",
       "│<span style=\"color: #008080; text-decoration-color: #008080\">        image_AUROC        </span>│<span style=\"color: #800080; text-decoration-color: #800080\">    0.9887908697128296     </span>│\n",
       "│<span style=\"color: #008080; text-decoration-color: #008080\">       image_F1Score       </span>│<span style=\"color: #800080; text-decoration-color: #800080\">    0.9726775884628296     </span>│\n",
       "│<span style=\"color: #008080; text-decoration-color: #008080\">       pixel_AUPIMO        </span>│<span style=\"color: #800080; text-decoration-color: #800080\">    0.7428419829089654     </span>│\n",
       "└───────────────────────────┴───────────────────────────┘\n",
       "</pre>\n"
      ],
      "text/plain": [
       "┏━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┓\n",
       "┃\u001b[1m \u001b[0m\u001b[1m       Test metric       \u001b[0m\u001b[1m \u001b[0m┃\u001b[1m \u001b[0m\u001b[1m      DataLoader 0       \u001b[0m\u001b[1m \u001b[0m┃\n",
       "┡━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━┩\n",
       "│\u001b[36m \u001b[0m\u001b[36m       image_AUROC       \u001b[0m\u001b[36m \u001b[0m│\u001b[35m \u001b[0m\u001b[35m   0.9887908697128296    \u001b[0m\u001b[35m \u001b[0m│\n",
       "│\u001b[36m \u001b[0m\u001b[36m      image_F1Score      \u001b[0m\u001b[36m \u001b[0m│\u001b[35m \u001b[0m\u001b[35m   0.9726775884628296    \u001b[0m\u001b[35m \u001b[0m│\n",
       "│\u001b[36m \u001b[0m\u001b[36m      pixel_AUPIMO       \u001b[0m\u001b[36m \u001b[0m│\u001b[35m \u001b[0m\u001b[35m   0.7428419829089654    \u001b[0m\u001b[35m \u001b[0m│\n",
       "└───────────────────────────┴───────────────────────────┘\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "[{'pixel_AUPIMO': 0.7428419829089654,\n",
       "  'image_AUROC': 0.9887908697128296,\n",
       "  'image_F1Score': 0.9726775884628296}]"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# will output the AUPIMO score on the test set\n",
    "engine.test(datamodule=datamodule, model=model)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Individual AUPIMO Scores (Detailed)\n",
    "\n",
    "AUPIMO assigns one recall score per anomalous image in the dataset.\n",
    "\n",
    "It is possible to access each of the individual AUPIMO scores and look at the distribution."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Collect the predictions and the ground truth."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "ckpt_path is not provided. Model weights will not be loaded.\n",
      "F1Score class exists for backwards compatibility. It will be removed in v1.1. Please use BinaryF1Score from torchmetrics instead\n",
      "Metric `AUPIMO` will save all targets and predictions in buffer. For large datasets this may lead to large memory footprint.\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "e8116b80da39406e966c2099ecb2fdb1",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Predicting: |          | 0/? [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "predictions = engine.predict(dataloaders=datamodule.test_dataloader(), model=model, return_predictions=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Compute the AUPIMO scores."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Metric `AUPIMO` will save all targets and predictions in buffer. For large datasets this may lead to large memory footprint.\n"
     ]
    }
   ],
   "source": [
    "aupimo = AUPIMO(\n",
    "    # with `False` all the values are returned in a dataclass\n",
    "    return_average=False,\n",
    ")\n",
    "\n",
    "for batch in predictions:\n",
    "    anomaly_maps = batch[\"anomaly_maps\"].squeeze(dim=1)\n",
    "    masks = batch[\"mask\"]\n",
    "    aupimo.update(anomaly_maps=anomaly_maps, masks=masks)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "# `pimo_result` has the PIMO curves of each image\n",
    "# `aupimo_result` has the AUPIMO values\n",
    "#     i.e. their Area Under the Curve (AUC)\n",
    "pimo_result, aupimo_result = aupimo.compute()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Check the outputs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([1.0000, 0.9144, 0.4944, 0.2837, 0.2784, 0.8687, 1.0000, 0.7463, 0.2899,\n",
      "        0.8998, 1.0000, 0.9147, 0.6389, 0.9422, 0.9582, 0.9396, 0.9890, 0.5130,\n",
      "        0.9698, 0.9237, 0.5732, 0.4620, 0.9995, 0.9078, 0.5873, 1.0000, 1.0000,\n",
      "        1.0000, 0.3785, 0.6764, 0.4217, 0.9299, 0.7756, 0.4339, 0.8334, 0.9297,\n",
      "        0.9992, 0.5584, 0.9937, 0.7811, 0.4986, 0.7630, 0.5361, 0.7157, 0.1689,\n",
      "        0.3086, 0.3604, 0.2423, 0.2880, 0.6404, 0.5570, 0.3274, 0.7749, 0.6740,\n",
      "        0.5516, 1.0000, 0.2399, 0.9721, 0.5346, 0.4709, 1.0000, 0.9732, 0.8470,\n",
      "        0.8863, 0.0596, 0.0000, 0.5244, 0.0000, 1.0000, 1.0000, 1.0000, 0.0088,\n",
      "        0.9706, 1.0000,    nan,    nan,    nan,    nan,    nan,    nan,    nan,\n",
      "           nan,    nan,    nan,    nan,    nan,    nan,    nan,    nan,    nan,\n",
      "           nan,    nan,    nan,    nan,    nan,    nan,    nan,    nan,    nan,\n",
      "           nan,    nan,    nan,    nan,    nan,    nan,    nan, 0.9895, 0.8531,\n",
      "        0.9985, 0.9470, 1.0000, 1.0000, 0.9918, 0.9792, 1.0000, 1.0000, 0.8824,\n",
      "        1.0000, 0.9996, 1.0000, 1.0000, 1.0000, 1.0000, 1.0000],\n",
      "       dtype=torch.float64)\n"
     ]
    }
   ],
   "source": [
    "# the `nan`s are the normal images; they do not\n",
    "# have a score because recall is not defined for them\n",
    "print(aupimo_result.aupimos)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Statistics\n",
    "\n",
    "Compute statistics of the AUPIMO scores."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "MEAN\n",
      "aupimo_result.aupimos[~isnan].mean().item()=0.7428419829089654\n",
      "OTHER STATISTICS\n",
      "DescribeResult(nobs=92, minmax=(0.0, 1.0), mean=0.7428419829089654, variance=0.08757789538421837, skewness=-0.9285672286850366, kurtosis=-0.3299234749959594)\n"
     ]
    }
   ],
   "source": [
    "# ignore removing the `nan`s\n",
    "isnan = torch.isnan(aupimo_result.aupimos)\n",
    "\n",
    "print(f\"MEAN\\n{aupimo_result.aupimos[~isnan].mean().item()=}\")\n",
    "print(f\"OTHER STATISTICS\\n{stats.describe(aupimo_result.aupimos[~isnan])}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Plot\n",
    "\n",
    "Visualize the distribution of the AUPIMO scores."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAkYAAAHHCAYAAABa2ZeMAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjkuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8hTgPZAAAACXBIWXMAAA9hAAAPYQGoP6dpAABHeUlEQVR4nO3deVyVZeL+8eugrCKgiKAJgktuqZVLWo0raoqW6aiZlFtuqVk21tdpUcuyZbLFXBvDmkDLMkctNXMrjVwwTVNJc8FccAVUEBHu3x8ez29OuHAQzgH8vF8vXuO5n+c8z3W4G7t6tmMxxhgBAABAbq4OAAAAUFRQjAAAAKwoRgAAAFYUIwAAACuKEQAAgBXFCAAAwIpiBAAAYEUxAgAAsKIYAQAAWFGMABQba9askcVi0Zo1a2xj/fr1U3h4uFP2Hx4ern79+tlez5kzRxaLRZs3b3bK/lu1aqVWrVo5ZV/ArYpiBBQB06ZNk8Vi0T333HPV5QcOHJDFYtG//vWvqy7/17/+JYvFogMHDtjGWrVqJYvFYvspX768mjRpoo8//lg5OTm29fr16ydfX1+77V15b82aNa+6vxUrVti2++WXX+Za/ttvvyk6Olq33XabPD09VblyZfXp00e//fbbjX4VTrFz506NHz/e7vdVVBTlbMCtoLSrAwCQYmNjFR4ero0bN2rv3r2qUaNGgWy3SpUqmjRpkiTpxIkT+vTTTzVw4ED9/vvveuONN677Xi8vL+3du1cbN25U06ZNc+X18vLShQsXcr1vwYIF6t27t8qXL6+BAwcqIiJCBw4c0OzZs/Xll19q3rx5evjhhwvk80nSRx99ZFf08mLnzp2aMGGCWrVq5dDRpsTERLm5Fe5/T14v23fffVeo+wbAESPA5fbv36+ffvpJkydPVlBQkGJjYwts2/7+/oqOjlZ0dLSeeeYZrV+/XlWqVNGHH36orKys6763evXqqlWrlubOnWs3fuHCBX399deKiorK9Z4//vhDjz32mKpVq6Zff/1VEydO1MCBA/Xqq6/q119/VbVq1fTYY49p3759BfYZ3d3d5enpWWDb+ytjjDIyMiRJnp6ecnd3L7R93YiHh4c8PDxctn/gVkAxAlwsNjZW5cqVU1RUlP7+978XaDH6Kx8fHzVr1kznz5/XiRMnbrh+79699fnnn9sdkVm8eLHS09PVs2fPXOu//fbbSk9P16xZsxQUFGS3rEKFCpo5c6bOnz+vt95664b7/vPPP9W1a1eVKVNGFStW1DPPPKPMzMxc613tGqN58+apUaNGKlu2rPz8/FS/fn29//77ki5fF9SjRw9JUuvWrW2nBK9ctxQeHq7OnTtr+fLlaty4sby9vTVz5kzbsv+9xuiK9PR0DRkyRIGBgfLz89Pjjz+uM2fO2K1jsVg0fvz4XO/9323eKNvVrjE6fvy4Bg4cqODgYHl5ealhw4b65JNP7Nb531Oxs2bNUvXq1eXp6akmTZpo06ZNuTIBtzJOpQEuFhsbq27dusnDw0O9e/fW9OnTtWnTJjVp0qRQ9rdv3z6VKlVKAQEBN1z30Ucf1fjx47VmzRq1adNGkhQXF6e2bduqYsWKudZfvHixwsPD9be//e2q22vRooXCw8P1zTffXHe/GRkZatu2rZKSkvTUU0+pcuXK+s9//qNVq1bdMPOKFSvUu3dvtW3bVm+++aYkadeuXVq/fr1GjRqlFi1a6KmnntIHH3ygf/7zn6pTp44k2f5XunzKrHfv3hoyZIgGDRqkWrVqXXefI0aMUEBAgMaPH6/ExERNnz5dBw8etF0snld5yfa/MjIy1KpVK+3du1cjRoxQRESE5s+fr379+iklJUWjRo2yWz8uLk5nz57VkCFDZLFY9NZbb6lbt27at2+fS4+EAUUJxQhwoYSEBO3evVtTpkyRJN1///2qUqWKYmNjC6QYZWdn6+TJk5KkkydPavr06dqyZYu6dOkiHx+fG76/Zs2aaty4seLi4tSmTRulpKTo22+/1UcffZRr3dTUVB05ckQPPfTQdbfZoEEDLVq0SGfPnlXZsmWvus6sWbP0+++/64svvrAdQRk0aJAaNmx4w8zffPON/Pz8tHz5cpUqVSrX8mrVqulvf/ubPvjgA7Vr1+6qd3nt3btXy5YtU4cOHW64P+nyKa6VK1faykXVqlX13HPPafHixXrwwQfztI28Zvtfs2bN0q5du/TZZ5+pT58+kqShQ4eqZcuWevHFFzVgwAC733FSUpL27NmjcuXKSZJq1aqlhx56SMuXL1fnzp3znBMoyTiVBrhQbGysgoOD1bp1a0mXT7f06tVL8+bNU3Z29k1vf/fu3QoKClJQUJDq1KmjKVOmKCoqSh9//HGet/Hoo49qwYIFunjxor788kuVKlXqqhdPnz17VpKuWXauuLI8LS3tmut8++23qlSpkv7+97/bxnx8fDR48OAb5g0ICND58+e1YsWKG657LREREXkuRZI0ePBguyMuw4YNU+nSpfXtt9/mO0NefPvttwoJCVHv3r1tY+7u7nrqqad07tw5rV271m79Xr162UqRJNuRvYK85gso7ihGgItkZ2dr3rx5at26tfbv36+9e/dq7969uueee5ScnKyVK1c6vM2/nrYJDw/XihUr9P3332vdunU6duyYlixZogoVKuR5m4888ohSU1O1dOlSxcbGqnPnzlctP1fGrhSka8lLgTp48KBq1KiR6/Pc6JSWJD355JO6/fbb1bFjR1WpUkUDBgzQsmXLbvi+/xUREeHQ+n99rIGvr68qVapU6LfcHzx4UDVr1sx1p9yVU28HDx60Gw8LC7N7faUk/fV6KOBWxqk0wEVWrVqlo0ePat68eZo3b16u5bGxsWrfvr2ky7fOS7LdHfVX6enpdutdUaZMGUVGRt5UzkqVKqlVq1Z65513tH79en311VdXXc/f31+VKlXSr7/+et3t/frrr7rtttvk5+d3U7mupWLFitq6dauWL1+upUuXaunSpYqJidHjjz+e66Lka/H29i6UbFdTEEcG8+pqpxaly3feAbiMI0aAi8TGxqpixYqaP39+rp/evXvr66+/thWhoKAg+fj4KDEx8arbSkxMlI+Pj0NHghzx6KOP6scff5Sfn586dep0zfU6d+6s/fv3a926dVdd/uOPP+rAgQM3vJ6latWq+uOPP3L9C/tan/+vPDw81KVLF02bNk1//PGHhgwZok8//VR79+6VlPvI2s3as2eP3etz587p6NGjdnfLlStXTikpKXbrXbx4UUePHrUbcyRb1apVtWfPnlzPcdq9e7dtOQDHUIwAF8jIyNCCBQvUuXNn/f3vf8/1M2LECJ09e1aLFi2SdPm/9Nu3b6/FixcrKSnJbltJSUlavHix2rdvf80jAjfr73//u8aNG6dp06Zd9zk6Y8aMkbe3t4YMGaJTp07ZLTt9+rSGDh0qHx8fjRkz5rr769Spk44cOWL3VO0rjwG4kb/u183NTQ0aNJAk2+3+ZcqUkaRcRSW/Zs2aZfdcqOnTp+vSpUvq2LGjbax69er64Ycfcr3vr0eMHMnWqVMnHTt2TJ9//rlt7NKlS5oyZYp8fX3VsmXL/Hwc4JbGqTTABa7clXWtO5aaNWtme9hjr169JEmvv/66mjVrprvvvluDBw9WeHi4Dhw4oFmzZslisej1118vtLz+/v5XfQbPX9WsWVOffPKJ+vTpo/r16+d68vXJkyc1d+5cVa9e/brbGTRokD788EM9/vjjSkhIUKVKlfSf//wnT3fSPfHEEzp9+rTatGmjKlWq6ODBg5oyZYruvPNO27U3d955p0qVKqU333xTqamp8vT0VJs2ba76CIK8uHjxotq2bauePXsqMTFR06ZN0/333283v0888YSGDh2q7t27q127dtq2bZuWL1+e6yifI9kGDx6smTNnql+/fkpISFB4eLi+/PJLrV+/Xu+9994NL4QHcBUGgNN16dLFeHl5mfPnz19znX79+hl3d3dz8uRJ29iuXbtMr169TMWKFU3p0qVNxYoVzSOPPGJ27dqV6/0tW7Y09erVu2GWvn37mjJlyjj83tWrVxtJZv78+bmW/frrr6Z3796mUqVKxt3d3YSEhJjevXub7du33zDPFQcPHjQPPvig8fHxMRUqVDCjRo0yy5YtM5LM6tWr7fJXrVrV9vrLL7807du3NxUrVjQeHh4mLCzMDBkyxBw9etRu+x999JGpVq2aKVWqlN02q1ataqKioq6aqWrVqqZv37621zExMUaSWbt2rRk8eLApV66c8fX1NX369DGnTp2ye292drZ5/vnnTYUKFYyPj4/p0KGD2bt3b65tXi9by5YtTcuWLe3WTU5ONv379zcVKlQwHh4epn79+iYmJsZunf379xtJ5u233871mSSZcePGXfXzArciizFcdQcAACBxjREAAIANxQgAAMCKYgQAAGBFMQIAALCiGAEAAFhRjAAAAKxK/AMec3JydOTIEZUtW7bAvwYAAAAUDmOMzp49q8qVK+f6ouTCVOKL0ZEjRxQaGurqGAAAIB8OHTqkKlWqOG1/Jb4YXXkk/v79+1W+fHkXp7m1ZWVl6bvvvlP79u3l7u7u6ji3NOaiaGE+ig7moug4ffq0IiIinP7VNiW+GF05fVa2bFn5+fm5OM2tLSsrSz4+PvLz8+MvHBdjLooW5qPoYC6KjitfzOzsy2C4+BoAAMCKYgQAAGBFMQIAALCiGAEAAFhRjAAAAKwoRgAAAFYUIwAAACuKEQAAgBXFCAAAwIpiBAAAYEUxAgAAsKIYAQAAWFGMAAAArChGAAAAVqVdHQAAABSspKQknTx50tUxbkpaWppL9ksxAgCgBElKSlKt2nV0ISPd1VFuipeXl0v2SzECAKAEOXnypC5kpCuw87NyDwx1dZx8s5zar6NL3nf6filGAACUQO6BofIMqeHqGPlmcjJdsl8uvgYAALCiGAEAAFhRjAAAAKwoRgAAAFYUIwAAACuKEQAAgBXFCAAAwIpiBAAAYEUxAgAAsKIYAQAAWFGMAAAArChGAAAAVhQjAAAAK4oRAACAFcUIAADAimIEAABgRTECAACwohgBAABYUYwAAACsKEYAAABWFCMAAAArihEAAIAVxQgAAMCKYgQAAGBFMQIAALCiGAEAAFhRjAAAAKwoRgAAAFYUIwAAACuKEQAAgBXFCAAAwIpiBAAAYFVkitEbb7whi8Wip59+2jZ24cIFDR8+XIGBgfL19VX37t2VnJzsupAAAKBEKxLFaNOmTZo5c6YaNGhgN/7MM89o8eLFmj9/vtauXasjR46oW7duLkoJAABKOpcXo3PnzqlPnz766KOPVK5cOdt4amqqZs+ercmTJ6tNmzZq1KiRYmJi9NNPP+nnn392YWIAAFBSlXZ1gOHDhysqKkqRkZGaOHGibTwhIUFZWVmKjIy0jdWuXVthYWGKj49Xs2bNrrq9zMxMZWZm2l6npaVJkrKyspSVlVVInwJ5ceX3zzy4HnNRtDAfRUdJmIucnBx5e3vLq7RFHqWMq+PkW46LGopLi9G8efO0ZcsWbdq0KdeyY8eOycPDQwEBAXbjwcHBOnbs2DW3OWnSJE2YMCHX+OrVq+Xj43PTmXHzVqxY4eoIsGIuihbmo+go7nMxd+5c65+yXZrjZqSnh+rRGOfv12XF6NChQxo1apRWrFghLy+vAtvu2LFjNXr0aNvrtLQ0hYaGqnXr1goMDCyw/cBxWVlZWrFihdq1ayd3d3dXx7mlMRdFC/NRdJSEudi2bZtatGih4EffkEdwNVfHybeco4dcsl+XFaOEhAQdP35cd999t20sOztbP/zwgz788EMtX75cFy9eVEpKit1Ro+TkZIWEhFxzu56envL09Mw17u7uXmz/IS9pmIuig7koWpiPoqM4z4Wbm5syMjJ04ZKRyba4Ok6+mUuu2a/LilHbtm21fft2u7H+/furdu3aev755xUaGip3d3etXLlS3bt3lyQlJiYqKSlJzZs3d0VkAABQwrmsGJUtW1Z33HGH3ViZMmUUGBhoGx84cKBGjx6t8uXLy8/PTyNHjlTz5s2veeE1AADAzXD5XWnX8+6778rNzU3du3dXZmamOnTooGnTprk6FgAAKKGKVDFas2aN3WsvLy9NnTpVU6dOdU0gAABwS3H5Ax4BAACKCooRAACAFcUIAADAimIEAABgRTECAACwohgBAABYUYwAAACsKEYAAABWFCMAAAArihEAAIAVxQgAAMCKYgQAAGBFMQIAALCiGAEAAFhRjAAAAKwoRgAAAFYUIwAAACuKEQAAgBXFCAAAwIpiBAAAYEUxAgAAsKIYAQAAWFGMAAAArChGAAAAVhQjAAAAK4oRAACAFcUIAADAimIEAABgRTECAACwohgBAABYUYwAAACsSufnTVlZWTp27JjS09MVFBSk8uXLF3QuAAAAp8vzEaOzZ89q+vTpatmypfz8/BQeHq46deooKChIVatW1aBBg7Rp06bCzAoAAFCo8lSMJk+erPDwcMXExCgyMlILFy7U1q1b9fvvvys+Pl7jxo3TpUuX1L59ez3wwAPas2dPYecGAAAocHk6lbZp0yb98MMPqlev3lWXN23aVAMGDNCMGTMUExOjH3/8UTVr1izQoAAAAIUtT8Vo7ty5edqYp6enhg4delOBAAAAXOWm70pLS0vTwoULtWvXroLIAwAA4DIOF6OePXvqww8/lCRlZGSocePG6tmzpxo0aKCvvvqqwAMCAAA4i8PF6IcfftDf/vY3SdLXX38tY4xSUlL0wQcfaOLEiQUeEAAAwFkcLkapqam25xYtW7ZM3bt3l4+Pj6KiorgbDQAAFGsOF6PQ0FDFx8fr/PnzWrZsmdq3by9JOnPmjLy8vAo8IAAAgLM4/OTrp59+Wn369JGvr6/CwsLUqlUrSZdPsdWvX7+g8wEAADiNw8XoySefVNOmTXXo0CG1a9dObm6XDzpVq1aNa4wAAECxlq/vSmvcuLEaNGig/fv3q3r16ipdurSioqIKOhsAAIBTOXyNUXp6ugYOHCgfHx/Vq1dPSUlJkqSRI0fqjTfeKPCAAAAAzuJwMRo7dqy2bdumNWvW2F1sHRkZqc8//7xAwwEAADiTw6fSFi5cqM8//1zNmjWTxWKxjderV09//PFHgYYDAABwJoePGJ04cUIVK1bMNX7+/Hm7ogQAAFDcOFyMGjdurG+++cb2+koZ+ve//63mzZsXXDIAAAAnc/hU2uuvv66OHTtq586dunTpkt5//33t3LlTP/30k9auXVsYGQEAAJzC4SNG999/v7Zu3apLly6pfv36+u6771SxYkXFx8erUaNGhZERAADAKfL1HKPq1avro48+KugsAAAALuVwMUpLS7vquMVikaenpzw8PG46FAAAgCs4XIwCAgKue/dZlSpV1K9fP40bN872dSEAAADFgcPFaM6cOXrhhRfUr18/NW3aVJK0ceNGffLJJ3rxxRd14sQJ/etf/5Knp6f++c9/FnhgAACAwuJwMfrkk0/0zjvvqGfPnraxLl26qH79+po5c6ZWrlypsLAwvfbaaxQjAABQrDh8ruunn37SXXfdlWv8rrvuUnx8vKTLd65d+Q41AACA4sLhYhQaGqrZs2fnGp89e7ZCQ0MlSadOnVK5cuVuPh0AAIATOXwq7V//+pd69OihpUuXqkmTJpKkzZs3a/fu3fryyy8lSZs2bVKvXr0KNikAAEAhc7gYPfjgg0pMTNTMmTOVmJgoSerYsaMWLlyo8PBwSdKwYcMKNCQAAIAz5OsBj+Hh4Zo0aVJBZwEAAHCpfBUjSUpPT1dSUpIuXrxoN96gQYObDgUAAOAKDhejEydOqH///lq6dOlVl2dnZ990KAAAAFdw+K60p59+WikpKdqwYYO8vb21bNkyffLJJ6pZs6YWLVpUGBkBAACcwuEjRqtWrdJ///tfNW7cWG5ubqpataratWsnPz8/TZo0SVFRUYWREwAAoNA5fMTo/PnzqlixoiSpXLlyOnHihCSpfv362rJlS8GmAwAAcCKHi1GtWrVst+k3bNhQM2fO1OHDhzVjxgxVqlSpwAMCAAA4i8PFaNSoUTp69Kgkady4cVq6dKnCwsL0wQcf6PXXX3doW9OnT1eDBg3k5+cnPz8/NW/e3O6i7gsXLmj48OEKDAyUr6+vunfvruTkZEcjAwAA5InD1xhFR0fb/tyoUSMdPHhQu3fvVlhYmCpUqODQtqpUqaI33nhDNWvWlDFGn3zyiR566CH98ssvqlevnp555hl98803mj9/vvz9/TVixAh169ZN69evdzQ2AADADeX7OUZX+Pj46O67787Xe7t06WL3+rXXXtP06dP1888/q0qVKpo9e7bi4uLUpk0bSVJMTIzq1Kmjn3/+Wc2aNbvZ6AAAAHYcLkbGGH355ZdavXq1jh8/rpycHLvlCxYsyFeQ7OxszZ8/X+fPn1fz5s2VkJCgrKwsRUZG2tapXbu2wsLCFB8ff81ilJmZqczMTNvrtLQ0SVJWVpaysrLylQ0F48rvn3lwPeaiaGE+io6SMBc5OTny9vaWV2mLPEoZV8fJt5ybPnSTPw7v9umnn9bMmTPVunVrBQcHy2Kx3FSA7du3q3nz5rpw4YJ8fX319ddfq27dutq6das8PDwUEBBgt35wcLCOHTt2ze1NmjRJEyZMyDW+evVq+fj43FRWFIwVK1a4OgKsmIuihfkoOor7XMydO9f6p+L70OX09FA9GuP8/TpcjP7zn/9owYIF6tSpU4EEqFWrlrZu3arU1FR9+eWX6tu3r9auXZvv7Y0dO1ajR4+2vU5LS1NoaKhat26twMDAgoiMfMrKytKKFSvUrl07ubu7uzrOLY25KFqYj6KjJMzFtm3b1KJFCwU/+oY8gqu5Ok6+5Rw95JL9OlyM/P39Va1awf2iPTw8VKNGDUmXL+betGmT3n//ffXq1UsXL15USkqK3VGj5ORkhYSEXHN7np6e8vT0zDXu7u5ebP8hL2mYi6KDuShamI+iozjPhZubmzIyMnThkpHJvrmzOq5kLrlmvw7frj9+/HhNmDBBGRkZhZFHOTk5yszMVKNGjeTu7q6VK1faliUmJiopKUnNmzcvlH0DAIBbm8NHjHr27Km5c+eqYsWKCg8Pz9WoHXn69dixY9WxY0eFhYXp7NmziouL05o1a7R8+XL5+/tr4MCBGj16tMqXLy8/Pz+NHDlSzZs35440AABQKBwuRn379lVCQoKio6Nv+uLr48eP6/HHH9fRo0fl7++vBg0aaPny5WrXrp0k6d1335Wbm5u6d++uzMxMdejQQdOmTcv3/gAAAK7H4WL0zTffaPny5br//vtveuezZ8++7nIvLy9NnTpVU6dOvel9AQAA3IjD1xiFhobKz8+vMLIAAAC4lMPF6J133tFzzz2nAwcOFEIcAAAA18nXd6Wlp6erevXq8vHxyXXx9enTpwssHAAAgDM5XIzee++9QogBAADgevm6Kw0AAKAkylMxSktLs11wfeVLWa+FC7MBAEBxladiVK5cOR09elQVK1ZUQEDAVZ9dZIyRxWJRdnbx/cI6AABwa8tTMVq1apXKly8v6fK31AMAAJREeSpGLVu2vOqfAQAAShKHn2MEAABQUlGMAAAArChGAAAAVnkqRosWLVJWVlZhZwEAAHCpPBWjhx9+WCkpKZKkUqVK6fjx44WZCQAAwCXyVIyCgoL0888/S/r/zysCAAAoafJ0u/7QoUP10EMPyWKxyGKxKCQk5Jrr8oBHAABQXOWpGI0fP16PPPKI9u7dqwcffFAxMTEKCAgo5GgAAADOlecvka1du7Zq166tcePGqUePHvLx8SnMXAAAAE6X52J0xbhx4yRJJ06cUGJioiSpVq1aCgoKKthkAAAATubwc4zS09M1YMAAVa5cWS1atFCLFi1UuXJlDRw4UOnp6YWREQAAwCkcLkbPPPOM1q5dq0WLFiklJUUpKSn673//q7Vr1+rZZ58tjIwAAABO4fCptK+++kpffvmlWrVqZRvr1KmTvL291bNnT02fPr0g8wEAADhNvk6lBQcH5xqvWLEip9IAAECx5nAxat68ucaNG6cLFy7YxjIyMjRhwgQ1b968QMMBAAA4k8On0t5//3116NBBVapUUcOGDSVJ27Ztk5eXl5YvX17gAQEAAJzF4WJ0xx13aM+ePYqNjdXu3bslSb1791afPn3k7e1d4AEBAACcxeFiJEk+Pj4aNGhQQWcBAABwKYevMQIAACipKEYAAABWFCMAAAArh4pRdna2fvjhB6WkpBRSHAAAANdxqBiVKlVK7du315kzZworDwAAgMs4fCrtjjvu0L59+wojCwAAgEs5XIwmTpyof/zjH1qyZImOHj2qtLQ0ux8AAIDiyuHnGHXq1EmS9OCDD8pisdjGjTGyWCzKzs4uuHQAAABO5HAxWr16dWHkAAAAcDmHi1HLli0LIwcAAIDL5es5Rj/++KOio6N177336vDhw5Kk//znP1q3bl2BhgMAAHAmh4vRV199pQ4dOsjb21tbtmxRZmamJCk1NVWvv/56gQcEAABwlnzdlTZjxgx99NFHcnd3t43fd9992rJlS4GGAwAAcCaHi1FiYqJatGiRa9zf358nYgMAgGLN4WIUEhKivXv35hpft26dqlWrViChAAAAXMHhYjRo0CCNGjVKGzZskMVi0ZEjRxQbG6t//OMfGjZsWGFkBAAAcAqHb9f/v//7P+Xk5Kht27ZKT09XixYt5OnpqX/84x8aOXJkYWQEAABwCoeLkcVi0QsvvKAxY8Zo7969OnfunOrWrStfX9/CyAcAAOA0DhejKzw8PFS2bFmVLVuWUgQAAEoEh68xunTpkl566SX5+/srPDxc4eHh8vf314svvqisrKzCyAgAAOAUDh8xGjlypBYsWKC33npLzZs3lyTFx8dr/PjxOnXqlKZPn17gIQEAAJzB4WIUFxenefPmqWPHjraxBg0aKDQ0VL1796YYAQCAYsvhU2menp4KDw/PNR4RESEPD4+CyAQAAOASDhejESNG6NVXX7V9R5okZWZm6rXXXtOIESMKNBwAAIAz5elUWrdu3exef//996pSpYoaNmwoSdq2bZsuXryotm3bFnxCAAAAJ8lTMfL397d73b17d7vXoaGhBZcIAADARfJUjGJiYgo7BwAAgMs5fI0RAABASeXw7fqnTp3Syy+/rNWrV+v48ePKycmxW3769OkCCwcAAOBMDhejxx57THv37tXAgQMVHBwsi8VSGLkAAACczuFi9OOPP2rdunW2O9IAAABKCoevMapdu7YyMjIKIwsAAIBLOVyMpk2bphdeeEFr167VqVOnlJaWZvcDAABQXDl8Ki0gIEBpaWlq06aN3bgxRhaLRdnZ2QUWDgAAwJkcLkZ9+vSRu7u74uLiuPgaAACUKA4Xox07duiXX35RrVq1CiMPAACAyzh8jVHjxo116NChwsgCAADgUg4fMRo5cqRGjRqlMWPGqH79+nJ3d7db3qBBgwILBwAA4EwOF6NevXpJkgYMGGAbs1gsXHwNAACKPYeL0f79+wsjBwAAgMs5fI1R1apVr/vjiEmTJqlJkyYqW7asKlasqK5duyoxMdFunQsXLmj48OEKDAyUr6+vunfvruTkZEdjAwAA3JDDR4w+/fTT6y5//PHH87yttWvXavjw4WrSpIkuXbqkf/7zn2rfvr127typMmXKSJKeeeYZffPNN5o/f778/f01YsQIdevWTevXr3c0OgAAwHU5XIxGjRpl9zorK0vp6eny8PCQj4+PQ8Vo2bJldq/nzJmjihUrKiEhQS1atFBqaqpmz56tuLg42wMlY2JiVKdOHf38889q1qyZo/EBAACuyeFidObMmVxje/bs0bBhwzRmzJibCpOamipJKl++vCQpISFBWVlZioyMtK1Tu3ZthYWFKT4+/qrFKDMzU5mZmbbXV76mJCsrS1lZWTeVDzfnyu+feXA95qJoYT6KjpIwFzk5OfL29pZXaYs8ShlXx8m3HIcbSsGwGGMK5Le2efNmRUdHa/fu3fl6f05Ojh588EGlpKRo3bp1kqS4uDj179/fruhIUtOmTdW6dWu9+eabubYzfvx4TZgwIdd4XFycfHx88pUNAAA4V3p6uh599FGlpqbKz8/PafstsD5WunRpHTlyJN/vHz58uHbs2GErRfk1duxYjR492vY6LS1NoaGhat26tQIDA29q27g5WVlZWrFihdq1a5fr+VdwLuaiaGE+io6SMBfbtm1TixYtFPzoG/IIrubqOPmWc9Q1D5N2uBgtWrTI7rUxRkePHtWHH36o++67L18hRowYoSVLluiHH35QlSpVbOMhISG6ePGiUlJSFBAQYBtPTk5WSEjIVbfl6ekpT0/PXOPu7u7F9h/ykoa5KDqYi6KF+Sg6ivNcuLm5KSMjQxcuGZns4vt9puaSa/brcDHq2rWr3WuLxaKgoCC1adNG77zzjkPbMsZo5MiR+vrrr7VmzRpFRETYLW/UqJHc3d21cuVKde/eXZKUmJiopKQkNW/e3NHoAAAA1+VwMcrJySmwnQ8fPlxxcXH673//q7Jly+rYsWOSJH9/f3l7e8vf318DBw7U6NGjVb58efn5+WnkyJFq3rw5d6QBAIAC56Jrvi+bPn26JKlVq1Z24zExMerXr58k6d1335Wbm5u6d++uzMxMdejQQdOmTXNyUgAAcCtwuBhlZ2drzpw5WrlypY4fP57rCNKqVavyvK283BDn5eWlqVOnaurUqY5GBQAAcEi+HvA4Z84cRUVF6Y477pDFUnwv7AIAAPhfDhejefPm6YsvvlCnTp0KIw8AAIDLOPwlsh4eHqpRo0ZhZAEAAHAph4vRs88+q/fffz9P1wcBAAAUJw6fSlu3bp1Wr16tpUuXql69erkegLVgwYICCwcAAOBMDhejgIAAPfzww4WRBQAAwKUcLkYxMTGFkQMAAMDlHL7GCAAAoKTKUzF64IEH9PPPP99wvbNnz+rNN9/kYYwAAKBYytOptB49eqh79+7y9/dXly5d1LhxY1WuXFleXl46c+aMdu7cqXXr1unbb79VVFSU3n777cLODQAAUODyVIwGDhyo6OhozZ8/X59//rlmzZql1NRUSZLFYlHdunXVoUMHbdq0SXXq1CnUwAAAAIUlzxdfe3p6Kjo6WtHR0ZKk1NRUZWRkKDAwMNct+wAAAMWRw3elXeHv7y9/f/+CzAIAAOBS3JUGAABgRTECAACwohgBAABYUYwAAACsHC5G1apV06lTp3KNp6SkqFq1agUSCgAAwBUcLkYHDhxQdnZ2rvHMzEwdPny4QEIBAAC4Qp5v11+0aJHtz8uXL7e7VT87O1srV65UeHh4gYYDAABwpjwXo65du0q6/KTrvn372i1zd3dXeHi43nnnnQINBwAA4Ex5LkY5OTmSpIiICG3atEkVKlQotFAAAACu4PCTr/fv318YOQAAAFwuX18JsnLlSq1cuVLHjx+3HUm64uOPPy6QYAAAAM7mcDGaMGGCXnnlFTVu3FiVKlWSxWIpjFwAAABO53AxmjFjhubMmaPHHnusMPIAAAC4jMPPMbp48aLuvffewsgCAADgUg4XoyeeeEJxcXGFkQUAAMClHD6VduHCBc2aNUvff/+9GjRoIHd3d7vlkydPLrBwAAAAzuRwMfr111915513SpJ27Nhht4wLsQEAQHHmcDFavXp1YeQAAABwOYevMQIAACipHD5i1Lp16+ueMlu1atVNBQIAAHAVh4vRleuLrsjKytLWrVu1Y8eOXF8uCwAAUJw4XIzefffdq46PHz9e586du+lAAAAArlJg1xhFR0fzPWkAAKBYK7BiFB8fLy8vr4LaHAAAgNM5fCqtW7dudq+NMTp69Kg2b96sl156qcCCAQAAOJvDxcjf39/utZubm2rVqqVXXnlF7du3L7BgAAAAzuZwMYqJiSmMHAAAAC7ncDG6IiEhQbt27ZIk1atXT3fddVeBhQIAAHAFh4vR8ePH9cgjj2jNmjUKCAiQJKWkpKh169aaN2+egoKCCjojAACAUzh8V9rIkSN19uxZ/fbbbzp9+rROnz6tHTt2KC0tTU899VRhZAQAAHAKh48YLVu2TN9//73q1KljG6tbt66mTp3KxdcAAKBYc/iIUU5Ojtzd3XONu7u7Kycnp0BCAQAAuILDxahNmzYaNWqUjhw5Yhs7fPiwnnnmGbVt27ZAwwEAADiTw8Xoww8/VFpamsLDw1W9enVVr15dERERSktL05QpUwojIwAAgFM4fI1RaGiotmzZou+//167d++WJNWpU0eRkZEFHg4AAMCZ8vUcI4vFonbt2qldu3YFnQcAAMBl8nwqbdWqVapbt67S0tJyLUtNTVW9evX0448/Fmg4AAAAZ8pzMXrvvfc0aNAg+fn55Vrm7++vIUOGaPLkyQUaDgAAwJnyXIy2bdumBx544JrL27dvr4SEhAIJBQAA4Ap5LkbJyclXfX7RFaVLl9aJEycKJBQAAIAr5LkY3XbbbdqxY8c1l//666+qVKlSgYQCAABwhTwXo06dOumll17ShQsXci3LyMjQuHHj1Llz5wINBwAA4Ex5vl3/xRdf1IIFC3T77bdrxIgRqlWrliRp9+7dmjp1qrKzs/XCCy8UWlAAAIDCludiFBwcrJ9++knDhg3T2LFjZYyRdPmZRh06dNDUqVMVHBxcaEEBAAAKm0MPeKxataq+/fZbnTlzRnv37pUxRjVr1lS5cuUKKx8AAIDT5OvJ1+XKlVOTJk0KOgsAAIBLOfwlsgAAACUVxQgAAMCKYgQAAGBFMQIAALCiGAEAAFhRjAAAAKwoRgAAAFYUIwAAACuXFqMffvhBXbp0UeXKlWWxWLRw4UK75cYYvfzyy6pUqZK8vb0VGRmpPXv2uCYsAAAo8VxajM6fP6+GDRtq6tSpV13+1ltv6YMPPtCMGTO0YcMGlSlTRh06dNCFCxecnBQAANwK8vWVIAWlY8eO6tix41WXGWP03nvv6cUXX9RDDz0kSfr0008VHByshQsX6pFHHnFmVAAAcAsostcY7d+/X8eOHVNkZKRtzN/fX/fcc4/i4+NdmAwAAJRULj1idD3Hjh2TJAUHB9uNBwcH25ZdTWZmpjIzM22v09LSJElZWVnKysoqhKTIqyu/f+bB9ZiLooX5KDpKwlzk5OTI29tbXqUt8ihlXB0n33Jc1FCKbDHKr0mTJmnChAm5xlevXi0fHx8XJMJfrVixwtURYMVcFC3MR9FR3Odi7ty51j9luzTHzUhPD9WjMc7fb5EtRiEhIZKk5ORkVapUyTaenJysO++885rvGzt2rEaPHm17nZaWptDQULVu3VqBgYGFlhc3lpWVpRUrVqhdu3Zyd3d3dZxbGnNRtDAfRUdJmItt27apRYsWCn70DXkEV3N1nHzLOXrIJfstssUoIiJCISEhWrlypa0IpaWlacOGDRo2bNg13+fp6SlPT89c4+7u7sX2H/KShrkoOpiLooX5KDqK81y4ubkpIyNDFy4ZmWyLq+Pkm7nkmv26tBidO3dOe/futb3ev3+/tm7dqvLlyyssLExPP/20Jk6cqJo1ayoiIkIvvfSSKleurK5du7ouNAAAKLFcWow2b96s1q1b215fOQXWt29fzZkzR88995zOnz+vwYMHKyUlRffff7+WLVsmLy8vV0UGAAAlmEuLUatWrWTMta+Yt1gseuWVV/TKK684MRUAALhVFdlrjAAgL5KSknTy5ElXx7gpOTk5ro4AwIpiBKDYSkpKUq3adXQhI93VUW6Kt7e35s6dqz///FMRERGujgPc0ihGAIqtkydP6kJGugI7Pyv3wFBXx8m3UmlHJEmnTp2iGAEuRjECUOy5B4bKM6SGq2Pkm6V08b2lGihpiux3pQEAADgbxQgAAMCKYgQAAGBFMQIAALCiGAEAAFhRjAAAAKwoRgAAAFYUIwAAACuKEQAAgBXFCAAAwIpiBAAAYEUxAgAAsOJLZAEA+Itt27bJza14HjvYtWuXqyMUaxQjAACs/vzzT0lSixYtlJGR4eI0cAWKEQAAVqdOnZIklX9gpLL9Krs4Tf5k7Nus1B8/c3WMYotiBADAX7iXv02lK1R3dYx8yTp1yNURirXieQIVAACgEFCMAAAArChGAAAAVhQjAAAAKy6+htMV5+eDXFGhQgWFhYW5OsZNK+5zwfNaipakpCSdPHnS1TFuSmJionx9fV0dAy5EMYLTlKTng3h5+yhx965iW45K0lygaEhKSlKt2nV0ISPd1VFuire3t+bOnevqGHAhihGcpiQ8H0S6fCvsqSXv6OTJk8W2GJWUueB5LUXHyZMndSEjXYGdn5V7YKir4+Sb+XOrqyPAxShGcLri/HyQkqa4zwXPayl63AND5RlSw9Ux8u1S2hFXR4CLFd+LCwAAAAoYxQgAAMCKYgQAAGBFMQIAALCiGAEAAFhRjAAAAKwoRgAAAFYUIwAAACuKEQAAgBXFCAAAwIpiBAAAYHXLfFfa9u3b5efn5+oYN6VChQrF9ktLS6Jdu3a5OkK+JSYmytfX19Ux8BeJiYlycyue/71anP//APyvW6YYdezYURcuXHB1jJvi5e2jxN27KEculn3ujGSxKDo62tVR8s3b21tz5851dQxYZZ9PkVRVgwYNUkZGhqvjALe0W6YYlYscIhMY4eoY+ZZ16pBOLXlHJ0+epBi5WE7mOckYBXZ+Vu6Boa6Oky/mz62ujoD/kZN5XpJU/oGRyvar7OI0+ZOxb7NSf/zM1TGAm3bLFCP3cpVlCanh6hgoQdwDQ+VZTP+ZupR2xNURcBXu5W9T6QrVXR0jX7JOHXJ1BKBAFM+T2QAAAIWAYgQAAGBFMQIAALCiGAEAAFhRjAAAAKwoRgAAAFYUIwAAACuKEQAAgBXFCAAAwIpiBAAAYEUxAgAAsKIYAQAAWFGMAAAArChGAAAAVhQjAAAAK4oRAACAFcUIAADAimIEAABgRTECAACwohgBAABYlXZ1ADhm165dro6Qb4mJifL19XV1DAAAroliVExknzsjWSyKjo52dZR88/b21ty5c10dAwCAa6IYFRM5meckYxTY+Vm5B4a6Ok6+mD+3ujoCAADXRTEqZtwDQ+UZUsPVMfLlUtoRV0cAAOC6isXF11OnTlV4eLi8vLx0zz33aOPGja6OBAAASqAiX4w+//xzjR49WuPGjdOWLVvUsGFDdejQQcePH3d1NAAAUMIU+WI0efJkDRo0SP3791fdunU1Y8YM+fj46OOPP3Z1NAAAUMIU6WJ08eJFJSQkKDIy0jbm5uamyMhIxcfHuzAZAAAoiYr0xdcnT55Udna2goOD7caDg4O1e/fuq74nMzNTmZmZttepqamSJMuZJJnCi1ro3M4elZeXlyyn9svkZN74DUWQ29ljSk9Pl+X0QeVcvODqOPnGXBQdJWEupJIxH8xF0VFS5sJyJkmSZIyT/+1tirDDhw8bSeann36yGx8zZoxp2rTpVd8zbtw4I4kffvjhhx9++CkBP3/88YczKodNkT5iVKFCBZUqVUrJycl248nJyQoJCbnqe8aOHavRo0fbXqekpKhq1apKSkqSv79/oebF9aWlpSk0NFSHDh2Sn5+fq+Pc0piLooX5KDqYi6IjNTVVYWFhKl++vFP3W6SLkYeHhxo1aqSVK1eqa9eukqScnBytXLlSI0aMuOp7PD095enpmWvc39+ff8iLCD8/P+aiiGAuihbmo+hgLooONzfnXg5dpIuRJI0ePVp9+/ZV48aN1bRpU7333ns6f/68+vfv7+poAACghCnyxahXr146ceKEXn75ZR07dkx33nmnli1bluuCbAAAgJtV5IuRJI0YMeKap85uxNPTU+PGjbvq6TU4F3NRdDAXRQvzUXQwF0WHq+bCYoyz74MDAAAomor0Ax4BAACciWIEAABgRTECAACwohgBAABYlYhiNHXqVIWHh8vLy0v33HOPNm7caFs2evRolS9fXqGhoYqNjbV73/z589WlSxdnxy0RJk2apCZNmqhs2bKqWLGiunbtqsTERLt1Lly4oOHDhyswMFC+vr7q3r273VPMT58+rS5dusjX11d33XWXfvnlF7v3Dx8+XO+8845TPk9J8sYbb8hisejpp5+2jTEXznP48GFFR0crMDBQ3t7eql+/vjZv3mxbbozRyy+/rEqVKsnb21uRkZHas2ePbXlmZqYee+wx+fn56fbbb9f3339vt/23335bI0eOdNrnKa6ys7P10ksvKSIiQt7e3qpevbpeffVVu+/dYi4Kzw8//KAuXbqocuXKslgsWrhwod3yG/3upct/L/Xp00d+fn4KCAjQwIEDde7cOdvyAwcOqEWLFipTpoxatGihAwcO2L2/c+fO+uqrrxwP79QvICkE8+bNMx4eHubjjz82v/32mxk0aJAJCAgwycnJZtGiRSY4ONhs2rTJxMXFGS8vL3PixAljjDEpKSmmZs2a5uDBgy7+BMVThw4dTExMjNmxY4fZunWr6dSpkwkLCzPnzp2zrTN06FATGhpqVq5caTZv3myaNWtm7r33Xtvy0aNHm5YtW5rExETz9NNPm0aNGtmWxcfHm0aNGplLly459XMVdxs3bjTh4eGmQYMGZtSoUbZx5sI5Tp8+bapWrWr69etnNmzYYPbt22eWL19u9u7da1vnjTfeMP7+/mbhwoVm27Zt5sEHHzQREREmIyPDGGPMBx98YOrUqWN27Nhh3n77bRMUFGRycnKMMcbs27fP1KxZ06Smprrk8xUnr732mgkMDDRLliwx+/fvN/Pnzze+vr7m/ffft63DXBSeb7/91rzwwgtmwYIFRpL5+uuv7Zbf6HdvjDEPPPCAadiwofn555/Njz/+aGrUqGF69+5tW96tWzfzyCOPmN9//9307NnTdO/e3bZs3rx5pkuXLvnKXuyLUdOmTc3w4cNtr7Ozs03lypXNpEmTzJtvvml69eplW1axYkWzceNGY4wxgwcPNpMnT3Z63pLq+PHjRpJZu3atMeZy8XR3dzfz58+3rbNr1y4jycTHxxtjjOnYsaOZPn26McaYnTt3Gh8fH2OMMRcvXjQNGzY0mzZtcvKnKN7Onj1ratasaVasWGFatmxpK0bMhfM8//zz5v7777/m8pycHBMSEmLefvtt21hKSorx9PQ0c+fONcYYM2zYMPP8888bY4xJT083kszx48eNMZf/g2TBggWF+AlKjqioKDNgwAC7sW7dupk+ffoYY5gLZ/prMcrL737nzp1Gkt3fPUuXLjUWi8UcPnzYGGNMnTp1zNKlS40xl4tY3bp1jTHGnDlzxtSoUcMkJSXlK2+xPpV28eJFJSQkKDIy0jbm5uamyMhIxcfHq2HDhtq8ebPOnDmjhIQEZWRkqEaNGlq3bp22bNmip556yoXpS5bU1FRJsn3ZX0JCgrKysuzmpnbt2goLC1N8fLwkqWHDhlq1apUuXbqk5cuXq0GDBpKkt956S61atVLjxo2d/CmKt+HDhysqKsrudy4xF860aNEiNW7cWD169FDFihV111136aOPPrIt379/v44dO2Y3F/7+/rrnnnvs5mLdunXKyMjQ8uXLValSJVWoUEGxsbHy8vLSww8/7PTPVRzde++9WrlypX7//XdJ0rZt27Ru3Tp17NhREnPhSnn53cfHxysgIMDu757IyEi5ublpw4YNki7Pz/fff6+cnBx99913tr+3xowZo+HDhys0NDR/AfNVp4qIw4cPG0nmp59+shsfM2aMadq0qTHGmHHjxpnq1aubO+64wyxYsMBkZmaaO+64w2zevNlMmTLF3H777ebee+81O3bscMVHKBGys7NNVFSUue+++2xjsbGxxsPDI9e6TZo0Mc8995wx5vJ/IfTu3duEhYWZFi1amN9++838/vvvpmbNmubkyZNmyJAhJiIiwvTo0cOkpKQ47fMUR3PnzjV33HGH7TD0/x4xYi6cx9PT03h6epqxY8eaLVu2mJkzZxovLy8zZ84cY4wx69evN5LMkSNH7N7Xo0cP07NnT2PM5aN0Tz75pAkPDzeNGzc2P/74ozl16pSpVq2aSUpKMi+88IKpXr26ad++vfnzzz+d/hmLi+zsbPP8888bi8ViSpcubSwWi3n99ddty5kL59Ffjhjl5Xf/2muvmdtvvz3XtoKCgsy0adOMMcb8+eefJioqyoSGhpqoqCjz559/mrVr15rGjRubU6dOmR49epiIiAgzZMgQk5mZmee8xeIrQW7G+PHjNX78eNvrCRMmKDIyUu7u7po4caK2b9+uJUuW6PHHH1dCQoLrghZjw4cP144dO7Ru3TqH3ufv76+4uDi7sTZt2ujtt99WbGys9u3bp8TERA0aNEivvPIKF/9ew6FDhzRq1CitWLFCXl5e+doGc1EwcnJy1LhxY73++uuSpLvuuks7duzQjBkz1Ldv3zxtw93dXVOnTrUb69+/v5566in98ssvWrhwobZt26a33npLTz31VP4uLr0FfPHFF4qNjVVcXJzq1aunrVu36umnn1blypWZixLitttu05IlS2yvMzMz1aFDB33yySeaOHGiypYtq8TERD3wwAOaOXNmni+UL9an0ipUqKBSpUrZ3V0jScnJyQoJCcm1/u7du/XZZ5/p1Vdf1Zo1a9SiRQsFBQWpZ8+e2rJli86ePeus6CXGiBEjtGTJEq1evVpVqlSxjYeEhOjixYtKSUmxW/9acyNJMTExCggI0EMPPaQ1a9aoa9eucnd3V48ePbRmzZpC/BTFW0JCgo4fP667775bpUuXVunSpbV27Vp98MEHKl26tIKDg5kLJ6lUqZLq1q1rN1anTh0lJSVJku33nde/syRp9erV+u233zRixAitWbNGnTp1UpkyZdSzZ0/m4jrGjBmj//u//9Mjjzyi+vXr67HHHtMzzzyjSZMmSWIuXCkvv/uQkBAdP37cbvmlS5d0+vTpa87P66+/rvbt26tRo0Zas2aNunfvLnd3d3Xr1s2h+SnWxcjDw0ONGjXSypUrbWM5OTlauXKlmjdvbreuMUZDhgzR5MmT5evrq+zsbGVlZUmS7X+zs7OdF76YM8ZoxIgR+vrrr7Vq1SpFRETYLW/UqJHc3d3t5iYxMVFJSUm55kaSTpw4oVdeeUVTpkyRpFzzw9xcW9u2bbV9+3Zt3brV9tO4cWP16dPH9mfmwjnuu+++XI+t+P3331W1alVJUkREhEJCQuzmIi0tTRs2bLjqXFx5zMLMmTNVqlQp5sIB6enpcnOz/1dcqVKllJOTI4m5cKW8/O6bN2+ulJQUuzM5q1atUk5Oju65555c29y1a5fi4uL06quvSrrJv7ccPllYxMybN894enqaOXPmmJ07d5rBgwebgIAAc+zYMbv1Zs2aZXcr34YNG4yfn5+Jj483L7/8su1qduTNsGHDjL+/v1mzZo05evSo7Sc9Pd22ztChQ01YWJhZtWqV2bx5s2nevLlp3rz5Vbf36KOPmilTpthev/nmm6ZRo0Zm586dpmPHjubJJ58s9M9UkvzvNUbGMBfOsnHjRlO6dGnz2muvmT179pjY2Fjj4+NjPvvsM9s6b7zxhgkICDD//e9/za+//moeeuihXLcpX/HPf/7TPPvss7bXn3/+uQkLCzPbtm0zAwcONJ06dXLK5yqO+vbta2677Tbb7foLFiwwFSpUsF1XZwxzUZjOnj1rfvnlF/PLL78YSWby5Mnml19+sT0iJy+/+wceeMDcddddZsOGDWbdunWmZs2adrfrX5GTk2Puv/9+s3jxYtvYsGHDTFRUlNm5c6e56667zFtvvZXn7MW+GBljzJQpU0xYWJjx8PAwTZs2NT///LPd8mPHjpmqVavabvG7YsKECaZ8+fKmdu3aZsOGDc6MXOxJuupPTEyMbZ2MjAzz5JNPmnLlyhkfHx/z8MMPm6NHj+ba1rJly0zTpk1Ndna2bez8+fOmR48epmzZsqZt27YmOTnZGR+rxPhrMWIunGfx4sXmjjvuMJ6enqZ27dpm1qxZdstzcnLMSy+9ZIKDg42np6dp27atSUxMzLWd7du3mxo1atg9Gyw7O9sMGzbM+Pn5mSZNmpg9e/YU+ucprtLS0syoUaNMWFiY8fLyMtWqVTMvvPCC3UW4zEXhWb169VX/HdG3b19jTN5+96dOnTK9e/c2vr6+xs/Pz/Tv39+cPXs2175mzJhhd+DDGGOSk5NN27ZtTdmyZU2PHj3M+fPn85zdYsz/PAYUAADgFlasrzECAAAoSBQjAAAAK4oRAACAFcUIAADAimIEAABgRTECAACwohgBAABYUYwAQJe/cNpischisei99967qW21atXKtq2tW7cWSD4AzkExAnBD8fHxKlWqlKKionItW7NmjSwWS64vqZWk8PBwu5JxpSxYLBb5+/vrvvvu06pVq2zL+/Xrp65du9q9tlgsGjp0aK5tDx8+XBaLRf369bMbP3TokAYMGKDKlSvLw8NDVatW1ahRo3Tq1Kkbfs569erp6NGjGjx4sG1s9OjRKl++vEJDQxUbG2u3/vz589WlS5dc21mwYIE2btx4w/0BKHooRgBuaPbs2Ro5cqR++OEHHTly5Ka2FRMTo6NHj2r9+vWqUKGCOnfurH379l1z/dDQUM2bN08ZGRm2sQsXLiguLk5hYWF26+7bt0+NGzfWnj17NHfuXO3du1czZsywfbH06dOnr5utdOnSCgkJkY+PjyRp8eLFiouL03fffae33npLTzzxhE6ePClJSk1N1QsvvKCpU6fm2k758uUVFBSU598JgKKDYgTgus6dO6fPP/9cw4YNU1RUlObMmXNT2wsICFBISIjuuOMOTZ8+XRkZGVqxYsU117/77rsVGhqqBQsW2MYWLFigsLAw3XXXXXbrDh8+XB4eHvruu+/UsmVLhYWFqWPHjvr+++91+PBhvfDCCw5l3bVrl1q1aqXGjRurd+/e8vPz0/79+yVJzz33nIYNG5arnAEo3ihGAK7riy++UO3atVWrVi1FR0fr448/VkF9xaK3t7ck6eLFi9ddb8CAAYqJibG9/vjjj9W/f3+7dU6fPq3ly5frySeftG33ipCQEPXp00eff/65Q9kbNmyozZs368yZM0pISFBGRoZq1KihdevWacuWLXrqqafyvC0AxQPFCMB1zZ49W9HR0ZKkBx54QKmpqVq7du1Nbzc9PV0vvviiSpUqpZYtW1533ejoaK1bt04HDx7UwYMHtX79elumK/bs2SNjjOrUqXPVbdSpU0dnzpzRiRMn8pyxQ4cOio6OVpMmTdSvXz998sknKlOmjIYNG6YZM2Zo+vTpqlWrlu677z799ttved4ugKKrtKsDACi6EhMTtXHjRn399deSLl+D06tXL82ePVutWrXK1zZ79+6tUqVKKSMjQ0FBQZo9e7YaNGhw3fcEBQXZTuMZYxQVFaUKFSpcdd2COpp1xfjx4zV+/Hjb6wkTJigyMlLu7u6aOHGitm/friVLlujxxx9XQkJCge4bgPNRjABc0+zZs3Xp0iVVrlzZNmaMkaenpz788EP5+/vLz89P0uWLkQMCAuzen5KSIn9/f7uxd999V5GRkfL393foAuUBAwZoxIgRknTVC55r1Kghi8WiXbt26eGHH861fNeuXSpXrtxNXRS9e/duffbZZ/rll1/08ccfq0WLFgoKClLPnj01YMAAnT17VmXLls339gG4HqfSAFzVpUuX9Omnn+qdd97R1q1bbT/btm1T5cqVNXfuXElSzZo15ebmlutoyb59+5Samqrbb7/dbjwkJEQ1atRwuKA88MADunjxorKystShQ4dcywMDA9WuXTtNmzbN7g42STp27JhiY2PVq1cvWSwWh/Z7hTFGQ4YM0eTJk+Xr66vs7GxlZWVJku1/s7Oz87VtAEUHR4wAXNWSJUt05swZDRw4MNdRn+7du2v27NkaOnSoypYtqyeeeELPPvusSpcurfr16+vQoUN6/vnn1axZM917770FkqdUqVLatWuX7c9X8+GHH+ree+9Vhw4dNHHiREVEROi3337TmDFjdNttt+m1117L9/7//e9/KygoyPbcovvuu0/jx4/Xzz//rKVLl6pu3bq5jpgBKH44YgTgqmbPnm075fVX3bt31+bNm/Xrr79Kkt5//3317dtXzz//vOrVq6d+/fqpQYMGWrx4cb6P0FyNn5+f7dTd1dSsWVObN29WtWrV1LNnT1WvXl2DBw9W69atFR8fr/Lly+drv8nJyXrttdf0wQcf2MaaNm2qZ599VlFRUfriiy/s7poDUHxZTEFfqQgAxdD48eO1cOHCAvsKjwMHDigiIkK//PKL7rzzzgLZJoDCxxEjALDavn27fH19NW3atJvaTseOHVWvXr0CSgXAmThiBAC6/IDIK18ZEhQUdNVTiHl1+PBh2wXgYWFh8vDwKJCMAAofxQgAAMCKU2kAAABWFCMAAAArihEAAIAVxQgAAMCKYgQAAGBFMQIAALCiGAEAAFhRjAAAAKwoRgAAAFb/D60PVy7Nlq5qAAAAAElFTkSuQmCC",
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "fig, ax = plt.subplots()\n",
    "ax.hist(aupimo_result.aupimos.numpy(), bins=np.linspace(0, 1, 11), edgecolor=\"black\")\n",
    "ax.set_ylabel(\"Count (number of images)\")\n",
    "ax.yaxis.set_major_locator(MaxNLocator(5, integer=True))\n",
    "ax.set_xlim(0, 1)\n",
    "ax.set_xlabel(\"AUPIMO [%]\")\n",
    "ax.xaxis.set_major_formatter(PercentFormatter(1))\n",
    "ax.grid()\n",
    "ax.set_title(\"AUPIMO distribution\")\n",
    "fig  # noqa: B018, RUF100"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Cite Us\n",
    "\n",
    "AUPIMO was developed during [Google Summer of Code 2023 (GSoC 2023)](https://summerofcode.withgoogle.com/archive/2023/projects/SPMopugd) with the `anomalib` team from Intel's OpenVINO Toolkit.\n",
    "\n",
    "arXiv: [arxiv.org/abs/2401.01984](https://arxiv.org/abs/2401.01984) (accepted to BMVC 2024)\n",
    "\n",
    "Official repository: [github.com/jpcbertoldo/aupimo](https://github.com/jpcbertoldo/aupimo) (numpy-only API and numba-accelerated versions available)\n",
    "\n",
    "```bibtex\n",
    "@misc{bertoldo2024aupimo,\n",
    "      author={Joao P. C. Bertoldo and Dick Ameln and Ashwin Vaidya and Samet Akçay},\n",
    "      title={{AUPIMO: Redefining Visual Anomaly Detection Benchmarks with High Speed and Low Tolerance}}, \n",
    "      year={2024},\n",
    "      url={https://arxiv.org/abs/2401.01984}, \n",
    "}\n",
    "```"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "anomalib-dev",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.14"
  },
  "orig_nbformat": 4
 },
 "nbformat": 4,
 "nbformat_minor": 2
}