demos/OracleAutoMLx_train_model.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "7a8099ce",
   "metadata": {},
   "source": [
    "***\n",
    "# <font color=red>Building and Evaluating a Machine Learning Model using AutoMLx</font>\n",
    "<p style=\"margin-left:10%; margin-right:10%;\">by the <font color=teal> Oracle AutoMLx Team </font></p>\n",
    "\n",
    "***"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "87e8ccb6",
   "metadata": {},
   "source": [
    "AutoMLx Demo Notebook.\n",
    "\n",
    "Copyright © 2024, Oracle and/or its affiliates.\n",
    "\n",
    "Licensed under the Universal Permissive License v 1.0 as shown at https://oss.oracle.com/licenses/upl/"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "96f316fb",
   "metadata": {},
   "source": [
    "## Overview of this Notebook\n",
    "\n",
    "In this notebook, we will build and evaluate a machine learning model using Oracle AutoMLx. The first dataset is a binary classification dataset.\n",
    "We explore the various options provided by Oracle AutoMLx, allowing the user to specify different options in the training procedure. We then evaluate the model trained by AutoMLx.\n",
    "\n",
    "---\n",
    "## Prerequisites\n",
    "\n",
    "  - Experience level: Novice (Python and Machine Learning)\n",
    "  - Professional experience: Some industry experience\n",
    "---\n",
    "\n",
    "## Business Use\n",
    "\n",
    "Data analytics and modeling problems using Machine Learning (ML) are becoming popular and often rely on data science expertise to build accurate ML models. Such modeling tasks primarily involve the following steps:\n",
    "- Preprocessing the dataset (for example, cleaning, imputing, engineering features and normalization).\n",
    "- Picking an appropriate model for the given dataset and prediction task at hand.\n",
    "- Tuning the chosen model’s hyperparameters for the given dataset.\n",
    "\n",
    "All of these steps are significantly time consuming and heavily rely on data scientist expertise. Unfortunately, to make this problem harder, the best feature subset, model, and hyperparameter choice widely varies with the dataset and the prediction task. Hence, there is no one-size-fits-all solution to achieve reasonably good model performance. Using a simple Python API, AutoML can quickly jump-start the datascience process with an accurately-tuned model and appropriate features for a given prediction task.\n",
    "\n",
    "## Table of Contents\n",
    "\n",
    "- <a href='#train'> Train a Model using AutoMLx</a>\n",
    "- <a href='#quality'> Inspect the Model's Quality</a>\n",
    "- <a href='#prediction'> Make Predictions using the Model</a>\n",
    "- <a href='#evaluate'> Evaluate the Quality of a Model on a Given Dataset</a>\n",
    "- <a href='#save'> Save a model</a>\n",
    "- <a href='#load'> Load a model</a>\n",
    "- <a href='#regression'> Train a Regression Model using AutoMLx</a>\n",
    "- <a href='#ref'>References</a>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7388792f",
   "metadata": {},
   "source": [
    "<a id='train'></a>\n",
    "## Train a Model using AutoMLx\n",
    "\n",
    "Here we show how easy it is to use the AutoMLx train_model API to quickly and automatically train a model for a\n",
    "machine learning problem. We pass the data, with the name of the target to predict and task to the `train_model`\n",
    "function. This function will return the best, fully-trained model that AutoML could find for the given dataset.\n",
    "\n",
    "You can find the synthetic datasets used in this notebook at https://docs.oracle.com/en-us/iaas/tools/automlx/latest/data/\n",
    "\n",
    "The data argument can be a string, in which case it should be the path to a CSV file that contains your dataset.\n",
    "Alternatively, you can directly pass a pandas DataFrame.\n",
    "\n",
    "The task can be either `classification` or `regression`, or we can import and use Task.CLASSIFICATION and TASK.REGRESSION."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "7c2c9e15",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-09-11T15:34:57.226372Z",
     "iopub.status.busy": "2024-09-11T15:34:57.226020Z",
     "iopub.status.idle": "2024-09-11T15:35:37.239507Z",
     "shell.execute_reply": "2024-09-11T15:35:37.238734Z"
    }
   },
   "outputs": [],
   "source": [
    "from automlx import train_model\n",
    "\n",
    "model = train_model(\n",
    "    data = \"classification_train.csv\",  # path to dataset CSV file or a pandas DataFrame\n",
    "    target_to_predict = \"income_group\",  # name of the target column in the dataset\n",
    "    task = 'classification',  # type of problem you are interested in solving,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bb257e8d",
   "metadata": {},
   "source": [
    "That's it! The model is fully trained and ready to be used to make predictions or to be deployed."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0305ebd6",
   "metadata": {},
   "source": [
    "<a id='quality'></a>\n",
    "## Inspect the Model's Quality\n",
    "\n",
    "But how well can you expect your model to work? There are many ways to measure the quality of a machine learning\n",
    "model. AutoML automatically calculates two different types of scores for your model. The first is the model's\n",
    "training score, this tells you how well the model learned to predict the target on the data that was used for\n",
    "training the model. Generally, higher scores are better; however, sometimes a model may see patterns in your data\n",
    "that appeared by random chance. When this happens, your model typically won't perform well when deployed, because\n",
    "those same patterns aren't likely to appear in future data that the model encounters. For this reason, AutoML\n",
    "automatically reserves 20% of the training data as a stress test for your model. This data is not used to train the\n",
    "model; instead, it is used to estimate the future quality of your model on new data. Both scores can be accessed\n",
    "using `model.quality`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "edafca08",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-09-11T15:35:37.242750Z",
     "iopub.status.busy": "2024-09-11T15:35:37.242348Z",
     "iopub.status.idle": "2024-09-11T15:35:37.250812Z",
     "shell.execute_reply": "2024-09-11T15:35:37.250243Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>neg_log_loss</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Evaluated on 2024-09-11</th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>Measured quality on training data</th>\n",
       "      <td>-0.109292</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Estimate of future quality</th>\n",
       "      <td>-0.131209</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                   neg_log_loss\n",
       "Evaluated on 2024-09-11                        \n",
       "Measured quality on training data     -0.109292\n",
       "Estimate of future quality            -0.131209"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.quality"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f8f404e8",
   "metadata": {},
   "source": [
    "The more similar the two scores are the better. If there is a large gap between them, it may mean that the model learned to rely on spurious correlations. However, if the model quality is still good on the stress test data, then the gap may not be a cause for concern."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f3af70bc",
   "metadata": {},
   "source": [
    "<a id='prediction'></a>\n",
    "## Make Predictions using the Model\n",
    "\n",
    "We can now use this model to make predictions! The following method will return a new dataset that is identical to\n",
    "the provided dataset with an additional column that contains the model's predictions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "b07dfe49",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-09-11T15:35:37.253129Z",
     "iopub.status.busy": "2024-09-11T15:35:37.252741Z",
     "iopub.status.idle": "2024-09-11T15:35:37.278315Z",
     "shell.execute_reply": "2024-09-11T15:35:37.277716Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>age</th>\n",
       "      <th>education</th>\n",
       "      <th>sex</th>\n",
       "      <th>income_group</th>\n",
       "      <th>prediction for income_group</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>42</td>\n",
       "      <td>diploma</td>\n",
       "      <td>female</td>\n",
       "      <td>&lt;=50k</td>\n",
       "      <td>&lt;=50k</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>57</td>\n",
       "      <td>bachelors</td>\n",
       "      <td>male</td>\n",
       "      <td>&lt;=50k</td>\n",
       "      <td>&lt;=50k</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   age  education     sex income_group prediction for income_group\n",
       "0   42    diploma  female        <=50k                       <=50k\n",
       "1   57  bachelors    male        <=50k                       <=50k"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data_with_prediction = model.predict(\"classification_train.csv\")\n",
    "data_with_prediction.head(2)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c76224ae",
   "metadata": {},
   "source": [
    "We can also save the dataset with the predictions by passing desired path to a new CSV file the `output` parameter."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "e34e2cd8",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-09-11T15:35:37.280806Z",
     "iopub.status.busy": "2024-09-11T15:35:37.280213Z",
     "iopub.status.idle": "2024-09-11T15:35:37.335441Z",
     "shell.execute_reply": "2024-09-11T15:35:37.334832Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>age</th>\n",
       "      <th>education</th>\n",
       "      <th>sex</th>\n",
       "      <th>income_group</th>\n",
       "      <th>prediction for income_group</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>42</td>\n",
       "      <td>diploma</td>\n",
       "      <td>female</td>\n",
       "      <td>&lt;=50k</td>\n",
       "      <td>&lt;=50k</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>57</td>\n",
       "      <td>bachelors</td>\n",
       "      <td>male</td>\n",
       "      <td>&lt;=50k</td>\n",
       "      <td>&lt;=50k</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   age  education     sex income_group prediction for income_group\n",
       "0   42    diploma  female        <=50k                       <=50k\n",
       "1   57  bachelors    male        <=50k                       <=50k"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data_with_prediction = model.predict(\"classification_train.csv\", output='data_with_prediction.csv')\n",
    "data_with_prediction.head(2)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "41c871c0",
   "metadata": {},
   "source": [
    "<a id='evaluate'></a>\n",
    "## Evaluate the Quality of a Model on a New Dataset\n",
    "\n",
    "Inspecting `model.quality` (see above) is always a good idea to ensure that the model performed well when it was\n",
    "trained. However, it is never a bad idea to continue evaluating the model over time on new data as you collect it.\n",
    "This can be achieved using the `evaluate_model_quality` function. We just need to pass the model and the desired\n",
    "dataset to this function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "f9888267",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-09-11T15:35:37.337903Z",
     "iopub.status.busy": "2024-09-11T15:35:37.337402Z",
     "iopub.status.idle": "2024-09-11T15:35:37.367815Z",
     "shell.execute_reply": "2024-09-11T15:35:37.367248Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>neg_log_loss</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>classification_test.csv</th>\n",
       "      <td>-0.159003</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                         neg_log_loss\n",
       "classification_test.csv     -0.159003"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from automlx import evaluate_model_quality\n",
    "\n",
    "score = evaluate_model_quality(model, \"classification_test.csv\")\n",
    "\n",
    "score"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c1e0f9b6",
   "metadata": {},
   "source": [
    "You can always compare the new results to `model.quality` to see if the model is still performing as well as it\n",
    "was expected to when it was trained. If not, it may be time to call `train_model` again with your new data so that\n",
    "the model can learn any new trends that have appeared in your dataset."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fd2bb1bc",
   "metadata": {},
   "source": [
    "<a id='save'></a>\n",
    "## Save a model\n",
    "\n",
    "Once we are satisfied with the results, we can save the model, using the `save` method, by passing a desired file\n",
    "path."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "13a0f118",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-09-11T15:35:37.370214Z",
     "iopub.status.busy": "2024-09-11T15:35:37.369677Z",
     "iopub.status.idle": "2024-09-11T15:35:37.729471Z",
     "shell.execute_reply": "2024-09-11T15:35:37.728727Z"
    }
   },
   "outputs": [],
   "source": [
    "model.save('model.amlx')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "576419dc",
   "metadata": {},
   "source": [
    "<a id='load'></a>\n",
    "## Load a model\n",
    "\n",
    "We can also load a saved model using the `load_model` function by providing the path to the model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "cd59604c",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-09-11T15:35:37.732780Z",
     "iopub.status.busy": "2024-09-11T15:35:37.732180Z",
     "iopub.status.idle": "2024-09-11T15:35:38.189844Z",
     "shell.execute_reply": "2024-09-11T15:35:38.189225Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>neg_log_loss</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Evaluated on 2024-09-11</th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>Measured quality on training data</th>\n",
       "      <td>-0.109292</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Estimate of future quality</th>\n",
       "      <td>-0.131209</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                   neg_log_loss\n",
       "Evaluated on 2024-09-11                        \n",
       "Measured quality on training data     -0.109292\n",
       "Estimate of future quality            -0.131209"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from automlx import load_model\n",
    "\n",
    "loaded_model = load_model('model.amlx')\n",
    "\n",
    "loaded_model.quality"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e52e6e6f",
   "metadata": {},
   "source": [
    "<a id='load'></a>\n",
    "## Train a Regression Model using AutoMLx\n",
    "\n",
    "Here we show how to train a model for a regression task. We also cover some optional arguments that can be used to\n",
    "further control how AutoML works.\n",
    "  - metric: The name of the desired scoring metric. By default, this is `auto`, which means that an appropriate\n",
    "  metric is chosen based on the dataset and task.\n",
    "  - time_budget: The maximum time budget in seconds. By default, this is `None`, which means that AutoML should run\n",
    "  until it is done (that is, until it cannot find any better models for your data). Note that AutoML will sometimes\n",
    "  run for longer than your requested time budget. This is to ensure that we can always return a fully-trained model\n",
    "  that is ready to be deployed.\n",
    "  - test_data: Advanced users can pass in a custom dataset for stress testing the model. This will be used to estimate\n",
    "  the quality of the final model on future data. If not provided, the test scores are estimated automatically by\n",
    "  reserving 20% of the training data for evaluation of the final model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "d547e67f",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-09-11T15:35:38.192627Z",
     "iopub.status.busy": "2024-09-11T15:35:38.191980Z",
     "iopub.status.idle": "2024-09-11T15:35:58.691625Z",
     "shell.execute_reply": "2024-09-11T15:35:58.691014Z"
    }
   },
   "outputs": [],
   "source": [
    "from automlx import Task\n",
    "\n",
    "model_regression = train_model(\n",
    "    data = \"regression_train.csv\",\n",
    "    target_to_predict = \"income\",\n",
    "    task = Task.REGRESSION,\n",
    "    metric = 'auto',\n",
    "    time_budget=30,\n",
    "    test_data = \"regression_test.csv\"\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "923e97d8",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-09-11T15:35:58.694436Z",
     "iopub.status.busy": "2024-09-11T15:35:58.694050Z",
     "iopub.status.idle": "2024-09-11T15:35:58.699307Z",
     "shell.execute_reply": "2024-09-11T15:35:58.698835Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>neg_mean_squared_error</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Evaluated on 2024-09-11</th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>Measured quality on training data</th>\n",
       "      <td>-3.781031e+07</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Estimate of future quality</th>\n",
       "      <td>-3.829070e+07</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                   neg_mean_squared_error\n",
       "Evaluated on 2024-09-11                                  \n",
       "Measured quality on training data           -3.781031e+07\n",
       "Estimate of future quality                  -3.829070e+07"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model_regression.quality"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9536ecba",
   "metadata": {},
   "source": [
    "<a id='ref'></a>\n",
    "## References\n",
    "* Oracle AutoML http://www.vldb.org/pvldb/vol13/p3166-yakovlev.pdf"
   ]
  }
 ],
 "metadata": {
  "jupytext": {
   "formats": "ipynb,md,py:percent"
  },
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.19"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}