sdk/python/jobs/single-step/scikit-learn/train-hyperparameter-tune-deploy-with-sklearn/train-hyperparameter-tune-with-sklearn.ipynb

{
 "cells": [
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Train and hyperparameter tune with scikit-learn\n",
    "In this article, learn how to run your scikit-learn training scripts using the Azure Machine Learning (Azure ML) Python SDK v2.\n",
    "\n",
    "The example scripts in this article are used to classify iris flower images to build a machine learning model based on scikit-learn's [iris dataset](https://archive.ics.uci.edu/ml/datasets/iris).\n",
    "\n",
    "Whether you're training a deep learning PyTorch model from the ground-up or you're bringing an existing model into the cloud, you can use Azure Machine Learning to scale out open-source training jobs using elastic cloud compute resources. You can build, deploy, version, and monitor production-grade models with Azure Machine Learning.\n",
    "\n",
    "## Requirements\n",
    "In order to benefit from this article, you need to have:\n",
    "* an Azure subscription. If you don't have an Azure subscription, [create a free account](https://aka.ms/AMLFree) before you begin.\n",
    "* Run this code on either of these environments:\n",
    "   1. an Azure Machine Learning compute instance - no downloads or installation necessary\n",
    "      * Complete the [Quickstart: Get started with Azure Machine Learning](https://docs.microsoft.com/azure/machine-learning/quickstart-create-resources) to create a dedicated notebook server pre-loaded with the SDK and the sample repository.\n",
    "      * In the samples deep learning folder on the notebook server, find a completed and expanded notebook by navigating to this directory: * v2  > jobs > single-step > scikit-learn > train-hyperparameter-tune-deploy-with-sklearn* folder.\n",
    "   1. your own Jupyter Notebook server\n",
    "      * [Install the Azure Machine Learning SDK v2](https://docs.microsoft.com/python/api/overview/azure/ml/installv2?view=azure-ml-py)\n",
    "      * [Create a workspace configuration file](https://docs.microsoft.com/azure/machine-learning/how-to-configure-environment#workspace)\n",
    "      \n",
    "      You can also find a completed [Jupyter Notebook version](./train-hyperparameter-tune-with-sklearn.ipynb) of this guide on the GitHub samples page."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Connect to the workspace\n",
    "\n",
    "First, you'll need to connect to your Azure ML workspace. The workspace is the top-level resource for Azure Machine Learning, providing a centralized place to work with all the artifacts you create when you use Azure Machine Learning.\n",
    "\n",
    "We are using `DefaultAzureCredential` to get access to workspace. \n",
    "`DefaultAzureCredential` should be capable of handling most Azure SDK authentication scenarios. \n",
    "\n",
    "Reference for more available credentials if it does not work for you: [configure credential example](../../configuration.ipynb), [azure-identity reference doc](https://docs.microsoft.com/python/api/azure-identity/azure.identity?view=azure-python)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "name": "credential"
   },
   "outputs": [],
   "source": [
    "# Handle to the workspace\n",
    "from azure.ai.ml import MLClient\n",
    "\n",
    "# Authentication package\n",
    "from azure.identity import DefaultAzureCredential\n",
    "\n",
    "credential = DefaultAzureCredential()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you want to use a browser instead to login and authenticate, you can use the following code instead. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Handle to the workspace\n",
    "# from azure.ai.ml import MLClient\n",
    "\n",
    "# Authentication package\n",
    "# from azure.identity import InteractiveBrowserCredential\n",
    "# credential = InteractiveBrowserCredential()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the next cell, enter your Subscription ID, Resource Group name and Workspace name. To find subscription ID and resource group:\n",
    "\n",
    "1. In the upper right Azure Machine Learning Studio toolbar, select your workspace name.\n",
    "1. Copy the value for Resource group and subsccription ID into the code."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "name": "ml_client"
   },
   "outputs": [],
   "source": [
    "# Get a handle to the workspace\n",
    "ml_client = MLClient(\n",
    "    credential=credential,\n",
    "    subscription_id=\"<SUBSCRIPTION_ID>\",\n",
    "    resource_group_name=\"<RESOURCE_GROUP>\",\n",
    "    workspace_name=\"<AML_WORKSPACE_NAME>\",\n",
    ")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The result is a handler to the workspace that you'll use to manage other resources and jobs.\n",
    "\n",
    "> [!IMPORTANT]\n",
    "> Creating MLClient will not connect to the workspace. The client initialization is lazy, it will wait for the first time it needs to make a call (in the notebook below, that will happen during compute creation)."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Create a Compute Resource to run our job\n",
    "\n",
    "AzureML needs a compute resource for running a job. It can be single or multi-node machines with Linux or Windows OS, or a specific compute fabric like Spark.\n",
    "\n",
    "In this example, we provision a Linux [compute cluster](https://docs.microsoft.com/azure/machine-learning/how-to-create-attach-compute-cluster?tabs=python). See the [full list on VM sizes and prices](https://azure.microsoft.com/pricing/details/machine-learning/) .\n",
    "\n",
    "For this example we only need a basic cluster, let's pick a Standard_DS3_v2 model with 2 vCPU cores, 7 GB RAM and create an Azure ML Compute"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "name": "cpu_compute_target"
   },
   "outputs": [],
   "source": [
    "from azure.ai.ml.entities import AmlCompute\n",
    "\n",
    "# Name assigned to the compute cluster\n",
    "cpu_compute_target = \"cpu-cluster\"\n",
    "\n",
    "try:\n",
    "    # let's see if the compute target already exists\n",
    "    cpu_cluster = ml_client.compute.get(cpu_compute_target)\n",
    "    print(\n",
    "        f\"You already have a cluster named {cpu_compute_target}, we'll reuse it as is.\"\n",
    "    )\n",
    "\n",
    "except Exception:\n",
    "    print(\"Creating a new cpu compute target...\")\n",
    "\n",
    "    # Let's create the Azure ML compute object with the intended parameters\n",
    "    cpu_cluster = AmlCompute(\n",
    "        name=cpu_compute_target,\n",
    "        # Azure ML Compute is the on-demand VM service\n",
    "        type=\"amlcompute\",\n",
    "        # VM Family\n",
    "        size=\"STANDARD_DS3_V2\",\n",
    "        # Minimum running nodes when there is no job running\n",
    "        min_instances=0,\n",
    "        # Nodes in cluster\n",
    "        max_instances=4,\n",
    "        # How many seconds will the node running after the job termination\n",
    "        idle_time_before_scale_down=180,\n",
    "        # Dedicated or LowPriority. The latter is cheaper but there is a chance of job termination\n",
    "        tier=\"Dedicated\",\n",
    "    )\n",
    "\n",
    "    # Now, we pass the object to MLClient's create_or_update method\n",
    "    cpu_cluster = ml_client.compute.begin_create_or_update(cpu_cluster).result()\n",
    "\n",
    "print(\n",
    "    f\"AMLCompute with name {cpu_cluster.name} is created, the compute size is {cpu_cluster.size}\"\n",
    ")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Create a job environment for the job\n",
    "\n",
    "To run an AzureML job, you'll need an [environment](https://docs.microsoft.com/azure/machine-learning/concept-environments). An environment is the software runtime and libraries that you want installed on the compute  where you’ll be training. It is similar to your python emvironment on your local machine.\n",
    "\n",
    "AzureML provides many curated or readymade environments which are useful for common training and inference scenarios. You can also create your own “custom” environments using a docker image, or a conda configuration \n",
    "\n",
    "In this example, you'll create a custom conda environment for your jobs, using a conda yaml file.\n",
    "First, create a directory to store the file in."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "name": "make_env_folder"
   },
   "outputs": [],
   "source": [
    "import os\n",
    "\n",
    "dependencies_dir = \"./env\"\n",
    "os.makedirs(dependencies_dir, exist_ok=True)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, create the file in the dependencies directory. You can also optionally install Intel® Extension for Scikit-Learn in your yaml file for additional performance no your Intel hardware. More details can be found at the end of this section."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "name": "make_conda_file"
   },
   "outputs": [],
   "source": [
    "%%writefile {dependencies_dir}/conda.yaml\n",
    "name: sklearn-env\n",
    "channels:\n",
    "  - conda-forge\n",
    "dependencies:\n",
    "  - python=3.8\n",
    "  - pip=21.2.4\n",
    "  - scikit-learn=0.24.2\n",
    "  - scipy=1.7.1\n",
    "  - pip:  \n",
    "    - azureml-mlflow==1.42.0\n",
    "    - mlflow-skinny==2.3.2"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The specification contains some usual packages, that you'll use in your job (numpy, pip).\n",
    "\n",
    "\n",
    "Use the *yaml* file to create and register this custom environment in your workspace:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "name": "custom_environment"
   },
   "outputs": [],
   "source": [
    "from azure.ai.ml.entities import Environment\n",
    "\n",
    "custom_env_name = \"sklearn-env\"\n",
    "\n",
    "job_env = Environment(\n",
    "    name=custom_env_name,\n",
    "    description=\"Custom environment for sklearn image classification\",\n",
    "    conda_file=os.path.join(dependencies_dir, \"conda.yaml\"),\n",
    "    image=\"mcr.microsoft.com/azureml/openmpi4.1.0-ubuntu20.04:latest\",\n",
    ")\n",
    "job_env = ml_client.environments.create_or_update(job_env)\n",
    "\n",
    "print(\n",
    "    f\"Environment with name {job_env.name} is registered to workspace, the environment version is {job_env.version}\"\n",
    ")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### **[Optional] Install Intel® Extension for Scikit-Learn optimizations for more performance on Intel hardware**"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Want to speed up your scikit-learn scripts on Intel hardware? Try adding [Intel® Extension for Scikit-Learn](https://www.intel.com/content/www/us/en/developer/tools/oneapi/scikit-learn.html) into your conda yaml file. We will show you how to enable these optimizations later in this example:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "name": "make_sklearnex_conda_file"
   },
   "outputs": [],
   "source": [
    "%%writefile {dependencies_dir}/conda.yaml\n",
    "name: sklearn-env\n",
    "channels:\n",
    "  - conda-forge\n",
    "dependencies:\n",
    "  - python=3.8\n",
    "  - pip=21.2.4\n",
    "  - scikit-learn=0.24.2\n",
    "  - scikit-learn-intelex\n",
    "  - scipy=1.7.1\n",
    "  - pip:  \n",
    "    - azureml-mlflow==1.42.0\n",
    "    - mlflow-skinny==2.3.2"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The specification contains some usual packages, that you'll use in your job (numpy, pip), along with Intel® Extension for Scikit-Learn.\n",
    "\n",
    "\n",
    "Use the *yaml* file to create and register this custom environment in your workspace:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from azure.ai.ml.entities import Environment\n",
    "\n",
    "custom_env_name = \"sklearn-env\"\n",
    "\n",
    "job_env = Environment(\n",
    "    name=custom_env_name,\n",
    "    description=\"Custom environment for sklearn image classification\",\n",
    "    conda_file=os.path.join(dependencies_dir, \"conda.yaml\"),\n",
    "    image=\"mcr.microsoft.com/azureml/openmpi4.1.0-ubuntu20.04:latest\",\n",
    ")\n",
    "job_env = ml_client.environments.create_or_update(job_env)\n",
    "\n",
    "print(\n",
    "    f\"Environment with name {job_env.name} is registered to workspace, the environment version is {job_env.version}\"\n",
    ")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Data for training\n",
    "We will download and extract the data as part of our training script `train_iris.py`\n",
    "\n",
    "## Build the command job to train\n",
    "\n",
    "Now that you have all assets required to run your job, it's time to build the job itself, using the Azure ML Python SDK v2. We will be creating a `command` job.\n",
    "\n",
    "An AzureML `command` job is a resource that specifies all the details needed to execute your training code in the cloud: inputs and outputs, the type of hardware to use, software to install, and how to run your code. the `command` job contains information to execute a single command.\n",
    "\n",
    "## The training script\n",
    "\n",
    "We will use the training script - *train_iris.py* python file. This script dowloads data, trains a model and registers the model too.\n",
    "First, create a directory to store the file in."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "name": "make_src_folder"
   },
   "outputs": [],
   "source": [
    "import os\n",
    "\n",
    "src_dir = \"./src\"\n",
    "os.makedirs(src_dir, exist_ok=True)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, create the script file in the source directory. If you want to use Intel® Extension for Scikit-Learn optimizations as part of this script, take a look at the alternative script file found at the end of this section."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "name": "create_script_file"
   },
   "outputs": [],
   "source": [
    "%%writefile {src_dir}/train_iris.py\n",
    "# Modified from https://www.geeksforgeeks.org/multiclass-classification-using-scikit-learn/\n",
    "\n",
    "import argparse\n",
    "import os\n",
    "\n",
    "# importing necessary libraries\n",
    "import numpy as np\n",
    "\n",
    "from sklearn import datasets\n",
    "from sklearn.metrics import confusion_matrix\n",
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "import joblib\n",
    "\n",
    "import mlflow\n",
    "import mlflow.sklearn\n",
    "\n",
    "def main():\n",
    "    parser = argparse.ArgumentParser()\n",
    "\n",
    "    parser.add_argument('--kernel', type=str, default='linear',\n",
    "                        help='Kernel type to be used in the algorithm')\n",
    "    parser.add_argument('--penalty', type=float, default=1.0,\n",
    "                        help='Penalty parameter of the error term')\n",
    "\n",
    "    # Start Logging\n",
    "    mlflow.start_run()\n",
    "\n",
    "    # enable autologging\n",
    "    mlflow.sklearn.autolog()\n",
    "\n",
    "    args = parser.parse_args()\n",
    "    mlflow.log_param('Kernel type', str(args.kernel))\n",
    "    mlflow.log_metric('Penalty', float(args.penalty))\n",
    "\n",
    "    # loading the iris dataset\n",
    "    iris = datasets.load_iris()\n",
    "\n",
    "    # X -> features, y -> label\n",
    "    X = iris.data\n",
    "    y = iris.target\n",
    "\n",
    "    # dividing X, y into train and test data\n",
    "    X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=0)\n",
    "\n",
    "    # training a linear SVM classifier\n",
    "    from sklearn.svm import SVC\n",
    "    svm_model_linear = SVC(kernel=args.kernel, C=args.penalty)\n",
    "    svm_model_linear = svm_model_linear.fit(X_train, y_train)\n",
    "    svm_predictions = svm_model_linear.predict(X_test)\n",
    "\n",
    "    # model accuracy for X_test\n",
    "    accuracy = svm_model_linear.score(X_test, y_test)\n",
    "    print('Accuracy of SVM classifier on test set: {:.2f}'.format(accuracy))\n",
    "    mlflow.log_metric('Accuracy', float(accuracy))\n",
    "    # creating a confusion matrix\n",
    "    cm = confusion_matrix(y_test, svm_predictions)\n",
    "    print(cm)\n",
    "\n",
    "    registered_model_name=\"sklearn-iris-flower-classify-model\"\n",
    "\n",
    "    ##########################\n",
    "    #<save and register model>\n",
    "    ##########################\n",
    "    # Registering the model to the workspace\n",
    "    print(\"Registering the model via MLFlow\")\n",
    "    mlflow.sklearn.log_model(\n",
    "        sk_model=svm_model_linear,\n",
    "        registered_model_name=registered_model_name,\n",
    "        artifact_path=registered_model_name\n",
    "    )\n",
    "\n",
    "    # # Saving the model to a file\n",
    "    print(\"Saving the model via MLFlow\")\n",
    "    mlflow.sklearn.save_model(\n",
    "        sk_model=svm_model_linear,\n",
    "        path=os.path.join(registered_model_name, \"trained_model\"),\n",
    "    )\n",
    "    ###########################\n",
    "    #</save and register model>\n",
    "    ###########################\n",
    "    mlflow.end_run()\n",
    "\n",
    "if __name__ == '__main__':\n",
    "    main()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### **[Optional]** Enable Intel® Extension for Scikit-Learn optimizations for more performance on Intel hardware**"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you have installed Intel® Extension for Scikit-Learn (as demonstrated in the previous section), you can enable the performance optimizations by adding the two lines of code to the top of the script file, as shown below.\n",
    "\n",
    "To learn more about Intel® Extension for Scikit-Learn, visit the package's [documentation](https://intel.github.io/scikit-learn-intelex/)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "name": "create_sklearnex_script_file"
   },
   "outputs": [],
   "source": [
    "%%writefile {src_dir}/train_iris.py\n",
    "# Modified from https://www.geeksforgeeks.org/multiclass-classification-using-scikit-learn/\n",
    "\n",
    "import argparse\n",
    "import os\n",
    "\n",
    "# Import and enable Intel Extension for Scikit-learn optimizations\n",
    "# where possible\n",
    "\n",
    "from sklearnex import patch_sklearn\n",
    "patch_sklearn()\n",
    "\n",
    "# importing necessary libraries\n",
    "import numpy as np\n",
    "\n",
    "\n",
    "from sklearn import datasets\n",
    "from sklearn.metrics import confusion_matrix\n",
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "import joblib\n",
    "\n",
    "import mlflow\n",
    "import mlflow.sklearn\n",
    "\n",
    "def main():\n",
    "    parser = argparse.ArgumentParser()\n",
    "\n",
    "    parser.add_argument('--kernel', type=str, default='linear',\n",
    "                        help='Kernel type to be used in the algorithm')\n",
    "    parser.add_argument('--penalty', type=float, default=1.0,\n",
    "                        help='Penalty parameter of the error term')\n",
    "\n",
    "    # Start Logging\n",
    "    mlflow.start_run()\n",
    "\n",
    "    # enable autologging\n",
    "    mlflow.sklearn.autolog()\n",
    "\n",
    "    args = parser.parse_args()\n",
    "    mlflow.log_param('Kernel type', str(args.kernel))\n",
    "    mlflow.log_metric('Penalty', float(args.penalty))\n",
    "\n",
    "    # loading the iris dataset\n",
    "    iris = datasets.load_iris()\n",
    "\n",
    "    # X -> features, y -> label\n",
    "    X = iris.data\n",
    "    y = iris.target\n",
    "\n",
    "    # dividing X, y into train and test data\n",
    "    X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=0)\n",
    "\n",
    "    # training a linear SVM classifier\n",
    "    from sklearn.svm import SVC\n",
    "    svm_model_linear = SVC(kernel=args.kernel, C=args.penalty)\n",
    "    svm_model_linear = svm_model_linear.fit(X_train, y_train)\n",
    "    svm_predictions = svm_model_linear.predict(X_test)\n",
    "\n",
    "    # model accuracy for X_test\n",
    "    accuracy = svm_model_linear.score(X_test, y_test)\n",
    "    print('Accuracy of SVM classifier on test set: {:.2f}'.format(accuracy))\n",
    "    mlflow.log_metric('Accuracy', float(accuracy))\n",
    "    # creating a confusion matrix\n",
    "    cm = confusion_matrix(y_test, svm_predictions)\n",
    "    print(cm)\n",
    "\n",
    "    registered_model_name=\"sklearn-iris-flower-classify-model\"\n",
    "\n",
    "    ##########################\n",
    "    #<save and register model>\n",
    "    ##########################\n",
    "    # Registering the model to the workspace\n",
    "    print(\"Registering the model via MLFlow\")\n",
    "    mlflow.sklearn.log_model(\n",
    "        sk_model=svm_model_linear,\n",
    "        registered_model_name=registered_model_name,\n",
    "        artifact_path=registered_model_name\n",
    "    )\n",
    "\n",
    "    # # Saving the model to a file\n",
    "    print(\"Saving the model via MLFlow\")\n",
    "    mlflow.sklearn.save_model(\n",
    "        sk_model=svm_model_linear,\n",
    "        path=os.path.join(registered_model_name, \"trained_model\"),\n",
    "    )\n",
    "    ###########################\n",
    "    #</save and register model>\n",
    "    ###########################\n",
    "    mlflow.end_run()\n",
    "\n",
    "if __name__ == '__main__':\n",
    "    main()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Configure the Command\n",
    "Now that you have a script that can perform the desired tasks, you'll use the general purpose **command** to run this script.  \n",
    "\n",
    "* The inputs used in this command are number of epochs, learning rate, momentum and output directory\n",
    "* Use the compute created earlier to run this command.\n",
    "* Use the custome environment `sklearn-env` which was created earlier. \n",
    "* Configure the command line action itself - in this case, the command is `python train_iris.py`. You can access the inputs/outputs in the command via the `${{ ... }}` notation.\n",
    "* Configure some metadata like display name, experiment name etc. An experiment is a container for all the iterations one does on a certain project. All the jobs submitted under the same experiment name would be listed next to each other in Azure ML studio."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "name": "job"
   },
   "outputs": [],
   "source": [
    "from azure.ai.ml import command\n",
    "from azure.ai.ml import Input\n",
    "\n",
    "job = command(\n",
    "    inputs=dict(kernel=\"linear\", penalty=1.0),\n",
    "    compute=cpu_compute_target,\n",
    "    environment=f\"{job_env.name}:{job_env.version}\",\n",
    "    code=\"./src/\",\n",
    "    command=\"python train_iris.py --kernel ${{inputs.kernel}} --penalty ${{inputs.penalty}}\",\n",
    "    experiment_name=\"sklearn-iris-flowers\",\n",
    "    display_name=\"sklearn-classify-iris-flower-images\",\n",
    ")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Submit the job \n",
    "\n",
    "It's now time to submit the job to run in AzureML. This time you'll use `create_or_update`  on `ml_client.jobs`.\n",
    "\n",
    "Once completed, the job will register a model in your workspace as a result of training. You can view the job in AzureML studio by clicking on the link in the output of the next cell."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "name": "create_job"
   },
   "outputs": [],
   "source": [
    "ml_client.jobs.create_or_update(job)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## What happens during job execution\n",
    "As the job is executed, it goes through the following stages:\n",
    "\n",
    "* *Preparing*: A docker image is created according to the environment defined. The image is uploaded to the workspace's container registry and cached for later runs. Logs are also streamed to the job history and can be viewed to monitor progress. If a curated environment is used, the cached image backing that curated environment will be used.\n",
    "* *Scaling*: The cluster attempts to scale up if the cluster requires more nodes to execute the run than are currently available.\n",
    "* *Running*: All scripts in the `src` folder are uploaded to the compute target, data stores are mounted or copied, and the script is executed. Outputs from stdout and the ./logs folder are streamed to the job history and can be used to monitor the job."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Tune model hyperparameters\n",
    "Now that we've seen how to do a simple Scikit-learn training run using the SDK, let's see if we can further improve the accuracy of our model. We can optimize our model's hyperparameters using Azure Machine Learning's sweep capabilities.\n",
    "\n",
    "You will replace some of the parameters passed to the training job with special inputs from the `azure.ml.sweep` package – that way, you are defining the parameter space in which to search.\n",
    "\n",
    "Let's tune the `kernel` and `penalty` parameters. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "name": "job_for_sweep"
   },
   "outputs": [],
   "source": [
    "from azure.ai.ml.sweep import Choice\n",
    "\n",
    "# we will reuse the command_job created before. we call it as a function so that we can apply inputs\n",
    "# we do not apply the 'iris_csv' input again -- we will just use what was already defined earlier\n",
    "job_for_sweep = job(\n",
    "    kernel=Choice(values=[\"linear\", \"rbf\", \"poly\", \"sigmoid\"]),\n",
    "    penalty=Choice(values=[0.5, 1, 1.5]),\n",
    ")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Then you configure sweep on the command job, with some sweep-specific parameters like the primary metric to watch and the sampling algorithm to use.\n",
    "\n",
    "In this example we will use random sampling to try different configuration sets of hyperparameters to maximize our primary metric, `Accuracy`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "name": "sweep_job"
   },
   "outputs": [],
   "source": [
    "sweep_job = job_for_sweep.sweep(\n",
    "    compute=\"cpu-cluster\",\n",
    "    sampling_algorithm=\"random\",\n",
    "    primary_metric=\"Accuracy\",\n",
    "    goal=\"Maximize\",\n",
    "    max_total_trials=12,\n",
    "    max_concurrent_trials=4,\n",
    ")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now you can submit this job as before. This will now run a sweep job that sweeps over our train job."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "name": "create_sweep_job"
   },
   "outputs": [],
   "source": [
    "returned_sweep_job = ml_client.create_or_update(sweep_job)\n",
    "\n",
    "# stream the output and wait until the job is finished\n",
    "ml_client.jobs.stream(returned_sweep_job.name)\n",
    "\n",
    "# refresh the latest status of the job after streaming\n",
    "returned_sweep_job = ml_client.jobs.get(name=returned_sweep_job.name)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can monitor the job using the studio UI link presented when you run the job.\n",
    "\n",
    "## Find and register the best model\n",
    "Once **all the runs complete**, you can find the run that produced the model with the highest accuracy."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "name": "model"
   },
   "outputs": [],
   "source": [
    "from azure.ai.ml.entities import Model\n",
    "\n",
    "if returned_sweep_job.status == \"Completed\":\n",
    "\n",
    "    # First let us get the run which gave us the best result\n",
    "    best_run = returned_sweep_job.properties[\"best_child_run_id\"]\n",
    "\n",
    "    # lets get the model from this run\n",
    "    model = Model(\n",
    "        # the script stores the model as \"sklearn-iris-flower-classify-model\"\n",
    "        path=\"azureml://jobs/{}/outputs/artifacts/paths/sklearn-iris-flower-classify-model/\".format(\n",
    "            best_run\n",
    "        ),\n",
    "        name=\"run-model-example\",\n",
    "        description=\"Model created from run.\",\n",
    "        type=\"custom_model\",\n",
    "    )\n",
    "\n",
    "else:\n",
    "    print(\n",
    "        \"Sweep job status: {}. Please wait until it completes\".format(\n",
    "            returned_sweep_job.status\n",
    "        )\n",
    "    )"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can now register this model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "name": "register_model"
   },
   "outputs": [],
   "source": [
    "registered_model = ml_client.models.create_or_update(model=model)"
   ]
  }
 ],
 "metadata": {
  "description": {
   "description": "Train and tune a machine learning model using scikit-learn training scripts to build a to classify iris flower images."
  },
  "kernelspec": {
   "display_name": "Python 3.10 - SDK V2",
   "language": "python",
   "name": "python310-sdkv2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.12"
  },
  "orig_nbformat": 4,
  "vscode": {
   "interpreter": {
    "hash": "27e3d0e72e4fca658b4cea21737d79da5e68f90d3ccf7f33207fcc73892eee38"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}