workshop_minitorch.ipynb

{
 "cells": [
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Minitorch workshop\n",
    "\n",
    "#### Warning\n",
    "This workshop does not require any prior knowledge of pytorch to follow it. It's actualy a great introduction to Pytorch and Autograds.\n",
    "\n",
    "However it does require some knowledge of machine learning and a good base in **python**.\n",
    "\n",
    "### Summary\n",
    "\n",
    "* Tensors\n",
    "* Reminders about ML and numeric differentiation\n",
    "* Automatic Differentiation\n",
    "* Pytorch architecture\n",
    "* Base Layers\n",
    "* Optimizer\n",
    "* Loss functions\n",
    "* Make a model !\n",
    "\n",
    "#### PreInstall\n",
    "\n",
    "It is advised to setup a virtual env before using a jupyter notebook. To do so use this command in your terminal:\n",
    "```bash\n",
    "python3 -m venv .venv\n",
    "```\n",
    "After this, select it as your VSCode Env using `Ctrl+P` then searching \"python env\" and creating a new env and selecting the path to the bin in the newly created `.venv` folder.\n",
    "\n",
    "#### Dependencies\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 490,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requirement already satisfied: numpy in ./.venv/lib/python3.10/site-packages (1.26.2)\n",
      "Requirement already satisfied: torch in ./.venv/lib/python3.10/site-packages (2.1.1)\n",
      "Requirement already satisfied: filelock in ./.venv/lib/python3.10/site-packages (from torch) (3.13.1)\n",
      "Requirement already satisfied: typing-extensions in ./.venv/lib/python3.10/site-packages (from torch) (4.9.0)\n",
      "Requirement already satisfied: sympy in ./.venv/lib/python3.10/site-packages (from torch) (1.12)\n",
      "Requirement already satisfied: networkx in ./.venv/lib/python3.10/site-packages (from torch) (3.2.1)\n",
      "Requirement already satisfied: jinja2 in ./.venv/lib/python3.10/site-packages (from torch) (3.1.2)\n",
      "Requirement already satisfied: fsspec in ./.venv/lib/python3.10/site-packages (from torch) (2023.12.2)\n",
      "Requirement already satisfied: nvidia-cuda-nvrtc-cu12==12.1.105 in ./.venv/lib/python3.10/site-packages (from torch) (12.1.105)\n",
      "Requirement already satisfied: nvidia-cuda-runtime-cu12==12.1.105 in ./.venv/lib/python3.10/site-packages (from torch) (12.1.105)\n",
      "Requirement already satisfied: nvidia-cuda-cupti-cu12==12.1.105 in ./.venv/lib/python3.10/site-packages (from torch) (12.1.105)\n",
      "Requirement already satisfied: nvidia-cudnn-cu12==8.9.2.26 in ./.venv/lib/python3.10/site-packages (from torch) (8.9.2.26)\n",
      "Requirement already satisfied: nvidia-cublas-cu12==12.1.3.1 in ./.venv/lib/python3.10/site-packages (from torch) (12.1.3.1)\n",
      "Requirement already satisfied: nvidia-cufft-cu12==11.0.2.54 in ./.venv/lib/python3.10/site-packages (from torch) (11.0.2.54)\n",
      "Requirement already satisfied: nvidia-curand-cu12==10.3.2.106 in ./.venv/lib/python3.10/site-packages (from torch) (10.3.2.106)\n",
      "Requirement already satisfied: nvidia-cusolver-cu12==11.4.5.107 in ./.venv/lib/python3.10/site-packages (from torch) (11.4.5.107)\n",
      "Requirement already satisfied: nvidia-cusparse-cu12==12.1.0.106 in ./.venv/lib/python3.10/site-packages (from torch) (12.1.0.106)\n",
      "Requirement already satisfied: nvidia-nccl-cu12==2.18.1 in ./.venv/lib/python3.10/site-packages (from torch) (2.18.1)\n",
      "Requirement already satisfied: nvidia-nvtx-cu12==12.1.105 in ./.venv/lib/python3.10/site-packages (from torch) (12.1.105)\n",
      "Requirement already satisfied: triton==2.1.0 in ./.venv/lib/python3.10/site-packages (from torch) (2.1.0)\n",
      "Requirement already satisfied: nvidia-nvjitlink-cu12 in ./.venv/lib/python3.10/site-packages (from nvidia-cusolver-cu12==11.4.5.107->torch) (12.3.101)\n",
      "Requirement already satisfied: MarkupSafe>=2.0 in ./.venv/lib/python3.10/site-packages (from jinja2->torch) (2.1.3)\n",
      "Requirement already satisfied: mpmath>=0.19 in ./.venv/lib/python3.10/site-packages (from sympy->torch) (1.3.0)\n",
      "Requirement already satisfied: torchvision in ./.venv/lib/python3.10/site-packages (0.16.1)\n",
      "Requirement already satisfied: numpy in ./.venv/lib/python3.10/site-packages (from torchvision) (1.26.2)\n",
      "Requirement already satisfied: requests in ./.venv/lib/python3.10/site-packages (from torchvision) (2.31.0)\n",
      "Requirement already satisfied: torch==2.1.1 in ./.venv/lib/python3.10/site-packages (from torchvision) (2.1.1)\n",
      "Requirement already satisfied: pillow!=8.3.*,>=5.3.0 in ./.venv/lib/python3.10/site-packages (from torchvision) (10.1.0)\n",
      "Requirement already satisfied: filelock in ./.venv/lib/python3.10/site-packages (from torch==2.1.1->torchvision) (3.13.1)\n",
      "Requirement already satisfied: typing-extensions in ./.venv/lib/python3.10/site-packages (from torch==2.1.1->torchvision) (4.9.0)\n",
      "Requirement already satisfied: sympy in ./.venv/lib/python3.10/site-packages (from torch==2.1.1->torchvision) (1.12)\n",
      "Requirement already satisfied: networkx in ./.venv/lib/python3.10/site-packages (from torch==2.1.1->torchvision) (3.2.1)\n",
      "Requirement already satisfied: jinja2 in ./.venv/lib/python3.10/site-packages (from torch==2.1.1->torchvision) (3.1.2)\n",
      "Requirement already satisfied: fsspec in ./.venv/lib/python3.10/site-packages (from torch==2.1.1->torchvision) (2023.12.2)\n",
      "Requirement already satisfied: nvidia-cuda-nvrtc-cu12==12.1.105 in ./.venv/lib/python3.10/site-packages (from torch==2.1.1->torchvision) (12.1.105)\n",
      "Requirement already satisfied: nvidia-cuda-runtime-cu12==12.1.105 in ./.venv/lib/python3.10/site-packages (from torch==2.1.1->torchvision) (12.1.105)\n",
      "Requirement already satisfied: nvidia-cuda-cupti-cu12==12.1.105 in ./.venv/lib/python3.10/site-packages (from torch==2.1.1->torchvision) (12.1.105)\n",
      "Requirement already satisfied: nvidia-cudnn-cu12==8.9.2.26 in ./.venv/lib/python3.10/site-packages (from torch==2.1.1->torchvision) (8.9.2.26)\n",
      "Requirement already satisfied: nvidia-cublas-cu12==12.1.3.1 in ./.venv/lib/python3.10/site-packages (from torch==2.1.1->torchvision) (12.1.3.1)\n",
      "Requirement already satisfied: nvidia-cufft-cu12==11.0.2.54 in ./.venv/lib/python3.10/site-packages (from torch==2.1.1->torchvision) (11.0.2.54)\n",
      "Requirement already satisfied: nvidia-curand-cu12==10.3.2.106 in ./.venv/lib/python3.10/site-packages (from torch==2.1.1->torchvision) (10.3.2.106)\n",
      "Requirement already satisfied: nvidia-cusolver-cu12==11.4.5.107 in ./.venv/lib/python3.10/site-packages (from torch==2.1.1->torchvision) (11.4.5.107)\n",
      "Requirement already satisfied: nvidia-cusparse-cu12==12.1.0.106 in ./.venv/lib/python3.10/site-packages (from torch==2.1.1->torchvision) (12.1.0.106)\n",
      "Requirement already satisfied: nvidia-nccl-cu12==2.18.1 in ./.venv/lib/python3.10/site-packages (from torch==2.1.1->torchvision) (2.18.1)\n",
      "Requirement already satisfied: nvidia-nvtx-cu12==12.1.105 in ./.venv/lib/python3.10/site-packages (from torch==2.1.1->torchvision) (12.1.105)\n",
      "Requirement already satisfied: triton==2.1.0 in ./.venv/lib/python3.10/site-packages (from torch==2.1.1->torchvision) (2.1.0)\n",
      "Requirement already satisfied: nvidia-nvjitlink-cu12 in ./.venv/lib/python3.10/site-packages (from nvidia-cusolver-cu12==11.4.5.107->torch==2.1.1->torchvision) (12.3.101)\n",
      "Requirement already satisfied: charset-normalizer<4,>=2 in ./.venv/lib/python3.10/site-packages (from requests->torchvision) (3.3.2)\n",
      "Requirement already satisfied: idna<4,>=2.5 in ./.venv/lib/python3.10/site-packages (from requests->torchvision) (3.6)\n",
      "Requirement already satisfied: urllib3<3,>=1.21.1 in ./.venv/lib/python3.10/site-packages (from requests->torchvision) (2.1.0)\n",
      "Requirement already satisfied: certifi>=2017.4.17 in ./.venv/lib/python3.10/site-packages (from requests->torchvision) (2023.11.17)\n",
      "Requirement already satisfied: MarkupSafe>=2.0 in ./.venv/lib/python3.10/site-packages (from jinja2->torch==2.1.1->torchvision) (2.1.3)\n",
      "Requirement already satisfied: mpmath>=0.19 in ./.venv/lib/python3.10/site-packages (from sympy->torch==2.1.1->torchvision) (1.3.0)\n"
     ]
    }
   ],
   "source": [
    "import sys\n",
    "!{sys.executable} -m pip install numpy\n",
    "!{sys.executable} -m pip install torch\n",
    "!{sys.executable} -m pip install torchvision\n",
    "\n",
    "import torch as t\n",
    "import numpy as np\n",
    "import torchvision as tv\n",
    "import tests"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": []
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Tensors\n",
    "\n",
    "Tensors are a form of list of information that is kept in one dimension in memory. A tensors may have different types, shapes and strides.\n",
    "\n",
    "#### Types\n",
    "\n",
    "The type of tensor impacts the capacity of storage of each element but also its performance when computing operations.\n",
    "Its usually a tradeoff that people forget to think about. In fact, there are operations that can calculate 4 `float32` (32 bits each) at the same time when we can only compute 2 `float64` (64 bits each) at the same time.\n",
    "\n",
    "So the type is very important even if it doesn't always look like it.\n",
    "\n",
    "#### Shape\n",
    "\n",
    "A tensor's data is kept in one dimension but the shape has an importance in representing it in another dimension.\n",
    "The shape is used to perform operations without having to keep a multidimensionnal list.\n",
    "Its a smart way of making the memory easily use cache efficiently.\n",
    "\n",
    "#### Strides\n",
    "\n",
    "A stride is like a step in a dimension. The strides are each step in the 1 dimensional list that is a tensor.\n",
    "\n",
    "For example:\n",
    "```python\n",
    "import torch\n",
    "a = torch.tensor([[3, 4], [2, 5]])\n",
    "a.strides -> (2, 1)\n",
    "```\n",
    "It's great to easily manipulate the content on an array without really even touching it.\n",
    "\n",
    "#### In Pytorch\n",
    "\n",
    "To create a tensor, just use the tensor method\n",
    "```python\n",
    "import torch\n",
    "a = torch.tensor([4, 2, 4, 5])\n",
    "```\n",
    "\n",
    "A type can be precised:\n",
    "```python\n",
    "import torch\n",
    "b = torch.tensor([4, 2, 4, 5], dtype=torch.float32)\n",
    "```\n",
    "\n",
    "A list of methods to create tensors exists in [pytorch docs](https://pytorch.org/docs/stable/tensors.html)."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Reminder about ML and numeric differentiation\n",
    "\n",
    "#### Basics\n",
    "\n",
    "The goal of neural networks is to find a solution to a problem as subequations working together in a bigger equation.\\\n",
    "For example, a linear neural network is an acyclic graph of linear transformations.\\\n",
    "As such, each linear transformation is easily differentiable. By making use of the chain rule we can therefore create an auto differentiation tool.\n",
    "\n",
    "#### Reminder about Numeric differentiation\n",
    "\n",
    "Differentiation is the idea of finding the movement of a funtion by calculating said movement at a minuscule step. Therefore the formula for calculating it is:\n",
    "\n",
    "$ f'(x) = lim_{h->0}\\frac{f(x + h) - f(x)}{h}$\n",
    "\n",
    "Where h is the very small substep.\n",
    "The absolute differentiation of some functions are already available in differentiation tables that can be found on the internet."
   ]
  },
  {
   "attachments": {
    "image.png": {
     "image/png": "iVBORw0KGgoAAAANSUhEUgAABEoAAAEkCAYAAAA4gNRtAAAABHNCSVQICAgIfAhkiAAAIABJREFUeF7snQm8VdP+wFdKJRFC3SKhicJteIjKEFLKnOEZmvBE4uEZGjxEMv49MqXx4RkylzKLkgzVzVA0kIoGJEqTuP/929nn7rP32ufsc84+5+yzz3f1uZ/uWXsNv/Vd66y79m/91m9VKjeCIkAghwRWrFgRq836vW7durE4++85FIuqIBBHQDdOJYE1PhmnDJioEbCPeWmb87NzzDs/R40H7Sk+Arp5nzm/+MZBmFusG6MiL+M0zL2GbIVKoEqhCo7c4SdQVlamVi5fbi62Z86cqWYZP35DvXr1VKvWrc3kLVu2VHVKSlRpaanf7KSDgG8CMj7nzJ4dG6crjDH7/fff+8ov47SuMTbl/xLj/4OMsco49YWORDkmYI1zqVZ+X/7XOE9lvOtElrEvwfoeyO/Wd0F+5/ugo0ZcPgk453yRxe/6xBrv1vrk+C5dzKYwzvPZo9GsO9M1NGuTaI4LWpVbApWwKMkt8CjXJouP1yZPTlkp4peJpTxBceKXGOl0BKxF8mxDOTJxwgRdkozi7MoTWUSzgM4IJ5nTICDzsLXrmKqSOo3qEmaxv1iiTEyIiodZJGB9J16ZONG3IjwVcaz1CXN+KtRIayeQqzW01Mk4ZexBwB8BFCX+OJHKg4A1sWdr8eFRrRktOzq9L7iAF9FEkHgWIyAL5VdeecX3zmFQ6FhAB0WScnQEnLvjfnfGdWXlKs6uPGHBnivqxVePzPnZUognosmcn4gOz+wE8rmGZpwyFiGQnACKkuSMSKEhICaBr06alJUdeU11CaOsyb5n796xM5oJM/CwaAjIImTs6NGhGKcCHeVe0Qy9rDbUUo7kQ/GXrYaxaM8W2eIrd9yYMerRESNC0XBrXF83YEAo5EGIcBAI0xpaiLA2Cce4QIrwEUBREr4+CbVE6bx41q291VFryyYtVYnxe93aJWYb6+5WJ9bWFT+srPj9p+Vq1vzZ5uey+WWx+GS/sCBJRqi4nqezWJaxao1ToeUcq0GMUym3a7dumL4W13AMpLVBWkXJWJefkr/mY5mb7WPe/N02R8tn+/g3PxtztQRrvl7x0wojrsJZt/kwxYDCJEVgJI8RkO/HkJtvTomINedLJj/rk9nG2mS5Me5THesyrk/o2lX16NUrJflIHC0C6SpIdGuToNfQQpq1SbTGG63JnACKkswZFkUJlnmgn10a+4Teu1vmi4LlxsJ79ldlatIHk3wpTliQFMWQ9GxkKovlzm07m4vj0malqlXjlp5l+nlgH6d+FtFYQvmhSpp0F9ZCzq4MCWqc++kR+S4sX228TBoK8HRfLKUelCZ+aJNGviNDhwzx5XtEvhNdjHk/yDlfeiCV9Ukf48hwp86d6bgiIpDKJmM219CsTYpo0NHUQAigKAkEY7QL8bMzb03sohixdiazQSUVpYksslmQZKMXwlmmKEhGjRyZdLEsyhGxGuly2PFZbYg1VoeOG5qwHiyhEuIpyofy4ic3MflRTNsBWfNwLsZ3Oh1jfScs5Uk6FoP4NEmHfDTz+FWQlDYxFOHGnB/Exk0ikqmuTwYMHoyPtURAI/IsTGtoQcraJCIDi2bkhACKkpxgLsxKZBHSr2/fhMLLAqRL2y5Zf+nUCWFN9qMnjk5o7i0vovc98AD+S3QQIxAn43S0oSBJ5MTS2kXM9kJZh9Pv4hnFno5eccX5VfZZVCzFSOfDj8/YIiofpP1+N5yy8V1xEim+z/379Us45wuR3l17ZV054kXe78uo+IYYMGgQ6xMvkAUcXyhr6GTWUMy3BTwIET1jAihKMkYYzQKSHV+QBXrvrr3zoiBxEvezIGGid1KLxmc/C5F8LpadlCdNfzWhibaMU86xO6lF/3OqCpJcWUXlknw6ShPm9Vz2UDjq8mNFIt+PgT2vD4XAMq4nT5+sRk8c4ymPjGOsSzzxFOSDYUOHJnQiH6Y1tABOtjaRNBdedBE+dgpyNCJ0JgRQlGRCL6J5E+3U5HNnPhnuZAsSXkKTESys58mUJGFaLNvJ+rGEEodq3JJQWOMxHWlTUZDk6vhAOu0IOk+qShMUJkH3QDjLSzbny3dkYM8BWT3+my4ZGdOjJ4xRkz+YrC2CMazFUpCRydbQYdlkdML1szYRC6j7hg93ZuUzBCJLAEVJZLs2vYYlmuDDtDOfqHXJFiS8hCaiVxjPEo3TMC+W7XStRYmXDxNMsgtjLKYjpZ/jYlJumBXT6bQ7nTypKE244jIdwoWRJ5Gfh0KZ84W0jOdbxw71dEzPrn1hjEedlMnm9UJaQ8sFCl5rE1HqcZxdNwKIiyIBFCVR7NU02pRop6aQFiH2ps9aMFsNHXOb1n8JL6FpDJKQZEmkJLnv6v8UnK+GRAtnTLJDMugCFCOZSbZUJdZQ2XaMHWCTclZUou+KJYR8Z2R+xyIrZ92S9YqiNucLMDnq4PUiyq591odU4BUkW0MPv+q+wOvMdoGJNh2xgMo2fcoPC4HKNxohLMIgR34I+Jngd6hRMz/CZVBrSe0S1b60vZpaNlWt27AurqTly5eraVOnqvZHHKFq1iy8tmWApWCzyjg9/ZRTlPSdM4gy735jIdJkj8bOR6H/LN+tlk1bqnXr16mFyxbGybt27VpVZtx+suOOO6pGjQuvbaGHn0MBrfE7f/58z1pFQXLbJUMN30+dVSHOuZ4NC+iBMBE2nY0f3fdFqpHvjDB+1bgFi+9NQODzVIxcqTrguus8nbYWomLcQtl4z0bmFcWyc69bn8j4ZX2Sp4GXYrVyhPK6a67R5pK1SSEqSaQxMt823rOx2mG7msY172Vx7ZN5duGCBWr9+vWqtGVLbduJhEAUCGBREoVezKANiZQkhWImmKz5ybTimBAmIxiO5+3attUKUsgLEWeD5Ay7zukfuzdOUoX1OdGOuLSkUK328t0LieZ2SzaOWua7l9Kv3+t7E6U5P5GVlMz7zzz3XPoAyZl1AonW0IWsyHOCS2QBxXExJy0+R4kAFiVR6s0U25Jsgu9i7G5GISTVii9cqDp36RKFpka2DbJg1lmSyA78MGMHPipBLEtkl3GyYZZtD9buDTuMhdXTMsf2v/RS08JBF+SFb0Cv61Uf4wYxLEh0hBLHCbMOhtWgfGeW/7hCe8zSsi5p3LQpV7Amxhmqp8WgJBHgSS0KjTmE9UmohmZMmGRr6FaNo2NpkcgCaoVh5Yv1XjjHKFJlTgBFSeYMC7aEobfc4nr5FOeBQy+9teD8PCTrBGsxUtc4jjN1ztS45PICLn/wWIwko5if514LZtmtObNj9/wIlcVa5ciYHC1wHhkTZYkcF+t+5plZrJ2igyJgLaKl33RBLPa23tBRontMXAoE5DsjSsbGxtE75/wuxVhH2DATTwFqHpN6zfkDegxQl595WR4ly07VlsJPSncecZD1yapVq1S79u2zUzmlpkXAS0lSyMeAk4GwjrMvWLogTinNRk4ycjwvZAIoSgq59zKQ3WshEkUliR2TpRV37tjLYqRy5cqctcxgTGUjq9x08MrEia6io2TS6mqcESELZ51/HfOFD6WeDlmo4mR+HfXoo1qZZCH97NBnzBd7QnAEtloONvL0XyLfnVkzZzLPB4c8KyXJ/Kb77sicL9ZDUQ4yJ+g2c8QqivVJuHpet9FoHQmLsnWgtenIRk64xiPSZI8AipLssQ1tyV5Kkqi/fFodIlpxCc6dG1lE169fH6eZIRm5smC+5eabXdLITnxUjoW5GmeLkAVJowaNXMdwUOolopb/Z17zq0hmWZHkX8roSpBod15aLfM8ysZw9n+iXXo5nlYMQTZzJDjXJxxvCE/ve83xcozSWl+GR9rgJbE2csa/NT6ucDZygmdNifkngKIk/32QUwkS7dZE6TxlMqiycyNO1Jy3jIgXb86yJ6OX/eeJFsxyXKFYgiy6dDuM7IyHcwR4LaCtI43FoOALS89Y/n68bhVBWRKWntoqR6I5v1BvDUmXcN1dS1y3OnG8IV2awebzmuOLZaPRoinKEp0/NY6KBTveKC3/BFCU5L8PciaBXLXX87zzXPWJuWCx7NbYGy/XnunOWsp1rPiBcA2TnEaIA0ynb4co3XSQCsxEO4wo9VIhmd20XgtoGbdjB48uip3G7BJOvXTrTL3uKmFZ0MsVrMz1qXPNRo5ERxmyUV+Yy9x6jMzwt1M2Ne7qYPmbiJ+q/PUcG43x7L02cuSoGNbZ+Run1BwsARQlwfIMdWkDrrvO5by1WF8+paMSnbXEeVr+hrL4JXl3yhSXAPdfdV/R3gzitcMoyk+cELuGSs4jEilJim03POfwk1RovXTqlCWYiieBl6PHXr6oiuUogw5zouMN+CvREct+nNcGTjFuNFq0vTZyxDobJXT2xyQ1ZJ/ANtmvghrCQEA04WKubw9iDl7si/gSg4EsxpxBWMlLKCH3BB4dMcJVqZi1Sl8Va5C29+7WyziGE8/A8rdQrFzC0G6UJGHohcQyyPdnYM/rTR8xziDfIelDQv4IeM35xXQcWEdfxq387XMGnYNzZxo+B0tAlHnff/99XKHFvNFoByG39HVu2zmOjbASZgQIFDoBFCWF3oM+5R89cqQrpU5B4EpUBBGyGHMuoGWSHzt6dBG0PlxNHDZ0qEsg6ZtiXzALFC+l3tAhQ1zMiMgNAS8liVxjWuxK6Nz0QGq1iLLROddLCaIs0c09qZVO6nQI6F6m5KWLOX8rTa/1CeM1ndGWfh6dMq+Y/KUlImdt5DjTiEKPDUcnFT4XGgEUJYXWY2nI+5pxDttpTSKacBYiFTBFI+4M7NY7iWT3s/xBnThhgqsSebkhbCUg31n57tqDKPXkO07ILQGdlZ5IIP3T5bDjcysMtfkm4KUskbmH75FvjIEl1L2AMufH4/Van/ASGtgwTFiQTiklCtditnJ1AjOVJQ6LPTYcnZT4XIgEUJQUYq+lKPMojTVJ7xN5+bRjNHfrjV1ge5BJ/tVJk1KkTfJ0CYgzP2dw9onzeTF+1u1i6b7jxcgmV22WF5R+ffu6qsMU24UklBHyIu40FRdB5XvEy2fuuowXUH+svdYnWL3645dJKjZw/NPzUujJpgIBAoVKAEVJofacT7l15yoxa9XDa9m01LVbL7uMTPJ6XkHG6nbnxR8HO/NuyrJodr7kcR7YzSmbMTqlHkqSbBIPvmydskS+R7q+Db52SuQFNLUxIOsTfFSlxiyI1Lr5IMobOL+t/01NL/tAPTbhcXX7qDvUyOdHqTdnvKU2/745KU4vhR4bjknRkSDEBFCUhLhzghANs1b/FGWS79K2iyuDzr+LKxERGRHQMcaHjjdSnWk6Dv68eQX5RHbBnUcZpXx8kgRJOftlWefqdS+fOkuH7EtUXDUU2wtopr279WhD77hiRLGn+9uZaV3k30qgWDZwysvL1RsfvKnOH9hL7dxuN3V4jyPU+YN6q+vuG6QuvKmvOvYfnVWTbvurV95LbmHNhiPfnqgRQFEStR61tUfnJI1zlYk7XCwYnD4g8FWSmFmmT2Vn0fniKS8v+NDxJut1HpgXPG9mQTzx2gXX3UwRRH2UkV0C5g6ox61nWBJmj73XnI8FYWLmupdQ1ieJmWXyVGcJoZsvMqkjDHl/XferOu7iLuqxiU+o3//Yov5x+gXqzRGvqY/+N131PeMiU8Rvly9Rp15xhlq4ZGFCkb02HHUsExbEQwiEhACKkpB0RDbE0O0w63ais1F3IZep89/CJJ+9HtWds47iYiRogl7ngYOuh/IqCOh2wbmVqbBHiNetIsz52etXndNc5vzkvHkJTc4oyBRO5/LFsIHTo9t56uHBD6qOhxyl/ta8jXpw4HDV7YgTTKyb/9isnnr1maSIvTYck2YkAQRCSABFSQg7JQiRZMdGzDLtwemROoh6oliG7mYRp8VDFNudrzY52RbDYiQI1rJodlo/yXeenfAg6LrL8DLDRvnsZlVoMaJ0dB7BwT9V9npRt4mDBaE/3ryE+uOUaSq9RXb80adM6whL/qrbVlUdWrUzf+68cphLrFOOPikWN3fhPNdzXYTzGDu38+koEVcIBFCUFEIvpSGjbpe+tFn8taJpFFs0WXSTPC+gwXe/7Cw6FXpd2rqvag6+5miUiPVT7vpR5wuAXfDc8c9mTTr/D1Kfrs+zKUcxlC1/R51zPps4qfU865PUeKWTeubMma5sUT0atl317dS7Y942f3bbZTdXu0t2LYnFbVPZ32ujHBNzBm7ncxLhcyEQqFIIQiJj6gTYpU+dmT2HbpIXU+zSUvfkn1lNxZ179uzZLgC6IyWuRESYBCzrp7L5FdfvOb/7oMqcgM6ngljzsAueOVspYfaXs9WsubPVhk0bjZ8NasPGDWqLcVb+5I4nqVbNWgZTSZJS5CVo0geTlPO7JC/2zPtJ4KXwWHekiTk/BYBGUtYnqfFKJ7Xz76jzprl0yizUPJ8t/DwmeosmLXw1w7J4tc+noiCVv6V169b1VQaJIBAGAihKwtALWZDBuWPDLn1qkHWTvPMPZ2olklpHwMlUXj6FPcE/AdlddC5GeLnzz89PSp2Fns6ax09ZpHETGPro7erZN593PTim7TGuuGxGSJ/2v+vyuCpQkAdL3Dnny5En5vzUGLM+SY1Xqql1PnRaNsmNwjZVWbOdfsuWLcYVwaPNaqpUrqzOP+Ec31U61yaSEUWJb3xZT/jBBx+ouXPnqg0bjM2Jv37++OMPde6556rGjRtnvf5CqQBFSaH0VApy6s5WcuwmBYB/JXVO8pb/B3YXU2fplcOp0GtVpIsRLz7Eh4NAMTr1yyX5/946Rl1+zmXq6AuONW9dkFB5m8rq0AMOzqUYWGjlgLZzzmcTJz3orE/S4+Ynl87SNarHbpLxuH3MnWr+twvMZHdfdaeqt3u9ZFliz7F88o0qLwn/9a9/qffff99V91lnneWKK+YIf4fNiplQAbZ9+fLlLqkxEXchSRpRd7c6SdOQIH0CKPTSZ2fP6bUYCaZ0StHtLvbuGk2nfvnqbTkj367V4ap963YxEY5o016Jk8FcB/w/ZI+4bs6vW7vC/0H2aqZkCPgn4LR6cjpN919SYaec8dmHavDwG81GnNPlbNX/nH4pNUjncN7JNqUCSRwogbfffluJVUmVKhU2E9WrV1fNmjULtJ5CLwxFSaH3oEZ+50RUrJO8Bk1KUSW7uBdwuvPVKRVK4hiBQlHo/fnnn6pyaXXzp8Wp4fNRw2Iku18q3e6iTjmVXSkqSv/g0xnq+v8MNH/enPFWrqrNST2Lv1scq+eI1h1yUqezEl3fMu87KaX3WTfnF+tOfXoEK3IxTjMl6J3fafVUjJauMhefcvnpqtz4d9TfjlCjbhrhDSzBE53iWY7fEPJPoGrVqurQQw+NO2bTvn17ValSpfwLFyIJUJSEqDOCEiVMk/yS5UvUih/TnxQXLlmoVv+yOig0KZWjewF1sk2pQBLHESgkhd6f5X8q+flpzU+h7EXnQo5xGlw36cZpPn0q3DPuXjVs9J3mz3ervguuoXkuSb5bX9sVJcbiPB9BN+87x0A+5IpCnU6ObOKk36u6ccq8nz5PK6fudsNis3qSNXfnS7qpFT+tVG32b6Ve+s/zqlrVamnB1VlmoyhJC2VWMslG4MKFC2Nlt2tXYdWZlQoLsFAUJQXYaYlEDtMkP3/xV+rQc9up9j2OUstWLksktvbZLOMmhDbdD1FH9zlG/fjzj9o02Y4scZgFr9Aca8q2DMVSvvNlv1jaHUQ7dQs53VwQRF3FVIYs6JwvH84dslzyWPvbWvXC2y/Fqmxb2jaX1We1rhmffhgrf5tK2+TcP4m9cc4+ljHA9yn47mfOz4wp65PM+Olyz9HcxKez3tHljULc5t83q1OuOF19aazfm+zVWL360Ctqh+13SLtpOsvslayjtTzLy8u18dmMnD9/vvr9999jVbRtG501RVDccOYaFMksljNs6FC19777qpNPPFFV2267lGvKxyQvX/ijL+qklv9lTXJk72PUlNFvqj3q7OFL/pnzZqmOPY9Vv2xcq+Ys+Fz1u+1y9dQdT/jKG2Qi8XQ++YPJsSJlwYzXbjfhr776So0eNUr16t3b9/lG5wuou1Ri/BLQ7dr4zUs6bwI6/yT5mE8tCUc8O1L98ecf5sdda+6imjSIjmf66WUzYh3RwfBVku4Opndv+n+i62Nzcc/18C6IP/30k6pdu7YrXhfBnK+jkn4c6xN/7C6+6CJ19tlnqyOOOippBt3xsHxaEH4yd6a69JbUfIN4NXLQRQNVtyO7ej024y+6qa96b9Y0VWeX3dWkByeo2jv5+257FapjJ8dZO3Xu7JWlaONvv+021bBhQ3XSKaeo7dJ410sHnPgosYeDD86tA/V0ZM51nrQVJVOmTFHW2e3jjjtONW/eXCv7ggULTGcxa9asUS1atFBHH320Nh2R3gR+NthNvPde9cR//6v+blzbdMrJJ3sqTHTacO+Ss/dEzrgNueRG1fvfF5mVLFr2tRJlyTuj3lB71t0zYcV2JYkkbGCkH3b5rQnz5PIhihI97fenTlXyI6Z7PQyFyX777adPaMTqdmd1VhGeBfAgjoBu10bmAm5oymyghGnRLPPizQ8OiTVon732VrMNqzt7qLptNdV83/2TNnrpiqXqM0MBPe/rL83zyHuWNFBt9mup9t5jbyWmuHId5A7b11RndjpDbbNNvOGpXBc5rWy6+vDzj1SXdserAxq1cNX36YLP1Ceff6LWrP1FNdu7qTqu7bFxDuNcGYyIDz6tWLAd2abi2M3yH5arqTOnqW9XLDEU7fVVx4OPVrvX3l1XRFbjWNzr8d5x++3KGDSqZ58+zPl6RDmNZX3ixr3IOFowcMAAta9x5WnvXr18KUysUvJ9POyH1T+oj76Y6W5UGjHLViY+qjny+VFq3ITHzBvHJgx/Ue27xz5xtYgsYnFS35iHUwnCsGx+WSwLClM9vTU//6yGT5igHjPe9c47//ycKEymT58eE6Zp06aqVq1aeuGKODZlRckvv/yiTjZe1EVRYg/y+Ygj4s8U33HHHeraa6+NJROTHnunFDH3tJq+evVqNfy++9QTjz2mzjnvvIQKE3sFOo1uWgKkmKnXyT3NCbfH4D5mTktZIpYlXsoS0Z537HWs+nXjOjOPTMhvG8qVhvUbplh7MMl1O4vBlBzdUqZNm6bkx4/CxE4B1umPCd13XPeSn34N5BQC+Vg0n3t9DzV/yQL1saF4sAdZPLc685C4uIMat1Blz87y7KxFSxepAfcNVs+8/qw2TdcOXdRSw6+UWPHJ3N392NPjFCV3jr1b3fzAELVu83oz/3X/N0AN6HONurX/Lebnb5Z9o/rf/k818b1JceW33q+VaVFYs0ZNbb2m8mVWxTWFHdp0UJs2b1LX3TtA3fe/4YZ/oAqT5F1q7KRefvBFdXjLw7RlBRFp+X9gcZ+cZiWjb6YZV0zKNZOHH354UoWJvUTm/OR8E6WAXyI67meLjI1bU2FiWGiL9euRmo1bpx8ddynRjJHj8X3/slz5z7X3qL81b+Nq6GlXdlcLl36tvn9rietZKhEcYU9MS96zh99/v6kwObdHD3XSSSepGjVqJM6U5tOpxuamFcSRq9/w22+/qRkzZqglS5aoH3/8UTVo0ECJomX//fdX4iTWb/juu+/U558bmzbz5plZdt55Z/NHbt5p0qSJr2KCKCNRRSkpSjZv3qw6deqkPvyw4iyxVfjAgQPNlyMr3G90sl1JIvHyR5SQOYGfRev4l8Lk74bCRBRXlpmW8+Wobu26mVeYQQnnn3ieqly5sjp3QE+zlK+/+yZmWdLA2MW0h4+/+EQd0/u4mJKk3u4lasrIN1xa7QzECSQrJtj+MFoKE93imTOq/himkkq+6yt+St9xcip1FXLaQjoqsObXNeqpV5+JHbdJxv3gA+IVJ/b0L70zwTx7LrcYSGjWsKk67ZhT1V719lTvGRYbj7/yvzgFR3vjul7r2sA//vhD9TGsA2W30RlGvTBO3XzpTeqLr+eqTv/oYjoAdAaxhrl73P+pf/cd7Hxkfv504Wfq9z+2mL9vY1i4NGnQSHW88Dj1flm8WbA8X71+jXkUc/YzH5vpCeEhIMoSS2EiVoWyaLYCc35u+on1SXLOixYtUoOMd5ZEChOrFKcfmOSlF2aKGw1rxS3GPC/3ncjR+QeeeijWkI2bNhoWfzPU1NnTVeMG+6bcQGFYZvwjpEZAFCYPGO96j48blxWFiZQvx+atcNhhyTcfxBpo8ODB6r+GEkc2OJxh7733VuMMeZMpXd5880111llnKVmPeQVRmIgewalLsNIHUYZX3fb4lBQlogwRJUmjRo3U66+/roYPH67uueceszz54yjHa3baaSc1efJk1b9/fzN+hx12UGvXrjV/9+oE0QY9N368mYbgJvCNManrgihM5Ev0P8PC5O/nnKNOPvVUXbK8x51zwt9VlW2qqLOvO9dcoouy5CjDQes7o95UlrJElCRH9zwmtlNZt3Yd9c6jb6hGxoI5bOHVV1+Nm1zCJl8+5PnZsHbyCokWz155iIdAtgicY/xx7nT88aZV3u67Jz7C4dz5yvWieacdd1JbZm8wUTz8zAjV99aKs+q/Tv/Jt5O9SVMnxylJ+p5xkbrn6rtU9WrVzbIvPO0C9fnCL1TZV3Ni2A8vrdjY6Hdbf1NJUr1KVTV26Fi1o3Esp8ulJ5ppV65eaZ5pP71/d1OJ0aPbeeofZ1yoFi1ZpM4b2CtW3vTPKnyQxCL/+uUDm3+SNi3aqDOuPltNNxbmLZuVqsHGufraO9dWfYdcquZ+vXXXSeRcv2G9qrFddnbZRCzn4t45Fpxt4HMFAWvOP9RYePc2juTYFSZwyi4B1iduvnZnlfandoVJT+NIzlEdO7ozF0HMF4vmqlEvjjFbKmv0y4Zd4dnqWjUzP5rx66+/qvsMdwKEeAKLvv5ai8SaPROxAAAgAElEQVSuMJF1y8mGD5MgLEycRg/JHLm+9NJLpnJj48aNppyNjaNsp512mtpll13U888/b1qYfPPNN6pDhw5KLFW8btAZNmyYuv7662NtPeaYY1TXrl1Np7JjxoxRc+fONZ/JO660XReCKENXri4uJUWJQKlZs6aaOHGiEq2RKEMsRYkULn8cDzroINW9e3ezrgMOOMA8oiMaq6VLl3r6J/nhhx/UM08/rZOPOB8ETIWJobT63xNPuL48+bYoscQ/s/MZqpJhWXLWv87+S1my2FSWvG1YjKxcvco8bmOZc9fZxVCSjH5DNWnoz+zKB6K0k+iONHz80UdKfgipEbAWz22NxbPuBVXHOrUaiju106KEc8D68SDWEc89+6ya8PLLqmu3bupc4yywbjzqcudzjL5v+AWxgigQ/N5EINezn3J595glyWnHnKIeHDjc1bwu7TrFK0paVni/P/qQjuq5V59XLz/0onEbzSFqse0aX9mBvMyw8BAlyaALr1dD+t1klt32wEPVbaPuiCk31qzxVqZOt1mOfPTZVkuRtgcebNy4MEntWHNHs7xTjj45VpZ8/vnXn7OsKIm3xpR1Sg9jrBDiCSw3Nrq8wgzj/Lv8iMKkjkYpmc/vk5fMhR7P+iT1HhSFyeBBg9Texkua8+9mMYzR16e/4RtaOooSJ8N169bxzuebeEVCURo8aLzrWe4XMlWY2B25Vq9e3Tw64xVee+018/SCFS688EIlJ0eqVdt6bfSVV16punTpYhpRSLj44ovVnDlzzBMF9iB9P8A4/maFjz/+WLVpU3HMS8oR3YGlLDn00EPj8suHIMpwFZogIiVFyUWG52j5scJee+1lgrVMd8TZmWh55OxS/fr1lZjFiKZJtFTJNFUJZOSRTwKiMJGfsIYzjjvNsCx5Wp1+1ZkxZcnBZ7dVv677VW3cstkUe/eddlVvjXzNMAtvFtZmIFeGBD6wOY/KsCiyQyBtAnKU9PnnnlMTDedpXgoT56I57coCyGhXJhx2kHvx4FXF4OE3qs1/bJ1fxVJv1I0jtElrG3OvPRx2UIWipPuxp6rTDQWLOH6VsNDwdWIF2YGUHcnL/94vpiSxnu2xe72YcmM34xYFrzDDsB6xhxaNmqvXHp4cpwwq2S1ecbHbLrt5FZeVeNntEh8HhNQJiLKEEDwB5wto8DUUV4le1ttRp/DP8y5X8kMoDAJ2hclAQ8F3mHGBQjrB7i5DrECsv+/OssRBtFh8WEEuZRkxIn4dIQqRa665JqYo+eKLL5RcPey81OGNN94wj3ZJOPDAA+OUJBInzuOvvvpq1ds4uilBdwtPEGWYhfsMKSlKdGUKMEtRIueWJGy77bYmLL+7dLpyiUuPgJhjrV+/1dGelBA2nwWnGovtF+59LmYGvmrNj7GGynWXb4563detDenRIVcYCIgGurRVK/Wh41qyfMh2+9i71MjxoxJWbfl02Pp9Wqkan+B9m49V0JXnX676nnlxwnKDfhi273rQ7ctWeZbCRCxMTjCuYD/PMG2tU6eOWV29evXidhiX58kHjFiFyJFFK/h1ZDp/8Vdq5AujY/kuMcZkrR30ptMLly6MpTug0QGudPZFlNyWYw/i72TYFUPj4uTDIsO5qxVE+aELcqvN1zYLFUkzZshIl8WMOKK1QpO9Gquq2/p3GKerN1lcvvo6mVw8z4yAnzk/nRryMeenIyd5UiPAPJAaL11qGOqoZBYnRgrikLhtmr4/5Xa79957LyaE1zEZSfDII4/E/JGIImP06Io1hb0VTuercuutU1EijlutsHjxYvN91XmMSK7ylk0qeVeoW9ftZzOIMlKhn7GiRLRQDz1U4fRHKpezSqmcSRXFijhtIegJiIWOLOYTBfNLc8EFapox8GWHNMzhpKO6qev7XKuGjhoWJ+aDNz6ovWoyn23RTfDypbbMzfIpW5jqlt1WMYdLFITZycZ5xnP+/nfz6JJTUSKsc71D9q3xgrZwmd4HkFdb/KRfbRwLyHeYbxx55KiAuxes87XOJzKGXzQsTF6xKUzqlpS4TLGd+XLx+YM58Q7UDy9N7nRN5Hr6tYrbbcQW5OLuFRahTrntt+q0a1VhTeJMJ5/tPkXk8+ghj8b8nVjpf/z5R/OmMysc4uFo9oM58dYk/+pxpWqzf2tXtR9+WnHc8ZADDnY9JyLcBMQxsNP5X6HM+X7I5mPOZ33ip2e2+jpIlvKIo44yXzwHGM4jw2RJmEzuQnwu1gfixoEQT0CUBl7+dKyUloLkcOOGGi8LED9c5ZYZ+3ul16kPkec+wxemFUSJIadJdEHeVe1BFCHOUFpaGosSXzXnGP41n3zySSVHf6wgv4tPVK8QRBleZeviM1aU2M8WSQViemM30dFV6oxr3ry5mjAp/jpBZ5pi/nytwfR92xVOdhZN99vPnNwtbaAoSsIe3jc8Z98zbqsTYLus/7zzKtVqv9LQ3XLj5HmVYRbWqXNnZ3RRfxarsj49e2oZyKR3kuFoWBQku9SurU2Tr8idd9jJuIK6XtLqv1v5fSyNn/S77LhL0jKznUAUV+s4KpAyZrvCpJZDgb/8p+UplxdEhvfLKq7OlSvTnTeGedXx4tsvxx4dVtpWeR1Xmb94vvpkbsW1wskUMe/PqThKUdr0INMfiTO8N7Pi2kF51s7jOl+5TcEe+p19ibMotfXWhQplUYfW6ZkauwpOEOHsazlG3Lq1W4GToIiieDTTOHK92rge0iuIgvwUY/4vMayz/u/uu72S5Sx+t5138zXnpypQGOZ8kZn1ibvnjjUs3zds2OoU2/nUUpDIJRWE3BAQi81njE0JQjyB6/71r7jbY+1PRUEijrHlmE0mChKrTLt/Eon729/+pu2OF198Ua22XdbQw7iu2CvYb9CRNJZlrj293IYjf0utMqV88Uny1FNP+f77GkQZXm3QxWesKNlnn33iyt1jjz109RAXMAFRkPQxFCTOs2klxg6oPYTNHH/arPdVxws6xc7MV61c1bgWcrPps+S7ld+po3p3VO+OflvtvcfeARNLr7jlq90vRnUcjNMrOfq5REEiC2S5kWlnY2K0Bx1DYZ1ri5Jb+9+i5CdREBPFyi23arvFx8Oy1xcnSp63Z2H7rucNREAVi8Lkx1Wr4krLF+PpNkeu7Xxak6z+ZbWa9eXsmPxHtD7Ck8zI5+NNadu1rLjxxplp6YqlaonxY4WzupzpTGJ+fueTKbH45vvu76mkmT674gpgceCqUwJNN6xO7Efgjmzj3RatMGlEOvu6oeHA/t8335xGSdHO4rW4r2rM/6cZ8//ZxvwvC+OyMvf1oPmY82+65N9KfqIQWJ+k14vyotnhyCPNTUangsRpRehUmKZXY3HncjIUxgR/BJrIu55xI1NQChKr1uk2v1GihKlVS38kV263tYJYAnVMcDOU5YDVSu88diPxcjPuBOPUgyg7ZG0tYeHChaavkhtvvNG0JBHrw0QhiDISle98to0zItXPNzsWDuLBlpA9AqIguePOO9Uo44yYU0kiterOc+nMM7MnoXfJcoWkXUlSsmtdNef5T9QrD7xs3N2+1UngUlGWXHBs3K0K3iVm/0nZl+7FnY5x9iUpnBpEQXK2YT0y3tgxuPSyy1xKEq9xuuKHlYXTSCSNPIFdjVs6nJZj8vKc6/l00+ZNasanFX9X2/p05LpsVfxtJAc2PVDbZ1sMhdB/JzweeyYWU3vV05vWSiLnsZvTjNtodOGdj96NRR/ZpoMuidr8+2Y147MKS5FTO+qvuH9vZoWlZL3dS3JybbxTUYI1ibYLXZGiIJH5/9m/5n9RkkjQ/d1kznfhSymC9UlKuMydeLEgGfPf/6pbhw51KUl0pTnnAV0a4hITgGFiPrqnoiCRdz3xB5LpMRtd+Xb/JKK08ApyW6UVxPJDfJR4BTnOYw9OnyXWs8OMW9BEqWI/QiPPRFFyuOFzRY7kJAtBlJGsDut5YrVNklJeNs5y33TT1qsAraR2L7pJsvM4BQKiILnAMLtK5rgnLDv1zqaJkuTYC46PWZLIYlyuBm7SoLF5w81L/3lOnXT5qaZlybfLl5jKkimj3ky4YHfWkavPugVfruoOcz2iIDnV8EEii2SnBYkfuWfPn626HHa8n6SkcRCYtaDCcsB61ML4o8bOjXuovPPWW0quCPYK4oRcrgyWm3Ak7WuTJ8clzfUuuFiF2K0pDj5AbyLrbM/3tuNi8mx3j1tiXpoy0byi3QqHH5TY/4n9GJBc5d6ogdtk/YfVP5g34Vihg4cFiLTtz7884EvaQw88JJbH/suUj21KlwSWMdrMaUTqvk/M+4lBioJE5v+/yxFLhwWhV07mfC8y6cczTt3sREEiF0+cb+zMOy1InKlFITpr5sxYNC/5TkKpf3YyROnszdDrtIB3jtSfyO2ociW2FUTpoAs/Gkcq7WulFi1a6JLF4uTmWys0aNAgzu+IM6NYsXxk+CoUY4tbbqmw6pa404y/I3KzTbIQRBnJ6pDnaStKxOusNEZC9+7d1fjx483fv/nmG7XKMFfmxhsTRyDhkksv9XSe46xA90dSdh1aNW7pTJqzz+8aZ9WPvbCTccRmi1lng7p7qrdHvRHni6TbkV3Vc/eMV6dd2d1Uliz+/lt1tChLRr+l9jTS5ys4d4/lFgyCm4CMu/GGE2e/TpklvfNGEXepxPgloNtZvNH4A6SbD/yWGdV04u9Jd17driCpWnXrjSoHtXTPm5PffzWn86n4D7GHA40bafyE1b+ujktWs8b2rmzr1q9T1907IC4+maPUabajMocaR2V0QY7K2EOHVvqjPPZjN5K+1X5u3uIA9H3b0aP2rbLvn0T3fdKNBV3biy1OFCRnGcdrkilIZC5q5XgJLTZWQbeX9Yk/oiNGjlRydM5P0P3NFMVpPtfQfuQOaxqUzv57pm+/fr7f9fyX6k754YfxzuG9HLnK+7w96I7SWM9F8WK/jaazDz+OcpHLkCFD1AknnKA6deoUsyR58803lZxO8fKbYpcpiDLchOJjvG1oEuT87rvv1DHHHGN6MBeTnXHjxsU5l5kyZUost2iu5M5nQvoEvDwM60rUTfLOP6a6fNmKm/LJe3FKkn3qNzR8kLylddh6SseT1NN3PRUTRa6MPLL3MWrZymXZEi9pubLjZQ/s0OuRyflGv0oSqwQnSydrfU3E6gjovuO6uUCXt9jjREFyleFE7SlD2S874paSRLhYL3d2Rrkep99+vyRW/baVq6jtHQqPV96bpC66qa/LSmbXnXaN69qffv7J1dWX3HqZ69anZns3jaVbuKTiymCJFMWK3e9JWw8LkI8+qzgqtFdJA1XXOGYpQY75fGsowa1gv/HmoMYHqhrb1Yg9s375fNEXaovNAujQgyqsTuYvWeBKH0TEpA/irYjkBZ/vk57s4BtuUP2MBb5fKxJ7Kbn+LulbULixTn7Ov6mF27JgJferJJFaUYgGy153vA7GesapvOvpS/AXa7f8kPWOWGbognNDqWHDhrpkZtxjjz0W92zAgPgNGM+MxoNDDz1UvfDCC3FJ7EqXRHmtZ0GU4VWPL0WJOLW71rgya9iwYWrJEuNYhHHGb+XKlWqHHXYwPdVut912cd5qxYutBDHZ6dKlixKHr3KfMiE3BGRRZw/OP6a5kUIpUZIcd9HxMUuSRnvsa1qINDSUJV6h+7GnqqeGVZyX//q7bwwHr8eajl7zETAZzB510SLbQz78P2Svdbkt2fkdd84BuZWmMGqzK0jE6bBdQWJvgdNMWMapbpcsW61es3ZNrGixyltsKJCt8Nr7rxtHFk9Tjz4/Sj37evwtAnuWxDtWf/PDt2L5yo3jLlfffY16bOITSvyHVK+y1YJGElj+SZ55dbxq0m1/ddVd/4rl+/Cziit6JfJgD0XJ4u8Xx/I0/MvfifhaOfWqM1Szbs3VzHlbb9iZ/mmFI1evI0X29kqhDUu2+k956OmHVVOjrNvH3hWrK4hfROnIvO+fpOzo+Q26OT+X3yW/chZKOsZpbnpKrAgJ6RFwrk2kFJTO6bEMKtecOXNiRR1yyCExQwe5nnj58ooLLPbdd9+4KmvUcG9kSAK5weaeeypuMr3wwguVHL2xh+uvv97cTBWdgS7I0Tj7GsyppJE8QZShqztZnC9FiUC94447TCFF42UpPZ43TO2towjHHXdcrC65E/nBBx9UZ5xxhpoxY4aqbVwJWr9+/WSy8DwgAmFYjMjO4UmXnhxTkjTZq4l6d4y/YzRndj5DPXbrmBiNhcsWqf63XxkQHf/FjJ5QIYOViwneP790Uuq8+KdTTrHlYcHsv8d33W03dbVx5frTzz5r3srkpSCxSnQ6dJX4XC6cnUcP/3FzXzX8yQdVz0F91PGXdFV//PmHOrvTmap7p9PjIDTes7ESaw4r3PPfe9WdY+9WjxvKkaP6HKPuNj7LMchH/v2Q2rhlcyzdo8+OVIPu/7c689pzzGOQO26/Y+yZXO1uDwc3bxP32fpg91o/b9E8NcIos9PFXdSEd19RNavXVNW2rWZalny/qmJRdvABHmVVjn8R/7/H/6OuMP4eXDK0v1ldrRoV8mmFSTFSN++zA5oiRJJnnQDjNDuIw2BFmJ2W5adUp6KETZz89IO9VrsyxHqHX7t2rZL3+GbNmpm30EiQZ3LTjRXsChZ7eVdccYWS/FYYNGiQ/bH5+6uvvqrWrFmj3n77bdczidi8ebMSowwrOG/UDaoMbeVJIn0pSpz3LUuZI40zf3L8xgrnnXdeXFWXGn41RJEiGihx+uqliUoiH48LlEAVY5dpzK2jzLts9t9nP+O4zZuq3u7+/Xuc2/UcNebmR83W71N/b3XvNcHuGvrBqjvOoHtp8lMWadwEdC8fuXwBdUtUmDG6BTMKPe++/O/jj6uTTzlF+d0Jz/fC+bjDjo1rzOsz3lKXDbtCjZuw1dS1z8m91H9vHe3yRi/KiqvO/2csryg9rvm/69V5A3sp8RvVuMG+aurYd0yH2gc3r7BC/M//hqtbR95m5rul303q330Hx8qYZvMVUtr0INcxICvhiUd0i+VZteZH9Y8hl5h1yo01b419Q7Vo1FzNmBN/TvrgA/T+To4++EhVxbZYu/mRW5XIKH9bRtzwoLr4jItidQXxy2THsRtZLDq98wdRTzGWwZwfXK/r1ieM02D4hmGzMZiW5L8UNnHy3wdOCbbfvsJfmdxq88gjj6gOHToo+V3e1a3NI3GCfOWVFZvUYjDx9ddfx4oTxYa861vHbuRGHLEYcVqTiHWIdT38o48+qt555504keSWm8svv1yJpauEOnXqmCdX7CGIMpwc/H72pSgRL7RWkMWXNLSPcQOLPYgW6hpjl84KAlicuYjTGHlGyB2BsCxGTj3mFMNp65vqHeN2G+uMeioUep7Uw7gN53njuM6beXHoiiY8ld5KPW2+X0BTlzicOXQLZhR63n3lV0FiL0G3cNYpqLxrTf9J8333V48MfjDueIyU1qxhU/XEbePUyJseUaKY1oXL/n6puvmSf/91+frWFKJg6H/2JWrG4++rBn9ZnFzV82q1u82nSYeWh6uJ97+oBl54fVyxc7+uuMnGy5GrZOjcrpM6qHGFh3y5HUcUOrOe+kgd2HirM9ovbGWJIqT5PvvrmmD6LfnnuZfHnslV8nJcaMqYt9WFp12gzZNupK5P+1wQbB3pyhaFfMz5wfUi65PgWDpL0t0ema9NnKrbVhyLdMqZ6ueqHn8nUi3Hb3rdfKp7P/FbHumCIXDyySfHClq2bJm6+OKLTUWG+CCRd3a7ouMGwwfVHntsPcYrR3Pkyt8zzzxTnWpY40p6OT0iQeb2mcZtUfLMGWbN2nrU1oqXYzaiCOnRo4eSG3fEx+HDDz9sPpZTK5ONmwarVasWV0wQZTjl8vu5kqHB2arCSZBDHLJ++eWXpuACqWbNmp6pBbqY7ci5bvFhQsgPgf6GczX7FWd1a9dVzw59Jj/CFGCtk6a/qoaOGxon+YUXXaR6GNfLEYIjIFevDjFuZ7GH+67+T6g8zP/555+qWpvtTYeSdWvXUcvfXhocgABKavePDnGliGnrfcOHB1AyRVgEVqxYoU43rFDsQebU+6+6T5UY/+ci/Lb+NzV/yXy1+peflfj92GePfeKcqCeSYeOmjerTBZ8p2ejYf+/9VPVq1V3JxYfIB59+qA5qcoDaecedXc9Tjdj8+2ZV9tUcs86WTUt9y+pVz/zFX6nvflhhWL+08bRk8crrN975XZJ8zxpO5rDQ8kswebpCmPOTtyK/KVifZJe/13zPGjo17v3u7q/K5pfFZZr2QYVfqtRKI3VQBH766SfVpk0btXjxYrPInXbayVRwiMXIjju6j7LKsRpxzvrAAw/ErD4knxzLkZtpRNkhz8XNhi6Ib1NRhIgliShidGoHyXuKscYSXyc63UEQZehk8xPn63pgccDidX2QsxLRPFnaJ+czPueOgOyAOu+C54oz//yduzWSk116//z8ptTtLuT6+tVksoo54e+zNiRLlpfnuh0bp/VDXgSLWKXyoiyK0kdHjIi1TEyKhf/AnvFWF9lqutx207KZ+/pcP/WJYuTgFn9LmLRa1WqmpUZQQXZCk9WZSl1NDAsa+clWuHXs1uNG9vKlz1GSBEu8EOb8YFscfGmsT4Jnai/RsnxiDZ0+Z7F0dSpJunarOJKZfsnkzJSAKCXmzZtnWpHIMZwWLVok3MgQxcX999+v7r33XiVXBovzVjGYkNty7D5MvOQSC5WhQ7duPG/atMl0GPvjjz+qjRs3mhspcu2wWJUkCkGUkaj8RM98KUoSFcCzcBLQLUaGjrkNqxKf3eU8p871kD7BpZhMtyDRLQJTLLZoks9yXF8tDUehl53uF652RYnUImMVBXR2eOeyVFnUO+d8qR8LwuB7gTk/c6bOscr6JHOmzhJ6G0fu7IoSeR62TRynzGH6PHl6/BXrItvxxi2ohHAQqF69unktbypBlCKNGjVKJYsrrZxMkSM78pNuCKKMVOr25aMklQJJGw4CurPAub7WMhwkUpeCXfrUmWWSw2kBIeNUt7ubSR1RzCsv6OzY5K5nZU4dbJzXtQcZq6KAJhQ2gVvHxh+zlNY4+7qwWxgu6Znz0+8P1ifps0slpzjGdd7QIopxnU+wVMotlrSjJ7pvjcTZcLH0frTaiaIkWv0Z1xrnYkQejn7ZPXlFGEFaTdNN8DoLnbQKJ5OLgI4tCxIXJleE7rvMjo0LU6ARMladi2cUe4EiznlhopR1Khzlphsss7LXFcz56bPVrU8Yq+nzTJTTuYa2jlsmysMzpd3okmOMBAgUIgEUJYXYaz5llj+ezkW9LAhlJ5qgJ6CzZOCcup5VULGW/wd7eSxIEtMVZ37Olzv5rrNjk5hbpk9lrIpJtjOIKbz0CaGwCMjfQucxBmnBgMEVVyIXVosKQ1ov6yydtURhtCg3UnqtT3JTe/HV4qXQYw3tPRY4xujNhieFSQBFSWH2m2+pdYt6TMX1+Jjg9VxyEStKPdnFtQfL/0Mu6i+0OkZPHO0SWfdddyUiImMCoozS7Y5Jn2CWnTHenBUgLzv976q4dtiqWPoWhWP2u0FeQpnz/XNmfeKfVVApvRR6+boqOKh2ZbMcjjFmky5l54MAipJ8UM9hnbpzluzW6zuACV7PJRexsiDp49ipl3HKgsRNXywXhI09iDd5Xu7crLIVo7PWkz65zLgOEWVJtqgHV670kU5JIlZZOHANjnOikrzmfN2RwkTlFMsz1if56WndcUuxQsOqxN0fOr9pHGN0cyKmsAigKCms/kpL2gGDBrnyTTImekIFASb4/I8GFiT++kBnTYJvEn/sgkolL3m6eRVlSVCEs1uO7qVTatT1aXYlKe7SdXM+x4PdY4L1iZtJrmJkrnf6KpG6Uei5e0DHhGOMbk7EFBYBFCWF1V9pSSsTvfP+chwQxqNkgk9raAWaiQVJcpxyhh9rkuSccpFCxuvwhx5yVbV1bh2KZYmLTDgi+hlWP07/PiKZ9KX0KSF3BLzmfI4Hx/cB65PcjUldTToLQplD8EtVQQu/abqRQ1wUCKAoiUIv+mhDz969XanEBwQT/VYP3c6Fs5gLcpTBNWSyHuG1IMHJnzJNfXU3Hui+21nvKCowCcgcoVOWyHzCMZxwDRL5/pw+4AxPJQnzfX76Szfns5FT0Rc6xR6Ou3M/VnU+wPBLtbUfZG4dOs59xbqOWe57jhohkBkBFCWZ8SuY3LJzM/iGG+LklcVIsU/0MsFz60G4hrHuj6soCIpZqSfjVOdTgRuZ8j92vZy7Wsdwinnc5r93tkpgfX+c1ljyVBRdKEny21O6I0/yd7nYFeS6IzfSU7q/kfntwejX7uXvTxTixR50Fk8o84p9VESn/ZVvNEJ0mkNLEhGoucMOatrUqWrt2rWxZOs2rFNTy6aqMzp2T5Q1ks+8Xj5lgu+lscCJJIQQNkqUeqtWrVLz58+Pk27BsgWqUYNGqqR2SQilzq5Ici2k8yVPxun1Awdmt2JK90Wg1LjBo3LlymrWzJlx6WV+lXFbc7sdVOM9G/kqi0TBEvCa56UWlCTBsk63tJo1a2q/P+J0t1i/O17jVo5Rn969+NZr6Y6tIPO1bNVKu4aeZVhndzmsc5BVFUxZOosnEX6gsTHLUcaC6UYETUAARUkCOFF7JIuR9kccwUT/V8d2H3Cmq4vl5fO+4cNd8UTklkCjxo3VwoUL1fLly2MVy0vn7K/KVPvS9mqHGjVzK1Aea2Mhkkf4KVQtypL69eur9959Ny6XqYyeM9X0WdLBGLuE3BEQBeP9z9yvrRAliRZL3iLrlpSo9evXxynILUVjsc35XkoSWZ/cettteeujYq9Y1tCNmzZVk155JQ6FbGIU4/zutTZhbi32b0q02o+iJFr9mbQ1TPRbEckE79yhlyfjn38+KUMSZJ+AjJO8I3IAACAASURBVFOv3ZsFSxcUze4NC5Hsj7UgaxAFX6s2bVwLaalj4bKFSm4bK1arqCA5JytLXjRFSTJtzjRtUhbyWix5jZQ5X74/XlavxaQs8bIgZBMnr0PUrFysJHQKcZnfJbRs2jL/QuZAApljx0wY66qJudWFhIgCJ4CipMA7MB3xi32iT/TyialgOiMqO3ksC6hnnn46rgJRcBWDqSsLkeyMq2yXKnOIKEvKZs+OO+Yo9VpWUevWryuaBXW2eTvLF58wAx4aqFWES1oW8k5i4fmcyOq1WBTkXusTjjKEZ5yKQk931LJYjop5WTxxLCw8YxRJgiOAoiQ4lgVVktdEP9u4rUFCVLXiXosQFs/hHL6ycNbt0FvKEhmnUTyG43VkAP854RynTqlEWSLHHJ3HxySdqSyRqyWxLnFiy+izLN4vu/tyrXNuKVi+O2IxiDI8I8xZz5zI6lW+M1G1LJGX7OsfGsCtTFkfYcFUkOioWJQV4ZYi2kmRY2FOInyOCgEUJVHpyTTaoZvopRhZxEdtx95aRFvmkXZccnPI8Z2L0xFXGsMm51nkxUa3eyPKEnFEHLWjDKLM0x0ZwH9OzodeRhVax8dq1arlcvIqBYvCZLJh/SAvSI33bBxJhV9GAH1mFn73GX5IxBeJMNUFmeNxfKwjE844L6tX09+PMedHzcGrrE96D+mjtYJiEyecYzTRUTFZQ0fRZ4msTca/Pd7VIaxNXEiIiBABFCUR6sxUm+I10Us58hIald0by0xQt4jm5pBUR01+0oujTN1NONZRhigsnC2/CmV/WXXZSbMQyc+4y7RWmWNl7HodxZHyRXkrL38Lli5EYZICcHkRGf/WePOYjU4BLkXVq1dP3XbHHSjCU+AalqReVq+Wg9eo7Np7HWOQfuAoQ1hGo14Or6NiklrmJNlwjIrVq5c1trSVY2H68UFsNAigKIlGP6bdCmuid3qblwKjsHvjZSYo7ePlM+1hk5eMsnDW7c5HYeFsLZZ1DoYZp3kZboFWah3F0Y1fa65FYeIPuV1BYh0V1eWUl8z/GDeYcdRGR6cw4hJdux2FXXt5+dQ5xJTeESuofv37F0ZHFbGU1hpad8wyClavsjaRGyK91ib3P/igatSoURGPAJoedQIoSqLewz7aZ1mW6BbxhfoSap331ZkJChIxZ+3Vu7cPOiQJCwFrd17ncd60LPnL3LXQjjGgzAvLCMuuHNb47dK1q9Z3idQu49iuMKm5fU1VUrsku4IVSOmyYJejSmJBkkhBIopF2eE8vXv3AmkZYiYiYFlkOa9klTzWrn3d3eoW1PckkT8SaZesTzgOnGhUhOuZzO1yS5/XhuPsr8pUIVpA+VmbSNsJEIgygUrlRohyA2lbagRemzxZDbn5Zm2murXrqt5dextXsx6vfR6WyNETxqjRE8d4isOZX080BfOgrKxMDR0yRH3//fcumWWctmzSUg3seb3rWZgiZBEyeuJo7U6NyMk4DVNvBS+LzLWjRo7UjmF7bdZ47nz48apV4+K4etJqv7xQTp4+2TwGqtvRdPbKYENB0gl/U04skfi8YsUK1f/SS7XfF+s70rtbL0NhUje07ZXxLOuTycZ41gVR8vW+4AJVWlqqe0xcARAYN2aMenTECK2khbKGFqX00DG3ec65Yu3Uo1cvbRuJhEDUCKAoiVqPBtCeRC+hUrxM9gN6XR+6Rbu8eA4dN9STgCxCBgwahCm2J6HCeiAL56G33KJ1lGmN0y5tOytZPIcpJFuEiKwoScLUY9mTRcbwHOMaYS/ltLPmQlECOuVO5bO8TMoObCIlor088UNygmGlw8I9FcqFmVa+L2NHj1YTJ0zQNiDM349kGzgcsdR2aUFGyhq6X9++nrKHdQ3N2sSzy3hQxARQlBRx5ydqeqLdGytfaZNS1fvEXnlXmPiZ3FmEJOrtwn2WbOEsLQvLLo6M09Evj9Fe/2j1AMq8wh2LmUhuKUz8WJhYY1qspqJkZSKK7tmG80Ov3XYnXxQkTiLF8Vm+K2KN5bVrb30/wqIkT7aBI/KKP53rBgwojg4sklYW2ho62dpE5tsBgwdj7VQk45dmVhBAUcJo8CTgZ0EimTsbu/a5XrD73XVkMe3ZvZF64OcYQ752G/2+ALJYjtSQTKsxlsLklVde8bSUchZsjWtzLi6g4znWsRq5GUJ305OzndZn5nQvMsUVL3N+su+J9d3I9ffCGtuJjgBLb/HyGe0xG+Y1tJBnbRLt8UfrgiGAoiQYjpEuxc+uvQDI9qLEUo743XXkHGWkh6WrcdZLZrJjDPZxKoVkw++DLEBW/LQ8oa8cqwGiIOlpOBbmdg5XlxZ1RDpKEwEm41t+xAmsvCBma4yn0jnW3C3fiVQVI1KPpRwR/yN8T1IhH+201nfEjyVWrtYnfo6MyXjuY/giwZ9OtMen1bqwraH9rk2wcC2O8UkrExNAUZKYD09tBGSyT+QTwg5LFiUSLPNw+T2VF1JZWC9fvVyt+GGl+cLp15mf1MPOvL0niu93v7s4FhlrAS1jte5udVTJLiUpOQRM9yWQRUjxjc10Wyxn3kcbjl9nzZyZVhH2+VicXdY1lCgy1iWkOt51Ash3QIJ9zjbncGPuTsVaxFm2zOXHd+mCubcTDJ/jCPhVkluZrDlfvgulzUpT/g5Yc76U53fjxqqbDZziHbyprKGFUiabOtacLP6eUlVQ41S4eMcoLXcTQFHiZkJMEgKZLNqtBbv1v3X1pSyoJcjNBn5uN9CJyIunjkrxxvndxdERSjZOJU+6L4AsQnTEifNDIF0rk2RlW+Nd0tl/l8/OOdoqy5qn052vvWQS5UjLli3ZbfcCRLwngVSV5PaC/Mz5maxP2MDx7Laie5DNNbTAZG1SdEOKBmeRAIqSLMKNetF+zgjnggFHF3JBuXDrSHUXJ1stFQXJCSecwAtgtgAXWbl2pcmK5cu116YWAhI5hiDfjZKSEm6uKYQOKwAZs6VQTLXp+NNJlVhxpQ/LGpq1SXGNO1qbGgEUJanxIrWGQD4WJUzsmo4gKiGBfIxTFsoJu4SHARKwj28pNt1jOgGKpC3KrhjB54gWEZEBEsj1y6g1vjkyFmAnRryofKxNZA3d2vjhWvWIDy6alzEBFCUZI6QAOwFrwp89e7a5wxnUYt164TzIMMkuLS0FOgQyImCNUykk2c0JqVRkLZI5OpAKNdJmg4CMcflZaVibyP8zDf8mubQ8sb4L0jaxFmHuzkYvU2YqBLL1QipWrYzxVHqCtF4ErDFqzdlBrqFFOYICz4s88RDQE0BRoudCbEAEZLKXMMdQnEiwFChW8XO/+EJt3LhR7bfffmq7GjXM2w0kWIsOueGAWw4sWvyfLQK6cSp1ibJvw/r1at68eap69epq/+bNTRGc41TiUOBlq3coN0gC1li3lChStvy+3FCoSJAxbw+iXJFQ11B2WMEa//JZ5mrzuTFX1zF+Z86OYeKXkBOQcW99D6zvgDX+V61apZYtXapq1aql9m3UKDbnW2MexV/IOzci4unWJro52pqfWZtEpONpRmgIoCgJTVcUnyBfffWV6tOzp9nww9u3V7ffcUfxQaDFoSdw7TXXqPenTjXlHDV2rGratGnoZUZACEAAAhBIj4Bs3nQ//XT1808/qZ1r11bjn33WVJQTIAABCECguAhsU1zNpbVhIjB61KiYOPIiKooTAgTCREDGpKUkEbnsYzZMciILBCAAAQgEQ+ClF180lSQS5H/5TIAABCAAgeIjgKKk+Po8FC12voCKULyEhqJrEMJGwDkmUegxPCAAAQhEl4BYkzz++ONxDZTPEk+AAAQgAIHiIoCipLj6OzStdb6AimDyEvrll1+GRkYEKW4CMhbt1iQWDd3YLW5StB4CEIBANAjYrUmsFmFVEo2+pRUQgAAEUiWAoiRVYqTPmIDXC6gUPGb06IzLpwAIBEHAayyi0AuCLmVAAAIQCBcBnTWJJSFWJeHqK6SBAAQgkAsCKEpyQZk64giMtfkmcaLhJdRJhM/5IJBImSfyJBrD+ZCXOiEAAQhAIDMCL73wQsw3ibMk06rEeE6AAAQgAIHiIYCipHj6OhQtlRfQadOmJZSFl9CEeHiYAwLJxqCMYY6J5aAjqAICEIBADghs2LBBPeHwTeKsVp5LOgIEIAABCBQHARQlxdHPoWllshdQEZSX0NB0V1EK4keZJ2D8jOWiBEijIQABCBQYgZeNm21Wr16dUGp5LukIEIAABCBQHARQlBRHP4eilfPmzUtqTWIJyktoKLqsKIXwO/ZEoSdjmgABCEAAAoVLwI81idU6rEoKt5+RHAIQgECqBFCUpEqM9GkTGJeCo1ZeQtPGTMYMCKSizJNqUhnTGYhFVghAAAIQyBIBP9YkVtVYlWSpEygWAhCAQAgJoCgJYadEUaRUX0CFgd+d/Sjyok35IZDqmEOhl59+olYIQAACQRBIxZrEqg+rkiDIUwYEIACB8BOoVG6E8IuJhIVOYPPGjWrDpk1xzTjh+OPjPp/bo4c6++yz4+Jq1apV6E1H/gIi8Msvv8RJ++STT6rHx42Li3vl1VfjPm9XrZqqWr16AbUSUSEAAQhAQAhs2bJF/fbbb3EwBl53nSorK4vFlZaWqluHDYtLs/3226sqVaoAEQIQgAAEIkyAWT7CnRumpsmLZLKXyWrGCyeKkTD1WvHJ4hx/MiadwZnG+ZzPEIAABCBQGARE2eGc07epXDlOePnsTFMYrUNKCEAAAhDIhABHbzKhR14IQAACEIAABCAAAQhAAAIQgAAEIkUARUmkupPGQAACEIAABCAAAQhAAAIQgAAEIJAJARQlmdAjLwQgAAEIQAACEIAABCAAAQhAAAKRIoCiJFLdSWMgAAEIQAACEIAABCAAAQhAAAIQyIQAipJM6JEXAhCAAAQgAAEIQAACEIAABCAAgUgRQFESqe6kMRCAAAQgAAEIQAACEIAABCAAAQhkQgBFSSb0yAsBCEAAAhCAAAQgAAEIQAACEIBApAigKIlUd9IYCEAAAhCAAAQgAAEIQAACEIAABDIhgKIkE3rkhQAEIAABCEAAAhCAAAQgAAEIQCBSBFCURKo7aQwEIAABCEAAAhCAAAQgAAEIQAACmRBAUZIJPfJCAAIQgAAEIAABCEAAAhCAAAQgECkCKEoi1Z00BgIQgAAEIAABCEAAAhCAAAQgAIFMCKAoyYQeeSEAAQhAAAIQgAAEIAABCEAAAhCIFAEUJZHqThoDAQhAAAIQgAAEIAABCEAAAhCAQCYEUJRkQo+8EIAABCAAAQhAAAIQgAAEIAABCESKAIqSSHUnjYEABCAAAQhAAAIQgAAEIAABCEAgEwIoSjKhR14IQAACEIAABCAAAQhAAAIQgAAEIkUARUmkupPGQAACEIAABCAAAQhAAAIQgAAEIJAJARQlmdAjLwQgAAEIQAACEIAABCAAAQhAAAKRIoCiJFLdSWMgAAEIQAACEIAABCAAAQhAAAIQyIQAipJM6JEXAhCAAAQgAAEIQAACEIAABCAAgUgRQFESqe6kMRCAAAQgAAEIQAACEIAABCAAAQhkQgBFSSb0yAsBCEAAAhCAAAQgAAEIQAACEIBApAigKIlUd9IYCEAAAhCAAAQgAAEIQAACEIAABDIhgKIkE3rkhQAEIAABCEAAAhCAAAQgAAEIQCBSBFCURKo7aQwEIAABCEAAAhCAAAQgAAEIQAACmRBAUZIJPfJCAAIQgAAEIAABCEAAAhCAAAQgECkCKEoi1Z00BgIQgAAEIAABCEAAAhCAAAQgAIFMCKAoyYQeeSEAAQhAAAIQgAAEIAABCEAAAhCIFAEUJZHqThoDAQhAAAIQgAAEIAABCEAAAhCAQCYEUJRkQo+8EIAABCAAAQhAAAIQgAAEIAABCESKAIqSSHUnjYEABCAAAQhAAAIQgAAEIAABCEAgEwIoSjKhR14IQAACEIAABCAAAQhAAAIQgAAEIkUARUmkupPGQAACEIAABCAAAQhAAAIQgAAEIJAJARQlmdAjLwQgAAEIQAACEIAABCAAAQhAAAKRIoCiJFLdSWMgAAEIQAACEIAABCAAAQhAAAIQyIQAipJM6JEXAhCAAAQgAAEIQAACEIAABCAAgUgRQFESqe6kMRCAAAQgAAEIQAACEIAABCAAAQhkQgBFSSb0yAsBCEAAAhCAAAQgAAEIQAACEIBApAigKIlUd9IYCEAAAhCAAAQgAAEIQAACEIAABDIhgKIkE3rkhQAEIAABCEAAAhCAAAQgAAEIQCBSBFCURKo7aQwEIAABCEAAAhCAAAQgAAEIQAACmRCokklm8oaYwLhxSo0albmAffoo1aNH6uV06JA0z/BPP41LU//WW5UaMyY+33vvJS3HlaAA2u6SWRdB23VUvOPo961spk5Vqn17pfbZR6mxY7158QQCEIAABCAAAQhAAAIQ0BJAUaLFEoHIxYuVkhemTEPHjumV4KPuUmfJixYpJT+ZhgJoe6ZN9MxP2z3R+H6QxTHvW4ZUEzr73cf3L9UqSA8BCEAAAhCAAAQgAIFiIYCiJGo9Lbvq8tJ0441RaxntgQAEIAABCEAAAhCAAAQgAAEIZJ0AipKsI85xBXLcht3kHEOnOghAAAIQgAAEIAABCEAAAhCICgEUJVHpyUTtEH8F6YaGDdPL6aPOspkz48quX1KidqtXL7367LlEZh/1J60oi21PWne6CWh7uuQq8hVqv2feckqAAAQgAAEIQAACEIAABAwCKEqiPgxEYZCOU9BMufios1/btnG19LnoItWrV69Ma97qfDYdB7SZ17y1BB9tD6oqVznSbtruwpKTiHz3O9ZkOelmKoEABCAAAQhAAAIQiD4BrgeOfh/TQghAAAIQgAAEIAABCEAAAhCAAAR8EkBR4hMUySAAAQhAAAIQgAAEIAABCEAAAhCIPgEUJdHvY1oIAQhAAAIQgAAEIAABCEAAAhCAgE8CKEp8giIZBCAAAQhAAAIQgAAEIAABCEAAAtEngKIk+n1MCyEAAQhAAAIQgAAEIAABCEAAAhDwSQBFiU9QJIMABCAAAQhAAAIQgAAEIAABCEAg+gRQlES/j2khBCAAAQhAAAIQgAAEIAABCEAAAj4JoCjxCYpkEIAABCAAAQhAAAIQgAAEIAABCESfAIqS6PcxLYQABCAAAQhAAAIQgAAEIAABCEDAJwEUJT5BkQwCEIAABCAAAQhAAAIQgAAEIACB6BNAURL9PqaFEIAABCAAAQhAAAIQgAAEIAABCPgkUMVnOpJBAAIQgECYCfTpo1THjhUSNmwYZmmRDQIQgAAEIAABCEAAAqElgKIktF2DYBCAAARSINCjRwqJSQoBCEAAAhCAAAQgAAEIeBHg6I0XGeIhAAEIQAACEIAABCAAAQhAAAIQKDoCWJRErcsxv49aj9IeCEAAAhCAAAQgAAEIQAACEMghARQlOYSdk6owv88JZiqBAAQgAAEIQAACEIAABCAAgWgS4OhNNPuVVkEAAhCAAAQgAAEIQAACEIAABCCQBgEUJWlAIwsEIAABCEAAAhCAAAQgAAEIQAAC0SSAoiSa/UqrIAABCEAAAhCAAAQgAAEIQAACEEiDAIqSNKCRBQIQgAAEIAABCEAAAhCAAAQgAIFoEkBREs1+pVUQgAAEIAABCEAAAhCAAAQgAAEIpEGAW2/SgEaWYAhc9I9/xBVUWloaTMGUAoGACLRu2VJVdozTgIqmGAhAAAIQCCGBrl27qjZt2sQkq1u3bgilRCQIQAACEMg2gUrlRsh2JZQPAQhAAAIQgAAEIAABCEAAAhCAAAQKgQBHbwqhl5ARAhCAAAQgAAEIQAACEIAABCAAgZwQQFGSE8xUAgEIQAACEIAABCAAAQhAAAIQgEAhEEBRUgi9hIwQgAAEkhHo0EGpSpUqfuQzAQIQgAAEIAABCEAAAhBImQCKkpSRkQECEIAABCAAAQhAAAIQgAAEIACBqBJAURLVnqVdEIAABCAAAQhAAAIQgAAEIAABCKRMAEVJyshCngHz+5B3EOJBAAIQgAAEIAABCEAAAhCAQJgJoCgJc+8gGwQgAAEIQAACEIAABCAAAQhAAAI5JYCiJKe4qQwCEIAABCAAAQhAAAIQgAAEIACBMBNAURLm3kE2CEAAAhCAAAQgAAEIQAACEIAABHJKAEVJTnFTGQQgAAEIQAACEIAABCAAAQhAAAJhJlAlzMIhGwQgAIFcEJgzZ46aOXOm2rBhQ+xny5Yt6sQTT1StW7fOhQjUAQEIQAACEIAABCAAAQiEhACKkpB0BGJAAAL5I3Dbbbepp59+2iXA0Ucf7YojAgIQgAAEIAABCEAAAhCINgGO3kS7f2kdBCDgg8DYsWPV+++/r7bddttY6kqVKqlDDjnER26SQAACEIAABCAAAQhAAAJRIoCiJEq9SVsgAIG0CFSvXl0ddthh6qijjorlb9eunZJ4AgQgAAEIQAACEIAABCBQXARQlBRXf9NaCEAgAYGvv/469vSYY45JkJJHEIAABCAAAQhAAAIQgEBUCeCjJKo9m4d2/fHHH+qzzz6Lq1mOLxx00EFaadavX6/mz59vPtt+++1V48aN49Jt2rRJzZs3Ly6uatWqav/999eWRyQEMiGwbt06tXDhwlgRRx55ZCbFkRcCEIAABEJOYPHixWrSpElK1haiHG/YsKFWYlGi//rrr3HP9txzT1W7dm1teiIhAAEIQKDwCaAoKfw+DE0LREnSsmVLlzwbN25U1apVc8W/8sor6owzzjDjmzZtqr788su4NOIzomPHjq58n3/+uWrevLkrnggIZELgo48+imXfZptt8E+SCUzyQgACEAgxgUWLFqnu3bur2bNnx8378vnAAw90Sd6tWzc1d+7cuPgTTjhBTZw40ZWWCAhAAAIQiAYBjt5Eox9D0Yr99ttPvfDCCy6/DqtXr9bK984778Tily1b5kpz6KGHqvHjx7via9So4YojAgKZEpg+fXqsiPbt22uVe5nWQX4IQAACEMgvge+//1797W9/i1OSiER//vmn+te//qUV7qmnnlIdOnSIeyaWsAQIQAACEIguARQl0e3bnLdMrEZOPvlkddVVV8XV/dNPP2llef3112Pxv/32m/r999/j0olC5PTTT1f16tWLxR977LFq77331pZHJAQyITB16tRYdrt/klWrVqlnn31W3XXXXWrChAnKS/GXSd3khQAEIACB7BMoLy9Xp512mvr5559Vly5dTEvW0tLSWMWyLlmzZo1LkAMOOMDMZw/9+vVzpSMCAhCAAASiQwBFSXT6MjQtadWqVZwsP/74o0u2lStXKjF9tYcffvjBlW7Lli1q+fLlsfj+/fu70hABgUwJyOL5vffeixUj/klk7A0YMECVlJSYJtqy03jiiSeaZ9hnzJiRaZXB599nH2UIW/EjnwkQgAAEIBAjIHO9KEZkw+XJJ580j/06N3c+/PBDLTH7mqVZs2ZKLA8JEIAABCAQXQL4KIlu3+atZfs4XtB0O/Djxo1zybdixYo46xFJIAsWWdhIkAWNnAkmQCBoAuI0WHzpSBD/JLIIPv7449Vbb73lqmrt2rWqc+fOSsarzveOK0OuIsaOzVVN1AMBCECgIAnI/P7QQw/Fyd6pU6e4z+KvyhknCez+TK644oqCbD9CQwACEICAfwJYlPhnRUqfBOxHZSSL01JEjtjceeedrtLEysQZHnnkkVjUjTfeqOQWHQIEgiZg909y2GGHqTPPPNNUkrRu3do8biM3HsixMiuIafb//ve/oMWgPAhAAAIQyDGB3XbbTe21116xWufMmeOSQNYx06ZNM+N32WUXdf7557vSEAEBCEAAAtEigKIkWv2pVAjM73fdddc4qk4fJeKgVY7jbLvttsp+Bavs0NuDWKI88cQTZpRcCWzdkBO1LqM9+ScgNyxZQRbDb7/9tmrXrp2aMmWK6tq1q2mm/eCDD8YJ+vHHH+dfcCSAAAQgAIGMCRx++OGxMsrKylzljRw5MmbdetNNN6ntttvOlYYICEAAAhCIFgEUJdHqT6XE/N7w6B77yYM5vpi22pUldh8lcoxm6NChJvVrrrlGNWrUKNYDTkXJsGHDTC/0Eu6++27zSAQBAtkg8O6778YVK8e8XnvtNVWzZs1YvPgqqV+/fuzzkiVLsiEKZUIAAhCAQI4JHHLIIbEaxRfJpk2bYp//+OMPdd9995mfZW1z4YUX5lg6qoMABCAAgXwQ4M0zH9SLoM4GDRrEWmlXlMhxhS+++EJVqVJFXXnllUpMXq1gV5QsXrzYvGVEgviDEH8RBAhkg4BYPH3zzTdxRYsPHd011DvuuGMs3S+//JINcSgTAhCAAARyTKBNmzZxNc6fPz/2edKkSaZPKglDhgwJl2+qHHOiOghAAALFRABFSTH1dg7buscee8Rqk+tVJWzYsCHmXV5uEJFzvnZFif12G7E2EesTsSJ54IEHcig5VRUbAecNNnKzkn130c7DrsyrU6dOsaGivRCAAAQiScB+RbA0cO7cuWY7xar1hhtuMH9v3ry5uuCCC8zfCRCAAAQgEH0CKEqi38d5aaH9iIJlUSIOXMVhq/gmsa7jk+MMVrAUJW+88YYSPyYSbr75ZtM/BAEC2SJg908idXjdZiDm1z///HNMDPsYz5ZslAsBCEAAAtknIBaEcuTSCpZFyYgRI5Tls0QsDcUalgABCEAAAsVBAEVJcfRzzltpf4kU5YgcpREHaBLER0nt2rXN3+vWrRuT7bvvvlNy9arlTb5FixamHxMCBLJJYOrUqbHiZcx5KebkyJg9NGnSJJtiUTYEIAABCOSQwMEHHxyrTRQlcizzn//8pxkn/8staAQIQAACECgeAihKiqevc9pS+xXBcvTmsssuM01Y9913X3X55ZfHZLFblIiiRI7kWMcbHn/8cdP6hACBbBGQ/OdQYQAAG+NJREFUq6rtVwOfddZZnlW9+uqrcc9OOOEEz7Q8gAAEIACBwiJgV4TMmzfP3KjZuHGjknWK+CYhQAACEIBAcRHAhrC4+jtnrbUrSjZv3qwmTpxo1v3II4/EKT/sFiWSTp5LuPHGG9VBBx2UM3mpqDgJzJkzJ3azkhAQx8FeYcKECbFHcl11w4YNvZISDwEIQAACBUagVatWMYlnzpyp5EfCqFGj1Pbbb19grUFcCEAAAhDIlAAWJZkSJL+WgN1SxEpwzjnnqI4dO8alr1WrluvaX/E+P3DgQG25REIgSAJ2axIp1+s4jShUpk2bFqu6b9++QYpBWRCAAAQgkGcCBx54oEuCM888M6EC3ZWBCAhAAAIQiAyBSsbNIuWRaQ0NCQ0BceBqv9FGdmO+/fbbmG8Su6B77rmnWrZsmRkljtK++uortc8++4SmLQgSXQJnnHFGzHFw1apV1aZNm7SN7datW8wqaq+99lILFizgWJiWFJEQgAAECpeAWLmKXzUJ1atXV0uWLIlbyxRuy5AcAhCAAARSJYBFSarESO+LgDhrrVSpUiztww8/rFWSSAL7VcIjR45ESeKLMImCIPDuu+/GipGjX7/88our2NGjR8eUJPJw+PDhKElclIiAAAQgUPgEdt9991gjHnzwQZQkhd+ltAACEIBA2gTwUZI2OjImIiBKkrffflvJy2flypVdR27seWUx8sMPP5gvn0cddVSiYnkGgcAILF26VImjYXt48cUXVY8ePcwocT786KOPqosvvjiW5J577lFdu3YNTIZACzKurlTvvFNRpFyr/e9/B1oFhUEAAhCIKoHJkyerzz77zGzeySefrHr16hXVptIuCEAAAhDwQYCjNz4gkQQCEIgegWeeeUbJ+XNnEEetTZs2VZ9//rl5xMYKd955p7r66qudycPzuUMHpWxXHav27ZV6773wyIckEIAABEJKQI7Y7Lfffmr9+vVKjld++umnascddwyptIgFAQhAAAK5IIBFSS4oUwcEIBA6Ah999JEp096G5UW/fv3M4zXvGBYZc+fONX8kiGXUKaecou644w7zamsCBCAAAQhEi8Dq1atNq1dRkmyzzTbq5ZdfRkkSrS6mNRCAAATSIoBFSVrYQpwJ8/sQdw6ihYnAb7/9pmbNmqVat26tatSoYYq2bt06tXjxYtOZX4MGDUx/OXJ0rCACFiUF0U0ICQEI5I+AWAlOmTJFtWzZUskNe8uXL1ddunRRZWVlplBPPfWU1tIwfxJTMwQgAAEI5IsAipJ8kc9WvbwsZYss5UIg3AT47oe7f5AOAhDIO4EBAwao2267TbVo0UKdddZZ6vbbb1dr16415br//vtN60ICBCAAAQhAQAhw9IZxAAEIQAACEIAABCAQeQLTpk0z2yg+qAYNGhRr77Bhw1CSRL73aSAEIACB1AigKEmNF6khAAEIQAACEIAABAqMwJYtW9T06dPjpG7YsKF68skn1aGHHlpgrUFcCEAAAhDINgEUJdkmTPkQgAAEIAABCEAAAnkl8Pvvv6srr7xSff/996p+/fqqVatWqlu3bjEfVXkVjsohAAEIQCB0BFCUhK5LEAgCEIAABCAAAQhAIEgC2223nXmDGQECEIAABCDgh8A2fhKRBgIQgAAEIAABCEAAAhCAAAQgAAEIFAMBFCXF0Mu0EQIQgAAEIAABCEAAAhCAAAQgAAFfBFCU+MJEIghAAAIQgAAEIAABCEAAAhCAAASKgQCKkmLoZdoIAQhAAAIQgAAEIAABCEAAAhCAgC8CKEp8YSIRBCAAAQhAAAIQgAAEIAABCEAAAsVAAEVJMfQybYQABCAAAQhAAAIQgAAEIAABCEDAFwEUJb4wkQgCEIAABCAAAQhAAAIQgAAEIACBYiCAoqQYepk2QgACEIAABCAAAQhAAAIQgAAEIOCLAIoSX5hIBAEIQAACEIAABCAAAQhAAAIQgEAxEKhSDI2kjRCAAAQiT6BPH6XkxwoNG0a+yTQQAhCAAAQgAAEIQAAC2SCAoiQbVCkTAhCAQK4J9OiR6xqpDwIQgAAEIAABCEAAApEkwNGbSHYrjYIABCAAAQhAAAIQgAAEIAABCEAgHQKVyo2QTkbyhJTAuHHxgon5/RFHhFRYxIIABCAAAQhAAAIQgAAEIAABCISLAIqScPVH7qX59ttg6txrr/TKyWf9+axbaOWz/nzWTdvT+644c6X7nXOWw2cIQAACEIAABCAAAQhAII4AipJiHxAdOig1dWrmFKZMSd1y5d13lTryyMzrbt9eqffeS70c2p46M2cO+t1JJPHnfI/5xNLxFAIQgAAEIAABCEAAAhAwCOCjhGEAAQhAAAIQgAAEIAABCEAAAhCAAAT+IoCihKEAAQhAAAIQgAAEIAABCEAAAhCAAAT+IoCihKEAAQhAAAIQgAAEIAABCEAAAhCAAAT+IoCPkmIfCs5bctLl0aNH6jnFoaj4uMg0pHuzD23PlLxS9HtqDPM95lOTltQQgAAEIAABCEAAAhAoSgIoSoqy22k0BCAAAQhAAAIQgAAEIAABCEAAAjoCHL3RUSEOAhCAAAQgAAEIQAACEIAABCAAgaIkgKKkKLudRkMAAhCAAAQgAAEIQAACEIAABCCgI4CiREeFOAhAAAIQgAAEIAABCEAAAhCAAASKkgCKkqLsdhoNAQhAAAIQgAAEIAABCEAAAhCAgI4AihIdFeIgAAEIQAACEIAABCAAAQhAAAIQKEoCKEqKsttpNAQgAAEIQAACEIAABCAAAQhAAAI6AihKdFSIgwAEIAABCEAAAhCAAAQgAAEIQKAoCaAoKcpup9EQgAAEIAABCEAAAhCAAAQgAAEI6AigKNFRIQ4CEIAABCAAAQhAAAIQgAAEIACBoiSAoqQou51GQwACEIAABCAAAQhAAAIQgAAEIKAjgKJER4U4CEAAAhCAAAQgAAEIQAACEIAABIqSAIqSoux2Gg0BCEAAAhCAAAQgAAEIQAACEICAjkAVXSRxEIAABIIiMGPGDPXyyy+bxXXs2NH8IUAAAhCAAAQgAAEIQAACEAgrARQlYe0Z5IJARAjcdddd6rnnnjNb06xZs4i0imZAAAIQgAAEIAABCEAAAlElUKncCFFtHO2CAATyS2Dt2rVqp512Un/++acpyIIFC1SjRo3yKxS1QwACEIAABCAAAQhAAAIQSEAAHyUJ4PAIAhDIjMCjjz4aU5Jsv/32KEkyw0luCEAAAhCAAAQgAAEIQCAHBDh6kwPIxV7FH3/8oaZPn67EV0Xnzp1VixYtXEg+//xz9fHHHyuxQGjdurU65JBDVJUq/ofnsmXL1GeffabmzZunKlWqpPbaay/VqlUr1bBhQ/NFfdSoUWrHHXdU3bt3V9ts49YPbtmyRb3//vvqo48+Ul26dFHNmzd3yfjVV1+ZMi5fvlzVrl3bbMfBBx/sSmdFZCKTMPvggw/MHy9mn376qfrkk0/Ur7/+qpo2baqOPfZYFzN5NnXqVCWy77zzzqpdu3aqcePGnjJ7PSgrK1NffvmlWrp0qapcubLJ98gjjzQ5eIVZs2apgQMHxh6LjFKOPVStWlXtv//+XkXExacqQzb69LvvvlMyVmWcSRCm8iNHipo0aeKrHSSCAAQgAAEIQAACEIAABEJOQI7eECCQLQJ33313efXq1eV4V+xn0KBBseq+/vrr8q5du8Y9l7RHHHFE+c8//5xUrEWLFpUbyg9Xfqu+E044odx4ETefGwqS8t9//z2uTEOJUn7HHXeUGy/scWXccMMNsXTffvtt+UknneSqo1evXlr5MpXpnnvuKTesL+LqGzBgQKyu+fPnlxtKEZc8Bx54YPkvv/wSS3fvvfeW16hRw5VuxIgRWrl1kY888ki5oWxylSE8DYVJ+ejRo13ZzjnnnHJD2aXNYx8H8rv0TbKQqgzZ6NM33nij3FAKJWyToTApHzZsWLLm8BwCEIAABCAAAQhAAAIQCDkBFXL5EK9ACRi7+eU9e/bUvljKC6e8zM6ePbvcsPLQppGX6AsuuCBh642bVMoN65FY/n333bdclDDGcY/yc88911Vuhw4d4soTGXXppG5LRsOCpNzwseEqS9LIC7wzZCJTImYigzw3LF5cShS78kEUPKIM8mJvKThWr17tFD3u84YNG8p79OgRa/cuu+xSftlll5UPHjy4vE6dOrF44W9YWcTyrlmzxlRIORUiXp8vvPBCTznSkSEbfXrbbbfFteeYY44pFyXUnXfeGVPCWe27/vrrPdvDAwhAAAIQgAAEIAABCECgMAigKCmMfio4KS+++GLz5XLbbbctHz9+fPnkyZPjXjanTJkSs+IQhYhxxMRUAuy6665xL+E//vijtu2TJk2KU5L07du3XF6s7cE4GhNXp92SRdL16dPHfC7WJM8//3z5xIkT49KLFYHIL2lOP/308scffzxOaWIcBYmrL1OZkjF77bXXYvL885//LDeOtpSLdYhdCXHUUUeVX3rppWbc7rvvXv7YY4+Z6YzjO3Hp3n33XS1XiRTFh3FsKZbeOK5TvmrVqlh6yWuvc8KECdqyHn744bh0xjEgbTpdZLoyBN2nxlGwuHEmijN7MI5IxSlLXnrpJV1ziIMABCAAAQhAAAIQgAAECogAipIC6qxCEtW4Dra8Zs2a5YZfElNsOb5if7k2bj4xP990001xzRo+fHhcunfeecfV7BUrVpQb/kti6U499VRXGomQ4yr2OkVZYw/PPPNM+Q477FBuvfwavkfi0peUlJifb7zxxlg2q0w5dmI/xhOETE5mixcvjpNnn332MT8LI3to0KBBXDpJU69evfJvvvkmlkyUGXYWL774YlwZ1gexyLD6RtKXlpaWb968OS7tpk2b4sp64okntGXZrXUOOOAAbRpdZCYyBNmnIpso0CxucrRJF+T4kZVGxhABAhCAAAQgAAEIQAACEChsAihKCrv/Qi29HK+xgliQ2F/U5ferrrrKJf/MmTPj0olFhDNYVgNShviFkOMeuiC+Pux16tLZZTScnrpkFMsNexAlgfhOsfsCkedByWSX580333TJc+utt7qaKkdB7O0UCxlRstiDk+u0adNc5UiE/aVfjtAYTktd6QynuXH1iRJMF/bYY49YOrFy8RsylSGoPhV5b7755lgb5JjYb7/95mqGWDLdcsst5lEcAgQgAAEIQAACEIAABCBQ+AT8XytivIkRIJAKAbl9xgpy64o9yG00xku/qzjDCiUuzlBIxH02HJmaN9hY4corr1S1atVylSMRCxYsiMXvt99+2nR2GeUGGXuQ22EM/xRxcXJLi/zYQ5Ay2eWxblax6jIsGtS1114bV7d8MJQicXEPPPCAeSuNPRjWJXGfdTcPGUogZfjYiKW75pprzNtc7EHS9O/fPxYl/WhYtMSlkQ8rV65UcuuPFdq3b+9Ko4sIQoYg+tSSzbCoiYkpNwgZjmrVk08+qQwHxbF4+d1+u4+uXcRBAAIQgAAEIAABCEAAAoVDwH1PauHIjqQFRECuBrYHw+GqqlatmqsFhrVGXJzhODTus3G0Iu6z4dfDVYYVYa9TrrJNFuR6YHswjpRoZXSWky2Z5Eplexg3bpx5Na89GBYOauHChbGotm3bKsPni1PE/2/vXkJ16uI4jq+3DOQyQGQiSTq5REduHUWRDNyS3AoDt6SMOBkZyQSZGCgxELlFRnKLgYGSUo5LJiSnKCFFUWi//9/O3q19c/Z7PM/z7v34rtJr72ftvf/7s57B+/yt9V/h1sxRsyU8uUmjY8eOhQmOqFmh1fjvtvzGWQ2XcMtlWw4Vnrf6LU7jmNfS493V1ZXXLXOukTHo5v0d0ygwJXiskG0cpy1ZcraMyNkMnUzsnEAAAQQQQAABBBBAAIH2EGBGSXuMY+XfwgqAxjHOnTvX2XKR3Jit1kfivNUJSRxbYdj4eM6cOc6Kv+beR7NJbFed+LMyMxqiBIAu0kyCmTNn5t47fbJZMfnx2Ha7YUzplp4FU5Q4un37dnzpwoUL07cJj8+ePZs4rxk/vb294cwQJWOsJkv8uZIHumdeTOpkS3vivlZU1o0ZMyZx76KDRsagZ/R3TKP4bLchZ/VdnL4/tqQnPC2LGTNmOKtdE84ksXo5Ra/DeQQQQAABBBBAAAEEEKihAP+HX8NBq1vIHz58cC9fvozDXrVqVeEr3L9/P/GZFRaNjzXbpKenJz4u+sGvDidOnEjcR8mZ3zWrs+EUZ9S0xKJMa1ZMSk74SaP169fnhmO7BcXnteRkxYoVmX5avuQnjWxnnEwfnXj06FHivNUKSRwPHjzYrVmzxtkOQE72eTOCogvu3r0bX1tmNk/UuZEx9HdM0ziaDfPs2TO3bt06ZzsdxR8rUWI7HTnbHclZ/ZL0ZRwjgAACCCCAAAIIIIBATQVIlNR04OoUdnoZxvLlywvDtwKm8WednZ3OX3rj17xQp6LZDJr5YFvTxvfRrJO8Ohp+EH7CQedtJ53CGP0PmhVTOp6VK1fmxuPP3NAsnbx6LX4f3WT+/PmZe+k9tLwmaqr9ohk7mjkiP/0ZPXp0ZulP5kZ2QnVGbCeh+KO+klRRx0bGoHumDcuOad47dXR0ONu+2llxV2eFW+MuOqfEn5IlNAQQQAABBBBAAAEEEGgPARIl7TGOlX4Lv06EltKMHz8+N17Vx/B/YKdnnrx58yZx3ciRI3Pvo1oaKrwZtaIZFP7FtuNNfKjlFqrjUaY1K6Yy8VgtaWe7CcVhFtUB8ZefKGFkWwdnXu3p06eJc5s3b3aTJ0/O9CtzQrNXFFvUZs2aVeYy18gY9MAyhqUC+9VJNVn279/vlixZ4hYvXhx/x5Tc0/e27FKt//JM+iKAAAIIIIAAAggggEDrBSjm2nrzv+6J/g9WzRIpatqtJWpaRrJx48ZE13ShVy0FSbcvX74429I3cVoFTvtq/lKRefPm9dU9/rxZMfk1XYriUR0WFXONWlFCwr9X0XKlz58/J95ZyaLfNX/2SbqfdgHym4qflmmNjEHP6++Y9hWrZtpcuXIl0e3Jkyd9XcbnCCCAAAIIIIAAAgggUBMBEiU1Gai6hqllMP7uLePGjct9lY8fP7ojR47En+3ZsyezXGbEiBGJa/2aItEHO3fudKpN4Td/i9sXL15knq8f6P4P3aKZGZkL7UQzYlKyx6/FUhRPelecvBkN8vd3aNGP/KjJ6cePH+Ghtvn127t37xLH/sGlS5fc0KFDnXYFymu+v3bpSSe0VNdj+/bt7ufPn4nLGxnDn4ypgtI2ycOGDXPnz5/Pe0W3YMGCxDbRX79+ze3HSQQQQAABBBBAAAEEEKifAImS+o1ZrSJW8ctotxAF/vz580z8WqaxYcOGeHaEZjNoN5F0S++c4tcz0T26u7vd6dOnnWZgaJlE1MaOHRv+VbvTqDiskjB+SxeQ9ZMJ6RjSx82ISXUv/FYUj7+kSfVD8pYiqSCsvwwmSkZotoUsVJxVbcKECYlnnjt3Lv2qYVJl9+7dbvXq1WE9Ez+x5Xf+9OlTfKhkiJ84uXnzplu2bFm4rfDly5cTz2hkDH8ypgrq+vXrTu9x586djINO6P39XYDKLtXKvRknEUAAAQQQQAABBBBAoFIC1Cip1HC0XzDpWQ/64aldWKKio/oxum3bNnft2rXw5QcOHOi0VCSvKKl+2KvGydu3b8O+hw8fDou9quaGirdqiY8SBidPnkz88NeP8iFDhsRFONM7lKRj1NavZVszYvITIIqjKB5/SU3RrBPN6PCbZoFohsm+ffvCGR3REht5K2ly8eLFsPuhQ4fCoqxaqqPaMSrIq+RBVJNFy2mU9Mhr6eTRli1bnIrRqo7HqVOnwku0i492z/FbI2P4kzHV7JBodxt9dxSrX+dG9W/27t0bJ6D0HSxTByfPinMIIIAAAggggAACCCBQQQH712YaAk0TsB/DquqZ+DNq1Khg06ZNgRXEDGxZRvyZ7awS2GyK38Zy9OjRzP2i+9tsieD169fh9dOnT8/td+DAgcz9LRkQ950yZUrm875ONDom272mz3isNkri/Q4ePFgYpu0OlGthy5QCS5bE17169SqwRFVuX38MNXaW7Cp8nhVl/e09rFBsYDMycq9vVAx/Mqa2S1AmftviOPzOWr2bxGc2Wyl4+PBh7rtwEgEEEEAAAQQQQAABBOopoH8VpSHQNAElP6If2bZNbGCzPzI/Qm22R2DbrgZWmLRUHLbzSOYeu3btCqzOSXy91dEIrI5G4tlXr17Nvb8f444dO3L79HWykTGVice2vk0Y2M42hSFeuHAh0Xf27NnBmTNncvvbFr3BokWLMr4aw6VLlwY3btzIvS598vjx44Etf0rcx3Y7CmxJT7pr5rgRMZQxzDz41wlbKhRYjZLAljwFVlQ418Jq0wRbt24NbHZJ0W04jwACCCCAAAIIIIAAAjUV+EdxV3CiCyG1gUBvb2+iIKuWxliyJFw6YzM/wjoPHR0d4fKZ/9q+ffvmHj9+7AYMGOAmTpwYLtlJNy0dUa2KqVOnxktM0n0aeVzFmKL3k7nqw0ybNs0NHz68z9fW8hP119IoLXfScpp0Uda+bqIdebQzjwr1qjaKCvlqN6OyrRExlH1WUT99h2T3/v17p/GNvm95S8OK7sF5BBBAAAEEEEAAAQQQqJcAiZJ6jVetolW9i7Vr18Yx64fzoEGDavUOBIsAAggggAACCCCAAAIIIPB3CbDrzd813i19W80gidqkSZNIkrRUn4chgAACCCCAAAIIIIAAAgj0R4BESX/UuKaUgLagjVpnZ2epa+iEAAIIIIAAAggggAACCCCAwP8pQKLk/9Rv42drmU1PT0/8hrY7SBu/La+GAAIIIIAAAggggAACCCDQLgIkStplJCv2Hg8ePEhEpOPv379XLErCQQABBBBAAAEEEEAAAQQQQCApQKKEb0RTBO7du5e4761bt8IaJfovDQEEEEAAAQQQQAABBBBAAIGqCgyoamDEVW+Brq4u193dnXkJbRFLQwABBBBAAAEEEEAAAQQQQKCqAmwPXNWRIS4EEEAAAQQQQAABBBBAAAEEEGi5AEtvWk7OAxFAAAEEEEAAAQQQQAABBBBAoKoCJEqqOjLEhQACCCCAAAIIIIAAAggggAACLRcgUdJych6IAAIIIIAAAggggAACCCCAAAJVFSBRUtWRIS4EEEAAAQQQQAABBBBAAAEEEGi5AImSlpPzQAQQQAABBBBAAAEEEEAAAQQQqKoAiZKqjgxxIYAAAggggAACCCCAAAIIIIBAywVIlLScnAcigAACCCCAAAIIIIAAAggggEBVBUiUVHVkiAsBBBBAAAEEEEAAAQQQQAABBFouQKKk5eQ8EAEEEEAAAQQQQAABBBBAAAEEqipAoqSqI0NcCCCAAAIIIIAAAggggAACCCDQcgESJS0n54EIIIAAAggggAACCCCAAAIIIFBVARIlVR0Z4kIAAQQQQAABBBBAAAEEEEAAgZYLkChpOTkPRAABBBBAAAEEEEAAAQQQQACBqgr8C+G8ZAxyqc33AAAAAElFTkSuQmCC"
    }
   },
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Auto Differentiation\n",
    "\n",
    "I cited the chain rule earlier but i didn't go into details about it. Basically the chain rule is the following:\n",
    "\n",
    "*The differentiation of p(x) = f(g(x)) is:*\n",
    "$p'(x) = f'(g(x)) * g'(x)$\n",
    "\n",
    "Here's an example to show you the power of this rule:\n",
    "\n",
    "Taking the following equation:\n",
    "\n",
    "y = a * x + b\n",
    "\n",
    "We can differentiate it by using the chain rule.\n",
    "y = f(g(x)) where f(x) = a * x and g(x) = x + b.\n",
    "\n",
    "Now we can differentiate f(x) on one side using the result of g(x) and differentiate g(x) then multiply these results to obtain the derivative of y.\n",
    "\n",
    "Here the calculation is simple but this logic can be applied to more complex calculations.\n",
    "\n",
    "This is the logic **pytorch** uses when using the ``backward`` method on a tensor.\n",
    "\n",
    "\n",
    "#### My Auto Differentiation :\n",
    "\n",
    "We will now recreate this very useful tool that is the base of neural network optimization.\n",
    "\n",
    "In order to recreate it, we must create our tensor class that will keep track of our gradient.\n",
    "We will wrap a numpy array in order to keep our data efficiently.\n",
    "(If you want to go further, you can make your own library in C to handle tensors. However it might be too difficult for the purposes of this notebook).\n",
    "\n",
    "A DAG, or Directed Acyclic Graph is the shape we go for when creating a graph of the operations to backward on using the chain rule.\n",
    "\n",
    "![image.png](attachment:image.png)\n",
    "\n",
    "Complete the following code:\n",
    "\n",
    "<details>\n",
    "<summary>Click here for Hints</summary>\n",
    "\n",
    "1) `MyCrossBackward` is the class that holds the tensors necessary to derive each operations. It also holds the differentiation function.\n",
    "\n",
    "2) The backward function just calls the `MyCrossBackward` object set on the object when performing operations.\n",
    "\n",
    "3) In python there are ways to override operations on objects for the interpreter. Here's an example:\n",
    "\n",
    "```python\n",
    "class TestObject:\n",
    "\n",
    "    def __init__(self, data : int):\n",
    "        self.data = data\n",
    "\n",
    "    def __add__(self, other):\n",
    "        return TestObject(self.data, other)\n",
    "\n",
    "a = TestObject(3)\n",
    "b = TestObject(4)\n",
    "a + b # is equivalent to a.__add__(b)\n",
    "```\n",
    "\n",
    "4) __div__ might not be right, __truediv__ ;)   \n",
    "5) Only operations on leaves need to keep the gradients (it's the variable), while intermediate result tensors do not need to be kept. As such, only gradients on leaves need to be stacked\n",
    "6) Here's a table to help you when writing the backwards functions:\n",
    "\n",
    "| Function | Derivative |\n",
    "|----------|------------|\n",
    "| $x^{n}$  | $nx^{n-1}$ |\n",
    "| $ax$     | $a$        |\n",
    "| $x + a$  | $1$        |\n",
    "| $x/a$    | $1/a$      |\n",
    "| $1/x$    | $-1/x^{2}$ |\n",
    "\n",
    "</details>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 491,
   "metadata": {},
   "outputs": [],
   "source": [
    "from typing import Any, Callable\n",
    "import numpy as np\n",
    "\n",
    "class MyCrossBackward:\n",
    "    def __init__(self, *args, backward_function: Callable[..., None], name: str) -> None:\n",
    "        self.__data = list(args)\n",
    "        self.__bf = backward_function\n",
    "        self.__name = name\n",
    "\n",
    "    def __str__(self) -> str:\n",
    "        return self.__name\n",
    "\n",
    "    def __call__(self, x) -> Any:\n",
    "        return self.run(x)\n",
    "\n",
    "    def run(self, base) -> None:\n",
    "        return self.__bf(base, *self.__data)\n",
    "\n",
    "class MyTensor:\n",
    "    def __init__(self, data: list | np.ndarray | int | float, dtype: np.dtype = None, requires_grad: bool = False, crossOpBack: 'MyCrossBackward' = None) -> None:\n",
    "        assert isinstance(data, (list, np.ndarray, int)), \"The given data is of an invalid type\"\n",
    "\n",
    "        # Initialize the following properties\n",
    "        self.__data = np.array(data)\n",
    "        self.dtype = dtype if dtype is not None else self.__data.dtype\n",
    "        self.requires_grad = requires_grad\n",
    "        self.grad = None\n",
    "        self.backward_op = crossOpBack\n",
    "\n",
    "    @property\n",
    "    def data(self):\n",
    "        return self.__data\n",
    "\n",
    "    @property\n",
    "    def shape(self):\n",
    "        return self.__data.shape\n",
    "\n",
    "    @property\n",
    "    def strides(self):\n",
    "        return self.__data.strides\n",
    "\n",
    "    def __getitem__(self, index):\n",
    "        return self.__data[index]\n",
    "\n",
    "    def __setitem__(self, index, value):\n",
    "        self.__data[index] = value\n",
    "\n",
    "    def __add__(self, other):\n",
    "        return MyTensor(self.__data + other.data,\n",
    "                        requires_grad=self.requires_grad or other.requires_grad,\n",
    "                        crossOpBack=MyCrossBackward(self, other, backward_function=add_deriv, name=\"AddBackward\")\n",
    "                        if self.requires_grad or other.requires_grad else None)\n",
    "\n",
    "    def __mul__(self, other):\n",
    "        return MyTensor(self.__data * other.data,\n",
    "                        requires_grad=self.requires_grad or other.requires_grad,\n",
    "                        crossOpBack=MyCrossBackward(self, other, backward_function=mul_deriv, name=\"MulBackward\")\n",
    "                        if self.requires_grad or other.requires_grad else None)\n",
    "\n",
    "    def __truediv__(self, other):\n",
    "        return MyTensor(self.__data / other.data,\n",
    "                        requires_grad=self.requires_grad or other.requires_grad,\n",
    "                        crossOpBack=MyCrossBackward(self, other, backward_function=div_deriv, name=\"DivBackward\")\n",
    "                        if self.requires_grad or other.requires_grad else None)\n",
    "\n",
    "    def __pow__(self, other):\n",
    "        return MyTensor(self.__data ** other,\n",
    "                        requires_grad=self.requires_grad,\n",
    "                        crossOpBack=MyCrossBackward(self, other, backward_function=pow_deriv, name=\"PowBackward\")\n",
    "                        if self.requires_grad else None)\n",
    "\n",
    "    def __sub__(self, other):\n",
    "        return MyTensor(self.__data - other.data,\n",
    "                        requires_grad=self.requires_grad or other.requires_grad,\n",
    "                        crossOpBack=MyCrossBackward(self, other, backward_function=sub_deriv, name=\"SubBackward\")\n",
    "                        if self.requires_grad or other.requires_grad else None)\n",
    "\n",
    "    def backward(self, base=None) -> None:\n",
    "        if base is None:\n",
    "            base = MyTensor([1])\n",
    "        if self.backward_op:\n",
    "            self.backward_op(base)\n",
    "\n",
    "\n",
    "\n",
    "def add_deriv(base, *tensors):\n",
    "    \"\"\" the tensors we will consider are only of the same shape \"\"\"\n",
    "    left : MyTensor = tensors[0]\n",
    "    right : MyTensor = tensors[1]\n",
    "\n",
    "    if (left.requires_grad):\n",
    "        grad = MyTensor(base.data * np.ones_like(right.data))\n",
    "        if (left.backward_op == None):\n",
    "            left.grad = left.grad and left.grad + grad or grad \n",
    "        left.backward(grad)\n",
    "    \n",
    "    if (right.requires_grad):\n",
    "        grad = MyTensor(base.data * np.ones_like(left.data))\n",
    "        if (right.backward_op == None):\n",
    "            right.grad = right.grad and right.grad + grad or grad\n",
    "        right.backward(grad)\n",
    "\n",
    "def mul_deriv(base, *tensors):\n",
    "    \"\"\" tensors[0] BxC, tensors[1] BxC \"\"\"\n",
    "    left : MyTensor = tensors[0]\n",
    "    right : MyTensor = tensors[1]\n",
    "\n",
    "    if left.requires_grad:\n",
    "        grad = MyTensor(base.data.T * right.data)\n",
    "        if (left.backward_op == None):\n",
    "            left.grad = left.grad and left.grad + grad or grad\n",
    "        left.backward(grad)\n",
    "    \n",
    "    if right.requires_grad:\n",
    "        grad = MyTensor(left.data.T * base.data)\n",
    "        if (right.backward_op == None):\n",
    "            right.grad = right.grad and right.grad + grad or grad\n",
    "        right.backward(grad)\n",
    "\n",
    "def div_deriv(base, *tensors):\n",
    "    \"\"\" tensors of the same shape \"\"\"\n",
    "    \n",
    "    unpacked_tensors : list[MyTensor] = list(tensors)\n",
    "\n",
    "    zis_tensor : bool = isinstance(unpacked_tensors[0], MyTensor)\n",
    "    is_tensor : bool = isinstance(unpacked_tensors[1], MyTensor)\n",
    "\n",
    "    if (zis_tensor and unpacked_tensors[0].requires_grad):\n",
    "        grad = MyTensor(base.data * (1 / (unpacked_tensors[1].data if is_tensor else unpacked_tensors[1])))\n",
    "        if (unpacked_tensors[0].backward_op == None):\n",
    "            unpacked_tensors[0].grad = unpacked_tensors[0].grad and unpacked_tensors[0].grad + grad or grad\n",
    "        unpacked_tensors[0].backward(grad, forced = True)\n",
    "\n",
    "    if (is_tensor and unpacked_tensors[1].requires_grad):\n",
    "        grad = MyTensor(-base.data * ((unpacked_tensors[0].data if zis_tensor else unpacked_tensors[0].data) / np.power(unpacked_tensors[1].data, 2)))\n",
    "        if (unpacked_tensors[1].backward_op == None):\n",
    "            unpacked_tensors[1].grad = unpacked_tensors[1].grad and unpacked_tensors[1].grad + grad or grad\n",
    "        unpacked_tensors[1].backward(grad)\n",
    "\n",
    "def pow_deriv(base, *args):\n",
    "    \"\"\" tensor, power \"\"\"\n",
    "    tensor : MyTensor = args[0]\n",
    "    pw : int = args[1]\n",
    "\n",
    "    grad = MyTensor(base.data * (pw * np.power(tensor.data, pw - 1)))\n",
    "    if tensor.backward_op == None:\n",
    "        tensor.grad = tensor.grad and tensor.grad + grad or grad\n",
    "    tensor.backward(grad)\n",
    "\n",
    "def sub_deriv(base, *tensors):\n",
    "    \n",
    "    unpacked_tensors : list[MyTensor] = list(tensors)\n",
    "\n",
    "    if (unpacked_tensors[0].requires_grad):\n",
    "        grad = MyTensor(base.data * np.ones_like(tensors[1].data))\n",
    "        if (unpacked_tensors[0].backward_op == None):\n",
    "            unpacked_tensors[0].grad = unpacked_tensors[0].grad and unpacked_tensors[0] + grad or grad\n",
    "        unpacked_tensors[0].backward(grad)\n",
    "\n",
    "    if (unpacked_tensors[1].requires_grad):\n",
    "        grad = MyTensor(-1 * base.data * np.ones_like(tensors[0].data))\n",
    "        if (unpacked_tensors[1].backward_op == None):\n",
    "            unpacked_tensors[1].grad = unpacked_tensors[1].grad and unpacked_tensors[1] + grad or grad\n",
    "        unpacked_tensors[1].backward(grad)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's test your code ! (They are not easy to pass but are mandatory)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 492,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Checking attributes...\n",
      "Checking list properties...\n",
      "Checking operations...\n",
      "Checking backwards...\n"
     ]
    }
   ],
   "source": [
    "tests.test_tensor_class(MyTensor)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Pytorch Architecture\n",
    "\n",
    "Pytorch decided to go with a component style architecture working by modules. Each module is a mathematical tranformation of a data *x* to a result *y*.\n",
    "A module is a class that each Neural Network module inherits from.\n",
    "\n",
    "This class automatically regulates the parameters and adding of sub modules in the network by controlling the __setattr__ method of python classes.\n",
    "\n",
    "Please complete the code below"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 493,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "\n",
    "class MyTensor:\n",
    "    def __init__(self, data, dtype=None, requires_grad=False):\n",
    "        assert isinstance(data, (list, np.ndarray, int, float)), \"The given data is of an invalid type\"\n",
    "\n",
    "        # Convertir la liste ou le scalaire en un tableau NumPy\n",
    "        if isinstance(data, (list, int, float)):\n",
    "            data = np.array(data)\n",
    "\n",
    "        # Reste de l'initialisation\n",
    "        self.__data = np.array(data, dtype=dtype)\n",
    "        self.requires_grad = requires_grad\n",
    "\n",
    "    @property\n",
    "    def data(self):\n",
    "        return self.__data\n",
    "\n",
    "class Parameter:\n",
    "    def __init__(self, tensor: MyTensor) -> None:\n",
    "        self.tensor: MyTensor = tensor\n",
    "        self.tensor.requires_grad = True\n",
    "        self.__name: str = None\n",
    "\n",
    "    @property\n",
    "    def name(self):\n",
    "        return self.__name\n",
    "    \n",
    "    @name.setter\n",
    "    def name(self, x):\n",
    "        self.__name = x\n",
    "\n",
    "class MyModule:\n",
    "    def __init__(self) -> None:\n",
    "        self.__parameters: list[Parameter] = []\n",
    "        self.__sub_modules: list[MyModule] = []\n",
    "\n",
    "    def __setattr__(self, __name: str, __value: Any) -> None:\n",
    "        if isinstance(__value, Parameter):\n",
    "            __value.name = __name\n",
    "            self.__parameters.append(__value)\n",
    "\n",
    "        if isinstance(__value, MyModule) and __value != self:\n",
    "            self.__sub_modules.append(__value)\n",
    "\n",
    "        self.__dict__.__setitem__(__name, __value)\n",
    "\n",
    "    def __call__(self, x) -> Any:\n",
    "        \"\"\"This function calls the forward of its child.\"\"\"\n",
    "        return self.forward(x)\n",
    "\n",
    "    def parameters(self):\n",
    "        \"\"\"\n",
    "        This function returns the list of all parameters.\n",
    "        It should also include parameters from sub-modules.\n",
    "        \"\"\"\n",
    "        all_parameters = self.__parameters.copy()\n",
    "        for submodule in self.__sub_modules:\n",
    "            all_parameters.extend(submodule.parameters())\n",
    "        return all_parameters"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's test it ! (only 1 test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 494,
   "metadata": {},
   "outputs": [],
   "source": [
    "tests.test_my_module(MyModule, MyTensor, Parameter)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Base Layers\n",
    "\n",
    "In this section, we will recreate the 'Linear' and 'ReLU' layers. If you didn't know, they are the ones that are the most used because, according to the universal approximation theorem, any function can be represented by using x number of relu and linear layers.\n",
    "\n",
    "To make it short, relu is a function equivalent to max(x, 0) or mathematicaly\n",
    "$$\n",
    "\\mathrm{F}(x) = \\begin{cases}\n",
    "    0 & \\text{if } x < 0 \\\\ % & is your \"\\tab\"-like command (it's a tab alignment character)\n",
    "    x & \\text{otherwise.}\n",
    "\\end{cases}\n",
    "$$\n",
    "\n",
    "It is useful for a neural network to represent threshold.\n",
    "And Linear layers and basically linear transformations of vector of entry to another vector of another form.\n",
    "$$\n",
    "    y = ax + b\n",
    "$$\n",
    "\n",
    "The problem is that when passing from one linear layer to another, as we want to have n result from m entries, the weight is often of the form nxm.\n",
    "therefore:\n",
    "$$\n",
    "    nxm * mx1 -> nx1\n",
    "$$\n",
    "This transformation is known as *matrix multiplication*\n",
    "We will need to code this transformation before coding the Linear layer.\n",
    "\n",
    "#### Now, let's code\n",
    "\n",
    "Complete the code below\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 495,
   "metadata": {},
   "outputs": [],
   "source": [
    "def matmul_deriv(base : MyTensor, *tensors):\n",
    "    ...\n",
    "\n",
    "def matmul(a : MyTensor, b : MyTensor):\n",
    "    ...\n",
    "\n",
    "class Linear(MyModule):\n",
    "\n",
    "    def __init__(self, input_size : int, output_size : int) -> None:\n",
    "\n",
    "        # init the MyModule class\n",
    "        super().__init__()\n",
    "\n",
    "        # code here (init 2 private parameters: weight (a), bias (b)) based on the input_size and output_size\n",
    "        ...\n",
    "    \n",
    "    def forward(self, x : MyTensor):\n",
    "        \"\"\"this function may return the linear transformation of x by the layer \n",
    "\n",
    "        x * weight + bias\n",
    "\n",
    "        Args:\n",
    "            x (MyTensor): The tensor to transform\n",
    "        \"\"\"\n",
    "        ...\n",
    "\n",
    "def relu(x : MyTensor) -> MyTensor:\n",
    "    \"\"\" code the relu transformation here\n",
    "        Hint : relu is just a multiplication of a matrix by a mask of 0 and 1 isn't it?\n",
    "        *use np.where and np.matmul()*\n",
    "    \"\"\"\n",
    "    # code here\n",
    "    ...\n",
    "    # end of code\n",
    "\n",
    "class ReLU(MyModule):\n",
    "\n",
    "    def __init__(self) -> None:\n",
    "        \"\"\" The relu class is just a transformation without any weight \"\"\"\n",
    "        super().__init__()\n",
    "        pass\n",
    "\n",
    "    def forward(self, x : MyTensor):\n",
    "        return relu(x)\n",
    "        "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You know what time it is? Testing time !"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 496,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Testing Linear forward..\n"
     ]
    },
    {
     "ename": "AttributeError",
     "evalue": "'NoneType' object has no attribute 'data'",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mAttributeError\u001b[0m                            Traceback (most recent call last)",
      "Cell \u001b[0;32mIn[496], line 1\u001b[0m\n\u001b[0;32m----> 1\u001b[0m \u001b[43mtests\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mtest_linear\u001b[49m\u001b[43m(\u001b[49m\u001b[43mLinear\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mMyTensor\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mParameter\u001b[49m\u001b[43m)\u001b[49m\n",
      "File \u001b[0;32m~/Desktop/minitorch-mehdib13/tests.py:422\u001b[0m, in \u001b[0;36mtest_linear\u001b[0;34m(LinearModule, TensorClass, ParameterClass)\u001b[0m\n\u001b[1;32m    419\u001b[0m b\u001b[38;5;241m.\u001b[39mweight \u001b[38;5;241m=\u001b[39m t\u001b[38;5;241m.\u001b[39mnn\u001b[38;5;241m.\u001b[39mParameter(t\u001b[38;5;241m.\u001b[39mtensor([[\u001b[38;5;241m2\u001b[39m, \u001b[38;5;241m2\u001b[39m], [\u001b[38;5;241m2\u001b[39m, \u001b[38;5;241m2\u001b[39m], [\u001b[38;5;241m2\u001b[39m, \u001b[38;5;241m2\u001b[39m], [\u001b[38;5;241m2\u001b[39m, \u001b[38;5;241m2\u001b[39m]], dtype\u001b[38;5;241m=\u001b[39mt\u001b[38;5;241m.\u001b[39mfloat32))\n\u001b[1;32m    420\u001b[0m b\u001b[38;5;241m.\u001b[39mbias \u001b[38;5;241m=\u001b[39m t\u001b[38;5;241m.\u001b[39mnn\u001b[38;5;241m.\u001b[39mParameter(t\u001b[38;5;241m.\u001b[39mtensor([\u001b[38;5;241m2\u001b[39m, \u001b[38;5;241m2\u001b[39m, \u001b[38;5;241m2\u001b[39m, \u001b[38;5;241m2\u001b[39m], dtype\u001b[38;5;241m=\u001b[39mt\u001b[38;5;241m.\u001b[39mfloat32))\n\u001b[0;32m--> 422\u001b[0m y \u001b[38;5;241m=\u001b[39m \u001b[43ma\u001b[49m\u001b[43m(\u001b[49m\u001b[43mTensorClass\u001b[49m\u001b[43m(\u001b[49m\u001b[43m[\u001b[49m\u001b[38;5;241;43m3\u001b[39;49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[38;5;241;43m4\u001b[39;49m\u001b[43m]\u001b[49m\u001b[43m)\u001b[49m\u001b[43m)\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mdata\u001b[49m\n\u001b[1;32m    423\u001b[0m y_ \u001b[38;5;241m=\u001b[39m b(t\u001b[38;5;241m.\u001b[39mtensor([\u001b[38;5;241m3\u001b[39m, \u001b[38;5;241m4\u001b[39m], dtype\u001b[38;5;241m=\u001b[39mt\u001b[38;5;241m.\u001b[39mfloat32))\u001b[38;5;241m.\u001b[39mdetach()\u001b[38;5;241m.\u001b[39mnumpy()\n\u001b[1;32m    424\u001b[0m \u001b[38;5;28;01massert\u001b[39;00m (y \u001b[38;5;241m==\u001b[39m y_)\u001b[38;5;241m.\u001b[39mall(), \u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mLinear result is not equivalent to torch\u001b[39m\u001b[38;5;124m'\u001b[39m\u001b[38;5;124ms\u001b[39m\u001b[38;5;124m\"\u001b[39m\n",
      "\u001b[0;31mAttributeError\u001b[0m: 'NoneType' object has no attribute 'data'"
     ]
    }
   ],
   "source": [
    "tests.test_linear(Linear, MyTensor, Parameter)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Optimizer\n",
    "\n",
    "An optimizer is a class that takes the ``parameters()`` of a network when initializing and that has 2 methods: ``zero_grad()`` and ``step()``.\n",
    "\n",
    "``zero_grad``: sets all the gradients of the parameters of the module and sub_modules to ``None``.\n",
    "``step``: Apply the gradients with the optimizer's strategy.\n",
    "\n",
    "Different optimizers have different strategies, let's create the easiest one in this workshop: ``SGD`` or Stochastic Gradient Descent.\n",
    "\n",
    "SGD works by optimizing a loss function (function that is at its lowest when the network perform well), by making the parameters converge to the minimum by reducing them or augmenting them depending on their measured impact through ``differentiation`` (the impact is basically the gradients).\n",
    "\n",
    "So basically, with $\\displaystyle \\frac{\\partial L}{\\partial W}$ being the partial derivative of the Loss *L* to *W* being the weights, the W is updated by:\n",
    "$$\n",
    "    W = W - lr * \\displaystyle \\frac{\\partial L}{\\partial W}\n",
    "$$\n",
    "Where $ lr $ is the learning rate. You can see it has the speed of convergence. It is like speed in a video game: if you are fast you travel large distances quickly but its hard to go on a precise position. It's the same idea here. It's often seen as a tradeoff between training speed and precision.\n",
    "\n",
    "#### Let's code it\n",
    "\n",
    "Let's stop with the theory, let's apply it!\n",
    "\n",
    "Complete the code below:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "class SGD:\n",
    "\n",
    "    def __init__(self, parameters : list[Parameter], lr : float = 0.1e-3) -> None:\n",
    "        self.lr = lr\n",
    "        self.params = parameters\n",
    "\n",
    "    def zero_grad(self):\n",
    "        ...\n",
    "\n",
    "    def step(self):\n",
    "        ...\n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "TESSSSTING !"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "tests.test_sgd(SGD, MyTensor, Parameter)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Loss function\n",
    "\n",
    "In Pytorch, a loss function is basically a Python object that overrides the ``__call__`` method.\n",
    "\n",
    "We will now recreate the most known loss function that works for 90% of supervised ML problems : MSELoss aka Mean Squared Error.\n",
    "As its name implies, it's the mean of the error between a target value *ŷ* and the output of the network *y*.\n",
    "Mathematically:\n",
    "$$\n",
    "    \\frac{1}{n}\\sum_{i=0}^{n}(ŷ_{i} - y_{i})^2\n",
    "$$\n",
    "\n",
    "Maybe you understood it but you may not be able to make this one without the ``sum`` function and its backward. I'll drop this one for you ;)\n",
    "\n",
    "#### Ok, know let's code it\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from typing import Tuple\n",
    "\n",
    "def sum_deriv(base_tensor : MyTensor, *tensors):\n",
    "    from_tensor : MyTensor = tensors[0]\n",
    "    axis : Tuple[int] = tensors[1]\n",
    "\n",
    "    if (from_tensor.requires_grad):\n",
    "        k = base_tensor.data.copy()\n",
    "        if (axis is not None):\n",
    "            o = np.array(from_tensor.shape)\n",
    "            o[list(axis)] = np.ones(len(axis), dtype=int)\n",
    "            k = k.reshape(tuple(o))\n",
    "        k = np.broadcast_to(k, from_tensor.shape)\n",
    "        if (from_tensor.backward_op == None):\n",
    "            from_tensor.grad += MyTensor(k)\n",
    "        else:\n",
    "            from_tensor.grad = MyTensor(k)\n",
    "        from_tensor.backward(from_tensor.grad, forced=True)\n",
    "\n",
    "def sum(x : MyTensor, axis : Tuple[int]):\n",
    "    return MyTensor(np.sum(x.data, axis=axis), requires_grad=x.requires_grad,\n",
    "                    crossOpBack=MyCrossBackward(x, backward_function=sum_deriv, name=\"SumBackward\") if x.requires_grad else None)\n",
    "\n",
    "class MSELoss:\n",
    "\n",
    "    def __init__(self) -> None:\n",
    "        pass\n",
    "\n",
    "    def __call__(self, y, y_target) -> Any:\n",
    "        # code here\n",
    "        ..."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Testing again :/ (last time! i promise)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "tests.test_mseloss(MSELoss, MyTensor)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Make a Model !\n",
    "\n",
    "Ok, with what you have, you can now create a basic model.\n",
    "\n",
    "Let's see your creativity at work !"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "class SuperModel(MyModule):\n",
    "\n",
    "    ...\n",
    "\n",
    "m = SuperModel()\n",
    "\n",
    "loss_fct = MSELoss()\n",
    "\n",
    "optimizer = SGD(m.parameters())\n",
    "\n",
    "# use the one you would like\n",
    "dataset = ...\n",
    "\n",
    "epochs = 10\n",
    "for i in range(epochs):\n",
    "\n",
    "    for batch, target in dataset:\n",
    "\n",
    "        optimizer.zero_grad()\n",
    "\n",
    "        y = m(batch)\n",
    "        loss = loss_fct(y, target)\n",
    "\n",
    "        # calculate the gradients\n",
    "        loss.backward()\n",
    "\n",
    "        # apply them\n",
    "        optimizer.step()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Conclusion\n",
    "\n",
    "Well done, you have made a mini version of torch. With this concept in mind you may be able to see the possibility to extend this work to a library of your!"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.12"
  },
  "orig_nbformat": 4
 },
 "nbformat": 4,
 "nbformat_minor": 2
}