tutorials/Example_4_symbolic_regression.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "134e7f9d",
   "metadata": {},
   "source": [
    "# Example 4: Symbolic Regression"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2571d531",
   "metadata": {},
   "source": [
    "The symbolic space is very dense, which means getting the correct symbolic formula (if existing at all) is a hard task. We will show how sentitive symbolic regression is, especially in the presence of noise. This is good or bad:\n",
    "\n",
    "**Good**: One can easily find symbolic formulas that match with data quite well (within some tolerable epsilon). When one does not care about the exact symbolic formula, they might be happy with these approximate symbolic formulas that fit their data well. These approximate symbolic formulas provide some level of insight, have predictive power and are easy to compute.\n",
    "\n",
    "**Bad**: It's hard to find the exact formula. When one does care about the exact formula, we either care about (i) its generalizability in future cases (like Newton's gravity), or (ii) fitting the clean data or solving a PDE as precise as machine precision. For case (i), it is open-ended and requires case-by-case analysis. For case (ii), we can get a (hopefully) clear signal of the correctness of a symbolic formula by noticing the loss to go down to near machine precision. We will use an example to demonstrate this below."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7c308c65",
   "metadata": {},
   "source": [
    "## Part I: Automated vs manual symbolic regression (How can we know that we get the exact formula?)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "2075ef56",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(torch.Size([1000, 2]), torch.Size([1000, 1]))"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from kan import *\n",
    "# create a KAN: 2D inputs, 1D output, and 5 hidden neurons. cubic spline (k=3), 5 grid intervals (grid=5).\n",
    "model = KAN(width=[2,5,1], grid=5, k=3, seed=0)\n",
    "\n",
    "# create dataset f(x,y) = exp(sin(pi*x)+y^2)\n",
    "f = lambda x: torch.exp(torch.sin(torch.pi*x[:,[0]]) + x[:,[1]]**2)\n",
    "dataset = create_dataset(f, n_var=2)\n",
    "dataset['train_input'].shape, dataset['train_label'].shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "8aa1966d",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "train loss: 1.54e-01 | test loss: 1.30e-01 | reg: 2.02e+01 : 100%|██| 20/20 [00:15<00:00,  1.26it/s]\n"
     ]
    }
   ],
   "source": [
    "# train the model\n",
    "model.train(dataset, opt=\"LBFGS\", steps=20, lamb=0.01, lamb_entropy=10.);"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "943d9182",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZcAAAFICAYAAACcDrP3AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjYuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8o6BhiAAAACXBIWXMAAA9hAAAPYQGoP6dpAAAxuUlEQVR4nO3deXBUx4EG8K9HM5JG6D5ACAmEhE4EAowEGDtgwBKsbDCRy+sj63jZymazsV22K1upTTmbZLe8laNStuNNXN5KUgHMOj6EOQyLuA3GCItboAMJCYEAg67RNbpmpvcPPL08DltCT3pzfL+q/PEazahNXvO9Pl63kFJKEBER6chkdAWIiMj3MFyIiEh3DBciItIdw4WIiHTHcCEiIt0xXIiISHcMFyIi0h3DhYiIdMdwISIi3TFciIhIdwwXIiLSHcOFiIh0x3AhIiLdMVyIiEh3DBciItKd2egKEHkDKSVaW1vR3d2N0NBQxMTEQAhhdLWIPBZ7LkRfw2az4c0330RaWhri4uIwdepUxMXFIS0tDW+++SZsNpvRVSTySIInURLdWWlpKYqLi2G32wHc6L24uXstISEhKCkpQWFhoSF1JPJUDBeiOygtLUVRURGklHC5XHf9OZPJBCEEtm3bxoAhugnDhegWNpsNiYmJ6O3t/dpgcTOZTLBarWhqakJkZOToV5DIC3DOhegWa9euhd1uH1KwAIDL5YLdbse6detGuWZE3oM9F6KbSCmRlpaG+vp6DKdpCCGQkpKC2tpariIjAsOFSKOlpQVxcXEj+nxMTIyONSLyThwWI7pJd3f3iD7f1dWlU02IvBvDhegmoaGhI/p8WFiYTjUh8m4MF6KbxMTEIDU1ddjzJkIIpKamIjo6epRqRuRdGC5ENxFC4IUXXrinz7744ouczCf6Cif0iW7B91yIRo49F6JbREZGoqSkBEIImExf30Tcb+hv3LiRwUJ0E4YL0R0UFhZi27ZtsFqtEELcNtzlLrNardi+fTsKCgoMqimRZ2K4EN1FYWEhmpqa8MYbbyAlJUXzZykpKXjjjTdw+fJlBgvRHXDOhWgIpJQ4c+YMXn/9dbz88svIycnh5D3R12DPhWgIhBCIiopCeHg4oqKiGCxE34DhQkREumO4EBGR7hguRESkO4YLERHpjuFCRES6Y7gQEZHuGC5ERKQ7hgsREemO4UJERLpjuBARke4YLkREpDuGCxER6Y7hQkREumO4EA1RREQEVq9ejYiICKOrQuTxeJ4L0RA5nU7Y7XaEhIQgICDA6OoQeTSGCxER6Y7DYkREpDuz0RUgcnO5XLhw4QL6+vqMrorXS0pKQlhYmNHVID/GcCGP4XA4sHHjRgwODiIoKMjo6nit5uZmPPPMM8jJyTG6KuTHGC7kUYQQeOyxx5CWlmZ0VbyCy+XC6dOnkZiYiNjYWDidTrzzzjvgVCoZjeFCHicgIABmM2/NbyKlxKVLl1BSUoKgoCAsXboUc+bMMbpaRAAYLkRea2BgADt27EBvby96e3uxadMmLpEmj8HVYkReSEqJY8eOoba2VpXFx8cjOzvbwFoR/T+GC5GXkVKira0Ne/bsgcvlAgCYzWasWLECoaGhBteO6AaGC5GXcblc2LdvH1pbW1VZbm4uMjMzDawVkRbDhciLSCnR0NCA8vJyVRYeHo6HH36Y8y3kURguRF6kv78fpaWl6O/vB3Bj6fbixYsxfvx4CCEMrh3R/2O4EHkJKSWOHz+O8+fPq7KkpCTMnz8fJhObMnkW3pFEXkBKCZvNhr1796pJfIvFgsLCQlitVoNrR3Q7hguRF5BS4tNPP0VLS4sqy83NRUZGBofDyCMxXIg8nPtN/CNHjqiysLAwLFu2jJP45LEYLkQebnBwEDt37kRvb68qe+CBBziJTx6N4ULkwaSUqKysRHV1tSpLSEjA/fffz0l88mi8O4k8WE9PD3bt2gWn0wngxqaey5Yt45v45PEYLkQeyuVyoaysDFeuXFFlGRkZyMnJ4XAYeTyGC5EHklKitbUVBw8eVGezBAcH4+GHH4bFYjG4dkTfjOFC5IHc+4d1dHSosry8PEyZMoW9FvIKDBciDyOlxMWLF3H8+HFVFhUVhUWLFjFYyGswXIg8zODgIHbv3o2+vj4AN/YP+9a3voWYmBiGC3kNhguRB7nb0uP8/HwGC3kVhguRB7Hb7dizZ49m6fHSpUsREhJicM2IhofhQuQhpJQoLy9HU1OTKktPT+fSY/JKDBciD+A+uvjAgQNq6XFQUBCWLVvGpcfklRguRB5ASokDBw6gra1Nlc2ZMwfJycnstZBXYrgQGUxKicuXL+OLL75QZeHh4XjooYe4fxh5Ld65RAZzOp3YvXu3ZtfjhQsXIjY2lr0W8loMFyIDSSlRU1ODs2fPqrL4+Hjuekxej3cvkYH6+vqwa9cuOBwOAIDJZMKSJUu46zF5PYYLkUGklDh+/DguXryoylJSUpCbm8vhMPJ6DBciA0gp0dHRgX379sHlcgEALBYLHn74YQQGBhpcO6KRY7gQGUBKic8++wwtLS2qLDc3F6mpqey1kE9guBCNMSklvvzyS5SVlamy0NBQLFmyBAEBAQbWjEg/DBeiMeZ0OrFnzx50d3ersgULFiA+Pp69FvIZDBeiMSSlRF1dHU6fPq3K4uLisHDhQi49Jp/Cu5loDPX392Pnzp0YHBwEcGPp8UMPPYSIiAiDa0akL4YL0RiRUuLYsWO4cOGCKktOTsacOXM4HEY+h+FCNAaklLDZbLctPS4oKEBQUJDBtSPSH8OFaAxIKbF///7blh5PmzaNvRbySQwXolEmpcSlS5c0ux6HhYVh6dKlXHpMPovhQjTKBgcHUVpaqtn1+MEHH8SECRPYayGfxXAhGkVSSpw8eRI1NTWqLCEhgbsek8/j3U00StyT+Lt27YLT6QQAmM1mFBQUYNy4cQbXjmh0MVyIRonL5cLevXvR3NysynJycjB9+nQOh5HPY7gQjQIpJc6fP3/bJH5BQQHMZrOBNSMaGwwXolFgt9uxbds29Pf3AwCEEFi0aBH3DyO/wXAh0pnL5cKBAwc0h4AlJydzEp/8Cu90Ih1JKXHhwgUcOHAAUkoAQHBwMP7mb/4GVqvV4NoRjR2GC5GO7HY7tmzZonmn5f777+chYOR3GC5EOnE6ndi7d69mY8qkpCQsWbKEw2Hkd3jHE+lASomqqip89tlnqiw4OBiPPvoo32khv8RwIRohKSVaWlqwefPm21aHcWNK8lcMF6IRkFKir68PH3/8seZlybS0NCxatIjDYeS3eOcTjYDT6cTOnTtRVVWlyqKiorB69WquDiO/xnAhukculwtHjhzBwYMH1bJji8WClStX8mVJ8nsMF6J7IKVEZWUlPvnkEzgcDgCAyWTCkiVLMHPmTAYL+T2GC9EwSSlRX1+PDz74QPM+S25uLpYsWcIDwIjAcCEaFiklmpqa8N5776Gzs1OVp6amYvXq1QgMDDSwdkSeg+FCNERSSly+fBnr169HS0uLKp84cSKefPJJhIWFcTiM6Cvc+5toCNw9lvXr1+P69euqPCYmBk8//TRiY2MZLEQ3YbgQfQMpJRoaGrBhwwa0traq8qioKHznO99BYmIig4XoFgwXoq/hcrlQVVWF999/XzPH4g6W5ORkBgvRHTBciO7C6XSivLwcmzdv1qwKi4mJwTPPPIOpU6cyWIjuguFCdAspJQYGBrB3717s3bsXg4OD6s/i4+PxzDPPcCiM6BswXIhuIqVEV1cXNm/ejBMnTsDlcqk/mzp1Kp566inExcUxWIi+AcOF6CtSSly5cgUffPABGhsbVbkQAjk5OXj88ccRHh7OYCEaAoYLEW5M3FdUVODjjz+GzWZT5QEBAXjggQewYsUKBAUFMViIhojhQn7NPb+yf/9+7N27V53HAgBWqxVFRUWYP38+zGY2FaLhYIshvyWlRGtrKzZv3oyzZ89q5lfi4uJQXFyM9PR0nslCdA8YLuR3pJRwuVw4e/Ystm7dqjnkCwAyMjJQXFzMiXuiEWC4kF+RUqKjowO7d+/GkSNHNMuMzWYzFi5ciMLCQlitVgYL0QgwXMgvuOdWTp8+jV27dmn2BwOAiIgIPProo5g9ezZMJhODhWiEGC7k09yhUldXh3379qG+vl4ztyKEQHp6OlatWoWJEycyVIh0wnAhj9Td3Y3z588jLCwMYWFhsFqtCAwMhNlsVgFwpyBwHzfscDjQ3t6OmpoaHD16FE1NTXA6nZqfDQ0NxeLFi/HAAw9wmTGRzhgu5JGampqwbt06ADfmQoKDgzFu3DiEh4cjMjISERERCA0NRXBwMCwWCwBgYGAAXV1daGlpwZUrV3Dt2jX09fXd9t0BAQHIzs5GYWEhEhISuBqMaBQwXMgj2Ww21dNwOp3o7+9HR0cHrly5cs/fGRAQgMmTJ2PJkiXIzMzU9IKISF8MF/JIHR0dun1XcHAwUlNTMX/+fKSnpyMwMJChQjTKGC7kkWbPno3Y2Fi0t7ejra0NNpsNHR0d6OnpQV9fHxwOB1wul2ZyHgBMJhMsFgtCQ0MRHx+PjIwMpKenIy4ujqvAiMYQw4U80vjx4zF+/Hg1QS+lhNPpxMDAAPr6+mC329Hb24uBgQE4HA4AgMVigdVqRWhoqJqPEUIwUIgMwHAhjyKlxIULFzAwMDDkz7j3/ZJSwm63w2633/Yei79wOp2w2+1GV4OI4UKeQwiB+Ph4VFZWorKy0ujqeK3g4GCEhIQYXQ3yc0K6xx2IDObe84u35MiZTCYusSZDMVyIiEh3fLQhIiLdMVyIiEh3DBciItIdw4WIiHTHcCEaIqfTia6urtt2Vyai2zFciIbo6tWr+OlPf4qrV68aXRUij8dwISIi3TFciIhIdwwXIiLSHcOFiIh0x3AhIiLdMVyIiEh3DBciItIdw4WIiHTHcCEiIt0xXIiISHcMFyIi0h3DhYiIdMdwISIi3TFciIhIdwwXoiGQUqKtrQ2dnZ1oa2uDlNLoKhF5NCHZSojuymazYe3atXjrrbdw/vx5VZ6amooXXngB3/3udxEZGWlcBYk8FMOF6C5KS0tRXFwMu90OAJreihACABASEoKSkhIUFhYaUkciT8VwIbqD0tJSFBUVQUoJl8t1158zmUwQQmDbtm0MGKKbMFyIbmGz2ZCYmIje3t6vDRY3k8kEq9WKpqYmDpERfYUT+kS3WLt2Lex2+5CCBQBcLhfsdjvWrVs3yjUj8h7suRDdREqJtLQ01NfXD2tFmBACKSkpqK2tVfMxRP6M4UJ0k5aWFsTFxY3o8zExMTrWiMg7cViM6Cbt7e0j+nxXV5dONSHybmajK0BkpL6+Phw/fhyHDx/G4cOH8cUXX4zo+z788EPk5+cjKysLcXFxHCIjv8VhMfIrnZ2dKC8vR1lZGQ4fPoyTJ09iYGAAERERmDdvHubPn49f/epXaGpqGvZ3x8XF4ZlnnsGlS5cAADExMcjOzkZ2djaysrIwadIkhg35DfZcyKe1traqICkrK8OZM2fgcrkwfvx4LFiwAKtXr8aCBQuQmZkJk+nGKLHT6cTLL7887An9V199FS+++CK6u7tRXV2NyspKVFZW4tChQ3C5XAgLC1NBk52djSlTpiAgIGC0/tOJDMWeC/mUpqYmlJWVqUCpra0FAEyePBkLFizAggULMH/+fEydOvWuvQi933Pp6+vDuXPnUFlZiaqqKpw7dw6Dg4OwWq3IyMhQvZvU1FRYLJYR/fcTeQqGC3ktKSXOnz+v5kvKysrUcFZGRgbmz5+vwiQhIWFY3z3cN/S3b9+OgoKCIX334OAgzp8/j6qqKlRWVqK6uhq9vb2wWCxIS0tTvZv09HRYrdZh1ZvIUzBcyGs4nU5UVlZqhrlaWlpgMpkwc+ZMFSbz5s1DdHT0iH/fUPcW27hx45CD5U5cLhcaGxvVMFpVVRU6OzthMpmQkpKihtEyMzMRFhY2sv8oojHCcCGPNTAwgFOnTqkgOXLkCLq6uhAYGIg5c+aoYa65c+ciNDR0VOpgs9mwbt06/O53v7ttV+QXX3wR3/3udxEREaHr75RS4sqVKypoKisr0dLSAgBISkrSzNvoEaJEo4HhQh7Dbrfj2LFjapjr6NGj6O/vx7hx45CXl6fCZPbs2QgKChrTurnPc+nq6kJYWBiio6PHdOVXc3OzCprKykpcuXIFADBhwgRN2EyYMIEr0sgjMFzIMDabDUeOHFHDXKdPn4bD4UBUVBTmz5+vhrlycnJgNnNh481sNptakVZVVYULFy5ASomoqCgVNNnZ2UhKSmLYkCEYLjRmrl27ppkvqaqqgpQSEydO1KzkSktLU8uCaWh6enpQU1Ojejd1dXVwOp0YN24csrKyVOCkpKRw+TONCYYLjQopJS5evKjC5PDhw2hoaAAApKSkqCBZsGABn65HQX9/P+rq6tQw2rlz59Df34+goCCkp6ernk1aWhoCAwONri75IIYL6cLlcqG2tlb1Sg4fPoyrV69CCIGsrCzVM5k3bx4mTJhgdHX9jtPpRH19verZVFVVoaenBwEBAZg2bZqat8nMzERISIjR1SUfwHChe+JwOHDmzBnVKzly5Aja29thNpuRm5ureiX5+fk8QMsDSSlx6dIl1bOprKyEzWaDEALJyclqGC0rK0v31XDkHxguNCT9/f04fvy46pWUl5ejp6cHQUFByMvLU2Fy33338cnXC0kpce3aNc27NteuXQMAJCQkaPZIG8mRBOQ/GC50R93d3SgvL1c9kxMnTmBgYADh4eHIz89Xw1wzZ87kmL2Pam1tRVVVlRpKc2/IGRsbq1n+nJCQwDkzug3DhQAAbW1tmj25Kioq4HK5EBsbq5l8z8rK4mojP9XV1aVZ/lxfXw+Xy4Xw8HDNMFpycjJX+xHDxV9dvnxZhUlZWRlqamoAAImJiZplwampqXwqpTvq7e3VbMhZW1urNuTMzMxUgcMNOf0Tw8UPSClRX1+vWRbsHuJIT09XLyzOnz8fiYmJBteWvNXg4CDq6uo0G3L29fVpNuTMzs5Geno6goODja4ujTKGiw9yOp2orq7WLAtubm6GyWRCTk6O6pXMmzcPsbGxRleXfJTT6bxtQ86uri61Iad7GC0rK2vU9oYj4zBcfMDAwABOnz6tguTIkSPo7OxEYGAgZs2apYa58vLyuKsuGUZKicuXL2v2SGttbQVw47ydm+dtuCGn92O4eKHe3l4cPXpUzZeUl5ejr68PISEhmmXBc+bM4fADeSwp5W0bcl69ehUAEB8frwkbbsjpfRguXmLPnj04dOgQDh8+jFOnTsHhcCAyMlIzXzJjxgxOnJJXs9lsml0EGhsbIaVEdHS0Cpu8vDz2bLwAw8VLtLe3w+VywWKxqP8FBATwaY58msPhQFdXFzo7O9HV1YXu7m5kZGQgKirK6KrRN2C4eAmXywUhBMOE/JrT6YQQgu/ReAGGCxER6Y7xT0REuuPxfl9xOp04evQourq6jK6K18vNzeXmhl7K6XTi3Llz6O3tNboqXi8lJcWvdwTnsNhXent7sXLlSvT19WHcuHFGV8dr1dfX43e/+x2WL19udFXoHvT39+OnP/0pBgYGuIx9BK5evYrnn38eeXl5RlfFMOy5fEVKCZPJhH//93/H/fffb3R1RpXL5UJfXx+klAgODtZtI0qHw4GnnnpKl+8i4wgh8Nxzz2H69OlGV8UruPdYS01NRWhoKBwOB1577TX4+3M7w+UWAQEBCAoKMroaupNSwuFw4PDhw9iwYQPOnj0Lp9OJadOm4cknn8SyZcsQGBg4otVoXM3mO0wmE9+ZGgIpJaqrq/H6668jNjYW+fn5WLp0KdsBGC5+QUqJrq4u/PrXv8a7776rGU+vr6/H/v378cQTT+Df/u3fEB4ezoZBNEQulwuHDh3C4OAgrl69iu3btyM3N9foankEhouPk1Kiu7sbP/7xj7Fp0ya4XK7bfmZgYAAbNmzAwMAAfvnLX3LOiTTcwzt86LhdS0sLzpw5o66nTJmCqVOnGlgjz8GlyD5ucHAQv/nNbzTBEhAQgMzMTOTm5sJsvvF8IaVESUkJ/vKXv9wxgMg/DQ4O4tixY1w9dgdSSpw6dQqdnZ2qbN68eVwI8RWGiw+TUuKTTz7B2rVrVWAEBQXh5ZdfxpYtW/Dxxx/jX/7lX9Qck9PpxO9//3ucPXvW7ycjCejp6cH69evx29/+Fh9++CEGBweNrpJHcTgcOHLkiGorISEhmDt3rsG18hwMFx8lpcTFixfxy1/+En19fQBu9Fief/55vPTSS4iIiMC4cePwz//8z3j66afVkEdrayvefPNNDAwMGFl9Mlh/fz/efvtt/O///i8GBwexY8cOHDp0iA8dN2lubsb58+fV9bRp0xAfH29gjTwLw8VHuXshjY2NqqygoAA//OEPNauALBYLXnrpJaSlpamy3bt3a57IyP9YLBYkJiaqh47BwUF88MEHaG5u5n2BGw9vZ8+eRU9PD4Ab81Fz585Vw8zEcPFJUkocP34cJSUlqmzixIn4yU9+gpCQEM3PCiEwYcIE/PCHP1QNo7e3F3/84x/Ze/FjJpMJq1atQn5+viq7fv06SktLGS64sUrs+PHj6u/CarVixowZXPRwE4aLDxoYGMDvf/97dHd3A7jxD8UPfvADpKWl3fHmF0KgqKgIM2fOVGUHDx7EqVOn+A+JHwsODsZTTz2l2d7+wIEDuH79uoG18gw2mw11dXXqOjk5GRMmTDCwRp6H4eJjpJT44osv8Omnn6qy6dOn44knnvjap6rQ0FA899xz6m19u92O//mf/+HKMT8mhMDEiROxZMkSVdbe3u73cy9SStTW1qKjo0OVzZw5ky+d3oLh4mMGBwfx5z//WS0dNZvN+Md//Mdv3EBPCIGCggJMmzZNle3cuVMzZ0P+x2QyYfHixYiIiFBln332mZpr8EdSSpw+fVo9eFksFsyYMcPgWnkehosPkVKioqJC02vJycnB8uXLhzQWHBkZieLiYnXd2tqKrVu3+vVTKgHjx4/HnDlz1PWVK1dQXV3tt/dFX18fqqur1fX48eM1ix/oBoaLD3G5XNiwYYN6qjSZTHj22WcRFhY2pM8LIbBy5UrExsaqss2bN/MYAj9nMpnw4IMPqmEfp9OJw4cP++2Q6dWrV3Ht2jV1nZmZCavVamCNPBPDxYdcvHgRO3fuVNepqakoLCwc1hPV5MmT8dBDD6nrc+fOoby83G+fUunGQ0daWhoSExNVWUVFBWw2m3GVMoiUEpWVlejv7wdw4++Gq8TujOHiI6SU2LJlC5qbm1VZcXExYmJihvU9JpMJxcXFCAwMBHBjDudue5KR/wgODta8fd7e3o6qqiq/e+hwuVw4e/asug4NDcW0adMYLnfAcPERnZ2d2LRpk7qOjY3FqlWrhn3TCyFw3333aV6q/PTTT3HlyhW9qkpeyH1fuLcKklLi6NGjfvfQ0dXVhYaGBnWdmJg47Ac4f8Fw8QFSSpSVlaG2tlaVLVmyBFOmTLmn7wsLC0NRUZG6vn79Ovbv3+93T6mklZiYqBkaq66u1mza6OvcWyrdPByYlZXFt/LvguHiA5xOJzZu3Kg2FgwKCsLjjz8Ok+ne/u8VQmDFihVqIYCUElu3buUb+34uKChIc1ZJe3s7zp8/71cPHVVVVXA6nQBu7NWXnZ1tcI08F8PFBzQ1NeHgwYPqOisrC/fdd9+IxoFTU1M1y0+PHz+u2aSP/I8QArNmzdKsGjt58qSxlRpDDocDVVVV6jo8PBxTpkzhfMtdMFy8nJQSu3btQltbmyp79NFHR3zgV2BgIFauXKkaTldXF3bu3OlXT6l0u8mTJyMuLk5dV1ZW+s1ZL52dnWhqalLXU6ZMGfIyf3/EcPFyfX19mhcdo6Kihr38+E6EEFi0aBHGjx+vyrZv3w673T6i7yXvFhISgqysLHV97do1XL161cAajZ2LFy9q3vnKyMhQ2yXR7RguXkxKierqalRUVKiy/Px8JCcn6/L9CQkJWLhwobquqanB6dOn2XvxY0IIzJw5Uz289Pf3+8Xhcu62dvN8S0ZGBofEvgbDxcvt2LFD9Sbc26TrtXrFZDJh5cqV6vv6+vrwySef+Pw/JHR3QghMmzYNoaGhquzMmTPqH11f5XA4UFNTo64jIiKQlJRkYI08H8PFi3V3d2PXrl3qeuLEiVi4cKFuT1NCCOTn52sa0Z49ezTzO+R/oqOjMXnyZHXd0NDg80uSb51vmTx5MudbvgHDxUtJKXHq1CnNmRIPPvigZrJVD1FRUVi6dKm6vnTpEr744gv2XvyY2WzWLMHt7OzEhQsXfPqeuHTpkma+JTMzk/Mt34Dh4sV27Nih9jgym80oKiq653db7kYIgUceeQTBwcEAbgwPbNmyxe/ezKb/J4TA9OnT1XCp0+nUbIniazjfcm8YLl7KZrNh//796jopKWnE77bciXsCNzMzU5UdOnSI28H4uaSkJM0JldXV1eolXl/jcDg0W+xHRkZqdiqgO2O4eCEpJU6ePIkLFy6oskWLFmkau55CQkKwYsUKdd3c3Ix9+/b59DAIfb3Q0FBMnTpVXV++fBmtra0G1mj03DrfkpSUxPmWIWC4eCEpJUpLS9WTYmBgIFasWDFq3XQhBAoLCxEeHq5+/6ZNm9SQHPkfk8mE6dOnq+uenh7U19f73AOHez8xzrcMH8PFC3V0dODAgQPqesqUKZg1a9aojgGnpqYiLy9PXZ84ccIv3m+gOxNCIDMzUx3N4D7nxBdxvuXeMFy8jPv87osXL6qyb33rW6pXMVosFgtWr16tFgzY7XZ8/PHHDBc/Fh8fr1mdWFtb63O92Tu938L5lqFhuHihnTt3qiExi8WCgoKCUX+SEkJg8eLFmndeduzYgevXr4/q7yXPZbVakZqaqq6vXr2qOazOF/D9lnvHcPEytw6JJSUlabbjGE0xMTGac16ampqwY8cO9l78lBAC2dnZ6t7r6+vzqS34Od8yMgwXL+IeErt5ldiDDz6IyMjIMfn9QggUFxdrJvbfe+89dHd3j8nvJ88ihEBaWprmdMrKykqfCReA8y0jwXDxMjt37lSHdlksFl12QB4qIQQyMjKwaNEiVXbmzBns2bPHp/5BoaEbP368Zufsuro69PX1GVgj/dw638L3W4aH4eJFOjs7NUNiiYmJo75K7FZmsxnPPvus5o39P/7xj+jp6RmzOpDnCAoKwrRp09T1tWvXcO3aNQNrpJ/Ozk5cunRJXXO+ZXgYLl7kzJkzmiGxBx54YNRenLwb92aWCxYsUGUnTpzA5s2b2XvxQ7fOuwwMDKC2ttYn7oXGxkbNkC/nW4aH4eIl3CdO3ryX2FisEruToKAg/NM//ROsViuAG72Xt956i1vC+CH3Fvzue8FX5l24n9jIMVy8RFdXl2YvsUmTJmHOnDmG3OxCCNx///1Yvny5Kvvyyy/x+eefj3ldyHixsbGIj49X1/X19V5/YinnW0aO4eIFpJQ4e/Ys6uvrVdnChQsRHR1tWJ0sFgteeuklxMfHIycnB++88w4ee+wxw+pDxgkKCkJ6erq6bmlp8fqjj202m+b9lilTpnC+ZZgYLl5i9+7dmiGxsVwldidCCKSnp+O///u/8f7776OgoAAWi8Ww+pCxbp13OXfunNcOjUkp0dDQoHm/JSsrS/fjLHwd/7buwv0ClSecsNfd3Y29e/eq64kTJ47K9vrDZTKZkJ+fj9jYWMPrQsYRQiAlJQXjxo1TZWfPnvXqM3+qqqpU/S0WCzIzM3mPDxPD5Q7sdjvWr1+P1atX47/+678MPR/cPUF665BYTEyMYXW6GRscATeOPr55TqKhocFrX64dGBjQnN8SHR2NSZMmGVgj78RwuUVfXx9eeeUV/Ou//iuamprwl7/8BceOHTO0i79r1y71YlpAQIDhQ2JEt3I/3bu1t7drNlf1Ji0tLZqVj7f2ymhoGC63CAoKwpQpU1RvpaOjA7/61a8MW/3S09OjGRJLSEjA3LlzGS7kcbKzs9V7IA6HwyuXJEspUVdXp2nvOTk5nG+5B/wbu4UQAt/73vcwc+ZMVVZWVoZPPvlkzBuKe0js/PnzqmzBggUeMyRG5CaEQHJysuboh8rKSjgcDgNrNXxSSpw5c0a19eDgYKSnp/Nh7h4wXO4gJiYGP/rRjzQvCb799ttoa2sb87rcOiS2fPlyPkWRRwoPD9ccfXzx4kVD2sxI2O12zfst8fHxmnd4aOj4r9QdCCGwaNEiLFu2TJXV1NSM+RYn3d3d2LNnj7qeOHEi8vLy+BRFHikgIAAzZsxQ193d3V63FUxTU5PmTJrMzEy1jx4ND8PlLgIDA/GDH/wAoaGhAACXy4W1a9eivb19TH6/u3t+85DYwoULERsbOya/n2i43PuM3bwF/+nTp70mXKSUqKioUAfxmUwmzfA4DQ/D5S6EEJg5cyYKCgpU2blz57B9+/YxayylpaVqSMxsNmPFihXstZBHmzhxIiZMmKCuq6urvWYrGIfDgYqKCnUdERGB1NRUtrl7xHD5GmazGf/wD/+gliG6XC68++67Y7J+v7Ozk6vEyOtYrVZMnz5dXTc3N+PixYte0Xu5fv06Ghsb1XVqauqYHcTnixguX8Pde1m8eLEqO3PmDA4ePDiqjUVKiZMnT6KhoUGVLVq0iKvEyCvk5uaqJcmDg4M4efKksRUaAvf+fe5elhACs2fP5uKZEeDf3DewWCx49tln1Tjy4OAg3n33XXUa5GiQUuKTTz7RnDhZVFTEXgt5PCEEUlNTNecMnTp1Su2L56mcTqfmZemQkBBMnz6dbW4EGC7fwH041uzZs1XZ4cOHUVFRMWq9l9bWVuzbt09dT506FbNnz+aNTl4hPDxc87Z+U1OTZodhT9TS0oJz586p6+TkZM3xzTR8DJchsFqtePrpp1VX326347333huVjfmklDh06JBm+4mHH34YERERuv8uotFgMpk0G6v29/cbvoXS13Gvart5LnXu3Lnc5XuEGC5DIITAsmXLkJKSospKS0s152vrxel0YsuWLWr7GavViqKiIt1/D9FocS9Jvnlo7NixY2rlo6dxOBwoKyvTDInNmjWLIwUjxHAZoujoaHz7299W183NzaPyUmVjYyPKysrU9fTp05GVlcUbnbxKZGSkZtXYpUuX0NDQ4JG9l6amJtTW1qrrtLQ0vpWvA4bLMDz22GOaFVslJSWw2Wy6fb+UEjt27EBrayuAG0+Ajz76qNqGhshbmEwmzJ8/X7Nq7PDhwx4XLu5h6N7eXgA32tzChQthNpsNrpn3Y7gMkRACU6ZM0bxUWVdXh3379unWYHp6erBlyxZ1HR0djYKCAvZayOsIIZCVlaWZFD969Cg6OjoMrNXtOjo6NCMF0dHRyM3NZZvTAcNlGEwmE5588knVk3A6nXjvvfd0WWYppcTRo0dRWVmpyh588EFMnjx5xN9NZISwsDDMnTtXXbe0tODEiRMe03txt7lr166psvz8fM1cEd07hsswCCEwa9YsTYMpLy/HqVOnRtxgnE4nPvjgA827LY8//rgaViDyRg888IB6GJNSYt++faP6jthw2O127N69W7Vdq9WKxYsXs9eiE4bLMAUFBWmWJff29uKvf/3riJYlSylRX1+vebclIyMD8+bN441OXksIgcmTJyMrK0uV1dXVoaqqyvDei5QSx44d0+yCMWPGDEyePJltTicMl2ESQmDx4sWYNm2aKtu5cycaGxtH1GBKSkrU2RdCCBQXFyMsLGzE9SUyktlsxtKlSzUT+6WlpYYfItbd3Y2tW7eqJf+BgYFYvnw5J/J1xHC5B1FRUSguLlbXLS0t+Oijj+75+65du4aNGzeq6/j4eDzyyCN8giKv596f7+Z3xE6fPo3q6mrDei8ulwt79+7FhQsXVFlubi6X/OuM4XIPhBB47LHHNCthPvroI1y/fn3Y3yWlxKZNmzQvZBYVFSExMVGXuhIZLTg4GIWFhWoTyP7+fmzZssWQuRcpJS5duqQ5ttxqtWLlypXsteiM4XKPkpKS8Oijj6rrixcvYtOmTcN+GmtpacH69evV5yIjI/HMM8/wCYp8hhACeXl5mt5LRUUFjh49Oqa9Fykl7HY71q1bpzn0b9GiRUhPT2eb0xnD5R4JIfD000+rPb+klFi/fj1aWlqG/B1SSnz44Yea0yZXrFjBG518TkhIiKZ34HA48NFHH6G9vX3MAsbhcODDDz/E6dOnVVlCQgJWrVrFrfVHAf9G75EQApmZmVi+fLkqO3/+PD7++OMhNRZ39/zPf/6z+vmIiAisWbOGy4/J5wghcN9992HWrFmqrKmpCSUlJaM+uS+lxODgIDZt2oQdO3ao9hYUFITvfOc7iI2N5cPcKGC4jEBAQADWrFmD8PBwADcmCv/0pz9pdjS+G6fTiXfeeUcz17Jq1SpkZ2fzRiefFBgYiCeeeEK1Fykl9u7di4MHD47KDuPu32G327FhwwZNkJlMJqxcuVKzezPpi+EyAkII5OTk4JFHHlFljY2N+NOf/qSWON6JlBKff/45/vrXv6qyCRMm4Pvf/z57LeSzhBBITk7GY489plmavH79epw8eVLX4TEpJZxOJ2pqavDrX/8a27ZtU8EihMBDDz2EVatWsb2NIobLCAUEBOD73/8+YmNjAdy4qd99912Ul5ffsbFIKfHll1/iF7/4hTo/wmQy4Xvf+x5SUlL4FEU+zWQyoaCgAPn5+aqsq6sLf/jDH3Ds2LER92CklBgYGEBNTQ3+8Ic/4LXXXsPZs2dVWzSZTHjooYc0p8vS6GC4jJAQAunp6XjuuedUMHR0dODnP/85rl+/rgkYKSW6u7vxs5/9DGfOnFHlc+fOxd/93d9xUpH8QlBQEJ577jmkpaWpMpvNhrfeegtbt26F3W4fVi/G3Utpbm7Gnj178Nprr+E//uM/8Omnn6rdjt2/99vf/jbWrFmDkJAQPsiNMi7s1oHJZMKaNWuwZ88enDhxAgBw4sQJ/PjHP8ZvfvMb1atpa2vDL37xC2zdulV9NiYmBj//+c950iT5DSEEoqOj8fzzz+P1119XLzP29PRgw4YN+OKLL1BUVIQZM2Zg3LhxEEJogsAdPIODg2hra0NNTQ3Ky8tRVVWFzs7O24LJvaP5U089hVmzZsFkMjFYxgDDRSfR0dH42c9+hr//+79Xa+h37NiBK1eu4KmnnoLL5cL777+P06dPq5s/ODgYP/nJTzB79mze7ORXhBBISEjAK6+8grffflu9se9yuVBTU4Pa2lrExsZi2rRpmDp1KmJiYmCxWDAwMID29nZcuXIFjY2N+PLLL+/a0xFCYOLEiXj44YexePFihIaGsp2NIYaLToQQmDdvHl599VW8+uqr6O3thZQSp06dwqlTp277+cDAQLzwwgv427/9Ww6HkV9y/+P/ox/9CO+//z7279+v3tp3uVy4fv06rl+/js8//1z9/FCGy4KDg5Geno5FixZh9uzZCAsLY6gYgOGiI/d5Lw6HA//5n/9514ORwsPD8corr2DNmjWwWCxjXEsizyGEQHh4ONasWYP8/Hxs2rQJNTU1GBwcvO1nvy5YgoKCMGnSJMydOxf5+fmYNGkSzGYzQ8VADBedmc1mPPvss8jIyMDrr7+Oo0ePwm63AwDGjRuH/Px8vPjii8jPz+cySCLcCBiz2aw2j6yrq0NZWRkqKyvR3NyM/v5+zSoyk8mEwMBAREREqC39c3JykJCQgMDAQAaKh2C4jAL3+eHr1q1DQ0MDGhsbAQBTp05FcnIyGwDRHQghEBQUhOzsbGRnZ6O/vx82mw1tbW3o7u6Gw+GA2WxGaGgooqKiEB4ejuDgYE7QeyiGy01cLheOHTumehp6a2ho0BxO5GucTqdmQ0DyTlJKnDt3Tpfju/UihFBDyD09Pejp6TG4RnfndDrVO2z+jOHyFZPJhIyMDOzevRu7d+82ujpeKywsjMuqvZgQAklJSTh+/DiOHz9udHW8ltVqRWhoqNHVMJSQRp836iGklHA4HIYfv+oLzGYzV8B5KfcLiWwHIxcQEODX7YDhQkREuvPfWCUiolHDcCEiIt0xXIiISHcMFyIi0h3DxUu4j2rl+gvyZ+53SL7uMD7yDAwXL1FRUYFJkyahoqLC6KoQGaaxsRHPPfec2vWCPBfDhYiIdMdwISIi3TFciIhIdwwXIiLSHcOFiIh0x3AhIiLdMVyIiEh3DBciItIdw4WIiHTHcCEiIt0xXIiISHcMFyIi0h3DhYiIdMdwISIi3TFcvICUEu3t7XC5XGhvb+eZLuSXpJRoa2tDT08P2tra2A48HMPFg9lsNrz55ptIS0vD4sWL0draisWLFyMtLQ1vvvkmbDab0VUkGnU3t4O8vDzs2LEDeXl5bAceTkjGv0cqLS1FcXEx7HY7AGie0oQQAICQkBCUlJSgsLDQkDoSjTa2A+/FcPFApaWlKCoqgpQSLpfrrj9nMpkghMC2bdvYsMjnsB14N4aLh7HZbEhMTERvb+/XNig3k8kEq9WKpqYmREZGjn4FicYA24H345yLh1m7di3sdvuQGhQAuFwu2O12rFu3bpRrRjR22A68H3suHkRKibS0NNTX1w9rJYwQAikpKaitrVXj0ETeiu3ANzBcPEhLSwvi4uJG9PmYmBgda0Q09tgOfAOHxTxId3f3iD7f1dWlU02IjMN24BsYLh4kNDR0RJ8PCwvTqSZExmE78A0MFw8SExOD1NTUYY8XCyGQmpqK6OjoUaoZ0dhhO/ANDBcPIoTACy+8cE+fffHFFzmJST6B7cA3cELfw3B9PxHbgS9gz8XDREZGoqSkBEIImExf/3+P+83kjRs3skGRT2E78H4MFw9UWFiIbdu2wWq1QghxWzffXWa1WrF9+3YUFBQYVFOi0cN24N0YLh6qsLAQTU1NeOONN5CSkqL5s5SUFLzxxhu4fPkyGxT5NLYD78U5Fy/gPseiq6sLYWFhiI6O5qQl+R22A+/CcCEiIt1xWIyIiHTHcCEiIt0xXIiISHcMFyIi0h3DhYiIdMdwISIi3TFciIhIdwwXIiLSHcOFiIh0x3AhIiLdMVyIiEh3DBciItIdw4WIiHTHcCEiIt39H44ZBch4pz1lAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 500x400 with 4 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "model = model.prune()\n",
    "model(dataset['train_input'])\n",
    "model.plot()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "4942984c",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "function , r2\n",
      "sin , 0.9981093780355159\n",
      "gaussian , 0.9360582190339871\n",
      "tanh , 0.8616859029524302\n",
      "sigmoid , 0.8585390273680941\n",
      "arctan , 0.8428622193038047\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "('sin',\n",
       " (<function kan.utils.<lambda>(x)>, <function kan.utils.<lambda>(x)>),\n",
       " 0.9981093780355159)"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# sin appears at the top of the suggestion list, which is good!\n",
    "model.suggest_symbolic(0,0,0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "3f1c41a6",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "function , r2\n",
      "cosh , 0.9910665391502297\n",
      "x^2 , 0.9885210310683376\n",
      "gaussian , 0.9883627975330689\n",
      "sin , 0.9843196558672351\n",
      "x^4 , 0.9403353142717915\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "('cosh',\n",
       " (<function kan.utils.<lambda>(x)>, <function kan.utils.<lambda>(x)>),\n",
       " 0.9910665391502297)"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# x^2 appears in the suggestion list (usually not top 1), but it is fine!\n",
    "model.suggest_symbolic(0,1,0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "01ff562d",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "function , r2\n",
      "sin , 0.9995702405196035\n",
      "x^2 , 0.9992413667649066\n",
      "cosh , 0.9990483455142343\n",
      "gaussian , 0.9989441353410312\n",
      "tanh , 0.9986571504172722\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "('sin',\n",
       " (<function kan.utils.<lambda>(x)>, <function kan.utils.<lambda>(x)>),\n",
       " 0.9995702405196035)"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# exp not even appears in the list (but note how high correlation of all these functions), which is sad!\n",
    "model.suggest_symbolic(1,0,0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "232b710b",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "function , r2\n",
      "sin , 0.9995702405196035\n",
      "x^2 , 0.9992413667649066\n",
      "cosh , 0.9990483455142343\n",
      "gaussian , 0.9989441353410312\n",
      "tanh , 0.9986571504172722\n",
      "sigmoid , 0.998657149375774\n",
      "arctan , 0.9970617106973462\n",
      "x^3 , 0.9962099497478061\n",
      "x^4 , 0.9947572943342223\n",
      "exp , 0.9913715887470934\n",
      "1/x^4 , 0.9890801101893518\n",
      "1/x^3 , 0.9884748093165208\n",
      "1/x^2 , 0.9874565358732027\n",
      "1/x , 0.9853279073610555\n",
      "1/sqrt(x) , 0.9830898307444438\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "('sin',\n",
       " (<function kan.utils.<lambda>(x)>, <function kan.utils.<lambda>(x)>),\n",
       " 0.9995702405196035)"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# let's try suggesting more by changing topk. Exp should appear in the list\n",
    "# But it's very unclear why should we prefer exp over others. All of them have quite high correlation with the learned spline.\n",
    "model.suggest_symbolic(1,0,0,topk=15)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "51844d0f",
   "metadata": {},
   "source": [
    "Let's train more! The loss goes down and the splines should be more exact"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "324937fe",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "train loss: 4.74e-03 | test loss: 4.80e-03 | reg: 2.93e+00 : 100%|██| 20/20 [00:03<00:00,  6.47it/s]\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZcAAAFICAYAAACcDrP3AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjYuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8o6BhiAAAACXBIWXMAAA9hAAAPYQGoP6dpAAAyPklEQVR4nO3de1BUV4IG8O82zaN5yVNBRJCHqLwkPDWJii9iTJxJsomZvLPZqdqZnWR3sjtTtVu72ampyiY7VVMTN1ubyu5kJjo7GyfG7JisJiYm+FZEiSAgCggioAhKY0M3Dd199g/TZ7jxEZALtx/fr2r+OEeBE6av3z1vRQghQEREpCGD3g0gIiLfw3AhIiLNMVyIiEhzDBciItIcw4WIiDTHcCEiIs0xXIiISHMMFyIi0hzDhYiINMdwISIizTFciIhIcwwXIiLSHMOFiIg0x3AhIiLNMVyIiEhzRr0bQOQNhBC4cuUKBgcHER4ejtjYWCiKoneziDwWey5Et2E2m7Fp0yZkZmYiPj4e8+bNQ3x8PDIzM7Fp0yaYzWa9m0jkkRTeREl0c7t378YjjzwCq9UK4Hrvxc3dawkNDcX27dtRUVGhSxuJPBXDhegmdu/ejfXr10MIAZfLdcu/ZzAYoCgKdu7cyYAhGoPhQvQNZrMZc+bMgc1mu22wuBkMBphMJnR2diIqKmrqG0jkBTjnQvQNmzdvhtVqHVewAIDL5YLVasWWLVumuGVE3oM9F6IxhBDIzMzEuXPnMJFHQ1EUpKWlobm5mavIiMBwIVLp6+tDfHz8pL4+NjZWwxYReScOixGNMTg4OKmvt1gsGrWEyLsxXIjGCA8Pn9TXR0REaNQSIu/GcCEaIzY2Funp6ROeN1EUBenp6YiJiZmilhF5F4YL0RiKouDFF1+8o6996aWXOJlP9DVO6BN9A/e5EE0eey5E3xAVFYXt27dDURQYDLd/RNw79D/88EMGC9EYDBeim6ioqMDOnTthMpmgKMoNw13uOpPJhF27dmHt2rU6tZTIMzFciG6hoqICnZ2deOONN5CWlqb6s7S0NLzxxhvo6upisBDdBOdciMZBCIHa2lq89tpr+Pu//3vk5+dz8p7oNthzIRoHRVEQHR2NiIgIREdHM1iIvgXDhYiINMdwISIizTFciIhIcwwXIiLSHMOFiIg0x3AhIiLNMVyIiEhzDBciItIcw4WIiDTHcCEiIs0xXIiISHMMFyIi0hzDhYiINMcj94nGyWKx4KuvvkJBQQEiIiL0bg6RR2O4EI2T0+mE1WpFaGgoAgIC9G4OkUdjuBARkeY450JERJoz6t0AIjeXy4XW1lYMDw/r3RSvl5KSgsjISL2bQX6M4UIew+FwYOvWrRgdHUVISIjezfFKQghcvnwZzz//PBYvXqx3c8iPMVzIYwghoCgKHnvsMWRlZendHK9gs9nQ2dmJOXPmwGQywel0YtOmTXo3i4jhQp4nICAAgYGBejfD4wkhcObMGfz2t79FTEwMsrOzsWLFCiiKonfTiBguRN5KCIGTJ09idHQUPT096O/vR1FRkd7NIgLA1WJEXmtgYADNzc2ynJiYiNmzZ+vYIqI/YbgQeSEhBFpaWmA2m2Vdbm4ugoKC9GsU0RgMFyIv5B4Sc++BDgoKQm5uLudbyGMwXIi8kNlsxpkzZ2Q5MTERSUlJOraISI3hQuRlhBBoamrCtWvXZF1+fj6HxMijMFyIvIzT6URNTY0cEgsODkZ+fj6HxMijMFyIvMyVK1fQ2toqy3PnzkViYqKOLSK6EcOFyIsIIVBfX4/BwUFZV1BQAKORW9bIszBciLzI6OgoTpw4IcuhoaHIycnhkBh5HIYLkRfp6upCR0eHLGdkZCA+Pl7HFhHdHMOFyEsIIXDixAnY7XYAgKIoKCoq4q2Y5JEYLkReYmhoCLW1tbIcExODhQsXckiMPBLDhcgLuE9Avnz5sqzLzc3lhWDksRguRF7A6XTi6NGjcLlcAIDAwEAUFxez10Iei+FC5AV6enpw9uxZWZ47dy5SUlIYLuSxGC5EHk4IgerqalitVllXUlLC417IozFciDzc4OCgam9LVFQUj3shj8dwIfJgQgg0NDSoJvLz8vIQHR2tY6uIvh3DhciDjY6O4tChQ3IiPzg4GEuWLGGvhTwew4XIQwkh0NbWpjqkMj09nRP55BUYLkQeyuVy4eDBgxgdHQUAGAwG3H333TykkrwCw4XIAwkhcOnSJZw6dUrWJSYmIjs7m70W8goMFyIPJITAoUOHMDQ0JOuWLl2K0NBQHVtFNH4MFyIPdOXKFVRXV8tybGwsd+STV2G4EHkYIQSOHDkCs9ks60pKSjBjxgz9GkU0QQwXIg/T39+Pw4cPy3JkZCTuvvtuGAx8XMl78NNK5EHcvZYrV67IuqKiIl4IRl6H4ULkQfr7+3HgwAFZDg8Px7JlyzjXQl6H4ULkIdz7Wsb2WoqLi5GYmMhwIa/DcCHyEH19fapeS0REBFasWMFgIa/EcCHyAC6XC5WVlaoVYqWlpUhISGC4kFdiuBDpTAiBzs5OHDlyRNZFRUWhvLycwUJei+FCpDOn04lPP/1UtRt/+fLliIuLY7iQ12K4EOlICIHTp0/j5MmTsi4hIQH33HMP97WQV+Onl0hHVqsVH3/8serk47Vr13I3Pnk9hguRTtxLj9vb22VdZmYmzxAjn8BwIdKBEAIXL17E559/DiEEACAkJAQPPvgggoODdW4d0eQxXIh0MDo6io8++ggDAwOyrqysDBkZGey1kE9guBBNMyEEqqurUVtbK+tmzpyJ++67j5P45DP4SSaaRkII9PT04KOPPoLT6QQAGI1GPPDAA4iJiWGvhXwGw4VoGo2MjODDDz/E1atXZV1BQQGKiooYLORTGC5E08TlcmH//v2q4bC4uDh85zvfgdFo1LFlRNpjuBBNAyEEWlpasHPnTrhcLgBAYGAgvvvd72LmzJnstZDPYbgQTTEhBMxmM7Zu3ao64mXJkiUoLCxksJBPYrgQTbGRkRFs27YNFy5ckHUpKSnYsGEDAgICdGwZ0dRhuBBNIafTid27d+PEiROyLiIiAt/73vcQGRnJXgv5LIYL0RRxuVw4duwYdu/eLedZjEYjHnroIcybN4/BQj6N4UI0BYQQaGpqwrZt2zAyMgIAUBQFK1aswJIlS7hZknweP+FEGhNC4Pz589iyZQssFousz8vLw4MPPsh5FvILDBciDbkPpPzNb36DK1euyPrU1FQ88cQTMJlMHA4jv8BwIdKI+2iXd955BxcvXpT1M2fOxLPPPovo6GgGC/kNhguRBtzB8utf/xodHR2yPioqCs8//zySkpIYLORXeOYE0SQJIdDV1YV33nkHnZ2dsj4yMhLPPfcc0tPTGSzkdxguRJMghEBrayveffdd9PT0yPqIiAg899xzWLRoEYOF/BLDhegOuVwu1NXV4fe//z3MZrOsd/dYsrOzGSzktxguRBMkhIDD4cD+/fuxY8cO2Gw2+WexsbF49tlnsWDBAgYL+TWGC9EECCFgtVqxY8cOHDhwAA6HQ/5ZYmIinn/+eaSmpjJYyO8xXIjGyT1x/9577+Hs2bOqP5s/fz6efvppzJo1i8FCBIYL0bdyD4NVVVVhx44dqvkVg8GAkpISPProo4iIiGCwEH2N4UJ0G0II9Pb2YseOHaipqVENg4WEhGDdunVYtWoVgoKCGCxEYzBciG5CCAG73Y6jR4/ik08+Ud15DwCzZs3Cxo0bkZ2dzUMoiW6C4UI0hhACTqcTZ8+exc6dO9HS0iKPyweAgIAA3HXXXXjooYcQFxfH3grRLTBciHA9VFwuFzo6OvDZZ5+htrYWo6Ojqr8THR2NBx54AGVlZQgMDGSwEN0Gw4X8mrun0tHRgS+//BK1tbUYHh5W/Z3AwEAUFhbigQcewMyZMxkqROPAcCG/JITAyMgImpubsW/fPpw+fRp2u131dwwGA1JTU7F+/XosWrQIAQEBDBaicWK4kN8QQkAIgYGBAdTV1eHw4cPo6OhQrQBzS0xMxOrVq1FUVMQ7WIjuAMOFfJoQAgBgt9vR3t6O6upq1NXVqfaquCmKglmzZmH58uUoLS1FeHg4Q4XoDjFcyCe5lxJ3d3ejrq4OtbW1uHTp0k17KQaDAXPmzMGyZctw1113MVSINMBwIZ/gHvIaHBzEhQsX0NDQgNOnT6Onp+eGVV9uISEhyMzMxL333osFCxYgJCSEoUKkEYYLeR33UJfL5YLVakVPTw9aW1vR3NyMjo4ODAwMqPamjGUwGBAXF4eCggKUlJRg9uzZnKgnmgIMF/J47l7J8PAwBgYG0N3djfPnz6O9vR2XLl2CxWK56XCXm6IoiIyMRFZWFoqKipCZmYmwsDD5Z0SkPYYLeaShoSH09vbi0qVL6OzsRFdXF3p6enDt2rUblgzfjMFgQFRUFDIyMpCfn4/58+cjMjISiqIwUIimAcOFPNKnn36KPXv23LZHMpaiKDCZTEhISEBmZiYWLlyIuXPnIiwsjIFCpAOGC3mkmJiY2wZLQEAAwsLCMGvWLMybNw/p6elITk5GVFQUjEYjw4RIZwwX8kiJiYkICAiA0+mE0WhEaGgo4uLiMHv2bMydOxfJycmIj49HWFgYJ+SJPBDDhTxSYmIi7r//fsycORMJCQmIiYmByWSC0Xj9I8swIfJsDBfyKEIItLa2wm63Y+7cuQAAs9l80x31dCOXy4WhoSG9m0HEcCHPYTAYMHv2bJw6dQqnTp3SuzleKyQkBKGhoXo3g/ycItw70oh05j7+nibPYDDwhkzSFcOFiIg0x1cbIiLSHMOFiIg0x3AhIiLNMVyIiEhzDBeicXI6nbBYLFzRRjQODBeicers7MSPf/xjdHZ26t0UIo/HcCEiIs0xXIiISHMMFyIi0hzDhYiINMdwISIizTFciIhIcwwXIiLSHMOFiIg0x3AhIiLNMVyIiEhzDBciItIcw4WIiDTHcCEiIs0xXIiISHMMF6JxEELg6tWrsFgsuHr1KoQQejeJyKMpgk8J0S2ZzWZs3rwZb775JlpbW2V9eno6XnzxRTz77LOIiorSr4FEHorhQnQLu3fvxiOPPAKr1QoAqt6KoigAgNDQUGzfvh0VFRW6tJHIUzFciG5i9+7dWL9+PYQQcLlct/x7BoMBiqJg586dDBiiMRguRN9gNpsxZ84c2Gy22waLm8FggMlkQmdnJ4fIiL7GCX2ib9i8eTOsVuu4ggUAXC4XrFYrtmzZMsUtI/Ie7LkQjSGEQGZmJs6dOzehFWGKoiAtLQ3Nzc1yPobInzFciMbo6+tDfHz8pL4+NjZWwxYReScOixGNYTabJ/X1FotFm4YQeTmj3g0g0tPw8DBOnjyJqqoqVFVVobq6elLf73/+539QWlqKnJwczJw5k0Nk5Lc4LEZ+xWKx4Pjx4zh27BiqqqpQW1uL0dFRREZGori4GCUlJfiXf/kXdHZ2Tvh7x8XF4dFHH8X58+dlOScnBzk5OcjOzkZycjLDhvwGey7k065cuSKD5NixY2hsbITL5UJ8fDxKS0uxYcMGlJaWIisrCwbD9VHikZER/PjHP57whP4//dM/4aWXXoLFYkFjYyPq6+tRX1+P/fv3w+VyITIyEtnZ2TJw5s2bh4CAgKn6TyfSFXsu5FO6urpkmFRVVckjW5KTk1FaWorS0lKUlJQgNTX1lr0Irfe52Gw2nDlzRobNmTNnMDIyApPJhIULF8qwmT9/PgIDAyf130/kKRgu5LWEEDh37pzslVRVVaGrqwsAkJmZqQqTxMTECX3vie7Q37VrF9auXTuu7z06Oorm5mY0NDSgvr4eDQ0NsNlsCAwMRFZWlhxGW7hwIUwm04TaTeQpGC7kNZxOJ5qamlQ9kytXrsBgMCAnJwclJSUoLS1FcXExYmJiJv3zxnu22IcffjjuYLkZl8uFtrY22bNpaGjAwMAADAYDMjIy5FBadnY2IiIiJvcfRTRNGC7ksUZHR1FXVyd7JseOHcPg4CCCgoKwePFi2TO56667EB4ePiVtMJvN2LJlC/7t3/7thlORX3rpJTz77LOYMWOGpj9TCIGuri4ZNvX19ejt7QUApKSkyKDJycnhnhryWAwX8hhWqxVfffWV7JXU1NTAbrcjLCwMhYWFMkzy8/MRHBw8rW0be59LREQEYmJipnXl1+XLl+UwWn19vVzNlpCQoAqbxMRErkgjj8BwId0MDAygurpaDnOdOnUKDocD0dHRKCkpkcNcixYtgtHIhY1jmc1m1ZyN+7iamJgY1Yq0lJQUhg3pguFC0+by5cuqZcFNTU0QQiAhIUE1+Z6RkSGXBdP4DA0NobGxUQZOc3MzHA4HwsPDkZ2dLQMnPT2dQU3TguFCU0IIgQsXLqgm39vb2wEA8+bNk0FSWlqKOXPm8O1aY3a7HWfPnpXDaKdPn4bdbkdwcLBq+XNWVhaCgoL0bi75IIYLacLlcqGlpUW1LPjSpUtQFAULFiyQPZPi4mLMnDlT7+b6HYfDgdbWVtW8zdDQEIxGIzIzM+W8zaJFixAWFqZ3c8kHMFzojjgcDjQ2NqrO5Orv74fRaEReXp7slRQVFWm+moomTwiB8+fPq1ak9ff3y6sDxi5/5gVodCcYLjQudrsdJ0+elL2SEydOYGhoCMHBwSgsLJRhUlBQgNDQUL2bSxMkhMDFixdVe20uXboEAJgzZ47qjDT2PGk8GC50U4ODgzhx4oTsmZw8eRKjo6OIiIhAcXGxnDPJy8vjkSU+qq+vDw0NDXIozX0gZ3x8vAybnJwcJCUlcc6MbsBwIQDA1atX5UbFqqoqNDQ0wOVyITY2VrWSa8GCBTxs0U9ZLBbV8ueWlha4XC7MmDHjhgM5udqPGC5+qru7W7UsuLm5GQCQlJSkCpO0tDS+ldJN2Ww2nD59WobNmTNnMDo6itDQUCxatEgGTmZmJnu3fojh4geEEGhra1MtC3bv8M7IyJDzJSUlJUhKStK5teStRkZG0NzcrFr+bLPZEBQUJA/kdC9/5oGcvo/h4oOcTifOnDmjWhbc19cHg8GARYsWySApKSnh2VQ0ZZxOp+pAzvr6elgsFnkgpztsFi1axAM5fRDDxQe4D3h0B0l1dTUsFgsCAwORn58vh7mKioqm7IBHom/j3lg7dq9NX18fACA1NVU1b6PFqdakL4aLF7LZbDcc8Dg8PIzQ0FDVsuDFixcjJCRE7+YS3ZQQApcvX1b1bLq7uwEAiYmJqgM5ExISOPfnZRguXqKyshJHjx5FVVUV6urq4HA4MGPGDNUBjzk5OTw3irxaf3+/qmfT3t4uD+R092rKyso4nOsFGC5e4vLly3C5XAgKCkJwcDCCgoJgNBr5Nkc+zeFwYHBwEBaLBRaLBUNDQ8jMzOSpAV6A4eIlXC4XFEVhmJBfczqdUBSF+2i8AMOFiIg0x/gnIiLNcfb3a06nEzU1NbBYLHo3xevl5uYiPj5e72bQHXA6nWhqaoLVatW7KV4vIyMD0dHRejdDNxwW+9rw8DAefvhhDA8Pcy/IHXKfBPCrX/0Ka9as0bs5dAfsdjt+8pOfYGRkhLvoJ6Grqwsvv/wyysrK9G6Kbthz+ZoQAgaDAf/8z//s8x8Il8uF4eFhAIDJZNJskYDD4cAzzzwDvq94L/dz8P3vfx95eXlwOp0wm80wm808kPIWRkdHMTo6Kp8lh8OBV155Re9m6Y7h8g0BAQEIDg7WuxmaE0Lg2rVr+OSTT7Bz506cP38eBoMBaWlpePjhh7Fq1SqEhIRMKmi4ms13GAwGVFZWYu/evTh//jxCQkLwy1/+kkuAv0EIgWPHjmHLli3IyMhAXl4eioqK+ByA4eIXhBA4deoUfvazn6GmpgYul0v+2blz57B3716Ul5fjZz/7Ge+zJwDXXxS6urpQW1sLADAajejs7GS43ERdXR06OzvR2dmJI0eOYO7cuXo3ySOwj+vjhBA4dOgQvv/97+P48eOqYHFzOBz4/PPP8YMf/AAdHR0c1iIAQHZ2thwGc19rzc+Gmt1uR1NTkyzHx8cjMTFRxxZ5DoaLDxNCoK6uDn/3d3+HixcvynqTyYTCwkLk5eUhKChI1tfV1eGnP/0p+vv7+Y8IITU1FZGRkbLsvkCO/qS3t1f1bGVkZPCa768xXHyU+1DAf/iHf5CHAQLXP/y//vWvsXXrVmzbtg2/+MUvVOc0HT16FJs2bYLD4dCj2eRBoqOjkZycLMvt7e24du2aji3yLEIItLS0qJZtZ2dnc1j5awwXH+VwOLBp0yacOnVK1mVkZOCtt97CPffcg+DgYJhMJnz3u9/F66+/Lt9QhRB47733sHfvXvZe/JzRaMSiRYtk2Ww2o6OjQ8cWeRYhBOrr6+VzEhwcjKysLJ1b5TkYLj5ICIH9+/dj+/btsi4mJgavvfYa5s+fr3qzMhgMWL16NX74wx8iICAAwPU9P7/61a/Q398/7W0nz6EoCrKzs+XnwuFwoKGhgS8dX7Pb7Thz5owsz5o1CwkJCTq2yLMwXHzQwMAANm3aBJvNBuD68uof/ehHKC4uvmmXPSAgAM888wyWLl0q6xobG/GHP/yB/5D4uZSUFMyYMUOWGxoaOGT6tZ6eHtV8S2ZmJjeejsFw8TFCCPzxj39EXV2drCsrK8Pjjz9+2w1wYWFhePnll+XwmMvlwu9+9zt0dXUxYPxYVFQUUlNTZbm9vR1ms1m39ngKIQTOnDkDu90O4HovLzc3l/MtYzBcfExvby/effdduaonLCwMf/3Xf42wsLDbfp2iKMjPz8d3vvMdWdfV1YX333+f4eLHAgICkJOTI8vXrl1DW1ubji3yDO6VmO5nw2Qy3TDk7O8YLj5ECIEdO3aoHv5169ahsLBwXB/6gIAAPP/884iLi5Pfb9u2bbh06dKUtZk8m3vexX3DqcvlUv2j6q+GhobQ3Nwsy4mJiZg1a5aOLfI8DBcfcvXqVbz33nvywY+MjMSf//mfj/vqY0VRkJaWpuq9dHd346OPPvL7f0z8WXJysmq5emNjI0ZGRnRskf46OzvR29sry4sWLfLJY6Mmg+HiI4QQ+Oyzz3Du3DlZV1FRgQULFkyoq24wGPC9731PHhXu7r1wnN1/hYeHIzMzU5a/+Q+rvxFCqALWPd9CagwXH2Gz2fD++++r5lqeeuopuYx0ItLT01VH5rvPH2PvxT8ZDAbk5eXJstVqxZkzZ/z28+B0OlULZiIjI5GRkcH5lm9guPgAIQSOHz+u2jC5dOnSO94tbDAYsHHjRrms0ul0Ytu2bXJlDPkXRVGwcOFChISEALj+eautrfXbcDGbzaoRgpSUFMTExOjYIs/EcPEBTqcTH3zwgeymBwYG4vHHH0dgYOAdfT93N7+wsFDW1dTU8OBCP5aYmIjZs2fLclNTE4aGhnRskT6EEGhtbVUNE+fl5Y17XtOfMFx8wIULF3DgwAFZzsrKQllZ2aS66cHBwfizP/szOaxmtVqxY8cOhoufCgkJQXZ2tiz39vbiwoULOrZIP7W1tXL4OTAwkPtbboHh4uXcE/lXrlyRdRs2bJj0Vc2KomDZsmWqgwv37NmDvr6+SX1f8l75+flyI+7IyAhOnTrldy8bdrsd9fX1shwXF8f7W26B4eLlbDYbdu7cKcsxMTGoqKjQ5E0qNjYWFRUVstzV1YWDBw/63T8odP1lIyMjQ3UUTG1trd8dBXPx4kV0dXXJclZW1rduUPZXDBcv5j6V9fTp07JuyZIlqt7GZCiKggceeEDeT+FyufDxxx9jdHRUk+9P3iU6Ohrp6emy3NbW5lc9WSEEGhoaMDw8DOD681FQUHDbY5X8GX8rXkwIgV27dslVXEajERs2bLij5ce3kpWVpVrDf/z4cbS3t2v2/cl7BAQEID8/X5YtFotfLUl2uVz46quvZDksLAwLFy7kfMstMFy82MDAAPbu3SvLc+bMQUlJiaYf9uDgYKxfv15+z4GBAXzxxRd+8w8K/Yl7FaF7J7oQAjU1NX7zWejv78fZs2dlOSUlBfHx8Tq2yLMxXLyUEAInTpxQXd5UXl4ud9ZrRVEUlJeXq9bxf/bZZ3JogPxLUlKSaklyY2MjBgcHdWzR9HDfOjl2CfLixYvveLm/P2C4eCkhBD799FM5oRoUFISKioopGf9NSkpCaWmpLDc2NvrVcAj9iclkUu3W7+3tRVtbm89/Ftwvc2OXIOfn53NI7DYYLl6qr68Phw8fluW0tDTV0ehaCggIwLp162Rw2Ww27NmzZ0p+Fnm+goIC1e2UJ0+e1LdB08Bms6mWIM+aNYtLkL8Fw8ULud+ixt6CV15ejoiIiCn5eYqioKysTHWk+BdffOGXO7T9nXtJ8thh0pMnT/r80UDnz59XXT2Rm5vLJcjfguHihVwuF3bv3g2n0wng+qT76tWrp/RnxsXFYcmSJbLc0tKCxsbGKf2Z5JkiIyOxcOFCWe7o6FDt/fA17oUL7uOVDAbDuO9I8mcMFy905coVVFVVyXJGRsaUL4k0GAy477775HCI3W7Hnj17fH6snW5kMBhQVFQkP2/Dw8M4efKkz34W7Ha7aglydHQ0b50cB4aLlxFC4KuvvlJ10ZcvXz7lXXRFUVBYWKhaKbRv3z6/WClEaoqiYNGiRYiMjJR1J06c8Nnd+t3d3Th//rwsL1y4EFFRUfo1yEswXLyMEAJ79uxRDYmtWrVqWn52bGysamjs3LlzOH36tM++sdKtxcfHqy4Qa21tRU9Pj44tmhpCCJw8eRI2mw3A9WAtLi7mrvxx4G/Iy/T39+PIkSOyPG/evAnfNnmnFEXBmjVr5PHidrsdX3755ZT/XPI8AQEBqqGxoaEhn7zjxeFw4Pjx47IcERFxx/ck+RuGixcRQqCurg7d3d2y7t577530CcjjpSgK7rrrLiQkJMi6ffv2cdWYH1IUBfn5+fLcOSEEjh07JnvUvuLixYtoaWmR5fnz53NX/jgxXLzMl19+qdo4WV5ePq0/PyYmRrWhsrW1lRsq/VRiYiIyMjJk+ezZs+jt7dWxRdpy37hptVoB/GlITMuz+3wZw8WLXLt2DYcOHZLl5ORk5OTkTGsX3WAwYPXq1fIBGx4eVp1vRv7DaDSipKREli0Wi08NjTkcDlRVVcn/nvDwcCxevJhDYuPEcPEiTU1NqrPEysrKVCt2poOiKCgqKsLMmTNl3b59+3jWmB9SFAWLFy+WKxWFEDh69KjPDI1dunQJzc3NspyZmanaSEy3x3C5DSGE/J/ehBCorKyUG7mMRiNWrlypy1tUXFwcCgsLZfns2bNobW2d9naQ/pKSkpCWlibLTU1NPrFqzL3k3z2fqCgKSktL5WIW+nYMl5twuVzo7+/HgQMHsH37dnlYnZ6sVisOHDggywkJCbp10d1DY+7lmFarFfv375/2dpD+jEYjysrKZHlwcBBfffWVR7yQTcbo6CiOHDmiGhIrKCjgkNgEMFy+QQiBN998Ew8++CCee+45/PKXv1TdT6+XlpYW1aqVkpIS1flO08k9sRkbGyvr9u7dK3tV5D/cKwjdKxaFEDh8+LDXb6js6upSPW9ZWVkcEpsghss3KIqC4eFhdHR0wOFwoKenR/ejLYQQ2L9/v9zIZTAYsHLlSl03ciUkJKhuJWxsbFTtYib/kZiYiKysLFluaWnx6rPG3Muqx64SW7p0KVeJTRDD5SZWrFiBoKAgANdXjOh986LdbletyIqNjVVtYNOD0WjEqlWrYDAYEBkZidzcXPZc/JTRaMTSpUtVGyrHrrLyNna7HUePHpXlqKgorhK7AwyXm1iwYAFSUlJk+ejRo7h27Zpu7Tl//jyamppkuaCgQLVaSw+KomDlypV4/fXXsWPHDmzevBnZ2dm6ton0oSgKCgoKMGPGDFl39OhRr1xBKIRAa2sr2tvbZV1ubi7i4uL0a5SXYrjcRGRkJO655x5Z7urqwqlTp3R5ExNC4ODBg7BYLACuP8irVq3yiC56QkICNm7ciPT0dNnTI/8UFxenuqGyvb0dra2tXtd7EULg0KFDquP177nnHp4ldgf4G7uFlStXyvuxR0ZGdNsoODo6isrKSlmOiopCWVmZR3TRFUXxiHaQ/tz/CLtfekZGRnDgwAGvC5dr167h2LFjsjxr1qxp36jsKxguN6EoCnJzc1XHyx84cECX4+W7u7tx6tQpWc7JyUFSUtK0t4PodhRFQU5OjurcuerqagwMDOjYqolxn903dp9OSUnJlN3w6usYLrfg7iG4tbW1qeY9poN7x7PZbJZ1Y3tURJ4kMjJSde7c5cuXvWrPi9PpxN69e+W+tuDgYNxzzz3stdwhhsstKIqiOkPLfbz8dD4oTqdTtVItPDwcd999Nz/s5JEURcG9996LkJAQANdfjvbu3es1e146OztRX18vy+np6UhLS+PzdocYLrfgXgHzzePl3Wvfp0NPTw9qampkecGCBUhNTZ22n080Uampqao9L42NjWhvb/f43ot7In/scS/Lly9HcHCwzi3zXgyX24iNjVWd+trS0oKzZ89Oy4MihEBVVZXqdICVK1fyw04eLTAwEOXl5fJt32azYd++fR4fLhaLRXW8UnR0NEpKSthrmQSGy20YDAasWbNGdbz82JVbU8nlcmHPnj1y/Dc0NBTLli3jh508mqIoKCwsVO3DOnz4MPr7+3Vs1e25D6kce6pASUmJ6ngjmjiGy224z9Aa+6B8+eWX0zI0dvnyZVRXV8vy/PnzVRczEXmqqKgoLF26VJYvX76MY8eOeWzvZXR0VPUiFxQUpPvxSr6Av71vERcXp1oB09zcPOU3L7rPNhp7q195eTlMJtOU/UwirSiKgvLyctUVyJ9//rlH7tgXQqClpQWNjY2ybsGCBcjIyOAowSQxXL6FwWDA2rVr5dCYzWbDnj17pvRnOp1OfPbZZ/JNymQy6XZ3C9FEKYqCuXPnqg42bW1t1e2Ui9txuVz47LPPZPC5h8J54sTkMVy+hXtobOxx219++eWUbqjs6elBVVWVLC9YsACZmZlT9vOItGY0GrF27Vp5uZbD4cCnn37qUcuShRDo7u5WPWtJSUkoLCzki5wGGC7jEBcXhyVLlshya2srGhoapuQtTAiBI0eOoK+vT9atWrWKQ2LkVdynXIydJ6ytrUVzc7PH9F6EENizZ488lNZ9bt90Xx3uqxgu42AwGLBu3Tr5Fma32/Hpp59OyUPicDjwySefyCGxsLAwDomRVwoJCUFFRYWcGB8eHsauXbs84mZXAOjr67vhKovly5fzWdMIw2UcFEVBUVER5syZI+sqKyun5NykCxcu4Pjx47KcnZ3NVWLklRRFQVlZGZKTk2XdsWPH0NbWpnvvxd1rGbuPbPny5TxaX0MMl3GKiorCihUrZPnChQuorq7W9CERQqCyslKeJaYoCtatW8eNk+S1IiIiUFFRobpI7P/+7/9077309fXh888/l89vVFQU1qxZw+XHGuJvcpwURcH999+vuqHy448/htPp1OxnDA8PY+fOnaoP/IoVK9hNJ6+lKAqWLVumOsn78OHDut714l4hNnap/7333qs6BZ0mj+EyTu4jxceem3To0CHN7goXQqCxsRENDQ2yrri4GHPnztXk+xPpJSoqCuvWrVP1Xv74xz9q+mI2XkII9PT0YPfu3TLcZsyYgfvvv5+9Fo3xtzkBYWFhuP/++2X5ypUr2LNnjyZvYEIIfPTRR7DZbACAgIAAbNiwQS4iIPJWiqJgxYoVqrmXqqqqKVtxeTsulwsff/zxDWf2zZkzhyMEGmO4TICiKKioqEBUVBSAGwNhMnp7e1WbM5OTk3m8PvmMGTNmYMOGDaqVY++//z7sdvu0tUEIgdbWVnzxxReyLjY2FuvXr+dzNgUYLhOUkpKi2vPS0NCAmpqaSb2BCSHwxRdfqIbYKioqEBMTM6m2EnkK910v8+fPl3X19fU4ePDgtPVeRkZG8Ic//EFugHbPo86aNYvhMgUYLhNkNBrxyCOPyNsgR0ZG8MEHH0xq/Nhms+GDDz6QK2giIiKwYcMGTdpL5CnCwsLw6KOPqhbFvP/+++jr65vygHFvTh67zH/u3Lm47777ONcyRfhbnSBFUVBaWqo6jqWyshJtbW139P3c97bU1dXJurKyMmRlZfFtinyK+zj+sQfBdnV1Ydu2bVM6uS+EQG9vL9577z15/IzRaMTGjRsxY8aMKfu5/o7hcgciIyPx0EMPyX/8+/v7sX379jtauz8yMoL//u//xsjICIDrly09/vjjsmdE5EuMRiMef/xxOW8JAF988cWkh5Zvx+FwYOvWrejs7JR1xcXFKCsr4wvcFGK43AFFUbB+/XrVYZY7duzAxYsXJ/R9hBCoqanBwYMHZV1ubi6WLFnCDz35JPeJyQ8//LBqcv83v/kNent7NQ8YIQQOHjyouuQvOjoaTz75JE8+nmIMlzs0e/ZsPPjgg7Lc3d2N7du3T+jhGBkZwTvvvKNafvzUU08hLCxM8/YSeQr3WX25ubmy7sKFC/jtb3+r6eoxIQTa2tqwefNmOTIQEBCAxx57DCkpKXyBm2IMlzukKAo2btwoV3QJIbB161Z0d3eP6+uFEDh06BD27dsn6xYuXIg1a9bwQ08+z2Qy4YUXXlCtiDx48CD+93//V5P5FyEEzGYz/uM//kO1E7+kpARr167lJP404G/4DimKgvT0dNWqrs7OTmzevHlcD8e1a9fw7//+7/JNzWg04oUXXuBx3+QXFEVBWloann76aTm/6HK5sG3bNlRWVk7q7DEhBKxWK9566y2cPn1a1iclJeGFF17gWX3ThOEyCQaDAc888wzi4+Nl3datW1FfX3/b4TGXy4Xf//73qKmpkXXFxcWqA/6IfJ37OuSxR8PY7Xb853/+Jw4dOnRHASOEgM1mw3/913/h8OHDsj48PBx/+Zd/iYSEBD5j04ThMgnut68nnnhCfmDNZjN+8Ytf3PKmSiEETpw4gbffflt1Z8tLL73EuRbyO0ajEU8++SRKSkpk3dDQEN58803s2bMHDodj3POYQgg5IjD2WKagoCA899xzWLx4MYNlGjFcJslgMODpp5++4UDLt99+G6Ojo6q/K4RAR0cH/vEf/xH9/f0ArgfUY489htLSUn7wye8oioKwsDD88Ic/RE5OjqwfGhrCW2+9hd/97ncYHBz81oBxuVxobW3Fq6++iv3798u/797PwuP0px9/2xqIj4/HT37yE3kVscvlwttvv413330XdrsdQggIIdDS0oK/+Zu/UY0D5+Tk4K/+6q8QEBCgV/OJdKUoCmJjY/G3f/u3qoAZGRnB9u3b8corr+Do0aOw2WyqkBFCwOVyobe3F1u3bsUrr7yiOgwzMDAQGzduxCOPPMIDYHXA37gG3GPHzz33nBzustvt+Nd//VecOHECq1evRkdHB95//33VarL4+Hj8/Oc/R3x8PHst5NcURUF8fDx++tOf4q233sLRo0flS9nZs2fx+uuvIyUlBXl5eUhNTUVwcDD6+/tx5swZ1NXVob+/XxU8YWFheOqpp7Bu3TpuSNYJw0UjRqMRP/rRj3DhwgV54dfIyAh27dqFXbt23fD3o6Ki8Oqrr+Kuu+5isBDhesDExMTg5ZdfxgcffKA6cdzhcKC1tRWtra3y795qqCw5ORl/8Rd/gYKCAo4I6IjhoqHw8HC8+uqrCAoKwkcffSTPMfqm5ORk/PznP0d5eTmDhWgMRVEQGhqKJ598Evn5+Xjvvfdw+vTpG56lmwVLREQEVq5ciYcffhixsbF8tnTGcNGQoiiIiorCa6+9huLiYrz77rtoa2vD6OgoFEVBdHQ01qxZgx/84AdITU3lh5/oFgICApCXl4esrCzU1dWhsrISjY2NMJvNqn1kwcHBSExMRHFxMcrLy5GUlMTeiodguGhMURSYTCY88cQTePDBB9Hc3Iyenh6EhYUhPT0diYmJMBgMDBaib6EoCkJCQlBcXIzCwkJYLBb09PSgv78fo6OjCA0NRXx8POLj4xESEiK/hjwDw2UMl8uFmpoaTW6WHCswMBAjIyM4ffq0aqWYr3E4HDCbzXo3gybJ5XKhqakJw8PDejfllgIDAzE6Ooru7u5xH7k0XZxOJywWi97N0B3D5WuKomD+/PmorKxUnaBKExMREcE7MryYwWBASkoKqqurUV1drXdzvFZoaCjCw8P1boauFDFdd4x6OCHEhHYD060ZjUZuWPNSQgg4nU4+BxoICAjw6+eA4UJERJrz31glIqIpw3AhIiLNMVyIiEhzDBciItIcw8VLuFwujIyMTOqGPiJv53Q6MTQ0pMlVyDS1GC5eorGxERkZGWhsbNS7KUS6aW9vx2OPPYb29na9m0LfguFCRESaY7gQEZHmGC5ERKQ5hgsREWmO4UJERJpjuBARkeYYLkREpDmGCxERaY7hQkREmmO4EBGR5hguRESkOYYLERFpjuFCRESaY7gQEZHmGC5eQAiB/v5+OJ1O9Pf3Qwihd5OIpp0QAlevXoXNZsPVq1f5HHg4hosHM5vN2LRpEzIzM7Fs2TJ0d3dj2bJlyMzMxKZNm2A2m/VuItGUG/sclJSUoLKyEiUlJXwOPJwiGP8eaffu3XjkkUdgtVoBQPWWpigKACA0NBTbt29HRUWFLm0kmmp8DrwXw8UD7d69G+vXr4cQ4rbXGhsMBiiKgp07d/LBIp/D58C7MVw8jNlsxpw5c2Cz2W77QLkZDAaYTCZ0dnYiKipq6htINA34HHg/zrl4mM2bN8NqtY7rgQIAl8sFq9WKLVu2THHLiKYPnwPvx56LBxFCIDMzE+fOnZvQShhFUZCWlobm5mY5Dk3krfgc+AaGiwfp6+tDfHz8pL4+NjZWwxYRTT8+B76Bw2IeZHBwcFJfb7FYNGoJkX74HPgGhosHCQ8Pn9TXR0REaNQSIv3wOfANDBcPEhsbi/T09AmPFyuKgvT0dMTExExRy4imD58D38Bw8SCKouDFF1+8o6996aWXOIlJPoHPgW/ghL6H4fp+Ij4HvoA9Fw8TFRWF7du3Q1EUGAy3/7/HvTP5ww8/5ANFPoXPgfdjuHigiooK7Ny5EyaTCYqi3NDNd9eZTCbs2rULa9eu1amlRFOHz4F3Y7h4qIqKCnR2duKNN95AWlqa6s/S0tLwxhtvoKuriw8U+TQ+B96Lcy5ewH2PhcViQUREBGJiYjhpSX6Hz4F3YbgQEZHmOCxGRESaY7gQEZHmGC5ERKQ5hgsREWmO4UJERJpjuBARkeYYLkREpDmGCxERaY7hQkREmmO4EBGR5hguRESkOYYLERFpjuFCRESaY7gQEZHm/h8BmzEh0DgRvgAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 500x400 with 4 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "model.train(dataset, opt=\"LBFGS\", steps=20);\n",
    "model.plot()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "fb0f6758",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "function , r2\n",
      "sin , 0.999987075018884\n",
      "gaussian , 0.921655835107275\n",
      "tanh , 0.8631397517896181\n",
      "sigmoid , 0.8594117556407576\n",
      "arctan , 0.8440367634049246\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "('sin',\n",
       " (<function kan.utils.<lambda>(x)>, <function kan.utils.<lambda>(x)>),\n",
       " 0.999987075018884)"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# sin appears at the top of the suggestion list, which is good!\n",
    "model.suggest_symbolic(0,0,0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "9a2406e8",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "function , r2\n",
      "x^2 , 0.9999996930603142\n",
      "cosh , 0.9999917592117541\n",
      "gaussian , 0.9999827145861027\n",
      "sin , 0.9980876045759569\n",
      "abs , 0.9377603078924529\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "('x^2',\n",
       " (<function kan.utils.<lambda>(x)>, <function kan.utils.<lambda>(x)>),\n",
       " 0.9999996930603142)"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# x^2 appears at the top of the suggestion list, which is good!\n",
    "# But note how competitive cosh and gaussian are. They are also locally quadratic.\n",
    "model.suggest_symbolic(0,1,0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "26dfe636",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "function , r2\n",
      "exp , 0.9999987580912774\n",
      "tanh , 0.9999187437583558\n",
      "cosh , 0.9999121147442106\n",
      "sigmoid , 0.9998776769631791\n",
      "gaussian , 0.9998535744392626\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "('exp',\n",
       " (<function kan.utils.<lambda>(x)>, <function kan.utils.<lambda>(x)>),\n",
       " 0.9999987580912774)"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# exp appears at the top of the suggestion list, which is good!\n",
    "model.suggest_symbolic(1,0,0)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a880bac4",
   "metadata": {},
   "source": [
    "The takeaway is that symbolic regression is very sensitive to noise, so if we want to extract exact symbolic formulas from trained networks, the networks need to be trained to quite high accuracy!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "0fd2e8b6",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "fixing (0,0,0) with sin, r2=0.999987075018884\n",
      "fixing (0,1,0) with x^2, r2=0.9999996930603142\n",
      "fixing (1,0,0) with exp, r2=0.9999987580912774\n"
     ]
    }
   ],
   "source": [
    "# now let's replace every activation function with its top 1 symbolic suggestion. This is implmented in auto_symbolic()\n",
    "model.auto_symbolic()\n",
    "\n",
    "# if the user wants to constrain the symbolic space, they can pass in their symbolic libarary\n",
    "# lib = ['sin', 'x^2', 'exp']\n",
    "# model.auto_symbolic(lib=lib)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a3d634fe",
   "metadata": {},
   "source": [
    "After retraining, we get (almost) machine precision! This is the winning signal that this formula is (very likely to be) exact!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "9fcecc80",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "train loss: 2.13e-10 | test loss: 2.78e-11 | reg: 2.93e+00 : 100%|██| 20/20 [00:01<00:00, 11.85it/s]\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZcAAAFICAYAAACcDrP3AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjYuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8o6BhiAAAACXBIWXMAAA9hAAAPYQGoP6dpAAArUklEQVR4nO3de3CV5Z0H8O/75gIJSUhCAgl3EgJyvyaIF1TWEiva1U11p90tdtvZ6c62sm2n0xlnV51uZ+x2Z3YWdKd2t11bcauWFqoo1HgLqNwRASFyCwImXAIJJyQ5uZ1znv3j5+N73pCEkLwn7/ue8/3MZOx5A+SBnvd83+f2ewyllAIREZGDTLcbQERE8YfhQkREjmO4EBGR4xguRETkOIYLERE5juFCRESOY7gQEZHjGC5EROQ4hgsRETmO4UJERI5juBARkeMYLkRE5DiGCxEROY7hQkREjmO4EBGR45LdbgCRHyil0NDQgJaWFmRkZGDUqFEwDMPtZhF5FnsuRH0IBAJYu3YtSkpKkJ+fjylTpiA/Px8lJSVYu3YtAoGA200k8iSDJ1ES9ayyshIVFRUIBoMApPei6V5Leno6NmzYgPLyclfaSORVDBeiHlRWVmLlypVQSiESifT660zThGEY2Lx5MwOGKArDhaibQCCA8ePHo62trc9g0UzTRFpaGmpra5GdnR37BhL5AOdciLp5/vnnEQwG+xUsABCJRBAMBrFu3boYt4zIP9hzIYqilEJJSQlOnTqFG7k1DMNAUVERTpw4wVVkRGC4ENlcvnwZ+fn5PX4vBUAhgPMAuvr4/aNGjYpR64j8g8NiRFFaWlp6/V4hgMc//29vmpubnW4SkS8xXIiiZGRkDOr3Z2ZmOtQSIn9juBBFGTVqFIqLi2943sQwDBQXFyM3NzdGLSPyF4YLURTDMPDoo48O6PeuXr2ak/lEn2O4EHXzyCOPID09HabZv9vDNE2kp6dj1apVMW4ZkX8wXIi6yc7OxoYNG2AYxnUDRu/Q37hxIzdQEkVhuBD1oLy8HJs3b0ZaWhoMw7hmuEtfS0tLw5YtW7BixQqXWkrkTQwXol6Ul5ejtrYWa9asQVFRke17RUVFWLNmDerq6hgsRD3gJkqiflBKIXDwIJJ/9jOEHnsM2fPmcfKeqA88LIyoHwzDQE5ODpCZCeTkAAwWoj5xWIyIiBzHcCEiIscxXIiIyHEMFyIichzDhYiIHMdwISIixzFciIjIcQwXIiJyHMOFiIgcx3AhIiLHMVyIiMhxDBciInIcw4WIiBzHkvtE/dXcDHz0EbBggVRHJqJeMVyI+iscBoJBID0dSEpyuzVEnsZwISIix3HOhYiIHMeTKMkzIpEIztfUINze7nZTfC930iRkZGW53QxKYAwX8oxQKITGl19GV1cXzOHD3W6OPymF5Pp64O/+Dhnz57vdGkpgDBfyDqUQNgzkPfwwxk6f7nZr/KGtDairA8aNA9LSEAqHcXTtWrdbRcRwIe9JSkpCckqK283wPqWAY8eA3/wGyM0FZs4E7roLhmG43TIiTugT+ZZSwIEDQFcXcPEisH070NrqdquIADBciPzr6lXgxAnrdWGhfBF5AMOFyI+UkmBparKuzZkDpKa61yaiKAwXIj/SQ2KRiLxOSZFw4XwLeQTDhciPmpqA48et12PHyheRRzBciPxGKQmW7kNiw4a51yaibhguRH4TiQAffighA0iozJ3LITHyFIYLkd80NAAnT1qvJ0zgkBh5DsOFyE+UAo4ckbNltPnzuUqMPIfhQuQnoZAMiWnp6VwlRp7EcCHyk3PngNOnrddFRcDo0a41h6g3DBciv1BKei0dHfLaMIDFi3kqJnkSw4XIL4JB2TipZWcDs2ZxSIw8ieFC5Ad6b8vFi9a1OXMAHghGHsVwIfKDcBjYtcte7qW0lL0W8iyGC5Ef1NfL2S3a+PHAlCkMF/IshguR1ykF7NtnP6ulrIx7W8jTGC5EXtfaKuGijRwpGyfZayEPY7gQeZlSQHW1fSJ/7lw51pjIwxguRF4WCgEffGBN5KemAkuXstdCnsdwIfIqpWQ3fk2Nda24GJg8meFCnsdwIfKqSAR4/32gs1NemyZw661AcrK77SLqB4YLkRcpBVy4ABw6ZF0rLARmz2avhXyB4ULkRUoBO3bYlx8vXSpVkIl8gOFC5EUNDcCePdbr3FzuyCdfYbgQeY1SwM6dQCBgXVuyRApVEvkEw4XIa65ckSExLTMTuOUWmdAn8gm+W4m8RM+1NDRY10pLeSAY+Q7DhchLrlyRTZNaRgawbBnnWsh3GC5EXqH3tXTvtRQWMlzIdxguRF5x+bK915KZCdx5J4OFfInhQuQFkQjw7rv2FWJLlwIFBQwX8iWGC5HblAI++0xOmtRycthrIV9juBC5LRQC/vxn+278O+4ARo1iuJBvMVyI3KTPa+leQ+y227ivhXyN714iNwWDwObNQFeXvDZNoLwcyMpyt11Eg8RwIXJLJAK8956c2aJNmwYsWsThMPI9hguRG5QCzp0D3n5b/jcADB8O3HcfMGyYu20jcgDDhcgNnZ3Apk3A1avWtVtuAaZOZa+F4gLDhWioKSXl9KMn8ceMkbkWTuJTnOA7mWgo6RMmX3sNCIflWnIycP/9sreFvRaKEwwXoqHU2Qls3CgFKrWFC+WLwUJxhOFCNFQiEWDbNvtwWF4e8Jd/Kb0XojjCcCEaCkoBJ08CW7ZIyABASgrw4INAfj57LRR3GC5EsaaUFKR8+WV7iZelSzkcRnGL4UIUa52dwB/+IMUptcmTga98BUhKcq1ZRLHEcCGKpXAYqKwEPvzQupaZCXzta1Lihb0WilMMF6JYiUSAvXslXPQ8S3Iy8MAD0nNhsFAcY7gQxYJSwLFjwPr1MiwGSJjceafsxOdmSYpzfIcTOU0p4OxZYN06oLnZuj53rmyW5DwLJQCGC5GT9A78554DLl+2rk+aBHz960BaGofDKCEwXIicohRQXw/87/9KxWNt9Gjgm99keRdKKAwXIifoYPn1r4EzZ6zr2dkSLOPGMVgoobDmBNFg6bNZnntO5lq0zEzgkUdYRp8SEsOFaDCUkpMkn3tO5lo0HSyzZjFYKCExXIgGKhIBjhwBXnjBXuU4KwtYtQqYM4fBQgmL4UJ0o5SSnffvvw+88goQDFrfy8mROZYZMxgslNAYLkQ3QikJk02bgPfeA0Ih63sFBRIsRUUMFkp4DBei/lIKqKsDXnoJOH7c/r2pU2UorKCAwUIEhgvR9SklPZRdu6THEghY3zNNYPFi4OGHWYiSKArDhagvev/Kq68C+/db594DwPDhwD33AHffDaSmMliIojBciHqiFNDRAezcCfz5z/bVYAAwZoz0VmbPZhFKoh4wXIii6ZVgJ04Ar78uRxPrcvmAFJ1cuFCOJ87LY2+FqBcMFyJAQkUpoLZWzl85cMAqla/l5gL33QfcfLOcy8JgIeoVw4USm1LSM6mrA7ZulRMjo/etAEBKikza33cfkJ/PUCHqB4YLJSa9AuzMGWDbNuDQoWtDxTDkxMj77gNmzpQhMQYLUb8wXChxKCX/bWkBqquB7dtlTqWr69pfO3q0rAJbsoRnsBANAMOF4p9SMn9SVwfs2yfzKZcvW2ETLS8PWLZMjiLmvhWiAWO4UHzSgXLhAnD4MHDwoEzW99RLMQzZWX/bbUBZGTByJEOFaJAYLhQf9Gqv1lYJkepq+bpw4dpVX1pKCjBlioTKnDnAiBEMFSKHMFzIn3SYBIOyg/7UKeDYMTmsq6nJvpM+mmHI6ZDz5sl8ysSJEjIMFSJHMVzI+3SQdHZKcFy4IKu8Pv0UOH9erkVXJ+5JZqYUl1y0CLjpJnltGAwVohhhuJA3tbQAly4BFy/KEcJ1ddJDCQSkLEtPk/HRTFPmToqKgLlzgenTpcdimgwUoiHAcCFvevNN4K23rt8jiZaWJjW/SkrksK5Jk4CMDPZQiFzAcCFvys3tO1hMUybg8/NlUn7qVJk/ycmR0iwAA4XIRQwX8qYxY2RHfDgsYZGeLsExdqyEyIQJstExI4N1vog8iOFC3jR2LLBypfRMxoyRYBkxgr0SIp9guJCnGEqhoaYG4Y4O6Z0AshqsqcndhvlEOBJBuLXV7WYQMVzIOwzTBMaOhfr4Y1z5+GO3m+NbKcOHIyk93e1mUIIzlLremk6ioaGUQri3zY90Q0zThMkTMslFDBciInIcH22IiMhxDBciInIcw4WIiBzHcCEiIscxXIj6KxwGmpt7L+dPRF9guBD1V20t8IMfyH+JqE8MFyIichzDhYiIHMdwISIixzFciIjIcQwXIiJyHMOFiIgcx3AhIiLHMVyIiMhxDBciInIcw4WIiBzHcCEiIscxXIiIyHEMFyIichzDhYiIHMdwIeoHpRQaGxvR3NyMxsZGKKXcbhKRpzFciPoQCASwdu1alJSUYMHChfj9+vVYsHAhSkpKsHbtWgQCAbebSORJhuIjGFGPKisrUVFRgWAwCACYoBQeB/BTAJ8ZBgAgPT0dGzZsQHl5uXsNJfIg9lyIelBZWYmVK1eira0NSqlrhsH0tba2NqxcuRKVlZUutZTImxguRN0EAgFUVFRAKYVIJNLnr41EIlBKoaKigkNkRFEYLkTdPP/88wgGg9cNFi0SiSAYDGLdunUxbhmRfzBciKIopfDMM88M6Pc+/fTTXEVG9DmGC1GUhoYG1NTUfBESBoBpAL4BYCOAMgBzAaR3+31KKdTU1KCxsXEom0vkWcluN4DIS1oCAcwDsAQSJGUARgIIAegCkAHgBwC+A+A0gMNRX00AmpubMWrUqKFvOJHHMFwosbW3AwcOALt3A7t3Y/zevdgEoB3AhwCeA7AbwEcAwgAKAUQATAcwG8AiAPd//kfVAch/8UVgyRJg9mxg9Gjg8yXLRImG+1wosTQ3A/v2AXv2SKAcPAh0dQFZWUBpKVRZGR546in8ubYWXf38I3MAzAFwR14efvLQQzDOnJFv5OVJyMyeDcyaBUyYwLChhMGeC8W3hgYrSPbsAaqrgUgEyM+XHsZXviL/nT4dME0YAJZ3duK1H/wA6Odz1xUA7xsGKh5/HMbq1RJg1dXA4cPy9d578jOzsiRkdOBMmQIkJcX0r0/kFvZcKL7U1Vlhsns3UFMj1ydMkBBZsgQoKwMmT+61FxEIBDB+/Hi0tbX1azmyaZpIS0tDbW0tsrOzr/0FbW3AsWNW2Bw7BnR2AmlpwIwZVthMmwakpAz8707kIQwX8i+lgFOnrF7J7t0SLgBQUmIPk8LCG/qj9Q79622kNE0ThmFgy5YtWLFiRf/+8K4u4MQJ4MgRCZsjRySAUlKkB6WH0WbMkAAi8iGGC/lHOAwcPWrvmTQ0AKYpH8hlZRImpaVAbu6gf1z32mLRt4oRVVts48aN/Q+WnkQiwKefWj2bI0eApib5e02dag2lzZoFZGYO6u9ENFQYLuRdXV3AoUNWz2TPHqClBUhNBebPt3omCxcCGRkxaUIgEMC6devw9NNPo0YPsQEoLi7G6tWr8cgjj2DkyJHO/lClpAemw+bwYeDSJfnepElW0MyeDXDZM3kUw4W8IxgEPvrI6pXs3w90dAAjRgCLFllhMm8eMGzYkDYt+jyXzMxM5ObmftF7GRL19dYw2uHDQG2tXC8osIdNYSFXpJEnMFzIPU1NwN691jDXxx8DoRCQkyNDXHqYa+ZMIJkLG20CAfuczalT0uPJzbWvSJs0iWFDrmC40NCpr7cvCz56VD4QCwrsk+9Tp8p8A/Vfa6ssf9aBc+KEBHVGhoSNDpziYgY1DQmGC8WGUsBnn9kn30+flu9NmWIFyZIlwPjxfLp2WkcHcPy4NYz2ySdybdgw+/Ln6dNlDovIYQwXckYkApw8aV8WfOGChMZNN1k9k9JSKYtCQysUkj0/0fM2ra3SiykpseZtZs6UOS6iQWK40MCEQjIMo3sle/cCV67Ih9XcuVavZPFiwOnVVDR4SgFnzthXpF25Ig8DRUX25c89bQwlug6GC/VPR4cUeNS9kg8/lCffYcNkJZcOkwULgPTuBenJ85QCzp+377W5cEG+N368vUYae57UDwwX6llLiwSI7pkcOCD7TjIzZWhLz5nMncuSJfHq8mUJGT2Upgty5udbYTN7NjBuHOfM6BoMFxKNjdZGxd275QMlEpFNetEruW66icUWE1Vzs33588mT8h4ZOfLagpxc7ZfwGC6J6tw5+7LgEyfk+rhx9jApKuJTKfWsrU1WoemwOXZMerfp6bIwQAdOSQl7twmI4ZIIlJLaVdHLgvUO76lTrfmSsjIJF6KB6OyUh5To5c9tbbLUWRfk1MufWZAz7jFc4lE4LE+R0cuCL1+WoYqZM60gKStjbSqKnXDYXpDz8GEZWtMFOXXYzJzJgpxxiOESD3SBRx0ke/fKTZySInW49DDX4sUxK/BIdF16Y230XpvLl+V7kyfb520cqGpN7mK4+FFb27UFHtvbZaw7elnw/PnA8OFut5aoZ0pJSaDons25c/K9wkJ7Qc6CAs79+QzDxS+qqoBduyRMDh2STYwjR9oLPM6ezbpR5G9Xrth7NqdPWwU5da/m5ps5nOsDDBe/qK+XZZ+pqbJxMTVVgoRPcxTPQiHZc9XcLF+trbL6jFUDPI/h4heRiAQJw4QSWTgs9wD30Xgew4WIiBzH+CciIsdx9vdz4XAYNfv3o6u52e2m+N7YOXOQk5/vdjNoAMLhMGqPHkU4GHS7Kb6XN3UqsnJy3G6Gaxgun+vq6kLtk0+is70dSdwLMiBKKaR++inwn/+JnC99ye3m0ACEQiHU//d/o6uzEyZ30Q9YSl0d8MMfIuvmm91uimsYLppSCJkmJjz5JEri/Q0RDsu+GNOUfTAOLRLoCoWwY9UqcBrPx5RC2DRR8Pd/jwlz58p7JRCQLxak7FlXl6xq+/xeCoVCOPjEE0j0u4Dh0k1yUhJShw1zuxnOUwq4dAl45RXgrbfk7I7UVCm9UVEB3HKLvB5M0HA1W9wwTRMpVVXA1q1San/4cOA//oNLgLtTSipjvPCClLSZMwdYvBgG7wOGS0KIRIDt24Gf/EQKC0b3LE6eBN54A7j/fuCxx+SsDt4YZBhAXR1w8KC8Tk6WYqcMl2sdOiRlbT77DNixA5g40e0WeQL7uPEuEgG2bAG++13g+HF7sGidncCGDcD3vic9Gg5rESClV/QwmD7Wmu8Nu44OKRKr5eVJ6RpiuMQ1paTH8i//ImPmWlaWlIu56SZ7uZhdu6T30tTEDxGSYpJZWdZrfYAcWS5dsuqhATI0xmO+ATBc4pdSMozx5JNyyqQ2fz7wu98BL74ovZUnnrB/gGzdCvzXf8lELiW2nBxgwgTr9enTwNWrrjXHc5QCamqA6GXbs2ZxWPlzDJd41dkpE7AnT1rX5s8HfvEL69z7zExg1Srgpz8FRoyQX6OUTE5u28beS6JLTgZmzLBeBwLA2bOuNceTPv7Yuk+GDZOD0AgAwyU+KQW8+y7w+uvWtYIC4Kmn5KTJ6Ccr05TJ/O98xxpfb2uTYLpyZWjbTd5iGFKFOClJXodCMjTGhw7R3m6fbxk9mvMtURgu8SgQAJ55RnovgDyBfv/7vXfZk5OBb38bWLrUunbkCPDyy/wgSXSTJsnRDtqRIxIyBFy8CFy4YL2eNo3HN0dhuMQbpYA//UlW9mi33w48+GDfY8EZGcCPfmTNvygFrFsnyysZMIkrO1sm9rUzZ+yLQxKVUtJraW+X14Yhe1w43/IFhku8uXxZ5kz0qp6MDFlifL0TKQ1D5mT+6q+sa+fOAS+9xHBJZElJMjSmNTUBn37qXnu8Qin7fMvw4dJzYbh8geEST5QCXnsNOHXKuvblLwMLFvTvTW+awDe/KRsptQ0bZDMdJSbDkOFUvWQ9EpFNg4n+wNHaKvvGtLFjgTFj3GuPBzFc4klTk32eJCtLwkJPyF6PYcgQSEWFde3iRRlmS/QPk0Q2YYL9WOHqams+L1HV1ckeF23GDFktRl9guMQLvULsxAnr2t13y5v+Rrrqpgn89V9bHyZKARs3Ag0NzraX/CMjQzYHarW19g/WRKOULGzQAWsYsryfbBgu8aKjA1i/3tr8mJYGfP3r/e+1RJs8GbjnHuv16dPAO++w95KoTNP+4RkM9l5KKBGEwzI0qGVmSvhyvsWG4RIPlJI3+/791rXSUvlAGMgbXvde9MbKSAT44x8lwCjxGIZUz9aLQpQCDhxI3HAJBOzzmpMnA7m5brXGsxgu8UAPXellkUlJEg4DHQM2DBlOW7LEunbwoH11DCWWwkKZtNaOHZNJ7USjS75EL8eeM8deo48AMFziw4ULMt+iTZkC3Hbb4LrpqanAV79qDau1t3NiP5ENHy69F62+XvZAJaKDB62l/ikpAx8hiHMMF79TCqiqklVd2sqVgz93wzCAW2+VoNKqquRDhRLTvHlWiaDOTuDw4cR72OjokMl8LS+P57f0guHidx0dwKZN9uXH997rzJNUdrZ9Yv/8eeC99xLvA4Xk/TR1qr2C9oEDiVcK5vx5WS2nTZ9uzU2SDcPFz5SSVTvRK1cWLQKKi5358w1DNmFmZMjrSESKYXZ1OfPnk7/k5gJFRdbr06cTa4m6XoIcXfJlwQKrN0c2/FfxuzfftCZWdYXjlBTn/vxp02Q4RNu/n+U/ElVSkpQI0q5elYn9ROnJRiLSW9PS0+XAPc639Ijh4mctLcBbb1mvCwoGP5HfXWqqzOHoP/PqVe55SVR6s6BehaiUPGwkynshELCXfJk0ScrsU48YLn6l97bU1FjXbr3VXhfMCYYBLFtmL/9RWWkNDVBiGTfOviS5ujoxliQrJdUvos84mj/f2VGCOMNw8Sul5EM++syWe++Nzfjv2LH2PS9Hj8pXojyxkiUtTfZ1aPX1MveSCO+F/fvtS5DnzeOQWB8YLn7V1CQrt7SJE4GFC2PzZk9Kkon96JMq337b+Z9D/rBggf10yujKEPEqGJRNxNqYMTIsRr1iuPiRUsBHH9k3sS1bZj8x0EmGIT2XggLr2rvvJsZwCNkZBlBSYi93cuBA/JcGOnPGfurk7NlcgnwdDBc/0kNieklwaiqwYkVsu+h5ecDNN1uvT54EPvkkdj+PvCsrS1ZJaWfPysFy8Uo/zOkhaNOUJf8cEusTw8WPAgFgxw7r9aRJsT9i1TSB8nJrOKSjQ4bGEmGsnexME1i82Hq/tbdLSZR4fS90dtqH/nJyeOpkPzBc/EY/RUWfDnnHHfad07FgGPK0VlhoXdu2jUNjiUifTpmZaV3bty9+d+ufOyfDYtpNN0nAUJ8YLn6jlPQY9I2cmiqHgg2F7kNjNTVcNZao8vLsB4jV1MRn3Tl9vEBbm7w2DKCsjLvy+4H/Qn7T1HTtkNisWUPTRTcM4Etfsg+NVVXF/ueS9yQny5lBWkuL7LuKtweNUAjYu9d6nZk5dPebzzFc/EQpqUQbXTjvtttiPySmGYYsd45eNbZ1qyzTpMRiGLKJUK+YUgrYs8c6CTVeXLhg36hcUuL8RuU4xXDxm3fesVaJpaQAy5cP7c/Py5NhAe3kSdm5TImnoMBeyPLYMeDSJffa4zSlZKGCnlfUQ2IDOTo8ATFc/KS5GfjgA+v1uHFDf1CRacocT/SGyq1b4284hK4vJcU+NHb1anydVhoKAbt3W3+fESO4K/8GMFz85JNPpNSGtnRp7DZO9sYwZBlq9NDAtm3xv4mOrqWHSdPT5bVSwK5d8TM0dvGivVBlSYl9SJj6xHDpi1LWl9v0iZPRtcTuvtudp6j8fFmWrB07Bpw6NfTtIPeNG2cfGjt6ND6GxnTF5+ghsSVL5L6jfmG49EQpWZW1axfw2mveCJdg0F5LrKBAJlTdCJekJOAv/sL62S0twPvvD307yH0pKfY5uKtXZR+WF+6ZwQiF5P7Xf4+MDKmpxiGxfmO4dKcU8OyzwAMPAN/4BvDzn7t/2p4+cfLkSetaWZm9vtNQ0k9x0T8/uldFiUMPk0avGtuxw/8bKuvq7Pfb9OlSrJL6jeHSnWFIeZWaGvmwPH9eNlG5/SS2dat1hkr3SXU3FBbaTyU8fFhqTFHiGTtWyqFox4/LfeNXell19JDYzTdzSOwGMVx6sny57HwH5AnM7ZMX29slXLT8fHttJzckJ1tDYxkZUhJD72KmxJKcDNxyi/V+bG2VD2e3H8gGqqMD2LnTej1yZOyOs4hjDJeezJhhP6th504ZS3ZLTY191cqiRe5v5DIM4K67gH/9V+BPfwL+7//sh0hR4jAMmY+IXrm4c6c/VxAqJYtToldlzpkj+7vohjBcepKVJTvftdpa99bvKyUT+bqL7oUhMW3sWGDVKhmPHj7c7daQm0aPlrIo2qlT8uW33otSwPbt9vL6t97qjfvNZ/gv1pvly63zsbu65HAsN3Sv35WbK5PpXuiiG4Y32kHuM03g9tutD+HOTtnw67dwaW6WIT1tzJjYH2cRpxguPTEM2fk+bpx17YMP5I031M6etR/KNX8+N3KR9+h7JnpF1e7d7g4n3yilpPhm9ImTpaVDV7svzjBcejNypOyA1z79dOhPXlRK9o/oG9QwZEiMq1bIizIz7Xte6uu9sdKyv8JhqTYRicjr1FTpjbHXMiAMl97oD3JdpK6zU4anhvJG6T4cpwOPb3byIsMAli0Dhg2T15GIrHL0y56Xc+dkblUrLpbqA7zfBoTh0htdUtzNkxc/+0y66Vr3oToiLzEMYMoU+56XI0fkfez13otSMvTd0iKvuwcl3TCGS19GjZLJc20oT17Ub/buQ2J6kQGRF6WmAnfeaT3tB4PyUOZ13UsY5eTIxkn2WgaM4dKX7icvtrcP3cmLXV1ynLEOMr08mm928jJdDiZ6X8j27VL1wqv0UcZ1dda10lJ5uKQBY7j0xTBkw2L0CpiqqqEZGqutlYOKtDlzgAkTYv9ziQYrN9e+GObCBW/v2A+FgLfeso4KSE2VrQjc2zIo/Ne7nrw86R5rJ0/KqrFY3ih6lVhTk7zWQ2K6JA2RlxmGfDinpclrpaQX7sUd+0rJPX3kiHVt+nQ5u4WjBIPCcLke0wTKy+1DY++8E9uf2dkJvPmmfUhs2TK+2ckfDAOYPNleDujECfkA91rvJRKRXkv3orB8kBs0hsv16KGx6FVjVVWx3VB55ox9SGz+fGDixNj9PCKnJSfLQ5nek9XVBbzxhreWJSsl1Zt37bKujR0r8y18kBs0hkt/5OVJ1VetpiZ2tcaUkr0B0avEVqzgKjHyF71jP/qUyo8+8la9MaVkFCJ6+Hn5cu7IdwjDpT8MA7jnHusprLMTqKyMzU3S0SFPeFpODofEyJ/S0uTBSL9329qALVusHfBuu3zZvvozJ8e+jJoGheHSH3poLHpoqqoKuHLF2Z+jlOyjqa62rpWWcuMk+ZNhSI9//Hjr2q5dMuzrdu9FKal+cfmyde2OO9w/yiKOMFz6a+RIearRamtjs7zyjTdk4xkgk4v33mstJiDym6wse++lpQXYvNn93ktDg33RzMiR0k4uP3YM/yX7yzCAlSutchDhMPDaa9baeCc0NcnKFa2w0H7CH5HfGIY8lEVX8v7gAykE61bvJRKRYKmvt67dfjtHCBzGcOkvwwBmzpTjfLWdO6UH4wSlgH377CfgLVvGbjr5X04O8OUv23svr77q7INZfykFXLx47VL/e+9lr8Vh/Ne8Eenp0nvRGhvtb9LBCIeBV16RJZuArLO//372Wsj/9Cqs6J7Bjh1DV6cvWiQCvP46cOmSde2uu2ReiPeaoxguN0IvC87JkddKAZs2OVMO5uxZGS7Qpk+X/S18w1M8yM6WhyXdO2hvB9avH9pd+0rJUujoTdC5ucB997HXEgP8F71REyfKmdra0aPAhx8O7glMKVmi2dgorw1DbsQRIwbXViKvMAxZjTV1qnXt4EHpwQxV76WrSwJNb4A2DBmuKyjgQ1wMMFxuVFISUFFhbWrs7AT+8IfBjR83NcmQmDZqlH2MmigeZGQAX/2qde+EQsDvfy8rt2IdMErJMug9e6xrEybIfcZeS0zwX/VGGYYc5Ro9sb9tmxS/G8gNohTw3nuy619bvty+N4AoHhiG7NsqLbWu1dYCf/xjbJcmKyUB9rvfWeVnkpOBhx+W4TqKCYbLQGRkSO9F9yyamqS7PZBwaW8HXnzR6vkMHw489BCfpig+paQAX/ua/UP9rbeA/ftj13sJhYCXX7av7Fy0iMv8Y4yfYANhGLJ0MXr1y6ZNcpzrjdBd9X37rGtlZcC8eXzTU3zSFZMfeMB6j7e3A889Jyu4nA4YpWReJ3oSPzsb+Nu/ZeXjGGO4DNTo0XKDaPX18nR0I937jg7gN7+ReRtAnur+5m94bjfFN115Irok/9mzwG9/6+zqMaWk1Mxvf2vdY6YpIwOTJ/MBLsYYLgNlmjJmO3q0dW39+v7vPNYHgu3YYV2bN49FKikxpKcD3/qWtawfkKX4r7zizOZKpeRo5V/8wr4Tv7RUjgLgsHPM8V94MCZOlLkX7dIl4Fe/6t/N0dwMPPusvdfyrW/JTUcU7wwDKC4GvvENa/VYOCwrL6uqBjfBr5RUYP6f/7EXgS0sBL79bZnXpJhjuAyGacrYbfTcy6uvSlmYvnovkYgswfzoI+taWZmsEmOvhRKFacp7/p577PMvv/oVsH37wAJGB8uvfy0jA/o+HDEC+Id/kMPAeI8NCYbLYI0fLz0O3c1ubQX+7d+klHdPAaMUcPgw8MtfWj2c9HTgu9+1zhwnShTJyfKAVlZmXWtpAZ55Bnj7bVnp1d9JfqXkkL1nn7WXZUpNBVatAhYsYLAMIYbLYBmGzL0sXGhdO3wY+Pd/l6ew6BtDKRn/ffJJe22jBx8ElizhG58Sj2FIr+If/xGYNcu63toqIfHCCxI21wuYSETmO3/2MxlW078+OVkm8O+5h0dXDDGGixOysoDHHrPW7isFbNgA/Pzn1o2hz+v+8Y+lXIw2bRqwerV1yiVRojEMqUrxox8Bs2db1zs75T564glZst/Wdu3DWiQiowTr1wOPP24/fjw5WSoCPPQQg8UF/ERzgj6p8p/+CXjqKalhFArJEsjqalmy3NoKvPSS7OTXsrOlF8PaRpToDEOOl/jxj6XHsmuX9VB2/LgMNU+cKCsqJ0+W5fqBgNT2+/hjqcsXHTzp6bKsf+VKa8EADSmGi1P05P65c7J3JRSSp6pdu+SruxEjgH/+ZymCyWAhkvsgNxf44Q+lJMymTdJbAeR+OnVKvvSv7W2obPx4WRW2aBF7LC5iuDhp2DDp2g8fLite2tt7/nWjRkmwPPgg19sTRTMMq9cxf7709qurrZpgWk/BkpEhZ7NUVAB5eXxocxnDxWlpacD3vy83xi9/KZP77e3yRs/IkHpG3/uejC0zWIh6lpQkO/inTQMOHZJJ+upqqeMXHTTDhsmw8uLFEiwTJsh9xWBxHcMlFlJSgLvvBm67Tbrxn30mk4tFRfLmT07mm5/oegxDRgFKS2WIq7lZVls2Nsq8Znq6zNPk51sbI3lfeQbDJYoZiaB2/3506XFepyQlSTe+psZeWj/OhEMhhAMB8Pb2NzMSQf3Rowj3NqzrBSkpEjDnzsmXh4TDYYSbmxP+PmC4fM4wDKhp04CqKpyvqnK7Ob6VmpmJlJEj3W4GDZBpmlCTJsHYuxeX9+51uzm+lZKejqSMDLeb4SpDqaE6Y9TblFIIhULgP8fgJScnw+R8ki8ppRAOh3kfOCApKSmh7wOGCxEROS5xY5WIiGKG4UJERI5juBARkeMYLkRE5DiGi19EIlIldjAn9BH5XTgsRWCdOAqZYorh4hfV1cDUqfZjW4kSzenTcn7S6dNut4Sug+FCRESOY7gQEZHjGC5EROQ4hgsRETmO4UJERI5juBARkeMYLkRE5DiGCxEROY7hQkREjmO4EBGR4xguRETkOIYLERE5juFCRESOY7gQEZHjGC4+oJTClStXEAqHceXKFSil3G4S0ZBTSqGxsRHBtjY0NjbyPvA4houHBQIBrF27FiUlJbh92TKcO3cOty9bhpKSEqxduxaBQMDtJhLFXPR9UFpWhqqqKpSWlfE+8DhDMf49qbKyEhUVFQgGgwCAmUphC4B7AVQbBgAgPT0dGzZsQHl5uXsNJYqh7vfBFKXwNIDVAD7lfeBp7Ll4UGVlJVauXIm2tjYopa7p/utrbW1tWLlyJSorK11qKVHs8D7wN/ZcPCYQCGD8+PFoa2tDJBL54vos4Iuey5GoX2+aJtLS0lBbW4vs7OyhbSxRjPR2HxQBX/RcTkX9et4H3sOei8c8//zzCAaDthuqL5FIBMFgEOvWrYtxy4iGDu8D/2O4eIhSCs8888yAfu/TTz/N1TMUF3gfxAeGi4c0NDSgpqbmhm8OpRRqamrQ2NgYo5YRDR3eB/GB4eIhLS0tg/r9zc3NDrWEyD28D+IDw8VDMjIyev1eDWQyv6aP35+Zmel0k4iGXF/3QS1kMr+2j9/P+8AbGC4eMmrUKBQXF8P4fP1+tHbIKrH2Hn6fYRgoLi5Gbm5urJtIFHN93QedkFVinT38Pt4H3sJw8RDDMPDoo48O6PeuXr26x5uRyG94H8QH7nPxmN7W9/eG6/spHvE+8D/2XDwmOzsbGzZsgGEYMM2+/+8xTROGYWDjxo28oSiu8D7wP4aLB5WXl2Pz5s1IS0uDYRjXdPP1tbS0NGzZsgUrVqxwqaVEscP7wN8YLh5VXl6O2tparFmzBkVFRbbvFRUVYc2aNairq+MNRXGN94F/cc7FB/Q5Fs3NzcjMzERubi4nLSnh8D7wF4YLERE5jsNiRETkOIYLERE5juFCRESOY7gQEZHjGC5EROQ4hgsRETmO4UJERI5juBARkeMYLkRE5DiGCxEROY7hQkREjmO4EBGR4xguRETkOIYLERE57v8Blv6jMrwdkEQAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 500x400 with 4 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "model.train(dataset, opt=\"LBFGS\", steps=20);\n",
    "model.plot()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "4eb022df",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/latex": [
       "$\\displaystyle 1.0 e^{1.0 x_{2}^{2} + 1.0 \\sin{\\left(3.14 x_{1} \\right)}}$"
      ],
      "text/plain": [
       "1.0*exp(1.0*x_2**2 + 1.0*sin(3.14*x_1))"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# obtaining symbolic formula\n",
    "formula, variables = model.symbolic_formula()\n",
    "formula[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "a8e794ba",
   "metadata": {
    "code_folding": []
   },
   "outputs": [
    {
     "data": {
      "text/latex": [
       "$\\displaystyle 1.0 e^{1.0 y^{2} + 1.0 \\sin{\\left(3.14 \\alpha \\right)}}$"
      ],
      "text/plain": [
       "1.0*exp(1.0*y**2 + 1.0*sin(3.14*\\alpha))"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# if you want to rename your variables, you could use the \"var\" argument\n",
    "formula, variables = model.symbolic_formula(var=['\\\\alpha','y'])\n",
    "formula[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "a1349079",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/latex": [
       "$\\displaystyle 3.14013671875 e^{1.0 y^{2} + 1.0 \\sin{\\left(3.14 \\alpha \\right)}} \\cos{\\left(3.14 \\alpha \\right)}$"
      ],
      "text/plain": [
       "3.14013671875*exp(1.0*y**2 + 1.0*sin(3.14*\\alpha))*cos(3.14*\\alpha)"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# one can even postprocess the formula (e.g., taking derivatives)\n",
    "from sympy import *\n",
    "diff(formula[0], variables[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4474d38d",
   "metadata": {},
   "source": [
    "When do we know the formula we guessed is wrong (not exact)? If the data is clean (no noise), we should see the training loss does not reach machine precision"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "22529272",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "r2 is 0.999993562134913\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "tensor(1.0000)"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# let's replace (0,1,0) with cosh\n",
    "model.fix_symbolic(0,1,0,'cosh')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "404dbd96",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "train loss: 1.26e-03 | test loss: 1.28e-03 | reg: 2.93e+00 : 100%|██| 20/20 [00:03<00:00,  6.54it/s]\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZcAAAFICAYAAACcDrP3AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjYuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8o6BhiAAAACXBIWXMAAA9hAAAPYQGoP6dpAAArX0lEQVR4nO3deXAc5Z0+8KdblyVL1mXZsnxLlg3yhQ/JgMEEAhbEBAgKbJEDL0slW1tZvFlqa7ObJVl+qWxStalUbEjthmzYBCcLAWJzxQYREh9cvrDxbWPLB4zwKWlkSaNrZt7fH1+/9LQsybLUo+6eeT5VKtMtyWqL6Xn6vb6voZRSICIicpDp9gUQEVHiYbgQEZHjGC5EROQ4hgsRETmO4UJERI5juBARkeMYLkRE5DiGCxEROY7hQkREjmO4EBGR4xguRETkOIYLERE5juFCRESOY7gQEZHjGC5EROS4VLcvgMgPlFJoaGhAa2srsrOzUVhYCMMw3L4sIs9iy4WoH8FgEKtWrUJ5eTmKioowdepUFBUVoby8HKtWrUIwGHT7Eok8yeBOlES9q62tRU1NDUKhEABpvWi61ZKVlYU1a9agurralWsk8iqGC1EvamtrsWzZMiilEI1G+/w60zRhGAbWrVvHgCGKwXAh6iEYDGLChAlob2/vN1g00zSRmZmJQCCAvLy8+F8gkQ9wzIWoh2eeeQahUGhAwQIA0WgUoVAIq1evjvOVEfkHWy5EMZRSKC8vx7Fjx3Alt4ZhGCgtLcWRI0c4i4wIDBcim/Pnz6OoqKjXz6UBGAfgFIDufr6/sLAwTldH5B/sFiOK0dra2ufnxgH494t/9qWlpcXpSyLyJYYLUYzs7OwhfX9OTo5DV0LkbwwXohiFhYUoKyu74nETwzBQVlaGgoKCOF0Zkb8wXIhiGIaBRx55ZFDfu2LFCg7mE13EcCHqYfny5cjKyoJpDuz2ME0TWVlZePDBB+N8ZUT+wXAh6iEvLw9r1qyBYRiXDRi9Qn/t2rVcQEkUg+FC1Ivq6mqsW7cOmZmZMAzjku4ufS4zMxPr16/H0qVLXbpSIm9iuBD1obq6GoFAACtXrkRpaantc6WlpVi5ciXq6+sZLES94CJKogFQSiG4ezdSf/QjhL/7XeTNncvBe6J+cLMwogEwDAP5+flAdjaQnw8wWIj6xW4xIiJyHMOFiIgcx3AhIiLHMVyIiMhxDBciInIcw4WIiBzHcCEiIscxXIiIyHEMFyIichzDhYiIHMdwISIixzFciIjIcQwXIiJyHEvuEw1Uayuwaxcwb55URyaiPjFciAYqEgHa24HMTCAlxe2rIfI0hgsRETmOYy5EROQ47kRJnhGNRnHq6FFEOjrcvhTfK5g8Gdm5uW5fBiUxhgt5RjgcRsNzz6G7qwvmiBFuX45vpZ45Azz8MLLnzXP7UiiJMVzIO5RCxDBQ9MADKJkxw+2r8YdQCKivByZMADIzEY5EcPBnPwMHUsltDBfyHNM0kZqW5vZleJ9SwOHDwNNPAwUFwKxZwM03wzAMt6+MiOFC5FtKAR9+CHR3A2fOAE1NQFWV21dFBICzxYj8q7lZWi5aSYl8EHkAw4XIj5QCjhyRgNFmzwbS0927JqIYDBciP1JKStFEo3KclgbMmQNwvIU8guFC5EfNzcBHH1nH48fLjDEij2C4EPmNniUWDFrn5sxhlxh5CsOFyG+iUWDHDgkZAMjIAObOZZcYeQrDhchvGhpkMF+bOFG6xYg8hOFC5CdKAfv2AS0t1rn582VAn8hDGC5EfhIOA9u3W8dZWZwlRp7EcCHyk/p64MQJ63jaNGDMGNcuh6gvDBciv1BKBvI7O+XYMIDKSu6KSZ7EcCHyi7Y2WTip5ecDFRXsEiNPYrgQ+YFe23LmjHVuzhyAG4KRRzFciPwgEgG2bLGXe6mqYquFPIvhQuQHZ88Chw5ZxxMnAlOnMlzIsxguRF6nFLBtm4y5aIsWsdwLeRrDhcjr2trsa1vy8oB589hqIU9juBB5mV6RHzuQP3eubGtM5GEMFyIvC4eBd96xBvLT04HrrmOrhTyP4ULkVUoBx44BR49a56ZN40A++QLDhcirolFg82agq0uOTRO44QYgNdXd6yIaAIYLkRcpBZw6BezZY50rKQFmz2arhXyB4ULkRUrJWEvs9OPrr5cqyEQ+wHAh8qKGBlnbohUWckU++QrDhchrlALefRcIBq1z114r61uIfILhQuQ1TU0SLtqoUTKQb/J2Jf/gq5XIS/RYS0ODda6qCigqcu+aiAaB4ULkJY2NwNtvW8c5OcDnPsexFvIdhguRV+h1LT1bLcXFDBfyHYYLkRcoBZw7d2mr5eabGSzkSwwXIi9QCnjrLfsMscWL2Woh32K4ELlNKeDjj4H337fO5eez1UK+xnAhcls4DKxfD4RC1rmbbwZGj2a4kG8xXIjcpPdr2b3bOldSAtx4I4OFfI3hQuSmtjbgj38Eurvl2DSBO+6QhZNEPsZwIXJLNAps2gScOGGdu+oqYOFCtlrI9xguRG5QCqivB/70J/lvAMjMBL74RdltksjnGC5EbujqAl5+GbhwwTq3eLHsNMlWCyUAhgvRcFMK2LLFvhFYcTFw++0sTkkJg69kouGkd5h89VUgEpFzqanAXXfJ2ha2WihBMFyIhlNnJ/CHP0hZfa2yEliwgMFCCYXhQjRcolFg40Z7d9iYMcDdd0vrhSiBMFyIhoNSwEcfyZqWaFTOpaUB994re7Ww1UIJhuFCFG9KSTfYs8/aS7zccAMwfz6DhRISw4Uo3jo7gd//HggErHNTp8ogfkqKe9dFFEcMF6J4ikSAN94Adu60zo0aBXz1q/InWy2UoBguRPESjcp6ljfesI+z1NQAU6YwWCihMVyI4kEp4OBB4PnnZTU+IGFyyy3AdddxsSQlPL7CiZymlBSj/M1vgNZW6/w110jtMI6zUBJguBA5Sa/Af/ppoKHBOj91qoyzZGayO4ySAsOFyClKAWfOAL/8JfDpp9b5sWOBhx5ieRdKKgwXIifoYHnqKeDjj63z+fnA3/wNMH48g4WSCmtOEA2V3pvlf/4H+OQT6/yoURIsLKNPSYjhQjQUSgHHjskYy+nT1vmcHOkKq6hgsFBSYrgQDVY0CuzdC6xeba9ynJsrwTJrFoOFkhbDhehKKQWEw8DmzcDatUB7u/W5ggLpCrv6agYLJTWGC9GVUApoa5MtijdvlpDRxo0DHn5Yph0zWCjJMVyIBkopGbB/9lkpnx9r+nRg+XLZrpjBQsRwIbospYDubuC994BXXgGam63PmSawaBFw//0sREkUg+FC1B+lZBbYSy8Bu3ZZ+94Dstr+C18Abr0VSE9nsBDFYLgQ9UYpoKNDWivr19tngwHS/fXAA8DMmSxCSdQLhgtRLKWkdXL4MPDaa8DRo1a5fECKTlZWyvbEhYVsrRD1geFCBEio6AH7118HPvzQKpWvFRYCd98tYyypqQwWon4wXCi5KSUtk0AA+POfgQ8+sK9bAWSDr6oqKZdfVMRQIRoAhgslJ70Q8vhxYONGYPfuS0PFNGXHyLvukjIuKSkMFqIBYrhQ8lBK/mxtBfbtA955R8ZUursv/driYuC224BrrwVGjGCoEF0hhgslPqVk/OSTT4AdO2RK8fnzVtjEGjMGuOkm4PrruW6FaAgYLpSYdKCcPg3s2SMD9IFA760U05SWypIlMljPUCEaMoYLJQY926utTTbrOnAA2L9fwqXnrC8tLQ0oLZVQmT0bGDmSoULkEIYL+ZMOk1BIdoCsq5O1KSdPAsGgfW1KLMOQ3SHnzgWuuw6YPJnTionigOFC3qeDpLMTuHABOHUKOHFCZnrV18u52OrEvcnJkeKSCxZIOfycHAkUhgpRXDBcyJtaW4Fz56Rbq75ePs6ckaKRHR29D8bHMk3ZtGvaNGmlzJgB5OXJeQYKUdwxXMib3ngDePPNy7dIYmVlAWPHSgulogKYNIktFCKXMFzImwoL+w8W05QB+LFjZaFjebmMn+TnyxgKwEAhchHDhbypuFhWxEciEhZZWbKFcEmJhMikSbImJTubA/JEHsRwIW8qKbFqeRUXS4tk5Ei2Soh8guFCnmIqhYa6OkQ6O4GJE+VkMCgfdFnRaBSR1lYwesltDBfyDMM0oUpKYOzZg6Y9e9y+HN9Ky8xESlaW25dBSc5Q6nJzOomGh1IKkdhthGnQTNOEyR0yyUUMFyIichwfbYiIyHEMFyIichzDhYiIHMdwISIixzFciAYqEpGCmpzRRnRZDBeigQoEgBUr5E8i6hfDhYiIHMdwISIixzFciIjIcQwXIiJyHMOFiIgcx3AhIiLHMVyIiMhxDBciInIcw4WIiBzHcCEiIscxXIiIyHEMFyIichzDhYiIHMdwISIixzFciAZAKYXGxka0tLaisbERSim3L4nI0xguRP0IBoNYtWoVysvLMW/+fLz44ouYN38+ysvLsWrVKgSDQbcvkciTDMVHMKJe1dbWoqamBqFQCAAwUSn8O4D/B+ATwwAAZGVlYc2aNaiurnbvQok8iC0Xol7U1tZi2bJlaG9vh1Lqkm4wfa69vR3Lli1DbW2tS1dK5E0MF6IegsEgampqoJRCNBrt92uj0SiUUqipqWEXGVEMhgtRD8888wxCodBlg0WLRqMIhUJYvXp1nK+MyD8YLkQxlFJ48sknB/W9TzzxBGeREV3EcCGK0dDQgLq6us9CwgBQDuBrANYCWAhgLoCsHt+nlEJdXR0aGxuH83KJPCvV7Qsg8pLWYBBzAFRd/KgEkAsgDCACYBSAbwP4WwDHAeyL+WgG0NLSgsLCwuG/cCKPYbhQcuvoAHbvBrZtA7ZuxYQdO/AygA4AHwD4NYBtAD6EhMs4AFEAVwGYBWnJ3HXxr6oHUPTss8CiRcCsWcCYMcDFKctEyYbrXCi5tLQAH3wAbN8ugbJ7N9DdDYwaBSxcCFVVhbt/+EO8EQige4B/ZQEkaD43ejQe//KXYZw8KZ8YPVpCRn9MnMiwoaTBlgsltsZGCZFt2yRQDhwAolGgqAioqgIee0z+nD4dME0YAD7f0YE//uM/AgN87moE8LZhoOZ734OxYoUE2IEDwL598rF5s/zMUaMkZGbOBGbPBqZOBVJS4vrPJ3ILWy6UWD791AqTbduAujo5P3GihIj+mDy5z1ZEMBjEhAkT0N7ePqDpyKZpIjMzE4FAAHl5eZd+QXs7cOiQBM3+/fLfXV1AZiZQUWG1bKZPB9LShvCPJ/IOhgv5l1LA8eOfjZdg+3agvl4+V15uD5Pi4iv6q/UK/cstpDRNE4ZhYP369Vi6dOnA/vLubuDIEatls3+/BFBaGjBjhrRqZs4Err5aAojIhxgu5B+RiDz16/GSbduAhgbANOXNWAdJZSWQnz/kH9eztljsrWLE1BZbu3btwIOlN9GohKQOm337gOZm+XdNm2Z1o1VUSNcakQ8wXMi7uruBPXus8ZLt24HWViA9HZg7V2ZlVVUB8+cDI0fG5RKCwSBWr16NJ554AnW6iw1AWVkZVqxYgeXLlyM3N9fZH6oUEAhYrZq9e4Fz5+Rzkyfbx2047Zk8iuFC3hEKAbt2Wa2SXbuAzk4gKwtYsECCZNEiYM4cICNjWC/ts/1cWlqQk5ODgoKCz1ovw+LsWXvLJhCQ88XFVjfarFnAuHGckUaewHAh9zQ3Azt2WGGyd690feXnS9dWZaWESUUFZ1X1FAxKq0aHzbFj0uIpKLBaNbNm9TtxgSieGC40fM6etY+XHD4sb4jFxfbxkmnTZLyBBq6tTaY/6260I0eAcBjIzpaw0YFTVgakcgUCxR/DheJDKeCTT6ww2boV0IsLp0yRFkllpQTKhAl8unZaZ6eEt27ZHDwo5zIyZBaanv581VUyhkXkMIYLOSMaBY4etQbft20DTp+W0JgxwxovWbhQyqLQ8AqHZc1P7LhNW5u0YsrLrXGbioq4TY6g5MJwocGJRKQbZutWCZIdO4CmJhkbmTPHGi9ZsABwejYVDZ1S0pLct0+60fbtk/9/hgGUllotm5kzgd4WhhJdBsOFBqaz0yrwuG2b1OcKhaSbZf58q4tr3jyZ3UX+ohRw6pS9ZXP6tHxuwgR7jTS2PGkAGC7Uu7Y2CRA9XqILPObkSNeWHoCfPZslSxLV+fPWWpt9+6wxs6Ii+/RnjplRLxguJJqa7OMl+/fLOEphob2MyowZnBacrC5ckK7QvXvl9XH0qLxGcnPt3WilpZztRwyXpHXqlL1a8JEjcn78eGvwvbJSKvfyqZR6094us9B0N9rhw9K6zcqyF+QsL2frNgkxXJKBUsCJE/ZqwXqF97Rp1nhJVRVQUuLqpZKPdXUBH31kdaMdOCABlJ4uLd7Y6c8syJnwGC6JKBKRmzw2TM6fl66KigprsWJlJWtTUfxEIlKQU89G279futZMU1ozuhtt5kwZy6OEwnBJBN3dcgPr8ZLt22XDqrQ0KfCoWyULFsiKbSI36IW1sTPSzp+Xz02ZYh+34UOP7zFc/Ki93SrwuH07sHOn7AWflSXTgnWYzJ0LjBjh9tUS9U4pe0HOvXtlszdACnDGTn8uLubYn88wXPxi40ZgyxYJlD17pMshN9c+XjJzJutGkb81Ndk3UTt+XEKosNCa+nzddWzZ+ADDxS/OnJFAyciwPlJT+TRHiS0clj18Wlrko61NxmtYNcDzGC5+EY1KkDBMKJlFInIPcB2N5zFciIjIcYx/IiJyHEd/L4pEIqjbuRPh1la3L8X3xs2ahfyiIrcvgwYhEokgcOgQIqGQ25fie6OnTcOo/Hy3L8M1DJeLuru7Uf+DH6CrowMp3M9iUJRSSD9xAvjpT5F/661uXw4NQjgcxtmnnkJ3VxdMrqIftLT6euDRRzHq2mvdvhTXMFw0pRA2TUz83vcwbdEit68mviIRWReTkuLoOpjucBjvP/QQOIznY0ohYpoo/sY3MHHOHHmtNDfLFOGpUzmQ3pvubvnIzAQMA+FwGLu//30k+13AcOkhJSUF6RkZbl+G85SS1dAvvwy89Zbs1ZGeLlve3nuvrB1ITx/abDTOZEsYpmkibcMGWV918qQ8hPz0p5wC3JNSsiXFb38rdfpmzwYqK2HwXmC4JIVoFHj3XeCHP5Tqx7Eti6NHgdpa4M47gX/5F2D0aIYEyWugvl728QFkTVUgwHDpzd698rsJBID335dSNsTZYgkvGgVefx1YsUKKWfbWZdXVBaxdCzzyiJTiZ7cWAbIiXneDhcNSXp+vDbvOTuDQIeu4qEhK1RDDJaEpBbz3HvD97wPBoHV+1Chr46/YcjFbtwL/9m9SuZZvIjRlirxWNL2BHFnOn5cHMq2sjNt8X8RwSVRKSbfG448DjY3W+blzgdWrgd/9DnjxReCxx+zlzjdtAn7+cxnIpeSWnw9MnGgdnzghDx4klJJu5dhp2zNnslv5IoZLourqkgHYujrr3Ny5Ehxz5kg5/pwc4OtfB37wA0BPv1YK+L//AzZvZusl2aWmyoQPralJSuaTZd8+6z7JyJDeAALAcElMSgEbNgDr11vniouB//gP2cY49snKNGUw/5vftPrX29uBn/1M3kwoeRmGbC6XkiLH4bDsLsmHDtHRIVs7a0VFslUAAWC4JKZgUFooXV1ynJoqA/oVFb032VNTgYcekunI2v79wPPP840k2U2deum4C7tMxdmz9vGW6dO5fXMMhkuiUQp45RWZ2aPdcANwzz399wVnZwOPPmqNvygl4zKBAAMmmeXl2afWnjhhnxySrJSS2ZcdHXJsGLLXDMdbPsNwSTTnz8uCLj2rJzsb+Na3Lr8S3zBkTOZLX7LOffop8PvfM1ySWUqKDFJrzc0SMMlOKVnfou+NESNkvIXh8hmGSyJRCli3Tnbv026/HbjmmoG96E0TWL5c+o61tWutrWcp+ehxFz1lPRKxv6kmq1DIPt4ybhwwdqx71+NBDJdE0txsb2nk5AAPPmgNyF6OYUgXSGzr5cwZ4KWX+GaSzCZPBgoKrOP9+6WWVjKrr5cxF+2qqxyt05cIGC6JQs8QO3rUOvf5z8tU0itpqpsm8Fd/Ze1RrpSES0ODs9dL/pGTI4sDtUBAul+TlVIypqknzBiGTO8nG4ZLoujslEWReiZPZibwwAMDb7XEmjwZqK62jk+eBP7yF7ZekpVp2t88W1svrVGXTKJRYM8e6zg7W4pWcrzFhuGSCJSSF/uuXda5hQtlgH4wL3jTBO67z1pYGY0Ca9ZIgFHyMQxpAetq4UpJQctkDZfmZvvi5EmTpOAr2TBcEoFSUkpfT4tMSQHuv19K6A+GfjOpqrLO7dnDgdxkNn68vSDjwYOy2DYZHT9uX2A8e7a9Rh8BYLgkhtOnZbxFmzoVWLx4aM309HSgpsbqVuvokPUzDJfkNGKEvRTM6dMyqJ1sdKtNdz+npkqXIbvELsFw8TulZEOnM2esc3fcMfR9NwwDuP56+wK6DRvsM2QoeRiGfUp7V5e9rlay0P9urbBQxijpEgwXv+vqAl57zT79+I47nHmSysuzD+yfPg28/XbyvaGQvJ7Ky+0VtD/8MPlKwZw5Yy/e2fN3Qp9huPiZLkGxd691bsEC+7TRoTAMCarsbDmORmWRZrKvcUhWhYXS5aodO5ZcxU2VksKdeqzJMIB586yCr2TD34rfvfkm0NYm/60rHKelOff3l5fbp6Hu3GmvAEDJIzVVZiBqzc19726aiKJRmZGp/72ZmX0XgyWGi6+1tgJ//rN1XFw89IH8ntLTgS98wfo7W1q45iVZGYbMjNKzEHu+2Sa6CxfsJV8mTmTJl34wXPxKF86LnW9//fX2umBOMAxgyRJ7+Y8337SmPVNymTTJ/oa6f39yTElWSu612F1d58wZ/HT/JMBw8SulgNpa+54td9wRn/7fceOARYus48OH5SNZnljJkpUlpeW106eBjz9273qG065d9inI8+a5ez0ex3Dxq+ZmmbmlTZwoL/Z49P+mpEh15didKt96y/mfQ97XcxC7q0tmjSX6g0ZHh73kS1GRTNPneEufGC5+pJTc0IGAde7GG4Hc3Pj8PMOQ1fqxK7Q3bLAmElDyMAzZtyR2HdWuXYk/gzAQsC8arajgFOTLYLj4kVIy7qFv6LQ0YOnS+D5FjR5t7xqrq7PvdknJIz9ftvTVTpyQ7rFEpR/mdG09w5Ap/2y19Ivh4kfBIPD++9bxlCkyiyeeL3bTlADT5WA6OzlrLFmZpv3NNRRK7Lpz4TDwwQfWcW7ulW9lkYQYLn6jn6Jim+hLlsS/ia6f1mK7xjZtYtdYMtL7xWdlybFSwI4dibta/9QpWTCqTZ9unz1JvWK4+I1SsrYlHJbjtDTZFGw4FBYC115rHR87xlljyWrcOKC01Dr+6KPE3FBOT/kPheTYMIDKysHtk5RkGC5+09xs7xKbPBmYOXN4muiGAdx6q71rLLYaMyWP1FRpyWrNzbLmJdEeNCIRYPt269+VlRX/LugEwXDxE6WkImvsLLHFi4dv1ophAPPn2xfRbd5sPdVR8tCvBb1vvFLA1q2yaj+RnDsnrTKttNTeNUx9Yrj4zYYN9llit9wyvD+/sNC+idjRo7LlLSWf8ePtWzIcOJBYhSx1l9iFC9a5ykpuDDZADBc/aWkB3n3XOh4/fvib6KYpYzyxCyo3b0687hC6vIwM2U5ba2pKrK6xSATYssVeqHL+fHaJDRDDxU8OHpQ1Bdq118Zv4WRf9Kyx2BpmmzZZawAoeejXQkaGHCsl44GJ0jV2/jxw6JB1XFoqD3Q0IAyX/ihlfbhNKXkTj60l9vnPu/MUVVQkT3DaRx+xDH+ymjTJvhPj/v324o5+pbczju0SW7TI2e0sEhzDpTdKyYtqyxb7Lo9uCoWk+0kbO1b21nAjXFJSZKxH/+zWVnudM0oeGRkyDqE1NUkNLi/cM0MRiUgrLHaWGFflXxGGS0/RKPCLXwD33AMsXw785Cfuz99XSgbNY8vrV1W5t5DLMOQpLvbnb9xotaooeei6c7Gzxt57z/8LKs+csXeJlZUBJSXuXY8PMVx6Mk2Zs3/8uMzKOn1amsduP4lt2mTtodJzUN0NxcX2XQn377fvLU7JY+JE+4LKAwdkCq9f6YoDra1ybBjAddexS+wKMVx6c/PN1iZA4bD7NbQ6OiRctKIi95voqalW11h2tlTKTYZNo+hS6eny5qu1tEgtLrcfyAaru1taX/r6s7Pdv998iOHSm6uvtg9SbtliH9gbbseO2RdyzZ/v/I6TV8owgM99Dnj8ceAPfwB++1v7JlKUPAxDpiRnZ8uxUsA77/izDL9SsvnZ0aPWuauv5sLJQWC49CYnR7YM1gIB96q+KiWD5bpApGlKCRY3u8S0khLg61+XVouejkrJadw44KqrrOMjR4CTJ/3ZennvPasL2jCAG25gLbFB8MA7lEfdcovVx9rd7V4Nra4u+8/Oz5cBVC800Q3DG9dB7ktNlQ3r9Ouho0PepP2mrU16KrTRo92blelzDJfeGIasfI9dMPXuu9KXPNxOnrRvynXNNfbaXkReYBjy2hw92jq3ZYs1KO4HSslkhNjtLObNY3n9QWK49CU3115e/sSJ4d95USl7qBmGzBJjbSPyooIC++La+np/lYOJRmXiTOx2FkuWsNUySAyXvvR8I+/qkrUcw3mjdHfLTDVNBx5f7ORFhgHcdJPVnRyJSJeuX8rBnD0rG/FpkybJeCLvt0FhuPTFMKSvtWd5+eHceVFPJNB6dtUReYlhyC6NU6da53bvBj791PutF734s7nZOnfjjVKskgaF4dKfwkJZia7V1Q3fzot6OqeeAq1bUlzIRV42YoS0XvTTfkuLvI69rr1dusT0vT1qlMwYZatl0Bgu/XFz58XubtnOOPbFfsMNfLGTtxmGdN3m5VnnNm92ZzLMQOlN+E6etM7Nm8e1LUPEcOmPLike2zW2cePwdI0FAlIAUJs9G5gwIf4/l2ioiorsxSwDAWDXLu92jUUiwJ/+ZA3k64rjXlhL5mP87V1Ob11jhw7F90bRXWK6/1d3iemSNERepmvf6ddrNAq8+aY3V+wrJS2W3butc2VlQEUFewmGiOFyOaYJLF1qdY11dEh3VTx1dcmTVGyXWOwCNSIvMwygvFzKpmgHD0oJI6+1XpQC3nrL6o3QD3K6yjMNGsPlcnTXWGz/64YN8e1D/vhje5fY3LlSeZbIL9LTgepqq2upsxN44w3vTUs+d84+4WDMGCnCyQe5IWO4DERhob3W2PHjMgAYj6cwpWRcJ3aW2G23cZYY+YthyILK2Iei7dvlwckrrRel5EGxqck6d9NNUmKJhozhMhCGIU9hsQsqa2vjc5N0dkr/tJafzy4x8qfsbOlSjt2x1Eutl6Ym6RLT93FurnSJkSMYLgPR21PYpk32Jx4nKCWTBQ4csM4tXMiFk+RPhiEPRrGzLd95xxuLKpWSe/j0aevc4sVS3ZkPco5guAxUbq5sIqYFAtLMd/omefNNIBSS/zZN4I47WO6b/KugQFoD+g07GPRG60Vfh75/R44Ebr+d048dxN/kQBmGvPj0viWRCPDHPzq7V/iFC9JM14qLObhI/qYXIsdWS96wQYpautV60WMtsdWPr78emDKF95qDGC4DZRgy933GDOvcli3SgnGC3rf7xAnr3JIl7u84STRUo0fbx16am4FXX3Wv9dLQAKxbZ2+13HknWy0O42/zSmRlAcuWWceNjfb1KEMRiQCvvGItNEtPlxc8n6TI70xTZjyOGWOd27xZthIe7tZLNAqsXw+cOWOdu/FGKbbJe81RDJcroZv4evMgpaRrzIlyMB9/LHu3aDNmcAc8ShyjR8uDmX49t7UBL744vKv2lZL7LHamZ24ucNddbLXEAX+jV2rSJPual4MHgZ07h/YEphTw+uvSEgLkBrzzTmmuEyUC/WA2ebJ1bseO+EyK6Us4LIEWDFrnbrtNZoHyIc5xDJcrlZICfOlL1qLG7m55wQ5lYP/CBeC116zjggJ7HzVRIhg1Cvjyl631Yt3dwHPP2d/s40Up4IMPZM8Wbdw44ItfZKslTvhbvVKGIRVfr7rKOvf228CRI4N7AlPK6n/Wbr6Z5V4o8RiGzH685hrr3IkTwMsvOzvrsielJMB+9ztZAA3IQ+J990n1DYoLhstgZGcDNTX22S9r1gwuXDo65OlN31wjRsjTHZ+mKBFlZABf+QqQkyPHSskAe7zKKQFyb734on0m5ty5MhuTvQNxw3ewwdDlYEpKrHOvvQZ88smV/T1KAVu3SnNdW7hQnuz4oqdEZBjAtGn2mZChEPD001LxwumAUQrYts2+YDInB/ja11j5OM4YLoM1Zgxwzz3W8dmzwAsvXNnc/c5O4De/sZrqaWnAV7/KfVsosaWkAHffbe9arqsDVq92dvaYUrIO7X//V+41QHoE7r1XtgTgA1xcMVwGyzSl+yp27r5ueg/k6UtvCPb++9a5OXNYpJKSQ3Y28PDDMsivbdjgXNULpWSizH//N3DqlHX+mmu4YHKY8Dc8FJMmyVOQdu4c8KtfDezmaGkBfvELe6vloYdkoSZRojMMWcv1la9Ys8fCYeDZZ+Whayir95WSscxf/cq+L9LYscA3vgFkZg7t2mlAGC5DYZrSjRU79vLqq1IWpr/WSzQqXWgffmidq6qSWWJstVCyME0Zu4wtbNneLq2NrVsHFzA6WH79a9kXSd+HmZnAN7/JNS3DiOEyVOPHS4tDN7Pb2oD//E/g/PneA0YpYP9+4KmnrBZOVhbwd3/HJypKPmlpwF//NTBvnnWupQVYtUrCIRIZ+CC/UrJnzFNPyaJkHU6pqTKAX1nJYBlGDJehMgzg/vtlvxdt3z7gJz+RJ6jYG0Mp6Tp7/HEJH+2ee6Tlwhc+JRvDkNlbjzxiH+BvaQF+/nNZm9LaevmAUQo4eRL48Y+lsrgOlpQU6bpetoxbVwwzhosTcnKAf/5nIC9PjpUCXnpJAkbfGErJwOJ3viPlYrTp04G//3ur35ko2RiGVP/+p38Crr7aOt/ZKZNkHn9cusna2y99WItG5UHthReAxx4Ddu+2viY1VYLlgQd4f7mAv3EnGAawYAHwD/8A/OhHMp0yHAaeeUZ2lbz7bukue/55+0r8vDzge9+TfVvYaqFkZhhyH3znO8B//ZdVc0zvzvrjH0tdsjlzZN+VjAxZdX/4sAzaNzbag2fECJkscNddVqkmGlYMF6eYpjwh1dfL2pVwWJ6qtm6Vj55GjgT+9V+lCCaDhUjug9GjpQXzwgsyLbmjQz4XDstamLo662v76iorKZFpzpWV7ApzEcPFSRkZwKOPyp9PP23dGD0VFgLf/a60aDjfnshiGDLB5cEHZRzzueek8ng4bP+63oJl5EiZcVlTI91sfGhzFcPFaZmZwLe/LYu1fvlLGdzv6JAX+siR0lL51reAmTMZLES9MQxpccyZI2th9uwB/vIXCZnmZnvQZGTI+pXKSqvgq2kyWDyA4RIPqakyd3/xYuD4cak5lpoqu91NnCj/zRc/Uf8MQ8ZOKitlTLOlRXaQbGqScc2sLGmhFBVZdcJ4X3kGwyWGGY2ifudOhNvbnf2LU1KkGX/smHwkqHA4jEhzM3h7+5sZjeLsoUOI9NWt6wVpaRIwn34qHx4SiUQQaWlJ+vuA4XKRYZpQ5eXAxo04tXGj25fjW+nZ2UiNrRdFvmKaJtTkyTC2b8f57dvdvhzfSsvKQkp2ttuX4SpDqeHaY9TblFIIh8Pgr2PoUlNTYXI8yZeUUohEIrwPHJCSkpLU9wHDhYiIHJe8sUpERHHDcCEiIscxXIiIyHEMFyIichzDxS+iUdm1cig79BH5XSQiRWCd2AqZ4orh4hcHDkgpjAMH3L4SIvccPw7cd5/8SZ7GcCEiIscxXIiIyHEMFyIichzDhYiIHMdwISIixzFciIjIcQwXIiJyHMOFiIgcx3AhIiLHMVyIiMhxDBciInIcw4WIiBzHcCEiIscxXIiIyHEMFx9QSqGpqQnd4TCampqglHL7koiGnVIKjY2NCLW3o7GxkfeBxzFcPCwYDGLVqlUoLy/HjUuWIBAI4MYlS1BeXo5Vq1YhGAy6fYlEcRd7H1QtWoSNGzeiatEi3gceZyjGvyfV1taipqYGoVAIAFChFF4D8EUABwwDAJCVlYU1a9agurravQsliqOe90GpUngCwAoAx3gfeBpbLh5UW1uLZcuWob29HUqpS5r/+lx7ezuWLVuG2tpal66UKH54H/gbWy4eEwwGMWHCBLS3tyMajX52fibwWctlf8zXm6aJzMxMBAIB5OXlDe/FEsVJX/dBGfBZy6Uu5ut5H3gPWy4e88wzzyAUCtluqP5Eo1GEQiGsXr06zldGNHx4H/gfw8VDlFJ48sknB/W9TzzxBGfPUELgfZAYGC4e0tDQgLq6uiu+OZRSqKurQ2NjY5yujGj48D5IDAwXD2ltbR3S97e0tDh0JUTu4X2QGBguHpKdnd3n5+ogg/l1fX4FkJOT4/QlEQ27/u6DTyCD+Z/08/28D7yB4eIhhYWFKCsrg3Fx/n6sDsgssY5evs8wDJSVlaGgoCDel0gUd/3dB12QB6yuXr6P94G3MFw8xDAMPPLII4P63hUrVvR6MxL5De+DxMB1Lh7T1/z+vnB+PyUi3gf+x5aLx+Tl5WHNmjUwDAOm2f//HtM0YRgG1q5dyxuKEgrvA/9juHhQdXU11q1bh8zMTBiGcUkzX5/LzMzE+vXrsXTpUpeulCh+eB/4G8PFo6qrqxEIBLBy5UqUlpbaPldaWoqVK1eivr6eNxQlNN4H/sUxFx/Q+1i0tLQgJycHBQUFHLSkpMP7wF8YLkRE5Dh2ixERkeMYLkRE5DiGCxEROY7hQkREjmO4EBGR4xguRETkOIYLERE5juFCRESOY7gQEZHjGC5EROQ4hgsRETmO4UJERI5juBARkeMYLkRE5Lj/D0NYrYYIe8NZAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 500x400 with 4 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# this loss is stuck at around 1e-3 RMSE, which is good, but not machine precision.\n",
    "model.train(dataset, opt=\"LBFGS\", steps=20);\n",
    "model.plot()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2318c655",
   "metadata": {},
   "source": [
    "## Part II: How hard (ill-defined) is symbolic regression, really?\n",
    "\n",
    "In part I, we show how people can use KANs for symbolic regression, but caveat that we need to train KANs to quite high precision. This is not a problem specific to KANs though; this issue originates from symbolic regression. The space of symbolic formulas is actually quite dense, so tiny noise can make one symbolic formula transit to another. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a4d76348",
   "metadata": {},
   "source": [
    "### 1D example: Adding noise to a bounded region sine"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "73dd7978",
   "metadata": {},
   "outputs": [],
   "source": [
    "def toy(bound=1., noise=0., fun=lambda x: torch.sin(torch.pi*x)):\n",
    "\n",
    "    num_pts = 101\n",
    "    x = torch.linspace(-bound,bound,steps=num_pts)\n",
    "    x = x[:,None]\n",
    "    y = fun(x) + torch.normal(0,1,size=(num_pts,)) * noise\n",
    "    dataset = {}\n",
    "    dataset['train_input'] = dataset['test_input'] = x\n",
    "    dataset['train_label'] = dataset['test_label'] = y\n",
    "    model = KAN(width=[1,1], grid=5, k=3, seed=0, grid_range=(-bound,bound))\n",
    "    model.train(dataset, opt=\"LBFGS\", steps=20)\n",
    "    model.suggest_symbolic(0,0,0)\n",
    "    model.plot()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "2e129909",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "train loss: 2.79e-03 | test loss: 2.79e-03 | reg: 3.12e-01 : 100%|██| 20/20 [00:01<00:00, 13.38it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "function , r2\n",
      "sin , 0.9999842278946689\n",
      "gaussian , 0.9184406012010798\n",
      "tanh , 0.8635381099424172\n",
      "sigmoid , 0.8601324746874981\n",
      "arctan , 0.845004037750832\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZcAAACuCAYAAAD6ZEDcAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjYuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8o6BhiAAAACXBIWXMAAA9hAAAPYQGoP6dpAAASl0lEQVR4nO3dXWyT1x3H8d9xEvJCAklMWi0D2jhztAChESPEvKRa14qsBRVVodvUTZomtZo0iV7scre7mTRtIpN2s1ab6DRpm6Ci7QINsIZoMEKBjfHSLiUOJS8gQgHTBCck4Tm7yPwooRCS+HHsh34/d49b+/m3kv3L/5znnGOstVYAAHgokO4CAACPHsIFAOA5wgUA4DnCBQDgOcIFAOA5wgUA4DnCBQDgOcIFAOA5wgUA4DnCBQDgOcIFAOA5wgUA4DnCBQDgOcIFAOC57HQXAPiBtVbXr1/X0NCQCgsLFQwGZYxJd1lAxqJzAaYRi8XU3NyscDissrIyVVRUqKysTOFwWM3NzYrFYukuEchIhsPCgPtrbW1VU1OT4vG4pInuJSHRtRQUFGjPnj1qbGxMS41ApiJcgPtobW3Vli1bZK2V4zgP/PcCgYCMMWppaSFggEkIF+AesVhMS5cu1fDw8LTBkhAIBJSfn6++vj4VFxenvkDAB5hzAe6xa9cuxePxGQWLJDmOo3g8rrfeeivFlQH+QecCTGKtVTgcVnd3t2bz1TDGKBQK6cKFCzxFBohwAab47LPPVFZWltT7g8GghxUB/sSwGDDJ0NBQUu8fHBz0qBLA3wgXYJLCwsKk3l9UVORRJYC/ES7AJMFgUJWVlbOeNzHGqLKyUqWlpSmqDPAXwgWYxBijHTt2zOm9r7/+OpP5wP8xoQ/cg3UuQPLoXIB7FBcXa8+ePTLGKBCY/iuSWKH/9ttvEyzAJIQLcB+NjY1qaWlRfn6+jDFfGO5KvJafn699+/Zp8+bNaaoUyEyEC/AAjY2N6uvr086dOxUKhab8s1AopJ07d6q/v59gAe6DORdgBqy1OnHihH7605/q17/+terq6pi8B6ZB5wLMgDFGJSUlys/PV0lJCcECPAThAgDwHOECAPAc4QIA8BzhAgDwHOECAPAc4QIA8BzhAgDwHOECAPAc4QIA8BzhAgDwHOECAPAcG1cCMxSPx/Xxxx+rurpaBQUF6S4HyGiECzBDjuNoZGREeXl5Dz1EDPiyI1wAAJ7LTncBwINYa8XfPjOXOAaA4wCQCQgXZKwzZ85o79696S7DN1asWKHt27enuwxAEuGCDDYwMKBbt27ppZde4q/xhzh79qw++eQTWWv5f4WMQLggoz322GPatGnTI/GDaa1VLBbTmTNn9Omnn8oYo8rKStXU1KioqCip/8bR0VEdO3bMw2qB5BAuwDwYHR3V3//+d/35z3/WlStX3LmkQCCgJ554Qj/84Q8ViUSUlZWV5koBb/A8JZBC1loNDw/rjTfe0G9+8xtdvnx5ykMKjuPo4sWL+sUvfqF33nlHd+/eTWO1gHcIFyCFxsbG9Ic//EHvvvuuxsfHJUnZ2dlatmyZvvrVr7rrZUZGRvT73/9ehw4dkuM46SwZ8ATDYkCKOI6j/fv367333nMDIxgM6rXXXlMkEpG1VgcOHNCuXbsUj8d1584dvfnmm6qoqFA4HH4k5pnw5UXnAqSAtVYXLlzQH//4R7djKS0t1c9+9jN961vf0sKFC1VYWKht27bpxz/+sRYsWCBJisVievPNNzUyMpLO8oGkES5ACiSGuW7duiVJys3N1U9+8hPV1NRM6UiysrK0efNmPf/88+7r//nPf3To0CEWkMLXCBfAY9ZatbW16fTp05ImVsy/8MILD3ykOisrS6+88oqWL18uaWI4bffu3bp+/fp8lg14inABPBaLxbR79253nmXZsmX67ne/+8DNLo0xKikp0SuvvKLs7Ilp0CtXrujAgQN0L/AtwgXwkLVWBw8eVF9fn6SJruR73/ueSktLp52gN8Zow4YNWrVqlfs5+/bto3uBbxEugIdu3ryplpYWt+Oorq6e8Q4Dubm5ampqcruXgYEBtbW10b3AlwgXwCPWWrW3t+vKlSuSJtazNDU1KS8vb0bvN8aotrZWK1ascD+vtbVVg4ODKasZSBXCBfDI7du39f7777udRlVVlb7xjW/Mar1Kbm6utm7d6s7P9Pb26uTJk3Qv8B3CBfCAtVanTp3SpUuXJE10IVu3bp1x15JgjFFdXd2UJ8fef/99jY2NeV4zkEqEC+CB8fFxtba2unuDLV26VPX19XNaZb9w4UI9++yz7vVHH32kaDRK9wJfIVyAJFlrdenSJZ07d8597ZlnnlFRUdGcPs8Yo6efflqLFy+WJN25c0cffPAB4QJfIVwADxw+fFjDw8OSpKKiIn3zm99Mam+wxx9/XGvXrnWvjx075q72B/yAcAGSNDg4qKNHj7rXtbW1Ki8vT+ozA4GAnn322SmPJf/rX/+ie4FvEC5AEqy1OnfunPv4cSIUHrQaf6aMMVqxYoU7sW+t1eHDhznvBb5BuABJcBxH7e3t7o9+eXn5FzannKuCggJt2rTJvZ4cYkCmI1yAJFy/ft3doFKS6uvr5zyRf6/EljAFBQWSpKGhIR0/fpyhMfgC4QLMkbVW//73v3Xz5k1J0oIFC9TQ0ODpIV/Lli1TVVWVe33kyBGNjo569vlAqhAuwBzdvXtX//jHP9xO4oknnlAoFPL0Hjk5OVMCKxqN6tKlS3QvyHiECzBHAwMD+vjjj93rDRs2zHpF/sMYY7R27VotWrRI0sQhZB0dHZ7eA0gFwgWYg8R2L4lNJfPy8hSJRFJy7v3jjz/ubmYpSR0dHRyDjIxHuABzcPfuXR07dswdngqFQlq2bFlK7hUIBKZs23/p0iVdvHgxJfcCvEK4AHNw9epV/fe//3WvI5GIFixYkJJ7JbbiLy4uliSNjo6qo6ODeRdkNMIFmKXEU2JDQ0OSJobE6urqUjIklhAMBt1TKiXp+PHjDI0hoxEuwCzdOyT25JNPpmxILCEQCGjjxo1ugPX19am7uzul9wSSQbgAs3Tt2jV1dna61/X19SkbEkswxmj16tUqKSmRNDE0xoJKZDLCBZgFa63OnDmjzz//XNLEyZGpHhJLKC0t1cqVK93rDz/8kKExZCzCBZgFx3GmTKYvX77c3Vwy1QKBgNavX+8GWW9vrz799NN5uTcwW4QLMAs3b97URx995F6vXbvW84WTD2KM0VNPPeUeIsbQGDIZ4QJfsNZqeHhYV69eTduPqbVW58+fd/cSy8nJ0bp16+ZlSCwhGAwyNAZfyE53AcDD9Pf368CBAzpx4oQcx9Evf/lLdzuU+WStnfKUWHl5uSoqKua1hsTQ2D//+U9Za9XT06Oenp55rQGYCToXZLyenh795S9/UVdXl3p6enThwoW0dC+3bt3SuXPn3Os1a9a42+HPl8RTY4lwvXPnjj788MN5rQGYCcIFGa+6ulpLliyRJI2Pj6dl40ZrrTo7O3Xt2jVJUnZ2tiKRyLzXIUmPPfaYqqurJUlZWVmKRqOcUImMw7AYMt6iRYv01FNP6eDBg5KkU6dOKR6Pa+HChfNWg7VWHR0dchxH0sQP/Ne+9rV5nW9JCAQCeu6551RWVqZIJKKvf/3rOnny5LzXAUyHzgUZLzHPkDiX/sqVK+rq6prXGm7fvj3lxMna2loVFhbOaw0Jxhg1NDRox44dqqurS8v8E/AwhAt8YcWKFQoGg5ImhsYmT6ynmrVWXV1dunr1qqSJsKuvr09L15IQCARkjElrDcB0CBf4QnFxsWpqatzrkydPKh6Pz9v9Ozo6ND4+LklasmSJqqur+WEHpkG4wBcCgYA2bNjg/qBfvnxZ0Wh0XrqXeDyuU6dOudc1NTXuQkYA90e4wBeMMVq5cqVKS0slSWNjY/Py1Ji1Vt3d3bp8+bJbx+QtWADcH+EC3ygpKZkyNHbixIl5GRrr6OjQ2NiYW8PKlSsJF+AhCBf4xr1nmvT396urqyulQ2PDw8NTHvNdtWqVu+09gAcjXOAbxhitWrXKfWpsbGxMx44dS+k9u7u71dvb695/48aN7iPRAB6Mbwl8paSkRKtXr3avT5w4odu3b6fkXom9xCYPidXU1DAkBswA4QJfCQQC2rRpk9s9XL58WZ2dnSkZGovH41P27Vq1apX7QAGA6REu8JXE0NjkvcaOHDniebgkFk729fW5950cagCmxzcFvrN48WKtWbPGvT558qR77LCXjhw5MmXh5OrVqxkSA2aIcIHvJPbWysrKkiQNDAzo7NmznnYvg4ODU4bEamtreUoMmAXCBb5jjFF1dbXKy8slTZxr397e7u5YnCxrrc6dO+fuJZaVlaWGhga6FmAWCBf4UmFhodavX+9enz59WgMDA558tuM4amtrc89I+cpXvsLCSWCWCBf4VkNDg3JzcyVJn3/+uY4fP+7J0NjAwMCU7fUjkYiKioqS/lzgy4RwgS8ZY1RRUaFwOCxpYiirra1No6OjSX1uYm3LrVu3JEm5ubl6+umn6VqAWSJc4FsLFizQM8884/7wd3V16ZNPPkmqexkZGdEHH3zgfkZVVZUqKys9qRf4MiFc4FvGGEUiEfcprtHRUR06dGjO4WKt1fnz5xWNRt3Pf+6555STk+NZzcCXBeECX1uyZInq6+vd646OjjlP7DuOo/3797trW8rKytJ+4iTgV4QLfM0Yo82bN7sT+zdv3lR7e/usuxdrrS5evDjlULCGhgbWtgBzRLjA14wxqqqqUnV1taSJkDh48KAGBwdn9TmO42jfvn3uJpiFhYVqbGykawHmiHCB7+Xk5GjLli3uvl89PT06evTojLsXa616e3vV3t7uvlZfX6/ly5cTLsAcES7wPWOM6urqVFFRIWkiLN59990Zb8XvOI727t3r7k+Wn5+vbdu2sUklkAS+PXgkFBQU6MUXX3Q7jWg0qsOHDz+0e7HWqrOzU21tbe5r69evVzgcpmsBkkC44JGQ2Mxycveye/duXb9+fdr3jYyMaNeuXYrH45Im5lq2b9/ubooJYG4IFzwyCgsL9fLLL7vB0N/fr7/+9a/uHmH3chxHLS0tU7Z6+fa3v61QKETXAiSJcMEjI3HGfW1trfva/v371dHR8YXhMWutTp8+rT/96U/ubsrLly9XU1MTXQvgAcIFj5S8vDz96Ec/0qJFiyRNDHv99re/1fnz592AcRxHZ8+e1a9+9SsNDQ2573v11VcVDAbTVjvwKMlOdwGAl4wxCofD+v73v6833nhD4+Pjunbtmn7+859r+/btqqqq0pkzZ/TOO++4m1MGAgG9/PLLWrduHcNhgEcIFzxyAoGAtm7dqv7+fr333nuy1urGjRv63e9+p6ysrClzMMYYPf/88/rOd77DcBjgIcIFj6ScnBy9+uqrysrK0t/+9jeNjY1J0pRgyc3N1bZt2/SDH/zA3T4GgDcIFzySjDHKy8vTa6+9ptWrV2vv3r2KRqMaGxtTXl6eqqqq9NJLL2nNmjV0LEAKEC7whbluo5+dna2NGzdq3bp1unHjhoaHh7Vw4UKVlJQoOzs7qc8G8GCECzJaZ2enmpubmWh/iP7+fi1evDjdZQAuY/mzDRmqt7d3ygJHTK+8vFxr1qwhiJERCBcAgOdYRAkA8BzhAgDwHOECAPAc4QIA8BzhAsyQ4ziKx+PuLsoAHoxwAWYoGo1q27Ztikaj6S4FyHiECwDAc4QLAMBzhAsAwHOECwDAc4QLAMBzhAsAwHOECwDAc4QLAMBzhAsAwHOECwDAc4QLAMBzhAsAwHOECzAD1lrduHFDw8PDunHjhjgdHJge4QJMIxaLqbm5WeFwWJFIREePHlUkElE4HFZzc7NisVi6SwQykrH8CQbcV2trq5qamhSPxyVpSrdijJEkFRQUaM+ePWpsbExLjUCmIlyA+2htbdWWLVtkrZ32cLBAICBjjFpaWggYYBLCBbhHLBbT0qVLNTw8PKNTJwOBgPLz89XX16fi4uLUFwj4AHMuwD127do1q+OME8cfv/XWWymuDPAPOhdgEmutwuGwuru7Z/VEmDFGoVBIFy5ccOdjgC8zwgWY5LPPPlNZWVlS7w8Ggx5WBPgTw2LAJENDQ0m9f3Bw0KNKAH8jXIBJCgsLk3p/UVGRR5UA/ka4AJMEg0FVVlbOet7EGKPKykqVlpamqDLAXwgXYBJjjHbs2DGn977++utM5gP/x4Q+cA/WuQDJo3MB7lFcXKw9e/bIGKNAYPqvSGKF/ttvv02wAJMQLsB9NDY2qqWlRfn5+TLGfGG4K/Fafn6+9u3bp82bN6epUiAzES7AAzQ2Nqqvr087d+5UKBSa8s9CoZB27typ/v5+ggW4D+ZcgBlInOcyODiooqIilZaWMnkPTINwAQB4jmExAIDnCBcAgOcIFwCA5wgXAIDnCBcAgOcIFwCA5wgXAIDnCBcAgOcIFwCA5wgXAIDnCBcAgOcIFwCA5wgXAIDn/gdc4Jzuj6QEnQAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 500x200 with 2 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# when the function is whole range \"bound=1.\"\" (captures a whole period of sine) and has zero noise \"noise=0.\"\n",
    "# it is quite clear the function is clear\n",
    "toy()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "b260de36",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "train loss: 9.30e-01 | test loss: 9.30e-01 | reg: 3.12e-01 : 100%|██| 20/20 [00:00<00:00, 40.68it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "function , r2\n",
      "sin , 0.9999842278898873\n",
      "gaussian , 0.9184406080128915\n",
      "tanh , 0.8635381682633535\n",
      "sigmoid , 0.8601325311561702\n",
      "arctan , 0.8450040982073312\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZcAAACuCAYAAAD6ZEDcAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjYuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8o6BhiAAAACXBIWXMAAA9hAAAPYQGoP6dpAAASl0lEQVR4nO3dXWyT1x3H8d9xEvJCAklMWi0D2jhztAChESPEvKRa14qsBRVVodvUTZomtZo0iV7scre7mTRtIpN2s1ab6DRpm6Ci7QINsIZoMEKBjfHSLiUOJS8gQgHTBCck4Tm7yPwooRCS+HHsh34/d49b+/m3kv3L/5znnGOstVYAAHgokO4CAACPHsIFAOA5wgUA4DnCBQDgOcIFAOA5wgUA4DnCBQDgOcIFAOA5wgUA4DnCBQDgOcIFAOA5wgUA4DnCBQDgOcIFAOC57HQXAPiBtVbXr1/X0NCQCgsLFQwGZYxJd1lAxqJzAaYRi8XU3NyscDissrIyVVRUqKysTOFwWM3NzYrFYukuEchIhsPCgPtrbW1VU1OT4vG4pInuJSHRtRQUFGjPnj1qbGxMS41ApiJcgPtobW3Vli1bZK2V4zgP/PcCgYCMMWppaSFggEkIF+AesVhMS5cu1fDw8LTBkhAIBJSfn6++vj4VFxenvkDAB5hzAe6xa9cuxePxGQWLJDmOo3g8rrfeeivFlQH+QecCTGKtVTgcVnd3t2bz1TDGKBQK6cKFCzxFBohwAab47LPPVFZWltT7g8GghxUB/sSwGDDJ0NBQUu8fHBz0qBLA3wgXYJLCwsKk3l9UVORRJYC/ES7AJMFgUJWVlbOeNzHGqLKyUqWlpSmqDPAXwgWYxBijHTt2zOm9r7/+OpP5wP8xoQ/cg3UuQPLoXIB7FBcXa8+ePTLGKBCY/iuSWKH/9ttvEyzAJIQLcB+NjY1qaWlRfn6+jDFfGO5KvJafn699+/Zp8+bNaaoUyEyEC/AAjY2N6uvr086dOxUKhab8s1AopJ07d6q/v59gAe6DORdgBqy1OnHihH7605/q17/+terq6pi8B6ZB5wLMgDFGJSUlys/PV0lJCcECPAThAgDwHOECAPAc4QIA8BzhAgDwHOECAPAc4QIA8BzhAgDwHOECAPAc4QIA8BzhAgDwHOECAPAcG1cCMxSPx/Xxxx+rurpaBQUF6S4HyGiECzBDjuNoZGREeXl5Dz1EDPiyI1wAAJ7LTncBwINYa8XfPjOXOAaA4wCQCQgXZKwzZ85o79696S7DN1asWKHt27enuwxAEuGCDDYwMKBbt27ppZde4q/xhzh79qw++eQTWWv5f4WMQLggoz322GPatGnTI/GDaa1VLBbTmTNn9Omnn8oYo8rKStXU1KioqCip/8bR0VEdO3bMw2qB5BAuwDwYHR3V3//+d/35z3/WlStX3LmkQCCgJ554Qj/84Q8ViUSUlZWV5koBb/A8JZBC1loNDw/rjTfe0G9+8xtdvnx5ykMKjuPo4sWL+sUvfqF33nlHd+/eTWO1gHcIFyCFxsbG9Ic//EHvvvuuxsfHJUnZ2dlatmyZvvrVr7rrZUZGRvT73/9ehw4dkuM46SwZ8ATDYkCKOI6j/fv367333nMDIxgM6rXXXlMkEpG1VgcOHNCuXbsUj8d1584dvfnmm6qoqFA4HH4k5pnw5UXnAqSAtVYXLlzQH//4R7djKS0t1c9+9jN961vf0sKFC1VYWKht27bpxz/+sRYsWCBJisVievPNNzUyMpLO8oGkES5ACiSGuW7duiVJys3N1U9+8hPV1NRM6UiysrK0efNmPf/88+7r//nPf3To0CEWkMLXCBfAY9ZatbW16fTp05ImVsy/8MILD3ykOisrS6+88oqWL18uaWI4bffu3bp+/fp8lg14inABPBaLxbR79253nmXZsmX67ne/+8DNLo0xKikp0SuvvKLs7Ilp0CtXrujAgQN0L/AtwgXwkLVWBw8eVF9fn6SJruR73/ueSktLp52gN8Zow4YNWrVqlfs5+/bto3uBbxEugIdu3ryplpYWt+Oorq6e8Q4Dubm5ampqcruXgYEBtbW10b3AlwgXwCPWWrW3t+vKlSuSJtazNDU1KS8vb0bvN8aotrZWK1ascD+vtbVVg4ODKasZSBXCBfDI7du39f7777udRlVVlb7xjW/Mar1Kbm6utm7d6s7P9Pb26uTJk3Qv8B3CBfCAtVanTp3SpUuXJE10IVu3bp1x15JgjFFdXd2UJ8fef/99jY2NeV4zkEqEC+CB8fFxtba2unuDLV26VPX19XNaZb9w4UI9++yz7vVHH32kaDRK9wJfIVyAJFlrdenSJZ07d8597ZlnnlFRUdGcPs8Yo6efflqLFy+WJN25c0cffPAB4QJfIVwADxw+fFjDw8OSpKKiIn3zm99Mam+wxx9/XGvXrnWvjx075q72B/yAcAGSNDg4qKNHj7rXtbW1Ki8vT+ozA4GAnn322SmPJf/rX/+ie4FvEC5AEqy1OnfunPv4cSIUHrQaf6aMMVqxYoU7sW+t1eHDhznvBb5BuABJcBxH7e3t7o9+eXn5FzannKuCggJt2rTJvZ4cYkCmI1yAJFy/ft3doFKS6uvr5zyRf6/EljAFBQWSpKGhIR0/fpyhMfgC4QLMkbVW//73v3Xz5k1J0oIFC9TQ0ODpIV/Lli1TVVWVe33kyBGNjo569vlAqhAuwBzdvXtX//jHP9xO4oknnlAoFPL0Hjk5OVMCKxqN6tKlS3QvyHiECzBHAwMD+vjjj93rDRs2zHpF/sMYY7R27VotWrRI0sQhZB0dHZ7eA0gFwgWYg8R2L4lNJfPy8hSJRFJy7v3jjz/ubmYpSR0dHRyDjIxHuABzcPfuXR07dswdngqFQlq2bFlK7hUIBKZs23/p0iVdvHgxJfcCvEK4AHNw9epV/fe//3WvI5GIFixYkJJ7JbbiLy4uliSNjo6qo6ODeRdkNMIFmKXEU2JDQ0OSJobE6urqUjIklhAMBt1TKiXp+PHjDI0hoxEuwCzdOyT25JNPpmxILCEQCGjjxo1ugPX19am7uzul9wSSQbgAs3Tt2jV1dna61/X19SkbEkswxmj16tUqKSmRNDE0xoJKZDLCBZgFa63OnDmjzz//XNLEyZGpHhJLKC0t1cqVK93rDz/8kKExZCzCBZgFx3GmTKYvX77c3Vwy1QKBgNavX+8GWW9vrz799NN5uTcwW4QLMAs3b97URx995F6vXbvW84WTD2KM0VNPPeUeIsbQGDIZ4QJfsNZqeHhYV69eTduPqbVW58+fd/cSy8nJ0bp16+ZlSCwhGAwyNAZfyE53AcDD9Pf368CBAzpx4oQcx9Evf/lLdzuU+WStnfKUWHl5uSoqKua1hsTQ2D//+U9Za9XT06Oenp55rQGYCToXZLyenh795S9/UVdXl3p6enThwoW0dC+3bt3SuXPn3Os1a9a42+HPl8RTY4lwvXPnjj788MN5rQGYCcIFGa+6ulpLliyRJI2Pj6dl40ZrrTo7O3Xt2jVJUnZ2tiKRyLzXIUmPPfaYqqurJUlZWVmKRqOcUImMw7AYMt6iRYv01FNP6eDBg5KkU6dOKR6Pa+HChfNWg7VWHR0dchxH0sQP/Ne+9rV5nW9JCAQCeu6551RWVqZIJKKvf/3rOnny5LzXAUyHzgUZLzHPkDiX/sqVK+rq6prXGm7fvj3lxMna2loVFhbOaw0Jxhg1NDRox44dqqurS8v8E/AwhAt8YcWKFQoGg5ImhsYmT6ynmrVWXV1dunr1qqSJsKuvr09L15IQCARkjElrDcB0CBf4QnFxsWpqatzrkydPKh6Pz9v9Ozo6ND4+LklasmSJqqur+WEHpkG4wBcCgYA2bNjg/qBfvnxZ0Wh0XrqXeDyuU6dOudc1NTXuQkYA90e4wBeMMVq5cqVKS0slSWNjY/Py1Ji1Vt3d3bp8+bJbx+QtWADcH+EC3ygpKZkyNHbixIl5GRrr6OjQ2NiYW8PKlSsJF+AhCBf4xr1nmvT396urqyulQ2PDw8NTHvNdtWqVu+09gAcjXOAbxhitWrXKfWpsbGxMx44dS+k9u7u71dvb695/48aN7iPRAB6Mbwl8paSkRKtXr3avT5w4odu3b6fkXom9xCYPidXU1DAkBswA4QJfCQQC2rRpk9s9XL58WZ2dnSkZGovH41P27Vq1apX7QAGA6REu8JXE0NjkvcaOHDniebgkFk729fW5950cagCmxzcFvrN48WKtWbPGvT558qR77LCXjhw5MmXh5OrVqxkSA2aIcIHvJPbWysrKkiQNDAzo7NmznnYvg4ODU4bEamtreUoMmAXCBb5jjFF1dbXKy8slTZxr397e7u5YnCxrrc6dO+fuJZaVlaWGhga6FmAWCBf4UmFhodavX+9enz59WgMDA558tuM4amtrc89I+cpXvsLCSWCWCBf4VkNDg3JzcyVJn3/+uY4fP+7J0NjAwMCU7fUjkYiKioqS/lzgy4RwgS8ZY1RRUaFwOCxpYiirra1No6OjSX1uYm3LrVu3JEm5ubl6+umn6VqAWSJc4FsLFizQM8884/7wd3V16ZNPPkmqexkZGdEHH3zgfkZVVZUqKys9qRf4MiFc4FvGGEUiEfcprtHRUR06dGjO4WKt1fnz5xWNRt3Pf+6555STk+NZzcCXBeECX1uyZInq6+vd646OjjlP7DuOo/3797trW8rKytJ+4iTgV4QLfM0Yo82bN7sT+zdv3lR7e/usuxdrrS5evDjlULCGhgbWtgBzRLjA14wxqqqqUnV1taSJkDh48KAGBwdn9TmO42jfvn3uJpiFhYVqbGykawHmiHCB7+Xk5GjLli3uvl89PT06evTojLsXa616e3vV3t7uvlZfX6/ly5cTLsAcES7wPWOM6urqVFFRIWkiLN59990Zb8XvOI727t3r7k+Wn5+vbdu2sUklkAS+PXgkFBQU6MUXX3Q7jWg0qsOHDz+0e7HWqrOzU21tbe5r69evVzgcpmsBkkC44JGQ2Mxycveye/duXb9+fdr3jYyMaNeuXYrH45Im5lq2b9/ubooJYG4IFzwyCgsL9fLLL7vB0N/fr7/+9a/uHmH3chxHLS0tU7Z6+fa3v61QKETXAiSJcMEjI3HGfW1trfva/v371dHR8YXhMWutTp8+rT/96U/ubsrLly9XU1MTXQvgAcIFj5S8vDz96Ec/0qJFiyRNDHv99re/1fnz592AcRxHZ8+e1a9+9SsNDQ2573v11VcVDAbTVjvwKMlOdwGAl4wxCofD+v73v6833nhD4+Pjunbtmn7+859r+/btqqqq0pkzZ/TOO++4m1MGAgG9/PLLWrduHcNhgEcIFzxyAoGAtm7dqv7+fr333nuy1urGjRv63e9+p6ysrClzMMYYPf/88/rOd77DcBjgIcIFj6ScnBy9+uqrysrK0t/+9jeNjY1J0pRgyc3N1bZt2/SDH/zA3T4GgDcIFzySjDHKy8vTa6+9ptWrV2vv3r2KRqMaGxtTXl6eqqqq9NJLL2nNmjV0LEAKEC7whbluo5+dna2NGzdq3bp1unHjhoaHh7Vw4UKVlJQoOzs7qc8G8GCECzJaZ2enmpubmWh/iP7+fi1evDjdZQAuY/mzDRmqt7d3ygJHTK+8vFxr1qwhiJERCBcAgOdYRAkA8BzhAgDwHOECAPAc4QIA8BzhAsyQ4ziKx+PuLsoAHoxwAWYoGo1q27Ztikaj6S4FyHiECwDAc4QLAMBzhAsAwHOECwDAc4QLAMBzhAsAwHOECwDAc4QLAMBzhAsAwHOECwDAc4QLAMBzhAsAwHOECzAD1lrduHFDw8PDunHjhjgdHJge4QJMIxaLqbm5WeFwWJFIREePHlUkElE4HFZzc7NisVi6SwQykrH8CQbcV2trq5qamhSPxyVpSrdijJEkFRQUaM+ePWpsbExLjUCmIlyA+2htbdWWLVtkrZ32cLBAICBjjFpaWggYYBLCBbhHLBbT0qVLNTw8PKNTJwOBgPLz89XX16fi4uLUFwj4AHMuwD127do1q+OME8cfv/XWWymuDPAPOhdgEmutwuGwuru7Z/VEmDFGoVBIFy5ccOdjgC8zwgWY5LPPPlNZWVlS7w8Ggx5WBPgTw2LAJENDQ0m9f3Bw0KNKAH8jXIBJCgsLk3p/UVGRR5UA/ka4AJMEg0FVVlbOet7EGKPKykqVlpamqDLAXwgXYBJjjHbs2DGn977++utM5gP/x4Q+cA/WuQDJo3MB7lFcXKw9e/bIGKNAYPqvSGKF/ttvv02wAJMQLsB9NDY2qqWlRfn5+TLGfGG4K/Fafn6+9u3bp82bN6epUiAzES7AAzQ2Nqqvr087d+5UKBSa8s9CoZB27typ/v5+ggW4D+ZcgBlInOcyODiooqIilZaWMnkPTINwAQB4jmExAIDnCBcAgOcIFwCA5wgXAIDnCBcAgOcIFwCA5wgXAIDnCBcAgOcIFwCA5wgXAIDnCBcAgOcIFwCA5wgXAIDn/gdc4Jzuj6QEnQAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 500x200 with 2 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# even with large noise, sine can be revealed, yeah!\n",
    "toy(noise=1.)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "b429397b",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "train loss: 9.30e-02 | test loss: 9.30e-02 | reg: 7.15e-01 : 100%|██| 20/20 [00:00<00:00, 43.08it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "function , r2\n",
      "sin , 0.9999916591202906\n",
      "arctan , 0.9999847147948822\n",
      "tanh , 0.999984517365484\n",
      "x , 0.9999796669306419\n",
      "abs , 0.9999796669306419\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZcAAACuCAYAAAD6ZEDcAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjYuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8o6BhiAAAACXBIWXMAAA9hAAAPYQGoP6dpAAANSUlEQVR4nO3dX0jd9R/H8df37J/HjuvkSYpYwY6cmyCILqqboBp4FiuKRgXbWG7NSe2Pbc5tDoM1dKnbmq6ytlzONSMGDnbhyroZURcVDSEIyjAoByPmPKUd/+yc8/ld2PF3jlPzz/d4vkefjxtB94X3hd89/Xw/3/P9WsYYIwAAbORK9wAAgPmHuAAAbEdcAAC2Iy4AANsRFwCA7YgLAMB2xAUAYDviAgCwHXEBANiOuAAAbEdcAAC2Iy4AANsRFwCA7YgLAMB2i9M9AJAJjDHq6elRf3+/PB6PfD6fLMtK91iAY7FyASYRCoVUX1+vQCCgvLw8rVy5Unl5eQoEAqqvr1coFEr3iIAjWbwsDBhfe3u71q5dq3A4LGlk9RIXX7VkZ2ertbVVwWAwLTMCTkVcgHG0t7drzZo1MsYoFotN+O9cLpcsy1JbWxuBARIQF2CMUCikFStWaGBgYNKwxLlcLrndbnV3d8vr9aZ+QCADsOcCjNHc3KxwODylsEhSLBZTOBzW2bNnUzwZkDlYuQAJjDEKBALq6urSdE4Ny7Lk9/vV2dnJXWSAiAuQ5Pr168rLy5vV8T6fz8aJgMzEZTEgQX9//6yO7+vrs2kSILMRFyCBx+OZ1fE5OTk2TQJkNuICJPD5fMrPz5/2vollWcrPz1dubm6KJgMyC3EBEliWpR07dszo2J07d7KZD/yLDX1gDD7nAsweKxdgDK/Xq9bWVlmWJZdr8lMk/gn9CxcuEBYgAXEBxhEMBtXW1ia32y3Lsm653BX/ntvt1qVLl1RQUJCmSQFnIi7ABILBoLq7u1VXVye/35/0M7/fr7q6Ol29epWwAONgzwWYAmOMLl++rNWrV+vzzz/X448/zuY9MAlWLsAUWJYlr9eb9BXAxIgLAMB2xAUAYDviAgCwHXEBANiOuAAAbEdcAAC2Iy4AANsRFwCA7YgLAMB2xAUAYDviAgCwHQ+uBKYoFotpeHhYS5cu/c/3vAALHXEBpsgYI2PMuO93AZCMuAAAbLc43QMAE4mvFDA18dUUqyo4AXGBY3311Vd6//330z1Gxnj00UdVUlKS7jEAScQFDvbHH3+op6dHr732Gn+Nj6O3t1dut1tZWVn65ptv9MMPP4zuCQHpRlzgaPfee6+ee+45/sNMYIxRV1eXioqK9PDDD+vgwYMaHBxUW1tbukcDRhEXIIMYY/Tzzz+rsLBQHR0d+u6775SVlaX8/Px0jwYk4WZ9IEMYY/Tjjz9q3bp16ujokCRFIhFduXJFQ0ND6R0OGIO4ABnAGKMrV65o/fr1+umnnySN3BW2evVqnTp1Sh6PJ80TAsmIC+Bwxhh9++232rBhgzo7OyWNhOXZZ59VY2Oj7rrrrjRPCNyKuAAOZozR119/rY0bN+q3336TJLlcLr344otqaGiQz+fjZgc4EnEBHMoYo8uXL6uwsFC///67pJGwrF+/XidOnNAdd9xBWOBYxAVwIGOMvvzyS23atElXr16VJC1atEibN2/W22+/reXLlxMWOBpxARwmFovps88+05YtW3Tt2jVJI2EpLi5WTU2NPB4PYYHjERfAQWKxmNra2rR161b9+eefkqTFixdr+/btqqysVHZ2NmFBRuBDlIBDxGIxXbx4Udu2bdONGzckSUuWLFFJSYkqKiqUlZWV5gmBqSMugAPEYjG1trZq586d6u3tlSQtXbpUpaWl2r9/v5YtW5bmCYHpIS5AmsViMZ0/f16vv/66QqGQpJGw7N27V2VlZYQFGYm4AGkUjUb16aefavfu3frrr78kScuWLdOBAwe0a9cuLV26NM0TAjNDXIA0iUaj+uSTT7R792719fVJkrKyslRRUaGSkhItWbIkzRMCM0dcgDSIRqM6d+6c9uzZkxSWgwcPatu2bYQFGY9bkYE5Fo1GdfbsWZWWlo6Gxe1269ChQ4QF8wYrF2AORaNRnTlzRvv27VN/f7+kkbBUVlaquLhYixdzSmJ+4DcZmCPRaFRNTU3at2+f/vnnH0lSdna2qqqqVFRURFgwr/DbDMyBSCSipqYm7d+/Pyksb731ll555RXCgnmHPRcgxSKRiD766KOksNx2222qrq4mLJi3+K0GUigSiej06dMqLy9XOByWNBKWmpoabdq0SYsWLUrzhEBqsHIBUiQSiaixsfGWsNTW1hIWzHusXIAUiIflwIEDSWE5cuSIXn75ZcKCeY+VC2Cz+KWwxLB4PB7CggWFuAA2Gm+PxePxqLa2lrBgQSEugE3G22NhxYKFirgANpjsUtjGjRsJCxYc4gLM0kSXwggLFjLiAsxC/AOS4+2xEBYsZMQFmKFoNDphWNhjwUJHXIAZSAxL4iNdampqCAsg4gJMW/zpxmOfFVZbW6vCwkLCAoi4ANMy3mPzCQtwK+ICTFHii77GXgojLEAy4gJMQTQaVXNz8y1hqa6u5iGUwDiIC/Af4u+837t37+irieMv+tq8eTNhAcZBXIBJxMNSVlaWFJbq6mrCAkyCuAATmCgs8RULb5AEJkZcgHFEo1GdO3fulrBUVVXxamJgCogLMEY8LHv27LklLEVFRYQFmALiAiRIDEtfX58kwgLMBHEB/hWNRtXS0pIUFrfbrcrKSsICTBNxAfT/sJSWliaFpaqqSlu3biUswDQRFyx4E4WlsrKSsAAzRFywoE0WluLiYsICzBBxwYJFWIDUIS5YkOJ3hREWIDWICxacaDSqjz/+mLAAKcRZhAUlHpbED0hyVxhgP1YuWDDizwojLEDqERcsCPEXfY19Vtjhw4cJC5ACnFGY9yKRyC3vvI+HZcuWLYQFSAHOKsxrkUhEp0+fVnl5ucLhsKTk97EQFiA1uCyGeSsSiejDDz9MCkv81cSEBUgtzi7MS5FIRCdPnlRFRYUGBgYkjYSlpqaGd94Dc4CVC+admzdvqqGhISksHo9HR48eJSzAHGHlgnnl5s2bevfdd/Xmm29qcHBQkpSTk6OjR49qw4YNhAWYI8QF88bw8LDq6+tVWVmpoaEhSdLy5ct17NgxrVu3jrAAc4i4YF4YHh7WsWPHVF1dPRqW22+/XcePH9dLL71EWIA5RlyQ8YaGhlRbW6sjR45oeHhYkuT1elVfX68XXnhBLhdbi8BcIy7IaIODgzp8+LCOHz+umzdvSpJyc3N14sQJPf/884QFSBPigoxkjNHg4KAOHTqkd955R5FIRJLk8/nU0NCgZ555hrAAaURckHGMMQqHw3rjjTd06tSp0bDk5eXpgw8+0FNPPUVYgDQjLsgoxhj19/ervLxcTU1NikajkqS7775bJ0+eVEFBgSzLSvOUAIgLMoYxRn///bfKysp07tw5xWIxSdI999yjxsZGPfHEE4QFcAjigoxgjFEoFNKuXbt0/vz50bDcd999amxs1GOPPUZYAAchLsgIPT092r59uy5evChjjCRp5cqVampq0iOPPEJYAIchLnC8a9eu6dVXX1V7e/toWAKBgM6cOaOHHnqIsAAOxC01cDRjjJqamvTFF1+MhuX+++9XS0sLYQEcjLjA0SzLUnFxsZ5++mlZlqUHH3xQLS0teuCBBwgL4GBcFoPj5ebm6r333pPf71dRUZH8fr8kja5kADgPcYGjff/999qxY4csy1IsFlNdXV26R3KkX3/9VXfeeWe6xwBGWYY//+BQv/zyiy5fvpzuMTJGfn6+nnzySS4XwhGICwDAdmzoAwBsR1wAALYjLgAA2xEXAIDtiAswRcYYxWIxPl8DTAFxAaaoo6ND2dnZ6ujoSPcogOMRFwCA7YgLAMB2xAUAYDviAgCwHXEBANiOuAAAbEdcAAC2Iy4AANsRFwCA7YgLAMB2xAUAYDviAgCwHXEBpsAYo97e3qSvACZGXIBJhEIh1dfXKxAIaNWqVRoeHtaqVasUCARUX1+vUCiU7hEBR7IMf4IB42pvb9fatWsVDoclKWm1YlmWJCk7O1utra0KBoNpmRFwKuICjKO9vV1r1qwZfUHYRFwulyzLUltbG4EBEhAXYIxQKKQVK1ZoYGBg0rDEuVwuud1udXd3y+v1pn5AIAOw5wKM0dzcrHA4PKWwSFIsFlM4HNbZs2dTPBmQOVi5AAmMMQoEAurq6prWHWGWZcnv96uzs3N0PwZYyIgLkOD69evKy8ub1fE+n8/GiYDMxGUxIEF/f/+sju/r67NpEiCzERcggcfjmdXxOTk5Nk0CZDbiAiTw+XzKz8+f9r6JZVnKz89Xbm5uiiYDMgtxARJYlqUdO3bM6NidO3eymQ/8iw19YAw+5wLMHisXYAyv16vW1lZZliWXa/JTJP4J/QsXLhAWIAFxAcYRDAbV1tYmt9sty7JuudwV/57b7dalS5dUUFCQpkkBZyIuwASCwaC6u7tVV1cnv9+f9DO/36+6ujpdvXqVsADjYM8FmAJjjG7cuKG+vj7l5OQoNzeXzXtgEsQFAGA7LosBAGxHXAAAtiMuAADbERcAgO2ICwDAdsQFAGA74gIAsB1xAQDYjrgAAGxHXAAAtiMuAADbERcAgO2ICwDAdv8DujkwD7uctTYAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 500x200 with 2 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# but when bound is small and there is noise, it starts to screw up (at least becomes less clear why we should prefer sine)\n",
    "toy(bound = 0.1, noise=0.1)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c2ec089e",
   "metadata": {},
   "source": [
    "### Phase diagram of symbolic regression (how fratcal/chaotic is my phase diagram?)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "29f51b09",
   "metadata": {},
   "source": [
    "#### mix three functions $f_1(x)={\\rm sin}(x)$, $f_2(x)=x^2$, and $f_3(x)={\\rm exp}(x)$ such that $f(x)=af_1(x)+bf_2(x)+(1-a-b)f_3(x)$. Symbolically regress $f(x)$."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "b76dfc4a",
   "metadata": {
    "code_folding": []
   },
   "outputs": [],
   "source": [
    "def mix(a, b, bound=1):\n",
    "    num_pts = 101\n",
    "    x = torch.linspace(-bound,bound,steps=num_pts)\n",
    "    x = x[:,None]\n",
    "    y = a * torch.sin(x) + b * x**2 + (1-a-b) * torch.exp(x)\n",
    "    dataset = {}\n",
    "    dataset['train_input'] = dataset['test_input'] = x\n",
    "    dataset['train_label'] = dataset['test_label'] = y\n",
    "    model = KAN(width=[1,1], grid=10, k=3, seed=0, grid_range=(-bound,bound))\n",
    "    model.train(dataset, opt=\"LBFGS\", steps=20)\n",
    "    return model.suggest_symbolic(0,0,0)[0]\n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "372aabd8",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "train loss: 2.40e-06 | test loss: 2.40e-06 | reg: 2.64e-01 : 100%|██| 20/20 [00:00<00:00, 29.47it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "function , r2\n",
      "cosh , 0.999997477547859\n",
      "exp , 0.9999670134850122\n",
      "sigmoid , 0.9999606621996252\n",
      "tanh , 0.9999524925435431\n",
      "1/x^4 , 0.9999517925552405\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "'cosh'"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "mix(a=0.2, b=0.0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "9166ca69",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "train loss: 2.77e-06 | test loss: 2.77e-06 | reg: 2.72e-01 : 100%|██| 20/20 [00:00<00:00, 43.39it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "function , r2\n",
      "exp , 0.9999999999827021\n",
      "cosh , 0.9999999999827017\n",
      "tanh , 0.999973163748351\n",
      "sigmoid , 0.9999497922899572\n",
      "1/x^4 , 0.9999369992759012\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "train loss: 2.52e-06 | test loss: 2.52e-06 | reg: 2.45e-01 : 100%|██| 20/20 [00:01<00:00, 17.30it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "function , r2\n",
      "cosh , 0.9999988787247418\n",
      "x^4 , 0.9999910879853997\n",
      "gaussian , 0.999967486241568\n",
      "tanh , 0.9999518786252838\n",
      "sigmoid , 0.999948450438625\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "train loss: 2.28e-06 | test loss: 2.28e-06 | reg: 2.18e-01 : 100%|██| 20/20 [00:00<00:00, 43.13it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "function , r2\n",
      "cosh , 0.9999946575638085\n",
      "x^3 , 0.9999164116905525\n",
      "gaussian , 0.9997468080512466\n",
      "x^4 , 0.9996076211798797\n",
      "tanh , 0.9995835694860234\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "train loss: 2.04e-06 | test loss: 2.04e-06 | reg: 1.94e-01 : 100%|██| 20/20 [00:00<00:00, 39.90it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "function , r2\n",
      "cosh , 0.9999854846669585\n",
      "x^3 , 0.9988138920172807\n",
      "gaussian , 0.9985227715662934\n",
      "x^2 , 0.998477650070286\n",
      "sin , 0.9981948138629363\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "train loss: 1.80e-06 | test loss: 1.80e-06 | reg: 1.71e-01 : 100%|██| 20/20 [00:00<00:00, 39.65it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "function , r2\n",
      "cosh , 0.9999662581221136\n",
      "x^2 , 0.9986097449347123\n",
      "sin , 0.998284128651733\n",
      "x^3 , 0.9936582971043266\n",
      "gaussian , 0.9936463187510403\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "train loss: 1.55e-06 | test loss: 1.55e-06 | reg: 1.51e-01 : 100%|██| 20/20 [00:00<00:00, 44.84it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "function , r2\n",
      "cosh , 0.9999418178114038\n",
      "x^2 , 0.9987944480619438\n",
      "sin , 0.9984323316332249\n",
      "gaussian , 0.9949686832586251\n",
      "tanh , 0.9764364382302457\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "train loss: 1.31e-06 | test loss: 1.31e-06 | reg: 1.36e-01 : 100%|██| 20/20 [00:00<00:00, 39.78it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "function , r2\n",
      "cosh , 0.9999041816268858\n",
      "x^2 , 0.9990436001283093\n",
      "sin , 0.9986633245000535\n",
      "gaussian , 0.9958810456319825\n",
      "tanh , 0.9380270364085883\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "train loss: 1.07e-06 | test loss: 1.07e-06 | reg: 1.29e-01 : 100%|██| 20/20 [00:00<00:00, 40.74it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "function , r2\n",
      "cosh , 0.9998655818685623\n",
      "x^2 , 0.9993505000566273\n",
      "sin , 0.9989811585960545\n",
      "gaussian , 0.9916259900602326\n",
      "x^4 , 0.9172564495092251\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "train loss: 8.32e-07 | test loss: 8.32e-07 | reg: 1.27e-01 : 100%|██| 20/20 [00:00<00:00, 44.57it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "function , r2\n",
      "x^2 , 0.9996700824962792\n",
      "sin , 0.9993888581205067\n",
      "cosh , 0.998561267814873\n",
      "gaussian , 0.9707186857583728\n",
      "abs , 0.9254006963892939\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "train loss: 6.00e-07 | test loss: 6.00e-07 | reg: 1.30e-01 : 100%|██| 20/20 [00:00<00:00, 44.38it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "function , r2\n",
      "x^2 , 0.9999132817985119\n",
      "sin , 0.9994936051757877\n",
      "gaussian , 0.9994851357951505\n",
      "cosh , 0.987913942212583\n",
      "abs , 0.933975094122013\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "train loss: 3.79e-07 | test loss: 3.79e-07 | reg: 1.38e-01 : 100%|██| 20/20 [00:00<00:00, 43.23it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "function , r2\n",
      "x^2 , 0.9999999998837575\n",
      "cosh , 0.9999099009608192\n",
      "gaussian , 0.9997105669072212\n",
      "sin , 0.9989290599804755\n",
      "abs , 0.93740817498461\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "train loss: 2.58e-06 | test loss: 2.58e-06 | reg: 2.68e-01 : 100%|██| 20/20 [00:00<00:00, 27.79it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "function , r2\n",
      "arctan , 0.9999798378098914\n",
      "cosh , 0.9999771001456361\n",
      "tanh , 0.9999633902076488\n",
      "sigmoid , 0.9999541433147963\n",
      "1/x^4 , 0.9999236487568766\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "train loss: 2.34e-06 | test loss: 2.34e-06 | reg: 2.40e-01 : 100%|██| 20/20 [00:00<00:00, 20.99it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "function , r2\n",
      "cosh , 0.9999937207935639\n",
      "x^4 , 0.9999862674170232\n",
      "gaussian , 0.9999638602636551\n",
      "sigmoid , 0.9999448387268524\n",
      "x^3 , 0.9999438604212395\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "train loss: 2.10e-06 | test loss: 2.10e-06 | reg: 2.14e-01 : 100%|██| 20/20 [00:00<00:00, 40.36it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "function , r2\n",
      "cosh , 0.9999949634057903\n",
      "x^3 , 0.9998843971854673\n",
      "gaussian , 0.9997599295163522\n",
      "tanh , 0.9996162757398458\n",
      "sigmoid , 0.9996147714224425\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "train loss: 1.86e-06 | test loss: 1.86e-06 | reg: 1.89e-01 : 100%|██| 20/20 [00:00<00:00, 50.75it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "function , r2\n",
      "cosh , 0.999982696239627\n",
      "x^2 , 0.9991234483879446\n",
      "sin , 0.9988859253126088\n",
      "x^3 , 0.9987772391387374\n",
      "gaussian , 0.998656639925584\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "train loss: 1.61e-06 | test loss: 1.61e-06 | reg: 1.66e-01 : 100%|██| 20/20 [00:00<00:00, 43.78it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "function , r2\n",
      "cosh , 0.9999905018303474\n",
      "x^2 , 0.9992854342507987\n",
      "sin , 0.998871759377929\n",
      "gaussian , 0.9985721495291437\n",
      "x^3 , 0.9939912670589373\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "train loss: 1.37e-06 | test loss: 1.37e-06 | reg: 1.45e-01 : 100%|██| 20/20 [00:00<00:00, 36.50it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "function , r2\n",
      "cosh , 0.9999796806444573\n",
      "x^2 , 0.999480532354729\n",
      "sin , 0.9991909206588246\n",
      "gaussian , 0.9964682981271429\n",
      "tanh , 0.9772535252536733\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "train loss: 1.13e-06 | test loss: 1.13e-06 | reg: 1.29e-01 : 100%|██| 20/20 [00:00<00:00, 46.90it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "function , r2\n",
      "cosh , 0.9999738670223917\n",
      "x^2 , 0.9996973599778907\n",
      "sin , 0.9994116683309964\n",
      "gaussian , 0.9973545677199919\n",
      "tanh , 0.9376915470857489\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "train loss: 8.85e-07 | test loss: 8.85e-07 | reg: 1.22e-01 : 100%|██| 20/20 [00:00<00:00, 43.78it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "function , r2\n",
      "x^2 , 0.9998957469235318\n",
      "cosh , 0.9998722722380837\n",
      "sin , 0.9996590283030754\n",
      "gaussian , 0.9939866829832639\n",
      "abs , 0.9232648879118915\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "train loss: 6.46e-07 | test loss: 6.46e-07 | reg: 1.20e-01 : 100%|██| 20/20 [00:00<00:00, 36.61it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "function , r2\n",
      "x^2 , 0.9999974059407801\n",
      "sin , 0.999834148899939\n",
      "cosh , 0.9962063866033833\n",
      "gaussian , 0.9754127817001195\n",
      "abs , 0.9311623016073874\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "train loss: 4.18e-07 | test loss: 4.18e-07 | reg: 1.24e-01 : 100%|██| 20/20 [00:00<00:00, 41.57it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "function , r2\n",
      "x^2 , 0.9999226429006036\n",
      "sin , 0.9998636426802294\n",
      "cosh , 0.9997752175292851\n",
      "gaussian , 0.999679958882009\n",
      "abs , 0.9372397600766894\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "train loss: 2.23e-07 | test loss: 2.23e-07 | reg: 1.33e-01 : 100%|██| 20/20 [00:00<00:00, 40.07it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "function , r2\n",
      "cosh , 0.9995568150249838\n",
      "gaussian , 0.9993467471612426\n",
      "sin , 0.9985870166410652\n",
      "x^2 , 0.9984339446779963\n",
      "abs , 0.9362895855226272\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "train loss: 2.41e-06 | test loss: 2.41e-06 | reg: 2.64e-01 : 100%|██| 20/20 [00:00<00:00, 50.70it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "function , r2\n",
      "cosh , 0.9999974775415001\n",
      "exp , 0.9999951134484997\n",
      "sigmoid , 0.9999606621811076\n",
      "tanh , 0.9999524925186463\n",
      "1/x^4 , 0.9999517925693375\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "train loss: 2.16e-06 | test loss: 2.16e-06 | reg: 2.36e-01 : 100%|██| 20/20 [00:00<00:00, 38.09it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "function , r2\n",
      "cosh , 0.9999987855303939\n",
      "x^3 , 0.9999772484086487\n",
      "x^4 , 0.9999697464110736\n",
      "gaussian , 0.9999593591292381\n",
      "sigmoid , 0.9999388880060787\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "train loss: 1.92e-06 | test loss: 1.92e-06 | reg: 2.09e-01 : 100%|██| 20/20 [00:00<00:00, 43.12it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "function , r2\n",
      "cosh , 0.99999935535384\n",
      "x^3 , 0.9998219698585193\n",
      "gaussian , 0.9997659958370352\n",
      "tanh , 0.9996382852712733\n",
      "sigmoid , 0.999636745445751\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "train loss: 1.66e-06 | test loss: 1.66e-06 | reg: 1.84e-01 : 100%|██| 20/20 [00:01<00:00, 17.83it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "function , r2\n",
      "cosh , 0.9999988957805453\n",
      "x^2 , 0.9996199310076759\n",
      "sin , 0.9994371092496096\n",
      "gaussian , 0.9987635279928482\n",
      "x^3 , 0.9986602848898886\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "train loss: 1.43e-06 | test loss: 1.43e-06 | reg: 1.60e-01 : 100%|██| 20/20 [00:00<00:00, 40.57it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "function , r2\n",
      "cosh , 0.9999954784528895\n",
      "x^2 , 0.9997669036986749\n",
      "sin , 0.9994959364448105\n",
      "gaussian , 0.9992839638818944\n",
      "x^3 , 0.9941048878141949\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "train loss: 1.19e-06 | test loss: 1.19e-06 | reg: 1.39e-01 : 100%|██| 20/20 [00:00<00:00, 40.14it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "function , r2\n",
      "x^2 , 0.9999057877413986\n",
      "cosh , 0.9997878856454342\n",
      "sin , 0.9997010374773055\n",
      "gaussian , 0.9978271729275431\n",
      "sigmoid , 0.9780411748220801\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "train loss: 9.43e-07 | test loss: 9.43e-07 | reg: 1.23e-01 : 100%|██| 20/20 [00:00<00:00, 42.49it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "function , r2\n",
      "x^2 , 0.9999916810769826\n",
      "sin , 0.9998540286212848\n",
      "gaussian , 0.9985646631915182\n",
      "cosh , 0.9978692658702253\n",
      "tanh , 0.9372183456949574\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "train loss: 7.05e-07 | test loss: 7.05e-07 | reg: 1.15e-01 : 100%|██| 20/20 [00:00<00:00, 46.45it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "function , r2\n",
      "x^2 , 0.9999411945915794\n",
      "sin , 0.999917846530661\n",
      "cosh , 0.9993168732975628\n",
      "gaussian , 0.9961111289682559\n",
      "abs , 0.9302392302167873\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "train loss: 4.75e-07 | test loss: 4.75e-07 | reg: 1.13e-01 : 100%|██| 20/20 [00:00<00:00, 48.74it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "function , r2\n",
      "sin , 0.9997864168657743\n",
      "x^2 , 0.9996579498087828\n",
      "cosh , 0.9942754283755333\n",
      "gaussian , 0.9798855202432393\n",
      "abs , 0.9369096690204137\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "train loss: 2.64e-07 | test loss: 2.64e-07 | reg: 1.18e-01 : 100%|██| 20/20 [00:00<00:00, 43.76it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "function , r2\n",
      "sin , 0.9992177278915768\n",
      "x^2 , 0.999137470112453\n",
      "gaussian , 0.999088490162276\n",
      "cosh , 0.9988973034727238\n",
      "abs , 0.9396624038444488\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "train loss: 1.78e-07 | test loss: 1.78e-07 | reg: 1.27e-01 : 100%|██| 20/20 [00:00<00:00, 45.43it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "function , r2\n",
      "x^2 , 0.998540782190707\n",
      "cosh , 0.998497783054037\n",
      "sin , 0.9981679237204564\n",
      "gaussian , 0.9980799911419045\n",
      "abs , 0.9348151361956722\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "train loss: 2.22e-06 | test loss: 2.22e-06 | reg: 2.60e-01 : 100%|██| 20/20 [00:00<00:00, 47.65it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "function , r2\n",
      "cosh , 0.9999930298639131\n",
      "exp , 0.9999836112405796\n",
      "tanh , 0.9999821515315185\n",
      "x^4 , 0.9999746028947676\n",
      "arctan , 0.9999713956244152\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "train loss: 1.98e-06 | test loss: 1.98e-06 | reg: 2.32e-01 : 100%|██| 20/20 [00:00<00:00, 47.28it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "function , r2\n",
      "cosh , 0.9999962889824672\n",
      "x^3 , 0.9999811150224204\n",
      "gaussian , 0.9999534848375812\n",
      "tanh , 0.9999336923279952\n",
      "sigmoid , 0.9999305175966535\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "train loss: 1.74e-06 | test loss: 1.74e-06 | reg: 2.05e-01 : 100%|██| 20/20 [00:00<00:00, 40.82it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "function , r2\n",
      "cosh , 0.9999962156472607\n",
      "x^2 , 0.9998260034206992\n",
      "gaussian , 0.9997664875004227\n",
      "x^3 , 0.9997117036934391\n",
      "sin , 0.9997113407524149\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "train loss: 1.49e-06 | test loss: 1.49e-06 | reg: 1.79e-01 : 100%|██| 20/20 [00:00<00:00, 47.64it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "function , r2\n",
      "cosh , 0.9999544661515833\n",
      "x^2 , 0.9999233768388777\n",
      "sin , 0.9998096068574465\n",
      "gaussian , 0.998846633386868\n",
      "x^3 , 0.9984457355669536\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "train loss: 1.25e-06 | test loss: 1.25e-06 | reg: 1.55e-01 : 100%|██| 20/20 [00:00<00:00, 39.89it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "function , r2\n",
      "x^2 , 0.9999875880501998\n",
      "cosh , 0.9999623957580879\n",
      "sin , 0.9998842247678139\n",
      "gaussian , 0.9997446918456595\n",
      "x^3 , 0.9939812278070211\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "train loss: 1.01e-06 | test loss: 1.01e-06 | reg: 1.33e-01 : 100%|██| 20/20 [00:00<00:00, 47.68it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "function , r2\n",
      "x^2 , 0.9999658213074126\n",
      "sin , 0.9998830954293118\n",
      "cosh , 0.9991428566544741\n",
      "gaussian , 0.9988921913945791\n",
      "tanh , 0.9787975107817283\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "train loss: 7.68e-07 | test loss: 7.68e-07 | reg: 1.16e-01 : 100%|██| 20/20 [00:00<00:00, 44.81it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "function , r2\n",
      "sin , 0.9999324078991241\n",
      "x^2 , 0.9997596795941941\n",
      "cosh , 0.9994865919109623\n",
      "gaussian , 0.9994862477413593\n",
      "tanh , 0.9365762760176262\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "train loss: 5.36e-07 | test loss: 5.36e-07 | reg: 1.08e-01 : 100%|██| 20/20 [00:00<00:00, 43.66it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "function , r2\n",
      "sin , 0.9997548044475728\n",
      "x^2 , 0.9992301995166885\n",
      "cosh , 0.9979137733537407\n",
      "gaussian , 0.9977997203751177\n",
      "abs , 0.9377138311216435\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "train loss: 3.19e-07 | test loss: 3.19e-07 | reg: 1.07e-01 : 100%|██| 20/20 [00:00<00:00, 41.27it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "function , r2\n",
      "sin , 0.9990790086224685\n",
      "x^2 , 0.998306797086686\n",
      "cosh , 0.9912197954445299\n",
      "gaussian , 0.983836484418377\n",
      "abs , 0.9422994090520702\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "train loss: 1.88e-07 | test loss: 1.88e-07 | reg: 1.12e-01 : 100%|██| 20/20 [00:00<00:00, 38.48it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "function , r2\n",
      "sin , 0.9974240586295893\n",
      "gaussian , 0.9973311148206122\n",
      "x^2 , 0.9971996559803431\n",
      "cosh , 0.9968911992725868\n",
      "abs , 0.9402041148610969\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "train loss: 2.92e-07 | test loss: 2.92e-07 | reg: 1.21e-01 : 100%|██| 20/20 [00:00<00:00, 40.66it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "function , r2\n",
      "cosh , 0.9964799073278351\n",
      "x^2 , 0.9963340407057836\n",
      "sin , 0.9959863127871974\n",
      "gaussian , 0.9956440035977147\n",
      "abs , 0.9297638765888608\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "train loss: 2.04e-06 | test loss: 2.04e-06 | reg: 2.57e-01 : 100%|██| 20/20 [00:01<00:00, 19.44it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "function , r2\n",
      "x^4 , 0.9999894819511871\n",
      "x^3 , 0.9999835048772354\n",
      "tanh , 0.9999809889835496\n",
      "cosh , 0.9999804113519436\n",
      "sigmoid , 0.9999728460390712\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "train loss: 1.80e-06 | test loss: 1.80e-06 | reg: 2.28e-01 : 100%|██| 20/20 [00:00<00:00, 36.75it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "function , r2\n",
      "cosh , 0.9999673825431549\n",
      "gaussian , 0.9999462038064604\n",
      "x^2 , 0.9999377574579958\n",
      "x^3 , 0.999934739879401\n",
      "tanh , 0.9999253640056137\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "train loss: 1.56e-06 | test loss: 1.56e-06 | reg: 2.00e-01 : 100%|██| 20/20 [00:00<00:00, 40.62it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "function , r2\n",
      "x^2 , 0.9999856957320528\n",
      "cosh , 0.9999806264360336\n",
      "sin , 0.9999333905161271\n",
      "gaussian , 0.9997629877627796\n",
      "tanh , 0.9996615330714775\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "train loss: 1.31e-06 | test loss: 1.31e-06 | reg: 1.74e-01 : 100%|██| 20/20 [00:00<00:00, 42.76it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "function , r2\n",
      "x^2 , 0.9999803474424048\n",
      "sin , 0.9999690179361935\n",
      "cosh , 0.9996537442008128\n",
      "gaussian , 0.9989097397461996\n",
      "tanh , 0.9985534483580678\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "train loss: 1.07e-06 | test loss: 1.07e-06 | reg: 1.50e-01 : 100%|██| 20/20 [00:00<00:00, 52.19it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "function , r2\n",
      "sin , 0.9999726154715255\n",
      "gaussian , 0.9999250404431426\n",
      "x^2 , 0.9998623405070776\n",
      "cosh , 0.9996561770567938\n",
      "tanh , 0.9942785952521871\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "train loss: 8.35e-07 | test loss: 8.35e-07 | reg: 1.28e-01 : 100%|██| 20/20 [00:00<00:00, 48.08it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "function , r2\n",
      "sin , 0.9999424641676066\n",
      "gaussian , 0.9999091837427161\n",
      "x^2 , 0.9995191008698547\n",
      "cosh , 0.9978922471122651\n",
      "tanh , 0.9795181094994256\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "train loss: 6.03e-07 | test loss: 6.03e-07 | reg: 1.10e-01 : 100%|██| 20/20 [00:00<00:00, 42.09it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "function , r2\n",
      "sin , 0.9998135112170027\n",
      "gaussian , 0.999787323738521\n",
      "x^2 , 0.9987606438444635\n",
      "cosh , 0.9982635098451202\n",
      "abs , 0.9422869570493372\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "train loss: 3.87e-07 | test loss: 3.87e-07 | reg: 1.01e-01 : 100%|██| 20/20 [00:00<00:00, 40.12it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "function , r2\n",
      "sin , 0.9992995819998103\n",
      "gaussian , 0.9987377926584199\n",
      "x^2 , 0.9973752641583749\n",
      "cosh , 0.995255140131929\n",
      "abs , 0.9455257642420224\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "train loss: 2.37e-07 | test loss: 2.37e-07 | reg: 1.00e-01 : 100%|██| 20/20 [00:00<00:00, 39.05it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "function , r2\n",
      "sin , 0.9974728070753829\n",
      "x^2 , 0.995416161237695\n",
      "gaussian , 0.9867522256462564\n",
      "cosh , 0.986582711416683\n",
      "abs , 0.9468472932133176\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "train loss: 2.80e-07 | test loss: 2.80e-07 | reg: 1.05e-01 : 100%|██| 20/20 [00:00<00:00, 43.89it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "function , r2\n",
      "sin , 0.9938672203947038\n",
      "gaussian , 0.9937913539713064\n",
      "x^2 , 0.9936057905336317\n",
      "cosh , 0.9932430068628105\n",
      "abs , 0.9381962721720656\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "train loss: 4.63e-07 | test loss: 4.63e-07 | reg: 1.16e-01 : 100%|██| 20/20 [00:00<00:00, 40.83it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "function , r2\n",
      "x^2 , 0.992730560931046\n",
      "sin , 0.9922829131306503\n",
      "gaussian , 0.9916990159565282\n",
      "cosh , 0.98563942691924\n",
      "abs , 0.9212244844946261\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "train loss: 1.86e-06 | test loss: 1.86e-06 | reg: 2.53e-01 : 100%|██| 20/20 [00:00<00:00, 41.47it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "function , r2\n",
      "cosh , 0.9999888112900659\n",
      "x^2 , 0.9999849501568968\n",
      "tanh , 0.9999760956345113\n",
      "x^3 , 0.9999758615432004\n",
      "sigmoid , 0.9999716645393338\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "train loss: 1.62e-06 | test loss: 1.62e-06 | reg: 2.24e-01 : 100%|██| 20/20 [00:00<00:00, 40.52it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "function , r2\n",
      "x^2 , 0.9999870715347138\n",
      "sin , 0.9999811871919927\n",
      "cosh , 0.9999553343368358\n",
      "gaussian , 0.9999383081018585\n",
      "tanh , 0.9999177958840636\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "train loss: 1.38e-06 | test loss: 1.38e-06 | reg: 1.96e-01 : 100%|██| 20/20 [00:00<00:00, 44.48it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "function , r2\n",
      "sin , 0.9999859303543726\n",
      "gaussian , 0.9999644813363999\n",
      "x^2 , 0.9999214589741492\n",
      "tanh , 0.9996703332150525\n",
      "sigmoid , 0.9996690041252987\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "train loss: 1.14e-06 | test loss: 1.14e-06 | reg: 1.69e-01 : 100%|██| 20/20 [00:00<00:00, 41.08it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "function , r2\n",
      "sin , 0.9999802936975296\n",
      "x^2 , 0.9997259928372615\n",
      "cosh , 0.9993475620341137\n",
      "gaussian , 0.9989573036832605\n",
      "tanh , 0.9986543793643053\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "train loss: 9.05e-07 | test loss: 9.05e-07 | reg: 1.44e-01 : 100%|██| 20/20 [00:00<00:00, 44.49it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "function , r2\n",
      "sin , 0.9999585866499787\n",
      "gaussian , 0.9999486876810368\n",
      "x^2 , 0.9992827278167005\n",
      "cosh , 0.9988740875766842\n",
      "tanh , 0.9946613426702609\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "train loss: 6.75e-07 | test loss: 6.75e-07 | reg: 1.22e-01 : 100%|██| 20/20 [00:00<00:00, 42.73it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "function , r2\n",
      "sin , 0.9998822169169899\n",
      "gaussian , 0.999878040103399\n",
      "x^2 , 0.99837407546255\n",
      "cosh , 0.9979460330489591\n",
      "tanh , 0.9802089053769921\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "train loss: 4.60e-07 | test loss: 4.60e-07 | reg: 1.04e-01 : 100%|██| 20/20 [00:00<00:00, 42.62it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "function , r2\n",
      "gaussian , 0.999597666676653\n",
      "sin , 0.9995937068570286\n",
      "x^2 , 0.99664915970953\n",
      "cosh , 0.9958802001000374\n",
      "abs , 0.9512847855771679\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "train loss: 2.98e-07 | test loss: 2.98e-07 | reg: 9.48e-02 : 100%|██| 20/20 [00:00<00:00, 46.20it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "function , r2\n",
      "sin , 0.9984509165908535\n",
      "gaussian , 0.9959578952120252\n",
      "x^2 , 0.9937818028920754\n",
      "cosh , 0.9907205081520056\n",
      "abs , 0.9534651069741744\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "train loss: 2.89e-07 | test loss: 2.89e-07 | reg: 9.37e-02 : 100%|██| 20/20 [00:00<00:00, 40.23it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "function , r2\n",
      "sin , 0.9943901136276602\n",
      "x^2 , 0.990205343640921\n",
      "gaussian , 0.987762675373181\n",
      "cosh , 0.9761222166037805\n",
      "abs , 0.9497419182649224\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "train loss: 4.40e-07 | test loss: 4.40e-07 | reg: 9.91e-02 : 100%|██| 20/20 [00:00<00:00, 43.16it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "function , r2\n",
      "gaussian , 0.9876449536124641\n",
      "sin , 0.9874566180838149\n",
      "cosh , 0.9873751146532993\n",
      "x^2 , 0.9871135812467379\n",
      "abs , 0.9320312251409515\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "train loss: 6.40e-07 | test loss: 6.40e-07 | reg: 1.10e-01 : 100%|██| 20/20 [00:00<00:00, 47.36it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "function , r2\n",
      "x^2 , 0.9873681256920457\n",
      "sin , 0.9867104677131169\n",
      "cosh , 0.9863604874844045\n",
      "gaussian , 0.9859360750741196\n",
      "x^4 , 0.9239128488596182\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "train loss: 1.69e-06 | test loss: 1.69e-06 | reg: 2.50e-01 : 100%|██| 20/20 [00:00<00:00, 38.33it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "function , r2\n",
      "sin , 0.999990707616756\n",
      "tanh , 0.9999723043366056\n",
      "sigmoid , 0.9999701323519831\n",
      "gaussian , 0.9999551813924956\n",
      "x^2 , 0.9999522532756852\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "train loss: 1.44e-06 | test loss: 1.44e-06 | reg: 2.20e-01 : 100%|██| 20/20 [00:00<00:00, 20.47it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "function , r2\n",
      "sin , 0.9999900693982379\n",
      "gaussian , 0.9999311426065755\n",
      "tanh , 0.9999145249443635\n",
      "sigmoid , 0.9999130489202535\n",
      "x^2 , 0.9998377031202675\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "train loss: 1.21e-06 | test loss: 1.21e-06 | reg: 1.92e-01 : 100%|██| 20/20 [00:00<00:00, 45.23it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "function , r2\n",
      "sin , 0.9999862820440971\n",
      "gaussian , 0.9997517753887365\n",
      "tanh , 0.9996841983206314\n",
      "sigmoid , 0.9996830107874256\n",
      "x^2 , 0.999585767134396\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "train loss: 9.77e-07 | test loss: 9.77e-07 | reg: 1.65e-01 : 100%|██| 20/20 [00:00<00:00, 55.67it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "function , r2\n",
      "sin , 0.9999745726295028\n",
      "x^2 , 0.9990823277390866\n",
      "gaussian , 0.9989945842715465\n",
      "cosh , 0.9988728127858241\n",
      "tanh , 0.9987539221752944\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "train loss: 7.49e-07 | test loss: 7.49e-07 | reg: 1.39e-01 : 100%|██| 20/20 [00:00<00:00, 40.60it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "function , r2\n",
      "sin , 0.9999370198218576\n",
      "x^2 , 0.9981115478889542\n",
      "cosh , 0.9974772652014616\n",
      "gaussian , 0.995731717057559\n",
      "tanh , 0.9950291700743682\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "train loss: 5.36e-07 | test loss: 5.36e-07 | reg: 1.16e-01 : 100%|██| 20/20 [00:00<00:00, 44.68it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "function , r2\n",
      "gaussian , 0.9998221482804251\n",
      "sin , 0.9998030639266311\n",
      "x^2 , 0.9962716348456105\n",
      "cosh , 0.9957027344640318\n",
      "tanh , 0.9808861574166389\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "train loss: 3.67e-07 | test loss: 3.67e-07 | reg: 9.76e-02 : 100%|██| 20/20 [00:00<00:00, 40.80it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "function , r2\n",
      "gaussian , 0.9993158902101864\n",
      "sin , 0.9992737234348714\n",
      "x^2 , 0.9929134959392898\n",
      "cosh , 0.9918352264630244\n",
      "abs , 0.9605168885411256\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "train loss: 3.18e-07 | test loss: 3.18e-07 | reg: 8.81e-02 : 100%|██| 20/20 [00:00<00:00, 37.24it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "function , r2\n",
      "sin , 0.9970616380122096\n",
      "gaussian , 0.9966420291540972\n",
      "x^2 , 0.9875208727310729\n",
      "cosh , 0.983348154979657\n",
      "abs , 0.9611248513995477\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "train loss: 4.31e-07 | test loss: 4.31e-07 | reg: 8.70e-02 : 100%|██| 20/20 [00:00<00:00, 43.04it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "function , r2\n",
      "sin , 0.9888254509108924\n",
      "x^2 , 0.981424420858435\n",
      "gaussian , 0.9756945249179574\n",
      "cosh , 0.9653653219988936\n",
      "abs , 0.949892059404306\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "train loss: 6.17e-07 | test loss: 6.17e-07 | reg: 9.30e-02 : 100%|██| 20/20 [00:00<00:00, 42.47it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "function , r2\n",
      "cosh , 0.9783739534751957\n",
      "gaussian , 0.9779770648401149\n",
      "sin , 0.9773944419005031\n",
      "x^2 , 0.9772100499611249\n",
      "abs , 0.9197221129892613\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "train loss: 8.36e-07 | test loss: 8.36e-07 | reg: 1.06e-01 : 100%|██| 20/20 [00:00<00:00, 45.89it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "function , r2\n",
      "cosh , 0.9847797381605126\n",
      "x^2 , 0.979881475095261\n",
      "sin , 0.9789559646121901\n",
      "gaussian , 0.9779176281844336\n",
      "x^4 , 0.9285991153021671\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "train loss: 1.52e-06 | test loss: 1.52e-06 | reg: 2.46e-01 : 100%|██| 20/20 [00:00<00:00, 43.84it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "function , r2\n",
      "sin , 0.9999930938197495\n",
      "gaussian , 0.9999927495837864\n",
      "tanh , 0.9999730056399402\n",
      "sigmoid , 0.9999716131493217\n",
      "arctan , 0.9999521920308199\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "train loss: 1.28e-06 | test loss: 1.28e-06 | reg: 2.17e-01 : 100%|██| 20/20 [00:00<00:00, 47.20it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "function , r2\n",
      "sin , 0.9999916138797103\n",
      "gaussian , 0.999926218584326\n",
      "tanh , 0.9999191528924947\n",
      "sigmoid , 0.9999179925256542\n",
      "arctan , 0.9998352652530494\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "train loss: 1.05e-06 | test loss: 1.05e-06 | reg: 1.88e-01 : 100%|██| 20/20 [00:00<00:00, 49.52it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "function , r2\n",
      "gaussian , 0.9999899760530466\n",
      "sin , 0.9999866880205311\n",
      "tanh , 0.9997096545485408\n",
      "sigmoid , 0.9997086093190996\n",
      "arctan , 0.9993950375881484\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "train loss: 8.25e-07 | test loss: 8.25e-07 | reg: 1.60e-01 : 100%|██| 20/20 [00:00<00:00, 50.16it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "function , r2\n",
      "sin , 0.9999712618941392\n",
      "gaussian , 0.9990277391373628\n",
      "tanh , 0.9988653756689869\n",
      "sigmoid , 0.9988643627333774\n",
      "x^2 , 0.9979566871598838\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "train loss: 6.15e-07 | test loss: 6.15e-07 | reg: 1.34e-01 : 100%|██| 20/20 [00:00<00:00, 44.33it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "function , r2\n",
      "gaussian , 0.9999414756950388\n",
      "sin , 0.9999185788926284\n",
      "x^2 , 0.9961768265523611\n",
      "tanh , 0.9954056127804757\n",
      "sigmoid , 0.9954045447762155\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "train loss: 4.42e-07 | test loss: 4.42e-07 | reg: 1.11e-01 : 100%|██| 20/20 [00:00<00:00, 36.47it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "function , r2\n",
      "gaussian , 0.9997728550720852\n",
      "sin , 0.9997168387512211\n",
      "x^2 , 0.9928623326890238\n",
      "cosh , 0.9921522541074415\n",
      "tanh , 0.9815839509974684\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "train loss: 3.64e-07 | test loss: 3.64e-07 | reg: 9.13e-02 : 100%|██| 20/20 [00:00<00:00, 51.37it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "function , r2\n",
      "gaussian , 0.9989709402248369\n",
      "sin , 0.9988544685797407\n",
      "x^2 , 0.9868026386831295\n",
      "cosh , 0.985345236582632\n",
      "abs , 0.9698770230380636\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "train loss: 4.35e-07 | test loss: 4.35e-07 | reg: 8.13e-02 : 100%|██| 20/20 [00:00<00:00, 40.11it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "function , r2\n",
      "gaussian , 0.9949371389273981\n",
      "sin , 0.9949107782356766\n",
      "x^2 , 0.977112955292746\n",
      "cosh , 0.9716178500832297\n",
      "abs , 0.967722640360993\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "train loss: 6.03e-07 | test loss: 6.03e-07 | reg: 8.00e-02 : 100%|██| 20/20 [00:00<00:00, 43.30it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "function , r2\n",
      "sin , 0.9790424824765466\n",
      "gaussian , 0.9715243025795919\n",
      "x^2 , 0.9671692490974791\n",
      "cosh , 0.9450165956953308\n",
      "abs , 0.9448625902159359\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "train loss: 8.07e-07 | test loss: 8.07e-07 | reg: 8.69e-02 : 100%|██| 20/20 [00:00<00:00, 43.17it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "function , r2\n",
      "cosh , 0.965461717507565\n",
      "x^2 , 0.9648932341203752\n",
      "sin , 0.963856710506799\n",
      "gaussian , 0.9636801162210671\n",
      "x^4 , 0.9002751610118467\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "train loss: 1.03e-06 | test loss: 1.03e-06 | reg: 1.03e-01 : 100%|██| 20/20 [00:00<00:00, 41.77it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "function , r2\n",
      "cosh , 0.9807409640082864\n",
      "x^2 , 0.9699721931104404\n",
      "sin , 0.9677166767908957\n",
      "gaussian , 0.9394860427747876\n",
      "x^4 , 0.9351858394996657\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "train loss: 1.36e-06 | test loss: 1.36e-06 | reg: 2.43e-01 : 100%|██| 20/20 [00:00<00:00, 44.90it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "function , r2\n",
      "gaussian , 0.9999978669498215\n",
      "sin , 0.9999956722754607\n",
      "tanh , 0.9999799263714397\n",
      "sigmoid , 0.9999789216022107\n",
      "arctan , 0.9999642054598574\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "train loss: 1.13e-06 | test loss: 1.13e-06 | reg: 2.13e-01 : 100%|██| 20/20 [00:00<00:00, 44.99it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "function , r2\n",
      "sin , 0.9999943906884494\n",
      "tanh , 0.9999343640823076\n",
      "sigmoid , 0.9999334248881454\n",
      "gaussian , 0.9999249439932951\n",
      "arctan , 0.9998684985487326\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "train loss: 9.03e-07 | test loss: 9.03e-07 | reg: 1.84e-01 : 100%|██| 20/20 [00:00<00:00, 45.74it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "function , r2\n",
      "sin , 0.9999896507243767\n",
      "tanh , 0.9997526274283742\n",
      "sigmoid , 0.9997517092998544\n",
      "gaussian , 0.9997514568009086\n",
      "arctan , 0.9994994651669644\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "train loss: 6.94e-07 | test loss: 6.94e-07 | reg: 1.56e-01 : 100%|██| 20/20 [00:00<00:00, 45.29it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "function , r2\n",
      "gaussian , 0.9999899485873852\n",
      "sin , 0.9999314322918655\n",
      "tanh , 0.9990030069449396\n",
      "sigmoid , 0.9990020609332764\n",
      "arctan , 0.9981096381936067\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "train loss: 5.19e-07 | test loss: 5.19e-07 | reg: 1.30e-01 : 100%|██| 20/20 [00:00<00:00, 36.49it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "function , r2\n",
      "gaussian , 0.9999536171672674\n",
      "sin , 0.9999094157336442\n",
      "tanh , 0.9958217326666846\n",
      "sigmoid , 0.9958206863529286\n",
      "x^2 , 0.9932659752647721\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "train loss: 4.21e-07 | test loss: 4.21e-07 | reg: 1.06e-01 : 100%|██| 20/20 [00:00<00:00, 49.72it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "function , r2\n",
      "gaussian , 0.9997544064398111\n",
      "sin , 0.9996390145601096\n",
      "x^2 , 0.9876782817075724\n",
      "cosh , 0.986829120472927\n",
      "tanh , 0.9823615338330297\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "train loss: 4.54e-07 | test loss: 4.54e-07 | reg: 8.52e-02 : 100%|██| 20/20 [00:00<00:00, 42.86it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "function , r2\n",
      "gaussian , 0.9985268503112813\n",
      "sin , 0.9983499410484818\n",
      "abs , 0.9785902377362155\n",
      "x^2 , 0.9771995020822951\n",
      "cosh , 0.9752846715582264\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "train loss: 5.93e-07 | test loss: 5.93e-07 | reg: 7.45e-02 : 100%|██| 20/20 [00:00<00:00, 40.78it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "function , r2\n",
      "gaussian , 0.991738617202277\n",
      "sin , 0.9916999095855713\n",
      "abs , 0.9717770596320756\n",
      "x^2 , 0.9601566417330277\n",
      "cosh , 0.9530802418693475\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "train loss: 7.85e-07 | test loss: 7.85e-07 | reg: 7.31e-02 : 100%|██| 20/20 [00:00<00:00, 44.58it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "function , r2\n",
      "sin , 0.9620464258065169\n",
      "gaussian , 0.9582876791641027\n",
      "x^2 , 0.9445008625317239\n",
      "cosh , 0.9430687242552336\n",
      "abs , 0.9308710215935037\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "train loss: 9.96e-07 | test loss: 9.96e-07 | reg: 8.13e-02 : 100%|██| 20/20 [00:00<00:00, 43.11it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "function , r2\n",
      "cosh , 0.9480750122921905\n",
      "x^2 , 0.9456250628531494\n",
      "sin , 0.9445956284973298\n",
      "gaussian , 0.9434289429080119\n",
      "x^4 , 0.9009688699792165\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "train loss: 1.22e-06 | test loss: 1.22e-06 | reg: 1.02e-01 : 100%|██| 20/20 [00:00<00:00, 37.36it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "function , r2\n",
      "cosh , 0.9766929896410047\n",
      "x^2 , 0.957514493788183\n",
      "sin , 0.9559851348570871\n",
      "x^4 , 0.9438827362855521\n",
      "gaussian , 0.9167827938895323\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "train loss: 1.20e-06 | test loss: 1.20e-06 | reg: 2.39e-01 : 100%|██| 20/20 [00:00<00:00, 46.26it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "function , r2\n",
      "sin , 0.9999984333249491\n",
      "sigmoid , 0.9999901865767792\n",
      "arctan , 0.9999829580110535\n",
      "gaussian , 0.9999746546393077\n",
      "tanh , 0.9999393123160958\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "train loss: 9.82e-07 | test loss: 9.82e-07 | reg: 2.10e-01 : 100%|██| 20/20 [00:00<00:00, 43.17it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "function , r2\n",
      "sin , 0.9999977984107461\n",
      "gaussian , 0.999996383890805\n",
      "tanh , 0.9999585593099201\n",
      "sigmoid , 0.9999577735635269\n",
      "arctan , 0.9999150645734015\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "train loss: 7.75e-07 | test loss: 7.75e-07 | reg: 1.80e-01 : 100%|██| 20/20 [00:00<00:00, 43.96it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "function , r2\n",
      "gaussian , 0.9999953796079631\n",
      "sin , 0.9999944856461577\n",
      "tanh , 0.9998140271258696\n",
      "sigmoid , 0.9998132139651115\n",
      "arctan , 0.9996263187286749\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "train loss: 5.99e-07 | test loss: 5.99e-07 | reg: 1.52e-01 : 100%|██| 20/20 [00:00<00:00, 39.44it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "function , r2\n",
      "sin , 0.9999799131978417\n",
      "tanh , 0.9991760870939851\n",
      "sigmoid , 0.999175206451367\n",
      "gaussian , 0.9991099365080184\n",
      "arctan , 0.9984657428834645\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "train loss: 4.86e-07 | test loss: 4.86e-07 | reg: 1.25e-01 : 100%|██| 20/20 [00:00<00:00, 56.80it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "function , r2\n",
      "gaussian , 0.9999683931970753\n",
      "sin , 0.9999135788767782\n",
      "tanh , 0.996308913203486\n",
      "sigmoid , 0.9963078955424317\n",
      "arctan , 0.9939227717854873\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "train loss: 4.85e-07 | test loss: 4.85e-07 | reg: 1.00e-01 : 100%|██| 20/20 [00:00<00:00, 44.41it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "function , r2\n",
      "gaussian , 0.9997689201745247\n",
      "sin , 0.9995873145216307\n",
      "abs , 0.9849023647718247\n",
      "tanh , 0.9833009867719864\n",
      "sigmoid , 0.9832997013656896\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "train loss: 5.93e-07 | test loss: 5.93e-07 | reg: 7.91e-02 : 100%|██| 20/20 [00:00<00:00, 47.72it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "function , r2\n",
      "gaussian , 0.9982643783383393\n",
      "sin , 0.9977962501289016\n",
      "abs , 0.9854353082947718\n",
      "x^2 , 0.9624423927207054\n",
      "cosh , 0.9600011409875004\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "train loss: 7.70e-07 | test loss: 7.70e-07 | reg: 6.75e-02 : 100%|██| 20/20 [00:00<00:00, 39.11it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "function , r2\n",
      "sin , 0.9870493767685123\n",
      "gaussian , 0.9870320319171259\n",
      "abs , 0.9705633369555351\n",
      "x^2 , 0.932694450295423\n",
      "cosh , 0.9237291936352401\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "train loss: 9.74e-07 | test loss: 9.74e-07 | reg: 6.60e-02 : 100%|██| 20/20 [00:00<00:00, 42.96it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "function , r2\n",
      "sin , 0.9329023836770476\n",
      "gaussian , 0.9251125534459796\n",
      "x^2 , 0.9090404451604721\n",
      "cosh , 0.9075530370365199\n",
      "abs , 0.902215514501183\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "train loss: 1.19e-06 | test loss: 1.19e-06 | reg: 7.68e-02 : 100%|██| 20/20 [00:00<00:00, 45.43it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "function , r2\n",
      "cosh , 0.936942764457975\n",
      "x^2 , 0.9195925986987522\n",
      "sin , 0.9179859981970137\n",
      "gaussian , 0.916197691863038\n",
      "x^4 , 0.9043637756591075\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "train loss: 1.42e-06 | test loss: 1.42e-06 | reg: 1.03e-01 : 100%|██| 20/20 [00:00<00:00, 42.15it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "function , r2\n",
      "cosh , 0.9746783379118565\n",
      "x^4 , 0.9547067294869719\n",
      "x^2 , 0.9426800819283715\n",
      "sin , 0.9405877648417602\n",
      "gaussian , 0.9047220443136558\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "train loss: 1.06e-06 | test loss: 1.06e-06 | reg: 2.36e-01 : 100%|██| 20/20 [00:00<00:00, 44.17it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "function , r2\n",
      "sin , 0.9999998722503725\n",
      "tanh , 0.9999959867450359\n",
      "sigmoid , 0.9999953377679788\n",
      "arctan , 0.9999901647899601\n",
      "gaussian , 0.9999791487269069\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "train loss: 8.56e-07 | test loss: 8.56e-07 | reg: 2.06e-01 : 100%|██| 20/20 [00:00<00:00, 50.53it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "function , r2\n",
      "sin , 0.9999997233652407\n",
      "tanh , 0.9999795491992056\n",
      "sigmoid , 0.9999788713253935\n",
      "arctan , 0.9999539297010896\n",
      "gaussian , 0.9999367314200862\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "train loss: 6.79e-07 | test loss: 6.79e-07 | reg: 1.77e-01 : 100%|██| 20/20 [00:00<00:00, 47.96it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "function , r2\n",
      "sin , 0.9999983352488726\n",
      "gaussian , 0.9999608456659727\n",
      "tanh , 0.9998803010371731\n",
      "sigmoid , 0.9998795700743581\n",
      "arctan , 0.9997532798065882\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "train loss: 5.56e-07 | test loss: 5.56e-07 | reg: 1.48e-01 : 100%|██| 20/20 [00:00<00:00, 47.37it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "function , r2\n",
      "sin , 0.9999886905327071\n",
      "gaussian , 0.9999526296594224\n",
      "tanh , 0.9993739474035951\n",
      "sigmoid , 0.9993731243691055\n",
      "arctan , 0.998837883424404\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "train loss: 5.28e-07 | test loss: 5.28e-07 | reg: 1.20e-01 : 100%|██| 20/20 [00:00<00:00, 47.09it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "function , r2\n",
      "gaussian , 0.9999347944113626\n",
      "sin , 0.9999299845850768\n",
      "tanh , 0.9968744391727294\n",
      "sigmoid , 0.9968734504088798\n",
      "arctan , 0.9949201894762312\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "train loss: 6.08e-07 | test loss: 6.08e-07 | reg: 9.51e-02 : 100%|██| 20/20 [00:00<00:00, 34.26it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "function , r2\n",
      "gaussian , 0.9997595833681479\n",
      "sin , 0.9993617636896577\n",
      "abs , 0.9889704797728257\n",
      "sigmoid , 0.9844737269360767\n",
      "tanh , 0.9844552121850878\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "train loss: 7.61e-07 | test loss: 7.61e-07 | reg: 7.30e-02 : 100%|██| 20/20 [00:00<00:00, 39.79it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "function , r2\n",
      "gaussian , 0.9979451890061467\n",
      "sin , 0.9972568796736283\n",
      "abs , 0.9882338451079785\n",
      "x^2 , 0.9400830044171552\n",
      "cosh , 0.9380462665721708\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "train loss: 9.51e-07 | test loss: 9.51e-07 | reg: 6.06e-02 : 100%|██| 20/20 [00:00<00:00, 42.43it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "function , r2\n",
      "sin , 0.9805383456413282\n",
      "gaussian , 0.9793534545721814\n",
      "abs , 0.9588682058685749\n",
      "x^2 , 0.8881789600114199\n",
      "cosh , 0.8769912910873774\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "train loss: 1.16e-06 | test loss: 1.16e-06 | reg: 5.94e-02 : 100%|██| 20/20 [00:00<00:00, 42.80it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "function , r2\n",
      "sin , 0.8848700744056857\n",
      "gaussian , 0.8765425797595888\n",
      "x^2 , 0.854821484894479\n",
      "cosh , 0.8535224650797281\n",
      "abs , 0.8493986649063162\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "train loss: 1.38e-06 | test loss: 1.38e-06 | reg: 7.70e-02 : 100%|██| 20/20 [00:00<00:00, 48.12it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "function , r2\n",
      "cosh , 0.9268644000446836\n",
      "x^4 , 0.9112716246650874\n",
      "x^2 , 0.8865324039130013\n",
      "sin , 0.8842948895377678\n",
      "gaussian , 0.8094804211038418\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "train loss: 1.60e-06 | test loss: 1.60e-06 | reg: 1.05e-01 : 100%|██| 20/20 [00:00<00:00, 44.20it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "function , r2\n",
      "cosh , 0.9740201843349593\n",
      "x^4 , 0.9673225582521513\n",
      "gaussian , 0.952288197814531\n",
      "tanh , 0.9497276520343576\n",
      "sigmoid , 0.9497237037538462\n"
     ]
    }
   ],
   "source": [
    "# let's do a phase diagram, which looks quite \"fractal\"\n",
    "num = 11\n",
    "a_arr = np.linspace(0,1,num=num)\n",
    "b_arr = np.linspace(0,1,num=num)\n",
    "sf_mat = np.empty((num,num), dtype='U8')\n",
    "\n",
    "for i in range(num):\n",
    "    for j in range(num):\n",
    "        a = a_arr[i]; b = b_arr[j]\n",
    "        sf_mat[i,j] = mix(a, b)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "7c60506b",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.image.AxesImage at 0x7fd438377a90>"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAaEAAAGdCAYAAAC7EMwUAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjYuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8o6BhiAAAACXBIWXMAAA9hAAAPYQGoP6dpAAAXb0lEQVR4nO3df2xV9f3H8VcpcFvk9gqY/pK2FtNvilQRW7YIVTBKE0AiMbip4IhsiYwCrU0cdLgh7EvvYFvTxM6SknyRfUmxmRvKkrnZ6GwlSCyFImELBCT0TtY0bHzvbXFepD3fPxYvu7QgznN539s+H8n5o6fHft451vvM6b2ek+Q4jiMAAAyMsh4AADByESEAgBkiBAAwQ4QAAGaIEADADBECAJghQgAAM0QIAGBmtPUAVxsYGNC5c+fk9XqVlJRkPQ4A4CtyHEe9vb3Kzs7WqFHXv9aJuwidO3dOOTk51mMAAL6mQCCgyZMnX/eYuIuQ1+uVJP33/6xRyjiP6Sxz/muN6foAbkyff7H1CPg3Fz/v16NvfBR5Pb+euIvQF3+CSxnnUapxhMZ700zXB3CDxiRbT4Ah3MhbKnwwAQBghggBAMwQIQCAGSIEADBDhAAAZogQAMAMEQIAmCFCAAAzRAgAYCZmEXrllVeUn5+vlJQUFRcX6/3334/VUgCABBWTCDU3N6uyslIbNmzQkSNH9MADD2j+/Pnq6uqKxXIAgAQVkwjV1tbqu9/9rr73ve9p6tSpqqurU05OjhoaGmKxHAAgQbkeoUuXLqmjo0NlZWVR+8vKynTgwIFBx4fDYYVCoagNADAyuB6h8+fPq7+/XxkZGVH7MzIy1N3dPeh4v98vn88X2XiWEACMHDH7YMLVt/B2HGfI23pXV1crGAxGtkAgEKuRAABxxvXnCd12221KTk4edNXT09Mz6OpIkjwejzwe2+cGAQBsuH4lNHbsWBUXF6ulpSVqf0tLi2bNmuX2cgCABBaTJ6tWVVXpmWeeUUlJie6//341Njaqq6tLK1eujMVyAIAEFZMIffvb39bf//53bd68WX/7299UVFSk3//+98rLy4vFcgCABBWTCEnSqlWrtGrVqlj9eADAMMC94wAAZogQAMAMEQIAmCFCAAAzRAgAYIYIAQDMECEAgBkiBAAwQ4QAAGaIEADATMxu2/N1BY5+X54Ur+0QU22XB+Jd38Z51iPgKuM3tXz5QbHWG5J+fWP3CuVKCABghggBAMwQIQCAGSIEADBDhAAAZogQAMAMEQIAmCFCAAAzRAgAYIYIAQDMECEAgBkiBAAwQ4QAAGaIEADADBECAJghQgAAM0QIAGCGCAEAzBAhAIAZIgQAMEOEAABmiBAAwAwRAgCYIUIAADNECABghggBAMwQIQCAGSIEADBDhAAAZogQAMAMEQIAmCFCAAAzo60HuJZlT4zWeO8Y6zGAQfo2zrMeAVeZXnm79Qhx42gc/H72fd5/w8dyJQQAMEOEAABmiBAAwAwRAgCYIUIAADNECABghggBAMwQIQCAGSIEADBDhAAAZogQAMAMEQIAmCFCAAAzrkfI7/dr5syZ8nq9Sk9P1+LFi3XixAm3lwEADAOuR6i1tVXl5eU6ePCgWlpadPnyZZWVlenixYtuLwUASHCuP0/oD3/4Q9TXO3fuVHp6ujo6OvTggw+6vRwAIIHF/KF2wWBQkjRx4sQhvx8OhxUOhyNfh0KhWI8EAIgTMf1gguM4qqqqUmlpqYqKioY8xu/3y+fzRbacnJxYjgQAiCMxjdDq1av10Ucfac+ePdc8prq6WsFgMLIFAoFYjgQAiCMx+3PcmjVrtG/fPrW1tWny5MnXPM7j8cjj8cRqDABAHHM9Qo7jaM2aNdq7d6/ee+895efnu70EAGCYcD1C5eXlampq0ptvvimv16vu7m5Jks/nU2pqqtvLAQASmOvvCTU0NCgYDGru3LnKysqKbM3NzW4vBQBIcDH5cxwAADeCe8cBAMwQIQCAGSIEADBDhAAAZogQAMAMEQIAmCFCAAAzRAgAYIYIAQDMxPyhdoBb+jbOsx4hrkyvvN16BMSh8ZtarEeQekPSr/Nu6FCuhAAAZogQAMAMEQIAmCFCAAAzRAgAYIYIAQDMECEAgBkiBAAwQ4QAAGaIEADADBECAJghQgAAM0QIAGCGCAEAzBAhAIAZIgQAMEOEAABmiBAAwAwRAgCYIUIAADNECABghggBAMwQIQCAGSIEADBDhAAAZogQAMAMEQIAmCFCAAAzRAgAYIYIAQDMECEAgBkiBAAwM9p6gGsZnfKmxqSkms7w+WdLTNeXpL6N86xHiBvTK2+3HgGIe/HwmtH3ef8NH8uVEADADBECAJghQgAAM0QIAGCGCAEAzBAhAIAZIgQAMEOEAABmiBAAwAwRAgCYIUIAADNECABghggBAMzEPEJ+v19JSUmqrKyM9VIAgAQT0wi1t7ersbFR99xzTyyXAQAkqJhFqK+vT0uXLtWOHTs0YcKEWC0DAEhgMYtQeXm5Fi5cqEceeeS6x4XDYYVCoagNADAyxOTJqq+99poOHz6s9vb2Lz3W7/dr06ZNsRgDABDnXL8SCgQCqqio0O7du5WSkvKlx1dXVysYDEa2QCDg9kgAgDjl+pVQR0eHenp6VFxcHNnX39+vtrY21dfXKxwOKzk5OfI9j8cjj8fj9hgAgATgeoQefvhhHTt2LGrfs88+q8LCQq1bty4qQACAkc31CHm9XhUVFUXtu+WWWzRp0qRB+wEAIxt3TAAAmInJp+Ou9t57792MZQAACYYrIQCAGSIEADBDhAAAZogQAMAMEQIAmCFCAAAzRAgAYIYIAQDMECEAgJmbcseE/8Tlzx7T52PSTGfo2zjPdH1Jml55u/UIiFMX7nzdegRNOL3EeoS4Eg//TqT7rQf4SrgSAgCYIUIAADNECABghggBAMwQIQCAGSIEADBDhAAAZogQAMAMEQIAmCFCAAAzRAgAYIYIAQDMECEAgBkiBAAwQ4QAAGaIEADADBECAJghQgAAM0QIAGCGCAEAzBAhAIAZIgQAMEOEAABmiBAAwAwRAgCYIUIAADNECABghggBAMwQIQCAGSIEADBDhAAAZogQAMAMEQIAmBltPcC15P/f95V2eYztEJW3266PuHThztetR0Cc+sc/+qxH0PhNLdYjSL0h6dd5N3QoV0IAADNECABghggBAMwQIQCAGSIEADBDhAAAZogQAMAMEQIAmCFCAAAzRAgAYIYIAQDMECEAgJmYROiTTz7RsmXLNGnSJI0bN0733nuvOjo6YrEUACCBuX4X7QsXLmj27Nl66KGH9NZbbyk9PV2nT5/Wrbfe6vZSAIAE53qEtm7dqpycHO3cuTOy74477nB7GQDAMOD6n+P27dunkpISPfHEE0pPT9eMGTO0Y8eOax4fDocVCoWiNgDAyOB6hD7++GM1NDSooKBAf/zjH7Vy5UqtXbtWv/rVr4Y83u/3y+fzRbacnBy3RwIAxKkkx3EcN3/g2LFjVVJSogMHDkT2rV27Vu3t7frggw8GHR8OhxUOhyNfh0Ih5eTk6Py7C5Q23vjJqsAQeLLqFRNOL7EeIa6cnvCq9Qhxoa83pJn35SkYDCotLe26x7p+JZSVlaW77rorat/UqVPV1dU15PEej0dpaWlRGwBgZHA9QrNnz9aJEyei9p08eVJ5eTf2vHEAwMjheoSef/55HTx4UDU1NTp16pSamprU2Nio8vJyt5cCACQ41yM0c+ZM7d27V3v27FFRUZF+8pOfqK6uTkuXLnV7KQBAgnP9/xOSpEcffVSPPvpoLH40AGAY4d5xAAAzRAgAYIYIAQDMECEAgBkiBAAwQ4QAAGaIEADADBECAJghQgAAMzG5YwLcEy+PDYiHW/bHy7nAFfHy7yQefj/xn+FKCABghggBAMwQIQCAGSIEADBDhAAAZogQAMAMEQIAmCFCAAAzRAgAYIYIAQDMECEAgBkiBAAwQ4QAAGaIEADADBECAJghQgAAM0QIAGCGCAEAzBAhAIAZIgQAMEOEAABmiBAAwAwRAgCYIUIAADNECABghggBAMwQIQCAGSIEADBDhAAAZogQAMAMEQIAmCFCAAAzRAgAYGa09QDX8n/5/6v+tDTTGSacXmK6frzMIEkX7nzdegQAwxBXQgAAM0QIAGCGCAEAzBAhAIAZIgQAMEOEAABmiBAAwAwRAgCYIUIAADNECABghggBAMwQIQCAGdcjdPnyZb344ovKz89XamqqpkyZos2bN2tgYMDtpQAACc71u2hv3bpV27dv165duzRt2jQdOnRIzz77rHw+nyoqKtxeDgCQwFyP0AcffKDHHntMCxculCTdcccd2rNnjw4dOuT2UgCABOf6n+NKS0v1zjvv6OTJk5Kko0ePav/+/VqwYMGQx4fDYYVCoagNADAyuH4ltG7dOgWDQRUWFio5OVn9/f3asmWLnnrqqSGP9/v92rRpk9tjAAASgOtXQs3Nzdq9e7eampp0+PBh7dq1Sz//+c+1a9euIY+vrq5WMBiMbIFAwO2RAABxyvUroRdeeEHr16/Xk08+KUm6++67dfbsWfn9fi1fvnzQ8R6PRx6Px+0xAAAJwPUroU8//VSjRkX/2OTkZD6iDQAYxPUroUWLFmnLli3Kzc3VtGnTdOTIEdXW1mrFihVuLwUASHCuR+jll1/Wj370I61atUo9PT3Kzs7Wc889px//+MduLwUASHCuR8jr9aqurk51dXVu/2gAwDDDveMAAGaIEADADBECAJghQgAAM0QIAGCGCAEAzBAhAIAZIgQAMEOEAABmiBAAwIzrt+1xy61nnlHa+DHWY5i7cOfr1iNIkiacXmI9QtycC1wRD78XSGxcCQEAzBAhAIAZIgQAMEOEAABmiBAAwAwRAgCYIUIAADNECABghggBAMwQIQCAGSIEADBDhAAAZogQAMAMEQIAmCFCAAAzRAgAYIYIAQDMECEAgBkiBAAwQ4QAAGaIEADADBECAJghQgAAM0QIAGCGCAEAzBAhAIAZIgQAMEOEAABmiBAAwAwRAgCYIUIAADNECABgZrT1ALi+CaeXWI8AADHDlRAAwAwRAgCYIUIAADNECABghggBAMwQIQCAGSIEADBDhAAAZogQAMAMEQIAmCFCAAAzRAgAYOYrR6itrU2LFi1Sdna2kpKS9MYbb0R933EcvfTSS8rOzlZqaqrmzp2r48ePuzUvAGAY+coRunjxoqZPn676+vohv79t2zbV1taqvr5e7e3tyszM1Lx589Tb2/u1hwUADC9f+VEO8+fP1/z584f8nuM4qqur04YNG/T4449Lknbt2qWMjAw1NTXpueee+3rTAgCGFVffEzpz5oy6u7tVVlYW2efxeDRnzhwdOHBgyH8mHA4rFApFbQCAkcHVCHV3d0uSMjIyovZnZGREvnc1v98vn88X2XJyctwcCQAQx2Ly6bikpKSorx3HGbTvC9XV1QoGg5EtEAjEYiQAQBxy9fHemZmZkv51RZSVlRXZ39PTM+jq6Asej0cej8fNMQAACcLVK6H8/HxlZmaqpaUlsu/SpUtqbW3VrFmz3FwKADAMfOUrob6+Pp06dSry9ZkzZ9TZ2amJEycqNzdXlZWVqqmpUUFBgQoKClRTU6Nx48bp6aefdnVwAEDi+8oROnTokB566KHI11VVVZKk5cuX69VXX9UPfvAD/fOf/9SqVat04cIFffOb39Tbb78tr9fr3tQAgGEhyXEcx3qIfxcKheTz+XT+3QVKGz/GehzEkQt3vm49Aq4y4fQS6xHiyukJr1qPEBf6ekOaeV+egsGg0tLSrnss944DAJghQgAAM0QIAGCGCAEAzBAhAIAZIgQAMEOEAABmiBAAwAwRAgCYcfUu2m744gYOvRc/N54E8aaXBx7GneQ+/jv9d32j+R2VpL6+XklXXs+vJ+5u2/PXv/6VB9sBwDAQCAQ0efLk6x4TdxEaGBjQuXPn5PV6r/kgvC8TCoWUk5OjQCDwpfctGu44F9E4H1dwLq7gXFzhxrlwHEe9vb3Kzs7WqFHXf9cn7v4cN2rUqC8t541KS0sb8b9QX+BcRON8XMG5uIJzccXXPRc+n++GjuODCQAAM0QIAGBmWEbI4/Fo48aN8ng81qOY41xE43xcwbm4gnNxxc0+F3H3wQQAwMgxLK+EAACJgQgBAMwQIQCAGSIEADAzLCP0yiuvKD8/XykpKSouLtb7779vPdJN5/f7NXPmTHm9XqWnp2vx4sU6ceKE9Vhxwe/3KykpSZWVldajmPjkk0+0bNkyTZo0SePGjdO9996rjo4O67FMXL58WS+++KLy8/OVmpqqKVOmaPPmzRoYGLAeLeba2tq0aNEiZWdnKykpSW+88UbU9x3H0UsvvaTs7GylpqZq7ty5On78uOtzDLsINTc3q7KyUhs2bNCRI0f0wAMPaP78+erq6rIe7aZqbW1VeXm5Dh48qJaWFl2+fFllZWW6ePGi9Wim2tvb1djYqHvuucd6FBMXLlzQ7NmzNWbMGL311lv685//rF/84he69dZbrUczsXXrVm3fvl319fX6y1/+om3btulnP/uZXn75ZevRYu7ixYuaPn266uvrh/z+tm3bVFtbq/r6erW3tyszM1Pz5s1Tb2+vu4M4w8w3vvENZ+XKlVH7CgsLnfXr1xtNFB96enocSU5ra6v1KGZ6e3udgoICp6WlxZkzZ45TUVFhPdJNt27dOqe0tNR6jLixcOFCZ8WKFVH7Hn/8cWfZsmVGE9mQ5Ozduzfy9cDAgJOZmen89Kc/jez77LPPHJ/P52zfvt3VtYfVldClS5fU0dGhsrKyqP1lZWU6cOCA0VTxIRgMSpImTpxoPImd8vJyLVy4UI888oj1KGb27dunkpISPfHEE0pPT9eMGTO0Y8cO67HMlJaW6p133tHJkyclSUePHtX+/fu1YMEC48lsnTlzRt3d3VGvpR6PR3PmzHH9tTTubmD6dZw/f179/f3KyMiI2p+RkaHu7m6jqew5jqOqqiqVlpaqqKjIehwTr732mg4fPqz29nbrUUx9/PHHamhoUFVVlX74wx/qww8/1Nq1a+XxePSd73zHerybbt26dQoGgyosLFRycrL6+/u1ZcsWPfXUU9ajmfri9XKo19KzZ8+6utawitAXrn4EhOM4//FjIYaD1atX66OPPtL+/futRzERCARUUVGht99+WykpKdbjmBoYGFBJSYlqamokSTNmzNDx48fV0NAwIiPU3Nys3bt3q6mpSdOmTVNnZ6cqKyuVnZ2t5cuXW49n7ma8lg6rCN12221KTk4edNXT09MzqOgjxZo1a7Rv3z61tbW59oiMRNPR0aGenh4VFxdH9vX396utrU319fUKh8NKTk42nPDmycrK0l133RW1b+rUqfrNb35jNJGtF154QevXr9eTTz4pSbr77rt19uxZ+f3+ER2hzMxMSf+6IsrKyorsj8Vr6bB6T2js2LEqLi5WS0tL1P6WlhbNmjXLaCobjuNo9erV+u1vf6t3331X+fn51iOZefjhh3Xs2DF1dnZGtpKSEi1dulSdnZ0jJkCSNHv27EEf1T958qTy8vKMJrL16aefDnroWnJy8oj4iPb15OfnKzMzM+q19NKlS2ptbXX9tXRYXQlJUlVVlZ555hmVlJTo/vvvV2Njo7q6urRy5Urr0W6q8vJyNTU16c0335TX641cHfp8PqWmphpPd3N5vd5B74XdcsstmjRp0oh7j+z555/XrFmzVFNTo29961v68MMP1djYqMbGRuvRTCxatEhbtmxRbm6upk2bpiNHjqi2tlYrVqywHi3m+vr6dOrUqcjXZ86cUWdnpyZOnKjc3FxVVlaqpqZGBQUFKigoUE1NjcaNG6enn37a3UFc/axdnPjlL3/p5OXlOWPHjnXuu+++EfmxZElDbjt37rQeLS6M1I9oO47j/O53v3OKioocj8fjFBYWOo2NjdYjmQmFQk5FRYWTm5vrpKSkOFOmTHE2bNjghMNh69Fi7k9/+tOQrxHLly93HOdfH9PeuHGjk5mZ6Xg8HufBBx90jh075vocPMoBAGBmWL0nBABILEQIAGCGCAEAzBAhAIAZIgQAMEOEAABmiBAAwAwRAgCYIUIAADNECABghggBAMwQIQCAmf8HQqOxdkrNcxQAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "classes = list(set(sf_mat.reshape(-1,)))\n",
    "n_class = len(classes)\n",
    "\n",
    "colors = np.random.rand(n_class,4)\n",
    "dic = {}\n",
    "for i in range(n_class):\n",
    "    dic[classes[i]] = colors[i]\n",
    "    \n",
    "\n",
    "img = np.zeros((num,num,4))\n",
    "for i in range(num):\n",
    "    for j in range(num):\n",
    "        img[i][j] = dic[sf_mat[i][j]]\n",
    "plt.imshow(img)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "16bfe1f1",
   "metadata": {},
   "source": [
    "Does this mean symbolic regression is screwed? The hope is that by incorporating reasonable inductive biases (hence reducing the symbolic search space), SR will become more robust."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "id": "39598bda",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "dict_keys(['x', 'x^2', 'x^3', 'x^4', '1/x', '1/x^2', '1/x^3', '1/x^4', 'sqrt', '1/sqrt(x)', 'exp', 'log', 'abs', 'sin', 'tan', 'tanh', 'sigmoid', 'sgn', 'arcsin', 'arctan', 'arctanh', '0', 'gaussian', 'cosh'])"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# we have used the default symbolic library whch contains the following functions\n",
    "SYMBOLIC_LIB.keys()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "id": "61234166",
   "metadata": {},
   "outputs": [],
   "source": [
    "# we may constrain to a smaller library (pass as parameter \"lib=lib\" in suggest_symbolic)\n",
    "lib = ['exp', 'x^2', 'sin']\n",
    "def mix(a, b, bound=1):\n",
    "    num_pts = 101\n",
    "    x = torch.linspace(-bound,bound,steps=num_pts)\n",
    "    x = x[:,None]\n",
    "    y = a * torch.sin(x) + b * x**2 + (1-a-b) * torch.exp(x)\n",
    "    dataset = {}\n",
    "    dataset['train_input'] = dataset['test_input'] = x\n",
    "    dataset['train_label'] = dataset['test_label'] = y\n",
    "    model = KAN(width=[1,1], grid=10, k=3, seed=0, grid_range=(-bound,bound))\n",
    "    model.train(dataset, opt=\"LBFGS\", steps=20)\n",
    "    return model.suggest_symbolic(0,0,0,lib=lib)[0]\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "id": "908b77ea",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "train loss: 2.17e-08 | test loss: 2.17e-08 | reg: 2.58e-01 : 100%|██| 20/20 [00:00<00:00, 45.44it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "function , r2\n",
      "exp , 0.9999999999999639\n",
      "x^2 , 0.9999841274399789\n",
      "sin , 0.9999195962429422\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "train loss: 2.02e-08 | test loss: 2.02e-08 | reg: 2.30e-01 : 100%|██| 20/20 [00:00<00:00, 43.80it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "function , r2\n",
      "exp , 0.9999940363902637\n",
      "x^2 , 0.9999842116986534\n",
      "sin , 0.9999055408903353\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "train loss: 1.87e-08 | test loss: 1.87e-08 | reg: 2.03e-01 : 100%|██| 20/20 [00:00<00:00, 30.85it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "function , r2\n",
      "x^2 , 0.9999842858440444\n",
      "exp , 0.9999260405306123\n",
      "sin , 0.999886385951903\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "train loss: 1.71e-08 | test loss: 1.71e-08 | reg: 1.76e-01 : 100%|██| 20/20 [00:00<00:00, 40.10it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "function , r2\n",
      "x^2 , 0.9999843962720248\n",
      "exp , 0.9999007440044174\n",
      "sin , 0.9998593570241779\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "train loss: 1.56e-08 | test loss: 1.56e-08 | reg: 1.50e-01 : 100%|██| 20/20 [00:00<00:00, 40.88it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "function , r2\n",
      "x^2 , 0.9999845742894306\n",
      "sin , 0.9998195334563984\n",
      "exp , 0.9996560303116143\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "train loss: 1.41e-08 | test loss: 1.41e-08 | reg: 1.24e-01 : 100%|██| 20/20 [00:00<00:00, 37.10it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "function , r2\n",
      "x^2 , 0.9999848839841157\n",
      "sin , 0.9997576089235269\n",
      "exp , 0.9990747506002051\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "train loss: 1.26e-08 | test loss: 1.26e-08 | reg: 9.83e-02 : 100%|██| 20/20 [00:00<00:00, 38.95it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "function , r2\n",
      "x^2 , 0.9999854490365007\n",
      "sin , 0.9996549423479186\n",
      "exp , 0.9970989669443501\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "train loss: 1.11e-08 | test loss: 1.11e-08 | reg: 7.39e-02 : 100%|██| 20/20 [00:00<00:00, 42.44it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "function , r2\n",
      "x^2 , 0.9999865646301423\n",
      "sin , 0.9994730789192812\n",
      "exp , 0.9895624276437505\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "train loss: 9.54e-09 | test loss: 9.54e-09 | reg: 5.21e-02 : 100%|██| 20/20 [00:00<00:00, 46.35it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "function , r2\n",
      "x^2 , 0.9999889651827465\n",
      "sin , 0.999120594481087\n",
      "exp , 0.9559414617900991\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "train loss: 8.03e-09 | test loss: 8.03e-09 | reg: 4.00e-02 : 100%|██| 20/20 [00:00<00:00, 48.92it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "function , r2\n",
      "x^2 , 0.9999924793158511\n",
      "sin , 0.9987167102775933\n",
      "exp , 0.7906888762229085\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "train loss: 6.57e-09 | test loss: 6.57e-09 | reg: 3.95e-02 : 100%|██| 20/20 [00:00<00:00, 34.37it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "function , r2\n",
      "x^2 , 0.9999982519802602\n",
      "sin , 0.9975488884124427\n",
      "exp , 0.2695851370154267\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "train loss: 2.04e-08 | test loss: 2.04e-08 | reg: 2.55e-01 : 100%|██| 20/20 [00:00<00:00, 35.39it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "function , r2\n",
      "x^2 , 0.9999897864586992\n",
      "exp , 0.999941148365934\n",
      "sin , 0.999933060918167\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "train loss: 1.88e-08 | test loss: 1.88e-08 | reg: 2.28e-01 : 100%|██| 20/20 [00:00<00:00, 36.69it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "function , r2\n",
      "x^2 , 0.9999903651856844\n",
      "exp , 0.9999481583538449\n",
      "sin , 0.9999215913152464\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "train loss: 1.73e-08 | test loss: 1.73e-08 | reg: 2.01e-01 : 100%|██| 20/20 [00:00<00:00, 42.49it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "function , r2\n",
      "x^2 , 0.9999910667256126\n",
      "exp , 0.9999745307384252\n",
      "sin , 0.9999059037181064\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "train loss: 1.58e-08 | test loss: 1.58e-08 | reg: 1.75e-01 : 100%|██| 20/20 [00:00<00:00, 40.01it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "function , r2\n",
      "x^2 , 0.9999919340015667\n",
      "sin , 0.9998836753524568\n",
      "exp , 0.9998682812651544\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "train loss: 1.43e-08 | test loss: 1.43e-08 | reg: 1.49e-01 : 100%|██| 20/20 [00:00<00:00, 44.22it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "function , r2\n",
      "x^2 , 0.9999930282260644\n",
      "sin , 0.9998507984084428\n",
      "exp , 0.9997376724009996\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "train loss: 1.28e-08 | test loss: 1.28e-08 | reg: 1.23e-01 : 100%|██| 20/20 [00:00<00:00, 42.51it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "function , r2\n",
      "x^2 , 0.9999944249680105\n",
      "sin , 0.9997993496969405\n",
      "exp , 0.9992295248908697\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "train loss: 1.12e-08 | test loss: 1.12e-08 | reg: 9.78e-02 : 100%|██| 20/20 [00:00<00:00, 34.00it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "function , r2\n",
      "x^2 , 0.9999962146964306\n",
      "sin , 0.9997131766871225\n",
      "exp , 0.9975441981216764\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "train loss: 9.73e-09 | test loss: 9.73e-09 | reg: 7.42e-02 : 100%|██| 20/20 [00:00<00:00, 48.14it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "function , r2\n",
      "x^2 , 0.9999983708153872\n",
      "sin , 0.9995575032145757\n",
      "exp , 0.9909835169112797\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "train loss: 8.22e-09 | test loss: 8.22e-09 | reg: 5.33e-02 : 100%|██| 20/20 [00:00<00:00, 37.91it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "function , r2\n",
      "x^2 , 0.9999990354001381\n",
      "sin , 0.9992662822226276\n",
      "exp , 0.960527319751894\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "train loss: 6.73e-09 | test loss: 6.73e-09 | reg: 4.04e-02 : 100%|██| 20/20 [00:00<00:00, 40.84it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "function , r2\n",
      "x^2 , 0.9999913264861179\n",
      "sin , 0.9988721445986375\n",
      "exp , 0.80011712829603\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "train loss: 5.20e-09 | test loss: 5.20e-09 | reg: 4.08e-02 : 100%|██| 20/20 [00:00<00:00, 40.37it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "function , r2\n",
      "x^2 , 0.9998163363479399\n",
      "sin , 0.9978431561059808\n",
      "exp , 0.2814546223704926\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "train loss: 1.90e-08 | test loss: 1.90e-08 | reg: 2.53e-01 : 100%|██| 20/20 [00:00<00:00, 43.20it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "function , r2\n",
      "exp , 0.9999999483032596\n",
      "x^2 , 0.9999941967836338\n",
      "sin , 0.9999450777771645\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "train loss: 1.75e-08 | test loss: 1.75e-08 | reg: 2.26e-01 : 100%|██| 20/20 [00:00<00:00, 36.22it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "function , r2\n",
      "exp , 0.9999968986658919\n",
      "x^2 , 0.9999950270576494\n",
      "sin , 0.9999359202086953\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "train loss: 1.60e-08 | test loss: 1.60e-08 | reg: 2.00e-01 : 100%|██| 20/20 [00:00<00:00, 46.59it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "function , r2\n",
      "x^2 , 0.9999959671234449\n",
      "exp , 0.9999689579268118\n",
      "sin , 0.9999233338974699\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "train loss: 1.45e-08 | test loss: 1.45e-08 | reg: 1.73e-01 : 100%|██| 20/20 [00:00<00:00, 39.89it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "function , r2\n",
      "x^2 , 0.9999970388609115\n",
      "exp , 0.9999277672142578\n",
      "sin , 0.9999054264402666\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "train loss: 1.30e-08 | test loss: 1.30e-08 | reg: 1.48e-01 : 100%|██| 20/20 [00:00<00:00, 43.91it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "function , r2\n",
      "x^2 , 0.9999981866112407\n",
      "sin , 0.9998787853748133\n",
      "exp , 0.9997940227934969\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "train loss: 1.14e-08 | test loss: 1.14e-08 | reg: 1.22e-01 : 100%|██| 20/20 [00:00<00:00, 43.28it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "function , r2\n",
      "x^2 , 0.9999993305928151\n",
      "sin , 0.999836805899482\n",
      "exp , 0.9993569516643317\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "train loss: 9.92e-09 | test loss: 9.92e-09 | reg: 9.77e-02 : 100%|██| 20/20 [00:00<00:00, 43.76it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "function , r2\n",
      "x^2 , 0.9999999767252477\n",
      "sin , 0.9997657565532695\n",
      "exp , 0.9979264009346674\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "train loss: 8.41e-09 | test loss: 8.41e-09 | reg: 7.48e-02 : 100%|██| 20/20 [00:00<00:00, 48.04it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "function , r2\n",
      "x^2 , 0.9999970458743549\n",
      "sin , 0.9996348080941417\n",
      "exp , 0.9922256134147542\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "train loss: 6.90e-09 | test loss: 6.90e-09 | reg: 5.51e-02 : 100%|██| 20/20 [00:00<00:00, 44.51it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "function , r2\n",
      "x^2 , 0.9999882011585163\n",
      "sin , 0.9993792035049228\n",
      "exp , 0.9648688511956083\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "train loss: 5.37e-09 | test loss: 5.37e-09 | reg: 4.22e-02 : 100%|██| 20/20 [00:00<00:00, 40.25it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "function , r2\n",
      "x^2 , 0.9999431478128625\n",
      "sin , 0.9989853642931659\n",
      "exp , 0.8099064637358544\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "train loss: 3.88e-09 | test loss: 3.88e-09 | reg: 4.35e-02 : 100%|██| 20/20 [00:00<00:00, 44.15it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "function , r2\n",
      "x^2 , 0.9991878132353741\n",
      "sin , 0.9979687146035487\n",
      "exp , 0.2948244018433414\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "train loss: 1.77e-08 | test loss: 1.77e-08 | reg: 2.51e-01 : 100%|██| 20/20 [00:00<00:00, 40.99it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "function , r2\n",
      "x^2 , 0.9999973837949323\n",
      "sin , 0.9999556108208976\n",
      "exp , 0.9999423526826876\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "train loss: 1.62e-08 | test loss: 1.62e-08 | reg: 2.25e-01 : 100%|██| 20/20 [00:00<00:00, 36.59it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "function , r2\n",
      "x^2 , 0.9999981726743682\n",
      "sin , 0.9999484842856431\n",
      "exp , 0.9999473753358622\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "train loss: 1.47e-08 | test loss: 1.47e-08 | reg: 1.98e-01 : 100%|██| 20/20 [00:00<00:00, 44.23it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "function , r2\n",
      "x^2 , 0.9999989595646115\n",
      "exp , 0.9999492978808466\n",
      "sin , 0.9999386215876175\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "train loss: 1.32e-08 | test loss: 1.32e-08 | reg: 1.72e-01 : 100%|██| 20/20 [00:00<00:00, 44.73it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "function , r2\n",
      "x^2 , 0.9999996254779309\n",
      "exp , 0.9999403199854738\n",
      "sin , 0.9999245063324657\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "train loss: 1.16e-08 | test loss: 1.16e-08 | reg: 1.47e-01 : 100%|██| 20/20 [00:00<00:00, 53.99it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "function , r2\n",
      "x^2 , 0.9999999902142925\n",
      "sin , 0.9999033788427966\n",
      "exp , 0.9998233862273559\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "train loss: 1.01e-08 | test loss: 1.01e-08 | reg: 1.22e-01 : 100%|██| 20/20 [00:00<00:00, 43.09it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "function , r2\n",
      "x^2 , 0.9999993945260922\n",
      "sin , 0.9998698461072781\n",
      "exp , 0.9994604176643853\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "train loss: 8.60e-09 | test loss: 8.60e-09 | reg: 9.78e-02 : 100%|██| 20/20 [00:00<00:00, 40.29it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "function , r2\n",
      "x^2 , 0.9999962624913747\n",
      "sin , 0.999812394838402\n",
      "exp , 0.9982470612489773\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "train loss: 7.09e-09 | test loss: 7.09e-09 | reg: 7.58e-02 : 100%|██| 20/20 [00:00<00:00, 39.31it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "function , r2\n",
      "x^2 , 0.9999855256919693\n",
      "sin , 0.9997042323555781\n",
      "exp , 0.9934084524703305\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "train loss: 5.56e-09 | test loss: 5.56e-09 | reg: 5.74e-02 : 100%|██| 20/20 [00:00<00:00, 48.21it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "function , r2\n",
      "x^2 , 0.9999509121969333\n",
      "sin , 0.9994826743156423\n",
      "exp , 0.9690407199009088\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "train loss: 4.05e-09 | test loss: 4.05e-09 | reg: 4.54e-02 : 100%|██| 20/20 [00:00<00:00, 39.78it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "function , r2\n",
      "x^2 , 0.9998306455096987\n",
      "sin , 0.9990671369324535\n",
      "exp , 0.8200102819209695\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "train loss: 2.55e-09 | test loss: 2.55e-09 | reg: 4.76e-02 : 100%|██| 20/20 [00:00<00:00, 36.39it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "function , r2\n",
      "x^2 , 0.9979550584268881\n",
      "sin , 0.9978931765377275\n",
      "exp , 0.30997176980589874\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "train loss: 1.64e-08 | test loss: 1.64e-08 | reg: 2.50e-01 : 100%|██| 20/20 [00:00<00:00, 40.35it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "function , r2\n",
      "exp , 0.9999995435772306\n",
      "x^2 , 0.9999993202720766\n",
      "sin , 0.9999646329733246\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "train loss: 1.49e-08 | test loss: 1.49e-08 | reg: 2.23e-01 : 100%|██| 20/20 [00:00<00:00, 41.62it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "function , r2\n",
      "x^2 , 0.999999776416721\n",
      "exp , 0.9999936654595362\n",
      "sin , 0.9999592519777621\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "train loss: 1.34e-08 | test loss: 1.34e-08 | reg: 1.97e-01 : 100%|██| 20/20 [00:00<00:00, 44.66it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "function , r2\n",
      "x^2 , 0.9999999292547446\n",
      "exp , 0.9999841395212624\n",
      "sin , 0.999951727445816\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "train loss: 1.19e-08 | test loss: 1.19e-08 | reg: 1.71e-01 : 100%|██| 20/20 [00:00<00:00, 44.92it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "function , r2\n",
      "x^2 , 0.9999996917196639\n",
      "sin , 0.9999408710455521\n",
      "exp , 0.9999069779386928\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "train loss: 1.03e-08 | test loss: 1.03e-08 | reg: 1.46e-01 : 100%|██| 20/20 [00:00<00:00, 47.71it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "function , r2\n",
      "x^2 , 0.9999982862472424\n",
      "sin , 0.999924500202756\n",
      "exp , 0.9998279543178681\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "train loss: 8.80e-09 | test loss: 8.80e-09 | reg: 1.21e-01 : 100%|██| 20/20 [00:00<00:00, 42.92it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "function , r2\n",
      "x^2 , 0.999994399261074\n",
      "sin , 0.9998982872842133\n",
      "exp , 0.9995430618744594\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "train loss: 7.28e-09 | test loss: 7.28e-09 | reg: 9.82e-02 : 100%|██| 20/20 [00:00<00:00, 44.87it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "function , r2\n",
      "x^2 , 0.9999846769079466\n",
      "sin , 0.9998527941960059\n",
      "exp , 0.998507365877467\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "train loss: 5.76e-09 | test loss: 5.76e-09 | reg: 7.72e-02 : 100%|██| 20/20 [00:00<00:00, 40.42it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "function , r2\n",
      "x^2 , 0.9999591064202273\n",
      "sin , 0.9997651030691422\n",
      "exp , 0.994405837621339\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "train loss: 4.24e-09 | test loss: 4.24e-09 | reg: 6.02e-02 : 100%|██| 20/20 [00:00<00:00, 42.98it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "function , r2\n",
      "x^2 , 0.9998837805081656\n",
      "sin , 0.9995754338853143\n",
      "exp , 0.972913947729953\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "train loss: 2.73e-09 | test loss: 2.73e-09 | reg: 5.00e-02 : 100%|██| 20/20 [00:00<00:00, 41.06it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "function , r2\n",
      "x^2 , 0.9996498645481074\n",
      "sin , 0.9991201921166716\n",
      "exp , 0.8303506225546471\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "train loss: 1.24e-09 | test loss: 1.24e-09 | reg: 5.35e-02 : 100%|██| 20/20 [00:00<00:00, 33.88it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "function , r2\n",
      "sin , 0.9975671382600958\n",
      "x^2 , 0.9959025139156165\n",
      "exp , 0.3272398323990947\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "train loss: 1.51e-08 | test loss: 1.51e-08 | reg: 2.48e-01 : 100%|██| 20/20 [00:00<00:00, 33.97it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "function , r2\n",
      "x^2 , 0.9999999273977364\n",
      "sin , 0.999972137087767\n",
      "exp , 0.9999416230314031\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "train loss: 1.36e-08 | test loss: 1.36e-08 | reg: 2.22e-01 : 100%|██| 20/20 [00:00<00:00, 40.23it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "function , r2\n",
      "x^2 , 0.9999998145393945\n",
      "sin , 0.9999682142467496\n",
      "exp , 0.9999659993562859\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "train loss: 1.21e-08 | test loss: 1.21e-08 | reg: 1.96e-01 : 100%|██| 20/20 [00:00<00:00, 44.89it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "function , r2\n",
      "x^2 , 0.9999990327979635\n",
      "sin , 0.9999626403806093\n",
      "exp , 0.9999479302673148\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "train loss: 1.05e-08 | test loss: 1.05e-08 | reg: 1.71e-01 : 100%|██| 20/20 [00:00<00:00, 41.24it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "function , r2\n",
      "x^2 , 0.9999971222660284\n",
      "sin , 0.9999545031184953\n",
      "exp , 0.9999322662928528\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "train loss: 9.01e-09 | test loss: 9.01e-09 | reg: 1.46e-01 : 100%|██| 20/20 [00:00<00:00, 43.63it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "function , r2\n",
      "x^2 , 0.9999928506457437\n",
      "sin , 0.9999421177337516\n",
      "exp , 0.9998097288010696\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "train loss: 7.49e-09 | test loss: 7.49e-09 | reg: 1.21e-01 : 100%|██| 20/20 [00:00<00:00, 44.46it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "function , r2\n",
      "x^2 , 0.9999822360404783\n",
      "sin , 0.9999220769608526\n",
      "exp , 0.9996077177342251\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "train loss: 5.96e-09 | test loss: 5.96e-09 | reg: 9.89e-02 : 100%|██| 20/20 [00:00<00:00, 53.97it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "function , r2\n",
      "x^2 , 0.999964802453253\n",
      "sin , 0.9998867957502443\n",
      "exp , 0.9987081345779585\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "train loss: 4.44e-09 | test loss: 4.44e-09 | reg: 7.88e-02 : 100%|██| 20/20 [00:00<00:00, 43.92it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "function , r2\n",
      "x^2 , 0.9999175944145272\n",
      "sin , 0.9998169466724585\n",
      "exp , 0.9952629708219127\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "train loss: 2.92e-09 | test loss: 2.92e-09 | reg: 6.33e-02 : 100%|██| 20/20 [00:00<00:00, 40.19it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "function , r2\n",
      "x^2 , 0.9997839395304963\n",
      "sin , 0.9996559816500792\n",
      "exp , 0.9765461190280487\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "train loss: 1.43e-09 | test loss: 1.43e-09 | reg: 5.57e-02 : 100%|██| 20/20 [00:00<00:00, 31.57it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "function , r2\n",
      "x^2 , 0.9993615297170999\n",
      "sin , 0.9992162433614376\n",
      "exp , 0.8408048845759444\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "train loss: 3.44e-10 | test loss: 3.44e-10 | reg: 6.15e-02 : 100%|██| 20/20 [00:00<00:00, 23.83it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "function , r2\n",
      "sin , 0.9969431394468133\n",
      "x^2 , 0.9927325368990941\n",
      "exp , 0.34705555016523576\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "train loss: 1.38e-08 | test loss: 1.38e-08 | reg: 2.47e-01 : 100%|██| 20/20 [00:00<00:00, 27.06it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "function , r2\n",
      "x^2 , 0.9999993835845823\n",
      "exp , 0.9999979009384041\n",
      "sin , 0.9999781529177164\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "train loss: 1.23e-08 | test loss: 1.23e-08 | reg: 2.21e-01 : 100%|██| 20/20 [00:00<00:00, 28.15it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "function , r2\n",
      "x^2 , 0.9999982596377558\n",
      "exp , 0.9999826642402296\n",
      "sin , 0.999975403004672\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "train loss: 1.07e-08 | test loss: 1.07e-08 | reg: 1.95e-01 : 100%|██| 20/20 [00:00<00:00, 28.02it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "function , r2\n",
      "x^2 , 0.9999960480978767\n",
      "sin , 0.9999713985665561\n",
      "exp , 0.9999678693890908\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "train loss: 9.22e-09 | test loss: 9.22e-09 | reg: 1.70e-01 : 100%|██| 20/20 [00:00<00:00, 35.18it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "function , r2\n",
      "x^2 , 0.999991537192374\n",
      "sin , 0.999965455283627\n",
      "exp , 0.9999527752253392\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "train loss: 7.70e-09 | test loss: 7.70e-09 | reg: 1.45e-01 : 100%|██| 20/20 [00:00<00:00, 30.27it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "function , r2\n",
      "x^2 , 0.9999840061625307\n",
      "sin , 0.9999562904170857\n",
      "exp , 0.999848705695026\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "train loss: 6.17e-09 | test loss: 6.17e-09 | reg: 1.22e-01 : 100%|██| 20/20 [00:00<00:00, 24.01it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "function , r2\n",
      "x^2 , 0.9999686012309674\n",
      "sin , 0.9999412814570412\n",
      "exp , 0.9996568552973389\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "train loss: 4.65e-09 | test loss: 4.65e-09 | reg: 9.98e-02 : 100%|██| 20/20 [00:00<00:00, 28.32it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "function , r2\n",
      "x^2 , 0.999936210582381\n",
      "sin , 0.999914407864863\n",
      "exp , 0.998905881229885\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "train loss: 3.14e-09 | test loss: 3.13e-09 | reg: 8.08e-02 : 100%|██| 20/20 [00:00<00:00, 37.47it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "function , r2\n",
      "x^2 , 0.9998600127039595\n",
      "sin , 0.9998596147774699\n",
      "exp , 0.9960274461980388\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "train loss: 1.63e-09 | test loss: 1.63e-09 | reg: 6.69e-02 : 100%|██| 20/20 [00:00<00:00, 39.89it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "function , r2\n",
      "sin , 0.9997236129916397\n",
      "x^2 , 0.9996456211560416\n",
      "exp , 0.9798815731174731\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "train loss: 3.91e-10 | test loss: 3.91e-10 | reg: 6.21e-02 : 100%|██| 20/20 [00:00<00:00, 37.63it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "function , r2\n",
      "sin , 0.9992448590848789\n",
      "x^2 , 0.9989434205375304\n",
      "exp , 0.851363652171008\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "train loss: 1.48e-09 | test loss: 1.48e-09 | reg: 6.99e-02 : 100%|██| 20/20 [00:00<00:00, 41.08it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "function , r2\n",
      "sin , 0.9959631103992738\n",
      "x^2 , 0.9880298934280559\n",
      "exp , 0.3699525677002401\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "train loss: 1.25e-08 | test loss: 1.25e-08 | reg: 2.46e-01 : 100%|██| 20/20 [00:00<00:00, 37.73it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "function , r2\n",
      "x^2 , 0.9999974755376599\n",
      "sin , 0.999982771618615\n",
      "exp , 0.9999378510072501\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "train loss: 1.10e-08 | test loss: 1.10e-08 | reg: 2.20e-01 : 100%|██| 20/20 [00:00<00:00, 46.47it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "function , r2\n",
      "x^2 , 0.9999950949412941\n",
      "sin , 0.9999809179819591\n",
      "exp , 0.9999775858852623\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "train loss: 9.43e-09 | test loss: 9.43e-09 | reg: 1.94e-01 : 100%|██| 20/20 [00:00<00:00, 48.24it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "function , r2\n",
      "x^2 , 0.9999909995161567\n",
      "sin , 0.99997811973606\n",
      "exp , 0.9999745650796973\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "train loss: 7.91e-09 | test loss: 7.91e-09 | reg: 1.69e-01 : 100%|██| 20/20 [00:00<00:00, 40.15it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "function , r2\n",
      "x^2 , 0.9999839054191747\n",
      "sin , 0.9999738708591638\n",
      "exp , 0.9999301819009739\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "train loss: 6.39e-09 | test loss: 6.39e-09 | reg: 1.45e-01 : 100%|██| 20/20 [00:00<00:00, 49.67it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "function , r2\n",
      "x^2 , 0.9999712511423499\n",
      "sin , 0.9999672116732772\n",
      "exp , 0.9998759476535651\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "train loss: 4.87e-09 | test loss: 4.87e-09 | reg: 1.22e-01 : 100%|██| 20/20 [00:00<00:00, 41.32it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "function , r2\n",
      "sin , 0.9999561449575607\n",
      "x^2 , 0.9999474340188654\n",
      "exp , 0.9996925240661405\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "train loss: 3.36e-09 | test loss: 3.36e-09 | reg: 1.01e-01 : 100%|██| 20/20 [00:00<00:00, 37.69it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "function , r2\n",
      "sin , 0.9999359791634311\n",
      "x^2 , 0.9998987103013962\n",
      "exp , 0.9990779156635234\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "train loss: 1.86e-09 | test loss: 1.86e-09 | reg: 8.32e-02 : 100%|██| 20/20 [00:00<00:00, 41.55it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "function , r2\n",
      "sin , 0.999893422872044\n",
      "x^2 , 0.9997851521886209\n",
      "exp , 0.9966235714622826\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "train loss: 5.79e-10 | test loss: 5.79e-10 | reg: 7.13e-02 : 100%|██| 20/20 [00:00<00:00, 39.52it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "function , r2\n",
      "sin , 0.9997784124877638\n",
      "x^2 , 0.9994637170628504\n",
      "exp , 0.9828804558908593\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "train loss: 1.33e-09 | test loss: 1.33e-09 | reg: 6.87e-02 : 100%|██| 20/20 [00:00<00:00, 37.55it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "function , r2\n",
      "sin , 0.9992376323441506\n",
      "x^2 , 0.9983591814967973\n",
      "exp , 0.862391351818434\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "train loss: 2.79e-09 | test loss: 2.79e-09 | reg: 7.94e-02 : 100%|██| 20/20 [00:00<00:00, 39.73it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "function , r2\n",
      "sin , 0.9944110485646586\n",
      "x^2 , 0.9816962474224649\n",
      "exp , 0.3965991904252684\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "train loss: 1.12e-08 | test loss: 1.12e-08 | reg: 2.45e-01 : 100%|██| 20/20 [00:00<00:00, 30.38it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "function , r2\n",
      "x^2 , 0.9999942564460803\n",
      "exp , 0.9999934337977243\n",
      "sin , 0.9999861648364792\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "train loss: 9.65e-09 | test loss: 9.65e-09 | reg: 2.19e-01 : 100%|██| 20/20 [00:00<00:00, 35.73it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "function , r2\n",
      "x^2 , 0.9999902923652078\n",
      "sin , 0.9999849518514359\n",
      "exp , 0.9999613659020306\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "train loss: 8.13e-09 | test loss: 8.13e-09 | reg: 1.94e-01 : 100%|██| 20/20 [00:00<00:00, 31.24it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "function , r2\n",
      "x^2 , 0.9999837274020729\n",
      "sin , 0.9999830308299852\n",
      "exp , 0.9999154398628454\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "train loss: 6.61e-09 | test loss: 6.61e-09 | reg: 1.69e-01 : 100%|██| 20/20 [00:00<00:00, 28.46it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "function , r2\n",
      "sin , 0.9999800294497047\n",
      "x^2 , 0.9999731551293094\n",
      "exp , 0.9999043625986571\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "train loss: 5.09e-09 | test loss: 5.09e-09 | reg: 1.45e-01 : 100%|██| 20/20 [00:00<00:00, 36.01it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "function , r2\n",
      "sin , 0.9999752362233539\n",
      "x^2 , 0.9999543237918428\n",
      "exp , 0.9998837024119729\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "train loss: 3.59e-09 | test loss: 3.59e-09 | reg: 1.23e-01 : 100%|██| 20/20 [00:00<00:00, 33.71it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "function , r2\n",
      "sin , 0.9999671472397126\n",
      "x^2 , 0.999920520242572\n",
      "exp , 0.9997162974965994\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "train loss: 2.11e-09 | test loss: 2.11e-09 | reg: 1.02e-01 : 100%|██| 20/20 [00:00<00:00, 29.70it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "function , r2\n",
      "sin , 0.9999521118045746\n",
      "x^2 , 0.9998517198959569\n",
      "exp , 0.9991994724291818\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "train loss: 8.18e-10 | test loss: 8.18e-10 | reg: 8.60e-02 : 100%|██| 20/20 [00:00<00:00, 27.73it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "function , r2\n",
      "sin , 0.9999191931580688\n",
      "x^2 , 0.9996922419099825\n",
      "exp , 0.9971630293162339\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "train loss: 1.23e-09 | test loss: 1.23e-09 | reg: 7.58e-02 : 100%|██| 20/20 [00:00<00:00, 27.04it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "function , r2\n",
      "sin , 0.9998214259046335\n",
      "x^2 , 0.9992365904461334\n",
      "exp , 0.9855351478460005\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "train loss: 2.62e-09 | test loss: 2.62e-09 | reg: 7.77e-02 : 100%|██| 20/20 [00:00<00:00, 24.95it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "function , r2\n",
      "sin , 0.9992094653374828\n",
      "x^2 , 0.997561488327335\n",
      "exp , 0.8738196605080718\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "train loss: 4.10e-09 | test loss: 4.10e-09 | reg: 9.07e-02 : 100%|██| 20/20 [00:00<00:00, 34.69it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "function , r2\n",
      "sin , 0.9922364482738998\n",
      "x^2 , 0.9865738449440605\n",
      "exp , 0.42783037973898724\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "train loss: 9.87e-09 | test loss: 9.87e-09 | reg: 2.44e-01 : 100%|██| 20/20 [00:00<00:00, 48.50it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "function , r2\n",
      "x^2 , 0.9999896926875822\n",
      "sin , 0.9999885782292838\n",
      "exp , 0.9999292279786546\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "train loss: 8.35e-09 | test loss: 8.35e-09 | reg: 2.18e-01 : 100%|██| 20/20 [00:00<00:00, 33.16it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "function , r2\n",
      "sin , 0.9999877872886065\n",
      "x^2 , 0.9999838323248553\n",
      "exp , 0.9999789165976448\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "train loss: 6.84e-09 | test loss: 6.84e-09 | reg: 1.93e-01 : 100%|██| 20/20 [00:00<00:00, 46.33it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "function , r2\n",
      "sin , 0.9999864664396515\n",
      "x^2 , 0.9999745891023756\n",
      "exp , 0.9999649555496989\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "train loss: 5.32e-09 | test loss: 5.32e-09 | reg: 1.69e-01 : 100%|██| 20/20 [00:00<00:00, 40.58it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "function , r2\n",
      "sin , 0.9999843410323541\n",
      "x^2 , 0.9999595824867687\n",
      "exp , 0.9999363993653761\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "train loss: 3.83e-09 | test loss: 3.83e-09 | reg: 1.46e-01 : 100%|██| 20/20 [00:00<00:00, 41.01it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "function , r2\n",
      "sin , 0.9999808836617524\n",
      "x^2 , 0.9999340658950667\n",
      "exp , 0.9998727806292239\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "train loss: 2.36e-09 | test loss: 2.36e-09 | reg: 1.24e-01 : 100%|██| 20/20 [00:00<00:00, 45.26it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "function , r2\n",
      "sin , 0.9999749630672443\n",
      "x^2 , 0.9998877158288519\n",
      "exp , 0.9997292201760375\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "train loss: 1.08e-09 | test loss: 1.08e-09 | reg: 1.04e-01 : 100%|██| 20/20 [00:00<00:00, 40.71it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "function , r2\n",
      "sin , 0.9999637514193707\n",
      "x^2 , 0.9997949968711753\n",
      "exp , 0.9992688918286982\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "train loss: 1.18e-09 | test loss: 1.18e-09 | reg: 8.91e-02 : 100%|██| 20/20 [00:00<00:00, 40.07it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "function , r2\n",
      "sin , 0.9999382587554829\n",
      "x^2 , 0.9995804083543193\n",
      "exp , 0.997562547933165\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "train loss: 2.49e-09 | test loss: 2.49e-09 | reg: 8.13e-02 : 100%|██| 20/20 [00:00<00:00, 39.32it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "function , r2\n",
      "sin , 0.9998545572827711\n",
      "x^2 , 0.9989551972006018\n",
      "exp , 0.9878365883665239\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "train loss: 3.94e-09 | test loss: 3.94e-09 | reg: 8.77e-02 : 100%|██| 20/20 [00:00<00:00, 41.76it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "function , r2\n",
      "sin , 0.999179162081282\n",
      "x^2 , 0.9964947779569694\n",
      "exp , 0.8855949580343588\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "train loss: 5.41e-09 | test loss: 5.41e-09 | reg: 1.03e-01 : 100%|██| 20/20 [00:00<00:00, 44.07it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "function , r2\n",
      "x^2 , 0.9896743802302694\n",
      "sin , 0.9893409020756886\n",
      "exp , 0.4646785423849215\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "train loss: 8.57e-09 | test loss: 8.57e-09 | reg: 2.43e-01 : 100%|██| 20/20 [00:00<00:00, 37.81it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "function , r2\n",
      "sin , 0.9999902713620692\n",
      "x^2 , 0.9999807202372036\n",
      "exp , 0.9997379079627818\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "train loss: 7.06e-09 | test loss: 7.06e-09 | reg: 2.18e-01 : 100%|██| 20/20 [00:00<00:00, 47.32it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "function , r2\n",
      "sin , 0.999989739250175\n",
      "x^2 , 0.9999756704785013\n",
      "exp , 0.999926363885015\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "train loss: 5.56e-09 | test loss: 5.56e-09 | reg: 1.93e-01 : 100%|██| 20/20 [00:00<00:00, 52.19it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "function , r2\n",
      "sin , 0.9999888074224221\n",
      "x^2 , 0.9999631678786869\n",
      "exp , 0.9999447765899155\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "train loss: 4.07e-09 | test loss: 4.07e-09 | reg: 1.69e-01 : 100%|██| 20/20 [00:00<00:00, 37.65it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "function , r2\n",
      "sin , 0.9999872737863882\n",
      "x^2 , 0.9999431335080622\n",
      "exp , 0.9999263142234696\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "train loss: 2.62e-09 | test loss: 2.62e-09 | reg: 1.46e-01 : 100%|██| 20/20 [00:00<00:00, 30.64it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "function , r2\n",
      "sin , 0.9999847406083843\n",
      "x^2 , 0.999909483042181\n",
      "exp , 0.9998436631999283\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "train loss: 1.34e-09 | test loss: 1.34e-09 | reg: 1.25e-01 : 100%|██| 20/20 [00:00<00:00, 28.07it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "function , r2\n",
      "sin , 0.9999803638564161\n",
      "x^2 , 0.999848886198266\n",
      "exp , 0.9997317587005629\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "train loss: 1.19e-09 | test loss: 1.19e-09 | reg: 1.06e-01 : 100%|██| 20/20 [00:00<00:00, 27.57it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "function , r2\n",
      "sin , 0.9999719462457353\n",
      "x^2 , 0.9997282247718177\n",
      "exp , 0.9992837377086745\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "train loss: 2.37e-09 | test loss: 2.37e-09 | reg: 9.29e-02 : 100%|██| 20/20 [00:00<00:00, 29.79it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "function , r2\n",
      "sin , 0.9999521133602743\n",
      "x^2 , 0.999448434641883\n",
      "exp , 0.9978756985305871\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "train loss: 3.79e-09 | test loss: 3.79e-09 | reg: 8.83e-02 : 100%|██| 20/20 [00:00<00:00, 43.22it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "function , r2\n",
      "sin , 0.99988019998444\n",
      "x^2 , 0.998618508248868\n",
      "exp , 0.9897729427792255\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "train loss: 5.25e-09 | test loss: 5.25e-09 | reg: 9.97e-02 : 100%|██| 20/20 [00:00<00:00, 39.55it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "function , r2\n",
      "sin , 0.9991252607520362\n",
      "x^2 , 0.995078459413144\n",
      "exp , 0.8974993831807665\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "train loss: 6.71e-09 | test loss: 6.71e-09 | reg: 1.16e-01 : 100%|██| 20/20 [00:00<00:00, 42.40it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "function , r2\n",
      "x^2 , 0.9896966425177599\n",
      "sin , 0.985121456003004\n",
      "exp , 0.508387788052642\n"
     ]
    }
   ],
   "source": [
    "# we can redo the analysis for a more contrained (bound) region. The phase diagram becomes even more \"fractal\"\n",
    "num = 11\n",
    "a_arr = np.linspace(0,1,num=num)\n",
    "b_arr = np.linspace(0,1,num=num)\n",
    "sf_mat = np.empty((num,num), dtype='U8')\n",
    "\n",
    "for i in range(num):\n",
    "    for j in range(num):\n",
    "        a = a_arr[i]; b = b_arr[j]\n",
    "        sf_mat[i,j] = mix(a, b, bound=0.3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "id": "759c31f7",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.image.AxesImage at 0x7fd49dd97160>"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAaEAAAGdCAYAAAC7EMwUAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjYuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8o6BhiAAAACXBIWXMAAA9hAAAPYQGoP6dpAAAWc0lEQVR4nO3dfWxW9d348Q9WuQBXqmCgNBasuUlQ8IGBWwR8itoEkcyYufk4olsiARUkcchw82GBDrcR8pOJd/3DsRiU3PemsmRuNm6CBo2IoMYtECaRTkeIC2kBXRnl/P7Yz/rrwOfr4tOW1ys5f/Rcx55PTsz1zrfXxTn9iqIoAgASHJM9AABHLxECII0IAZBGhABII0IApBEhANKIEABpRAiANMdmD/CfDh48GO+++25UV1dHv379sscB4HMqiiL27NkTdXV1ccwxn7zW6XERevfdd6O+vj57DAC+pNbW1jj55JM/8ZgeF6Hq6uqIiJg5b3OUStXJ0+S7Yts3s0cA+Fz2/aszLn/y9a7380/S4yL04Z/gSqXqKA0Qoa8cV5U9AsAX8lk+UvHFBADSiBAAaUQIgDQiBEAaEQIgjQgBkEaEAEgjQgCkESEA0lQsQg8++GA0NDTEgAEDYsKECfH8889X6lQA9FIVidDq1atj7ty5sXDhwti0aVOcd955MXXq1NixY0clTgdAL1WRCC1dujS++93vxve+97047bTTYtmyZVFfXx8rVqyoxOkA6KXKHqH9+/fHxo0bo7Gxsdv+xsbGWL9+/SHHd3R0RHt7e7cNgKND2SP03nvvRWdnZwwfPrzb/uHDh8fOnTsPOb6pqSlqamq6Ns8SAjh6VOyLCf95C++iKA57W+8FCxZEW1tb19ba2lqpkQDoYcr+PKGTTjopqqqqDln17Nq165DVUUREqVSKUqlU7jEA6AXKvhLq379/TJgwIVpaWrrtb2lpiUmTJpX7dAD0YhV5suq8efPihhtuiIkTJ8a5554bzc3NsWPHjpg5c2YlTgdAL1WRCH3729+Of/zjH3HffffF3//+9xg3blz87ne/i1GjRlXidAD0UhWJUETErFmzYtasWZX69QD0Ae4dB0AaEQIgjQgBkEaEAEgjQgCkESEA0ogQAGlECIA0IgRAGhECIE3FbtvzZV2x7ZvxleOqsscAoIKshABII0IApBEhANKIEABpRAiANCIEQBoRAiCNCAGQRoQASCNCAKQRIQDSiBAAaUQIgDQiBEAaEQIgjQgBkEaEAEgjQgCkESEA0ogQAGlECIA0IgRAGhECII0IAZBGhABII0IApBEhANKIEABpRAiANCIEQBoRAiCNCAGQ5tjsAT7Ok//1v1EaUJ09RrqrtlyWPQJAxVgJAZBGhABII0IApBEhANKIEABpRAiANCIEQBoRAiCNCAGQRoQASCNCAKQRIQDSiBAAacoeoaampjjnnHOiuro6hg0bFldccUVs2bKl3KcBoA8oe4TWrl0bs2fPjpdeeilaWlriwIED0djYGPv27Sv3qQDo5cr+PKHf//733X5+5JFHYtiwYbFx48Y4//zzy306AHqxij/Urq2tLSIihgwZctjXOzo6oqOjo+vn9vb2So8EQA9R0S8mFEUR8+bNiylTpsS4ceMOe0xTU1PU1NR0bfX19ZUcCYAepKIRuuWWW+L111+Pxx577GOPWbBgQbS1tXVtra2tlRwJgB6kYn+Ou/XWW2PNmjWxbt26OPnkkz/2uFKpFKVSqVJjANCDlT1CRVHErbfeGk888UQ899xz0dDQUO5TANBHlD1Cs2fPjlWrVsVTTz0V1dXVsXPnzoiIqKmpiYEDB5b7dAD0YmX/TGjFihXR1tYWF154YYwYMaJrW716dblPBUAvV5E/xwHAZ+HecQCkESEA0ogQAGlECIA0IgRAGhECII0IAZBGhABII0IApKn4Q+2+qCu2fTO+clxV9hgAVJCVEABpRAiANCIEQBoRAiCNCAGQRoQASCNCAKQRIQDSiBAAaUQIgDQiBEAaEQIgjQgBkEaEAEgjQgCkESEA0ogQAGlECIA0IgRAGhECII0IAZBGhABII0IApBEhANKIEABpRAiANCIEQBoRAiCNCAGQRoQASCNCAKQRIQDSHJs9wMd58r/+N0oDqlNnuGrLZannB/i8Tlxyc/YIceyeDyL+Z9NnOtZKCIA0IgRAGhECII0IAZBGhABII0IApBEhANKIEABpRAiANCIEQBoRAiCNCAGQRoQASFPxCDU1NUW/fv1i7ty5lT4VAL1MRSO0YcOGaG5ujjPPPLOSpwGgl6pYhPbu3RvXXXddPPzww3HiiSdW6jQA9GIVi9Ds2bNj2rRpcckll3zicR0dHdHe3t5tA+DoUJEnqz7++OPx6quvxoYNGz712Kamprj33nsrMQYAPVzZV0Ktra0xZ86cePTRR2PAgAGfevyCBQuira2ta2ttbS33SAD0UGVfCW3cuDF27doVEyZM6NrX2dkZ69ati+XLl0dHR0dUVVV1vVYqlaJUKpV7DAB6gbJH6OKLL4433nij274bb7wxxowZE/Pnz+8WIACObmWPUHV1dYwbN67bvuOPPz6GDh16yH4Ajm7umABAmop8O+4/Pffcc0fiNAD0MlZCAKQRIQDSiBAAaUQIgDQiBEAaEQIgjQgBkEaEAEgjQgCkOSJ3TPgirtj2zfjKcW52CtCXWQkBkEaEAEgjQgCkESEA0ogQAGlECIA0IgRAGhECII0IAZBGhABII0IApBEhANKIEABpRAiANCIEQBoRAiCNCAGQRoQASCNCAKQRIQDSiBAAaUQIgDQiBEAaEQIgjQgBkEaEAEgjQgCkESEA0ogQAGlECIA0IgRAGhECII0IAZDm2OwBAL6sE5fcnD0CX5CVEABpRAiANCIEQBoRAiCNCAGQRoQASCNCAKQRIQDSiBAAaUQIgDQiBEAaEQIgTUUi9M4778T1118fQ4cOjUGDBsXZZ58dGzdurMSpAOjFyn4X7d27d8fkyZPjoosuiqeffjqGDRsWf/3rX+OEE04o96kA6OXKHqElS5ZEfX19PPLII137TjnllHKfBoA+oOx/jluzZk1MnDgxrrrqqhg2bFiMHz8+Hn744Y89vqOjI9rb27ttABwdyh6ht956K1asWBGjR4+OP/zhDzFz5sy47bbb4le/+tVhj29qaoqampqurb6+vtwjAdBD9SuKoijnL+zfv39MnDgx1q9f37Xvtttuiw0bNsSLL754yPEdHR3R0dHR9XN7e3vU19fHn64aH185rqqcowF9lCer9ix79nwQ40+/Ldra2mLw4MGfeGzZV0IjRoyI008/vdu+0047LXbs2HHY40ulUgwePLjbBsDRoewRmjx5cmzZsqXbvq1bt8aoUaPKfSoAermyR+j222+Pl156KRYvXhzbtm2LVatWRXNzc8yePbvcpwKglyt7hM4555x44okn4rHHHotx48bFj3/841i2bFlcd9115T4VAL1c2f+dUETE5ZdfHpdffnklfjUAfYh7xwGQRoQASCNCAKQRIQDSiBAAaUQIgDQiBEAaEQIgjQgBkKYid0yAvsxjA6B8rIQASCNCAKQRIQDSiBAAaUQIgDQiBEAaEQIgjQgBkEaEAEgjQgCkESEA0ogQAGlECIA0IgRAGhECII0IAZBGhABII0IApBEhANKIEABpRAiANCIEQBoRAiCNCAGQRoQASCNCAKQRIQDSiBAAaUQIgDQiBEAaEQIgjQgBkEaEAEhzbPYA9A4nLrk5ewSgD7ISAiCNCAGQRoQASCNCAKQRIQDSiBAAaUQIgDQiBEAaEQIgjQgBkEaEAEgjQgCkKXuEDhw4EHfddVc0NDTEwIED49RTT4377rsvDh48WO5TAdDLlf0u2kuWLImHHnooVq5cGWPHjo1XXnklbrzxxqipqYk5c+aU+3QA9GJlj9CLL74Y3/jGN2LatGkREXHKKafEY489Fq+88kq5TwVAL1f2P8dNmTIlnn322di6dWtERLz22mvxwgsvxGWXXXbY4zs6OqK9vb3bBsDRoewrofnz50dbW1uMGTMmqqqqorOzMxYtWhTXXHPNYY9vamqKe++9t9xjANALlH0ltHr16nj00Udj1apV8eqrr8bKlSvjZz/7WaxcufKwxy9YsCDa2tq6ttbW1nKPBEAPVfaV0B133BF33nlnXH311RERccYZZ8Tbb78dTU1NMWPGjEOOL5VKUSqVyj0GAL1A2VdC77//fhxzTPdfW1VV5SvaAByi7Cuh6dOnx6JFi2LkyJExduzY2LRpUyxdujRuuummcp8KgF6u7BF64IEH4oc//GHMmjUrdu3aFXV1dXHzzTfHj370o3KfCoBeruwRqq6ujmXLlsWyZcvK/asB6GPcOw6ANCIEQBoRAiCNCAGQRoQASCNCAKQRIQDSiBAAaUQIgDQiBECast+2py85ccnN2SMAfC675/939gix91+dn/lYKyEA0ogQAGlECIA0IgRAGhECII0IAZBGhABII0IApBEhANKIEABpRAiANCIEQBoRAiCNCAGQRoQASCNCAKQRIQDSiBAAaUQIgDQiBEAaEQIgjQgBkEaEAEgjQgCkESEA0ogQAGlECIA0IgRAGhECII0IAZBGhABII0IApDk2e4CPc8K9N0Z19cDsMQCoICshANKIEABpRAiANCIEQBoRAiCNCAGQRoQASCNCAKQRIQDSiBAAaUQIgDQiBECazx2hdevWxfTp06Ouri769esXTz75ZLfXi6KIe+65J+rq6mLgwIFx4YUXxptvvlmueQHoQz53hPbt2xdnnXVWLF++/LCv33///bF06dJYvnx5bNiwIWpra+PSSy+NPXv2fOlhAehbPvejHKZOnRpTp0497GtFUcSyZcti4cKFceWVV0ZExMqVK2P48OGxatWquPnmm7/ctAD0KWX9TGj79u2xc+fOaGxs7NpXKpXiggsuiPXr1x/2v+no6Ij29vZuGwBHh7JGaOfOnRERMXz48G77hw8f3vXaf2pqaoqampqurb6+vpwjAdCDVeTbcf369ev2c1EUh+z70IIFC6Ktra1ra21trcRIAPRAZX28d21tbUT8e0U0YsSIrv27du06ZHX0oVKpFKVSqZxjANBLlHUl1NDQELW1tdHS0tK1b//+/bF27dqYNGlSOU8FQB/wuVdCe/fujW3btnX9vH379ti8eXMMGTIkRo4cGXPnzo3FixfH6NGjY/To0bF48eIYNGhQXHvttWUdHIDe73NH6JVXXomLLrqo6+d58+ZFRMSMGTPil7/8ZXz/+9+PDz74IGbNmhW7d++Or3/96/HMM89EdXV1+aYGoE/oVxRFkT3E/6+9vT1qampi05//T1RXD8weB6BX2T3/v7NHiL3/6oyL/mdTtLW1xeDBgz/xWPeOAyCNCAGQRoQASCNCAKQRIQDSiBAAaUQIgDQiBEAaEQIgTVnvol0OH97AYe/eD5InAeh99v6rM3uE2Pf/ZvgsN+Tpcbft+dvf/ubBdgB9QGtra5x88smfeEyPi9DBgwfj3Xffjerq6o99EN6naW9vj/r6+mhtbf3U+xb1da5Fd67HR1yLj7gWHynHtSiKIvbs2RN1dXVxzDGf/KlPj/tz3DHHHPOp5fysBg8efNT/D/Uh16I71+MjrsVHXIuPfNlrUVNT85mO88UEANKIEABp+mSESqVS3H333VEqlbJHSedadOd6fMS1+Ihr8ZEjfS163BcTADh69MmVEAC9gwgBkEaEAEgjQgCk6ZMRevDBB6OhoSEGDBgQEyZMiOeffz57pCOuqakpzjnnnKiuro5hw4bFFVdcEVu2bMkeq0doamqKfv36xdy5c7NHSfHOO+/E9ddfH0OHDo1BgwbF2WefHRs3bsweK8WBAwfirrvuioaGhhg4cGCceuqpcd9998XBgwezR6u4devWxfTp06Ouri769esXTz75ZLfXi6KIe+65J+rq6mLgwIFx4YUXxptvvln2OfpchFavXh1z586NhQsXxqZNm+K8886LqVOnxo4dO7JHO6LWrl0bs2fPjpdeeilaWlriwIED0djYGPv27cseLdWGDRuiubk5zjzzzOxRUuzevTsmT54cxx13XDz99NPx5z//OX7+85/HCSeckD1aiiVLlsRDDz0Uy5cvj7/85S9x//33x09/+tN44IEHskeruH379sVZZ50Vy5cvP+zr999/fyxdujSWL18eGzZsiNra2rj00ktjz5495R2k6GO+9rWvFTNnzuy2b8yYMcWdd96ZNFHPsGvXriIiirVr12aPkmbPnj3F6NGji5aWluKCCy4o5syZkz3SETd//vxiypQp2WP0GNOmTStuuummbvuuvPLK4vrrr0+aKEdEFE888UTXzwcPHixqa2uLn/zkJ137/vnPfxY1NTXFQw89VNZz96mV0P79+2Pjxo3R2NjYbX9jY2OsX78+aaqeoa2tLSIihgwZkjxJntmzZ8e0adPikksuyR4lzZo1a2LixIlx1VVXxbBhw2L8+PHx8MMPZ4+VZsqUKfHss8/G1q1bIyLitddeixdeeCEuu+yy5Mlybd++PXbu3NntvbRUKsUFF1xQ9vfSHncD0y/jvffei87Ozhg+fHi3/cOHD4+dO3cmTZWvKIqYN29eTJkyJcaNG5c9TorHH388Xn311diwYUP2KKneeuutWLFiRcybNy9+8IMfxMsvvxy33XZblEql+M53vpM93hE3f/78aGtrizFjxkRVVVV0dnbGokWL4pprrskeLdWH75eHey99++23y3quPhWhD/3nIyCKovjCj4XoC2655ZZ4/fXX44UXXsgeJUVra2vMmTMnnnnmmRgwYED2OKkOHjwYEydOjMWLF0dExPjx4+PNN9+MFStWHJURWr16dTz66KOxatWqGDt2bGzevDnmzp0bdXV1MWPGjOzx0h2J99I+FaGTTjopqqqqDln17Nq165CiHy1uvfXWWLNmTaxbt65sj8jobTZu3Bi7du2KCRMmdO3r7OyMdevWxfLly6OjoyOqqqoSJzxyRowYEaeffnq3faeddlr8+te/Tpoo1x133BF33nlnXH311RERccYZZ8Tbb78dTU1NR3WEamtrI+LfK6IRI0Z07a/Ee2mf+kyof//+MWHChGhpaem2v6WlJSZNmpQ0VY6iKOKWW26J3/zmN/HHP/4xGhoaskdKc/HFF8cbb7wRmzdv7tomTpwY1113XWzevPmoCVBExOTJkw/5qv7WrVtj1KhRSRPlev/99w956FpVVdVR8RXtT9LQ0BC1tbXd3kv3798fa9euLft7aZ9aCUVEzJs3L2644YaYOHFinHvuudHc3Bw7duyImTNnZo92RM2ePTtWrVoVTz31VFRXV3etDmtqamLgwIHJ0x1Z1dXVh3wWdvzxx8fQoUOPus/Ibr/99pg0aVIsXrw4vvWtb8XLL78czc3N0dzcnD1aiunTp8eiRYti5MiRMXbs2Ni0aVMsXbo0brrppuzRKm7v3r2xbdu2rp+3b98emzdvjiFDhsTIkSNj7ty5sXjx4hg9enSMHj06Fi9eHIMGDYprr722vIOU9bt2PcQvfvGLYtSoUUX//v2Lr371q0fl15Ij4rDbI488kj1aj3C0fkW7KIrit7/9bTFu3LiiVCoVY8aMKZqbm7NHStPe3l7MmTOnGDlyZDFgwIDi1FNPLRYuXFh0dHRkj1Zxf/rTnw77HjFjxoyiKP79Ne277767qK2tLUqlUnH++ecXb7zxRtnn8CgHANL0qc+EAOhdRAiANCIEQBoRAiCNCAGQRoQASCNCAKQRIQDSiBAAaUQIgDQiBEAaEQIgzf8FnJlyumuHwlcAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "classes = list(set(sf_mat.reshape(-1,)))\n",
    "n_class = len(classes)\n",
    "\n",
    "colors = np.random.rand(n_class,4)\n",
    "dic = {}\n",
    "for i in range(n_class):\n",
    "    dic[classes[i]] = colors[i]\n",
    "    \n",
    "\n",
    "img = np.zeros((num,num,4))\n",
    "for i in range(num):\n",
    "    for j in range(num):\n",
    "        img[i][j] = dic[sf_mat[i][j]]\n",
    "plt.imshow(img)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.16"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}