Application.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Nash-Q Learning Educational Software\n",
    "In this notebook you can interact with an educational software that simulates the Nash-Q learning algorithm in a multi-agent system.\n",
    "\n",
    "More in details the software allows you to specify a **Stochastic game** as an enviroment for the agents to explore. The agents will then learn to play the game by interacting with it and updating their knowledge of the environment trying to reach a Nash equilibrium.\n",
    "\n",
    "This notebook is devided in four main widgets:\n",
    "\n",
    "1. **Presets** : In this widget you can select a preset environment to play with.\n",
    "\n",
    "2. **Enviroment Editor** : In this widget you can edit the stochastic game enviroment. You can specify the number of agents, the number of games, the number of actions per player per game and the rewards and transition probability per action per game.\n",
    "\n",
    "3. **Nash Q Learning Editor** : In this widget you can edit the hyperparameters of the agents and train them on the enviroment.\n",
    "    \n",
    "4. **Learning Display** : In this widget you can view the learning process of the agents and the Nash equilibrium reached."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> **❗Note❗**: The widgets are **interactive** and they are all **linked together**. This means that if you change a parameter in one widget the other widgets will update accordingly.\n",
    "\n",
    "> Unfortunatly sometimes things can go wrong in unexpected ways, in that case you can always rerun the cell to reset the widgets or even restart the kernel."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Imports\n",
    "We developed a custom python package with all the necessary classes and functions to run the Nash-Q learning algorithm. The package is called `LearningNashQLearning` and it is fully available on PyP with an Open Source licence.\n",
    "\n",
    "Your are highly encouraged to check the source code of the package. That is located in this same repository in the folder `LearningNashQLearning`.\n",
    "\n",
    "Any form of contribution is encouraged and appreciated, feel free to open an issue or a pull request. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requirement already satisfied: LearningNashQLearning==0.11 in c:\\users\\federico rocca\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.10_qbz5n2kfra8p0\\localcache\\local-packages\\python310\\site-packages (0.11)\n",
      "Requirement already satisfied: numpy in c:\\users\\federico rocca\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.10_qbz5n2kfra8p0\\localcache\\local-packages\\python310\\site-packages (from LearningNashQLearning==0.11) (1.23.5)\n",
      "Requirement already satisfied: matplotlib in c:\\users\\federico rocca\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.10_qbz5n2kfra8p0\\localcache\\local-packages\\python310\\site-packages (from LearningNashQLearning==0.11) (3.7.1)\n",
      "Requirement already satisfied: seaborn in c:\\users\\federico rocca\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.10_qbz5n2kfra8p0\\localcache\\local-packages\\python310\\site-packages (from LearningNashQLearning==0.11) (0.13.2)\n",
      "Requirement already satisfied: networkx in c:\\users\\federico rocca\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.10_qbz5n2kfra8p0\\localcache\\local-packages\\python310\\site-packages (from LearningNashQLearning==0.11) (3.0)\n",
      "Requirement already satisfied: pygambit in c:\\users\\federico rocca\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.10_qbz5n2kfra8p0\\localcache\\local-packages\\python310\\site-packages (from LearningNashQLearning==0.11) (16.2.0)\n",
      "Requirement already satisfied: netgraph in c:\\users\\federico rocca\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.10_qbz5n2kfra8p0\\localcache\\local-packages\\python310\\site-packages (from LearningNashQLearning==0.11) (4.13.2)\n",
      "Requirement already satisfied: ipywidgets in c:\\users\\federico rocca\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.10_qbz5n2kfra8p0\\localcache\\local-packages\\python310\\site-packages (from LearningNashQLearning==0.11) (8.1.2)\n",
      "Requirement already satisfied: notebook in c:\\users\\federico rocca\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.10_qbz5n2kfra8p0\\localcache\\local-packages\\python310\\site-packages (from LearningNashQLearning==0.11) (7.2.0)\n",
      "Requirement already satisfied: ipympl in c:\\users\\federico rocca\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.10_qbz5n2kfra8p0\\localcache\\local-packages\\python310\\site-packages (from LearningNashQLearning==0.11) (0.9.3)\n",
      "Requirement already satisfied: ipython<9 in c:\\users\\federico rocca\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.10_qbz5n2kfra8p0\\localcache\\local-packages\\python310\\site-packages (from ipympl->LearningNashQLearning==0.11) (8.11.0)\n",
      "Requirement already satisfied: ipython-genutils in c:\\users\\federico rocca\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.10_qbz5n2kfra8p0\\localcache\\local-packages\\python310\\site-packages (from ipympl->LearningNashQLearning==0.11) (0.2.0)\n",
      "Requirement already satisfied: pillow in c:\\users\\federico rocca\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.10_qbz5n2kfra8p0\\localcache\\local-packages\\python310\\site-packages (from ipympl->LearningNashQLearning==0.11) (9.4.0)\n",
      "Requirement already satisfied: traitlets<6 in c:\\users\\federico rocca\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.10_qbz5n2kfra8p0\\localcache\\local-packages\\python310\\site-packages (from ipympl->LearningNashQLearning==0.11) (5.9.0)\n",
      "Requirement already satisfied: comm>=0.1.3 in c:\\users\\federico rocca\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.10_qbz5n2kfra8p0\\localcache\\local-packages\\python310\\site-packages (from ipywidgets->LearningNashQLearning==0.11) (0.2.2)\n",
      "Requirement already satisfied: widgetsnbextension~=4.0.10 in c:\\users\\federico rocca\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.10_qbz5n2kfra8p0\\localcache\\local-packages\\python310\\site-packages (from ipywidgets->LearningNashQLearning==0.11) (4.0.10)\n",
      "Requirement already satisfied: jupyterlab-widgets~=3.0.10 in c:\\users\\federico rocca\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.10_qbz5n2kfra8p0\\localcache\\local-packages\\python310\\site-packages (from ipywidgets->LearningNashQLearning==0.11) (3.0.10)\n",
      "Requirement already satisfied: contourpy>=1.0.1 in c:\\users\\federico rocca\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.10_qbz5n2kfra8p0\\localcache\\local-packages\\python310\\site-packages (from matplotlib->LearningNashQLearning==0.11) (1.0.7)\n",
      "Requirement already satisfied: cycler>=0.10 in c:\\users\\federico rocca\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.10_qbz5n2kfra8p0\\localcache\\local-packages\\python310\\site-packages (from matplotlib->LearningNashQLearning==0.11) (0.11.0)\n",
      "Requirement already satisfied: fonttools>=4.22.0 in c:\\users\\federico rocca\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.10_qbz5n2kfra8p0\\localcache\\local-packages\\python310\\site-packages (from matplotlib->LearningNashQLearning==0.11) (4.39.2)\n",
      "Requirement already satisfied: kiwisolver>=1.0.1 in c:\\users\\federico rocca\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.10_qbz5n2kfra8p0\\localcache\\local-packages\\python310\\site-packages (from matplotlib->LearningNashQLearning==0.11) (1.4.4)\n",
      "Requirement already satisfied: packaging>=20.0 in c:\\users\\federico rocca\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.10_qbz5n2kfra8p0\\localcache\\local-packages\\python310\\site-packages (from matplotlib->LearningNashQLearning==0.11) (23.0)\n",
      "Requirement already satisfied: pyparsing>=2.3.1 in c:\\users\\federico rocca\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.10_qbz5n2kfra8p0\\localcache\\local-packages\\python310\\site-packages (from matplotlib->LearningNashQLearning==0.11) (3.0.9)\n",
      "Requirement already satisfied: python-dateutil>=2.7 in c:\\users\\federico rocca\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.10_qbz5n2kfra8p0\\localcache\\local-packages\\python310\\site-packages (from matplotlib->LearningNashQLearning==0.11) (2.8.2)\n",
      "Requirement already satisfied: scipy in c:\\users\\federico rocca\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.10_qbz5n2kfra8p0\\localcache\\local-packages\\python310\\site-packages (from netgraph->LearningNashQLearning==0.11) (1.10.1)\n",
      "Requirement already satisfied: rectangle-packer in c:\\users\\federico rocca\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.10_qbz5n2kfra8p0\\localcache\\local-packages\\python310\\site-packages (from netgraph->LearningNashQLearning==0.11) (2.0.2)\n",
      "Requirement already satisfied: grandalf in c:\\users\\federico rocca\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.10_qbz5n2kfra8p0\\localcache\\local-packages\\python310\\site-packages (from netgraph->LearningNashQLearning==0.11) (0.8)\n",
      "Requirement already satisfied: jupyter-server<3,>=2.4.0 in c:\\users\\federico rocca\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.10_qbz5n2kfra8p0\\localcache\\local-packages\\python310\\site-packages (from notebook->LearningNashQLearning==0.11) (2.14.1)\n",
      "Requirement already satisfied: jupyterlab-server<3,>=2.27.1 in c:\\users\\federico rocca\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.10_qbz5n2kfra8p0\\localcache\\local-packages\\python310\\site-packages (from notebook->LearningNashQLearning==0.11) (2.27.2)\n",
      "Requirement already satisfied: jupyterlab<4.3,>=4.2.0 in c:\\users\\federico rocca\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.10_qbz5n2kfra8p0\\localcache\\local-packages\\python310\\site-packages (from notebook->LearningNashQLearning==0.11) (4.2.1)\n",
      "Requirement already satisfied: notebook-shim<0.3,>=0.2 in c:\\users\\federico rocca\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.10_qbz5n2kfra8p0\\localcache\\local-packages\\python310\\site-packages (from notebook->LearningNashQLearning==0.11) (0.2.4)\n",
      "Requirement already satisfied: tornado>=6.2.0 in c:\\users\\federico rocca\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.10_qbz5n2kfra8p0\\localcache\\local-packages\\python310\\site-packages (from notebook->LearningNashQLearning==0.11) (6.2)\n",
      "Requirement already satisfied: lxml in c:\\users\\federico rocca\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.10_qbz5n2kfra8p0\\localcache\\local-packages\\python310\\site-packages (from pygambit->LearningNashQLearning==0.11) (5.2.1)\n",
      "Requirement already satisfied: deprecated in c:\\users\\federico rocca\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.10_qbz5n2kfra8p0\\localcache\\local-packages\\python310\\site-packages (from pygambit->LearningNashQLearning==0.11) (1.2.14)\n",
      "Requirement already satisfied: pandas>=1.2 in c:\\users\\federico rocca\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.10_qbz5n2kfra8p0\\localcache\\local-packages\\python310\\site-packages (from seaborn->LearningNashQLearning==0.11) (2.0.1)\n",
      "Requirement already satisfied: backcall in c:\\users\\federico rocca\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.10_qbz5n2kfra8p0\\localcache\\local-packages\\python310\\site-packages (from ipython<9->ipympl->LearningNashQLearning==0.11) (0.2.0)\n",
      "Requirement already satisfied: decorator in c:\\users\\federico rocca\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.10_qbz5n2kfra8p0\\localcache\\local-packages\\python310\\site-packages (from ipython<9->ipympl->LearningNashQLearning==0.11) (5.1.1)\n",
      "Requirement already satisfied: jedi>=0.16 in c:\\users\\federico rocca\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.10_qbz5n2kfra8p0\\localcache\\local-packages\\python310\\site-packages (from ipython<9->ipympl->LearningNashQLearning==0.11) (0.18.2)\n",
      "Requirement already satisfied: matplotlib-inline in c:\\users\\federico rocca\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.10_qbz5n2kfra8p0\\localcache\\local-packages\\python310\\site-packages (from ipython<9->ipympl->LearningNashQLearning==0.11) (0.1.6)\n",
      "Requirement already satisfied: pickleshare in c:\\users\\federico rocca\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.10_qbz5n2kfra8p0\\localcache\\local-packages\\python310\\site-packages (from ipython<9->ipympl->LearningNashQLearning==0.11) (0.7.5)\n",
      "Requirement already satisfied: prompt-toolkit!=3.0.37,<3.1.0,>=3.0.30 in c:\\users\\federico rocca\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.10_qbz5n2kfra8p0\\localcache\\local-packages\\python310\\site-packages (from ipython<9->ipympl->LearningNashQLearning==0.11) (3.0.38)\n",
      "Requirement already satisfied: pygments>=2.4.0 in c:\\users\\federico rocca\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.10_qbz5n2kfra8p0\\localcache\\local-packages\\python310\\site-packages (from ipython<9->ipympl->LearningNashQLearning==0.11) (2.14.0)\n",
      "Requirement already satisfied: stack-data in c:\\users\\federico rocca\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.10_qbz5n2kfra8p0\\localcache\\local-packages\\python310\\site-packages (from ipython<9->ipympl->LearningNashQLearning==0.11) (0.6.2)\n",
      "Requirement already satisfied: colorama in c:\\users\\federico rocca\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.10_qbz5n2kfra8p0\\localcache\\local-packages\\python310\\site-packages (from ipython<9->ipympl->LearningNashQLearning==0.11) (0.4.6)\n",
      "Requirement already satisfied: anyio>=3.1.0 in c:\\users\\federico rocca\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.10_qbz5n2kfra8p0\\localcache\\local-packages\\python310\\site-packages (from jupyter-server<3,>=2.4.0->notebook->LearningNashQLearning==0.11) (4.4.0)\n",
      "Requirement already satisfied: argon2-cffi>=21.1 in c:\\users\\federico rocca\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.10_qbz5n2kfra8p0\\localcache\\local-packages\\python310\\site-packages (from jupyter-server<3,>=2.4.0->notebook->LearningNashQLearning==0.11) (23.1.0)\n",
      "Requirement already satisfied: jinja2>=3.0.3 in c:\\users\\federico rocca\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.10_qbz5n2kfra8p0\\localcache\\local-packages\\python310\\site-packages (from jupyter-server<3,>=2.4.0->notebook->LearningNashQLearning==0.11) (3.1.2)\n",
      "Requirement already satisfied: jupyter-client>=7.4.4 in c:\\users\\federico rocca\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.10_qbz5n2kfra8p0\\localcache\\local-packages\\python310\\site-packages (from jupyter-server<3,>=2.4.0->notebook->LearningNashQLearning==0.11) (8.0.3)\n",
      "Requirement already satisfied: jupyter-core!=5.0.*,>=4.12 in c:\\users\\federico rocca\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.10_qbz5n2kfra8p0\\localcache\\local-packages\\python310\\site-packages (from jupyter-server<3,>=2.4.0->notebook->LearningNashQLearning==0.11) (5.3.0)\n",
      "Requirement already satisfied: jupyter-events>=0.9.0 in c:\\users\\federico rocca\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.10_qbz5n2kfra8p0\\localcache\\local-packages\\python310\\site-packages (from jupyter-server<3,>=2.4.0->notebook->LearningNashQLearning==0.11) (0.10.0)\n",
      "Requirement already satisfied: jupyter-server-terminals>=0.4.4 in c:\\users\\federico rocca\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.10_qbz5n2kfra8p0\\localcache\\local-packages\\python310\\site-packages (from jupyter-server<3,>=2.4.0->notebook->LearningNashQLearning==0.11) (0.5.3)\n",
      "Requirement already satisfied: nbconvert>=6.4.4 in c:\\users\\federico rocca\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.10_qbz5n2kfra8p0\\localcache\\local-packages\\python310\\site-packages (from jupyter-server<3,>=2.4.0->notebook->LearningNashQLearning==0.11) (7.16.4)\n",
      "Requirement already satisfied: nbformat>=5.3.0 in c:\\users\\federico rocca\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.10_qbz5n2kfra8p0\\localcache\\local-packages\\python310\\site-packages (from jupyter-server<3,>=2.4.0->notebook->LearningNashQLearning==0.11) (5.10.4)\n",
      "Requirement already satisfied: overrides>=5.0 in c:\\users\\federico rocca\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.10_qbz5n2kfra8p0\\localcache\\local-packages\\python310\\site-packages (from jupyter-server<3,>=2.4.0->notebook->LearningNashQLearning==0.11) (7.7.0)\n",
      "Requirement already satisfied: prometheus-client>=0.9 in c:\\users\\federico rocca\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.10_qbz5n2kfra8p0\\localcache\\local-packages\\python310\\site-packages (from jupyter-server<3,>=2.4.0->notebook->LearningNashQLearning==0.11) (0.20.0)\n",
      "Requirement already satisfied: pywinpty>=2.0.1 in c:\\users\\federico rocca\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.10_qbz5n2kfra8p0\\localcache\\local-packages\\python310\\site-packages (from jupyter-server<3,>=2.4.0->notebook->LearningNashQLearning==0.11) (2.0.13)\n",
      "Requirement already satisfied: pyzmq>=24 in c:\\users\\federico rocca\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.10_qbz5n2kfra8p0\\localcache\\local-packages\\python310\\site-packages (from jupyter-server<3,>=2.4.0->notebook->LearningNashQLearning==0.11) (25.0.1)\n",
      "Requirement already satisfied: send2trash>=1.8.2 in c:\\users\\federico rocca\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.10_qbz5n2kfra8p0\\localcache\\local-packages\\python310\\site-packages (from jupyter-server<3,>=2.4.0->notebook->LearningNashQLearning==0.11) (1.8.3)\n",
      "Requirement already satisfied: terminado>=0.8.3 in c:\\users\\federico rocca\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.10_qbz5n2kfra8p0\\localcache\\local-packages\\python310\\site-packages (from jupyter-server<3,>=2.4.0->notebook->LearningNashQLearning==0.11) (0.18.1)\n",
      "Requirement already satisfied: websocket-client>=1.7 in c:\\users\\federico rocca\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.10_qbz5n2kfra8p0\\localcache\\local-packages\\python310\\site-packages (from jupyter-server<3,>=2.4.0->notebook->LearningNashQLearning==0.11) (1.8.0)\n",
      "Requirement already satisfied: async-lru>=1.0.0 in c:\\users\\federico rocca\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.10_qbz5n2kfra8p0\\localcache\\local-packages\\python310\\site-packages (from jupyterlab<4.3,>=4.2.0->notebook->LearningNashQLearning==0.11) (2.0.4)\n",
      "Requirement already satisfied: httpx>=0.25.0 in c:\\users\\federico rocca\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.10_qbz5n2kfra8p0\\localcache\\local-packages\\python310\\site-packages (from jupyterlab<4.3,>=4.2.0->notebook->LearningNashQLearning==0.11) (0.27.0)\n",
      "Requirement already satisfied: ipykernel>=6.5.0 in c:\\users\\federico rocca\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.10_qbz5n2kfra8p0\\localcache\\local-packages\\python310\\site-packages (from jupyterlab<4.3,>=4.2.0->notebook->LearningNashQLearning==0.11) (6.21.3)\n",
      "Requirement already satisfied: jupyter-lsp>=2.0.0 in c:\\users\\federico rocca\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.10_qbz5n2kfra8p0\\localcache\\local-packages\\python310\\site-packages (from jupyterlab<4.3,>=4.2.0->notebook->LearningNashQLearning==0.11) (2.2.5)\n",
      "Requirement already satisfied: tomli>=1.2.2 in c:\\users\\federico rocca\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.10_qbz5n2kfra8p0\\localcache\\local-packages\\python310\\site-packages (from jupyterlab<4.3,>=4.2.0->notebook->LearningNashQLearning==0.11) (2.0.1)\n",
      "Requirement already satisfied: babel>=2.10 in c:\\users\\federico rocca\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.10_qbz5n2kfra8p0\\localcache\\local-packages\\python310\\site-packages (from jupyterlab-server<3,>=2.27.1->notebook->LearningNashQLearning==0.11) (2.15.0)\n",
      "Requirement already satisfied: json5>=0.9.0 in c:\\users\\federico rocca\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.10_qbz5n2kfra8p0\\localcache\\local-packages\\python310\\site-packages (from jupyterlab-server<3,>=2.27.1->notebook->LearningNashQLearning==0.11) (0.9.25)\n",
      "Requirement already satisfied: jsonschema>=4.18.0 in c:\\users\\federico rocca\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.10_qbz5n2kfra8p0\\localcache\\local-packages\\python310\\site-packages (from jupyterlab-server<3,>=2.27.1->notebook->LearningNashQLearning==0.11) (4.22.0)\n",
      "Requirement already satisfied: requests>=2.31 in c:\\users\\federico rocca\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.10_qbz5n2kfra8p0\\localcache\\local-packages\\python310\\site-packages (from jupyterlab-server<3,>=2.27.1->notebook->LearningNashQLearning==0.11) (2.32.3)\n",
      "Requirement already satisfied: pytz>=2020.1 in c:\\users\\federico rocca\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.10_qbz5n2kfra8p0\\localcache\\local-packages\\python310\\site-packages (from pandas>=1.2->seaborn->LearningNashQLearning==0.11) (2023.3)\n",
      "Requirement already satisfied: tzdata>=2022.1 in c:\\users\\federico rocca\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.10_qbz5n2kfra8p0\\localcache\\local-packages\\python310\\site-packages (from pandas>=1.2->seaborn->LearningNashQLearning==0.11) (2023.3)\n",
      "Requirement already satisfied: six>=1.5 in c:\\users\\federico rocca\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.10_qbz5n2kfra8p0\\localcache\\local-packages\\python310\\site-packages (from python-dateutil>=2.7->matplotlib->LearningNashQLearning==0.11) (1.16.0)\n",
      "Requirement already satisfied: wrapt<2,>=1.10 in c:\\users\\federico rocca\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.10_qbz5n2kfra8p0\\localcache\\local-packages\\python310\\site-packages (from deprecated->pygambit->LearningNashQLearning==0.11) (1.14.1)\n",
      "Requirement already satisfied: idna>=2.8 in c:\\users\\federico rocca\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.10_qbz5n2kfra8p0\\localcache\\local-packages\\python310\\site-packages (from anyio>=3.1.0->jupyter-server<3,>=2.4.0->notebook->LearningNashQLearning==0.11) (3.4)\n",
      "Requirement already satisfied: sniffio>=1.1 in c:\\users\\federico rocca\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.10_qbz5n2kfra8p0\\localcache\\local-packages\\python310\\site-packages (from anyio>=3.1.0->jupyter-server<3,>=2.4.0->notebook->LearningNashQLearning==0.11) (1.3.1)\n",
      "Requirement already satisfied: exceptiongroup>=1.0.2 in c:\\users\\federico rocca\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.10_qbz5n2kfra8p0\\localcache\\local-packages\\python310\\site-packages (from anyio>=3.1.0->jupyter-server<3,>=2.4.0->notebook->LearningNashQLearning==0.11) (1.2.1)\n",
      "Requirement already satisfied: typing-extensions>=4.1 in c:\\users\\federico rocca\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.10_qbz5n2kfra8p0\\localcache\\local-packages\\python310\\site-packages (from anyio>=3.1.0->jupyter-server<3,>=2.4.0->notebook->LearningNashQLearning==0.11) (4.4.0)\n",
      "Requirement already satisfied: argon2-cffi-bindings in c:\\users\\federico rocca\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.10_qbz5n2kfra8p0\\localcache\\local-packages\\python310\\site-packages (from argon2-cffi>=21.1->jupyter-server<3,>=2.4.0->notebook->LearningNashQLearning==0.11) (21.2.0)\n",
      "Requirement already satisfied: certifi in c:\\users\\federico rocca\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.10_qbz5n2kfra8p0\\localcache\\local-packages\\python310\\site-packages (from httpx>=0.25.0->jupyterlab<4.3,>=4.2.0->notebook->LearningNashQLearning==0.11) (2022.12.7)\n",
      "Requirement already satisfied: httpcore==1.* in c:\\users\\federico rocca\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.10_qbz5n2kfra8p0\\localcache\\local-packages\\python310\\site-packages (from httpx>=0.25.0->jupyterlab<4.3,>=4.2.0->notebook->LearningNashQLearning==0.11) (1.0.5)\n",
      "Requirement already satisfied: h11<0.15,>=0.13 in c:\\users\\federico rocca\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.10_qbz5n2kfra8p0\\localcache\\local-packages\\python310\\site-packages (from httpcore==1.*->httpx>=0.25.0->jupyterlab<4.3,>=4.2.0->notebook->LearningNashQLearning==0.11) (0.14.0)\n",
      "Requirement already satisfied: debugpy>=1.6.5 in c:\\users\\federico rocca\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.10_qbz5n2kfra8p0\\localcache\\local-packages\\python310\\site-packages (from ipykernel>=6.5.0->jupyterlab<4.3,>=4.2.0->notebook->LearningNashQLearning==0.11) (1.6.6)\n",
      "Requirement already satisfied: nest-asyncio in c:\\users\\federico rocca\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.10_qbz5n2kfra8p0\\localcache\\local-packages\\python310\\site-packages (from ipykernel>=6.5.0->jupyterlab<4.3,>=4.2.0->notebook->LearningNashQLearning==0.11) (1.5.6)\n",
      "Requirement already satisfied: psutil in c:\\users\\federico rocca\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.10_qbz5n2kfra8p0\\localcache\\local-packages\\python310\\site-packages (from ipykernel>=6.5.0->jupyterlab<4.3,>=4.2.0->notebook->LearningNashQLearning==0.11) (5.9.4)\n",
      "Requirement already satisfied: parso<0.9.0,>=0.8.0 in c:\\users\\federico rocca\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.10_qbz5n2kfra8p0\\localcache\\local-packages\\python310\\site-packages (from jedi>=0.16->ipython<9->ipympl->LearningNashQLearning==0.11) (0.8.3)\n",
      "Requirement already satisfied: MarkupSafe>=2.0 in c:\\users\\federico rocca\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.10_qbz5n2kfra8p0\\localcache\\local-packages\\python310\\site-packages (from jinja2>=3.0.3->jupyter-server<3,>=2.4.0->notebook->LearningNashQLearning==0.11) (2.1.2)\n",
      "Requirement already satisfied: attrs>=22.2.0 in c:\\users\\federico rocca\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.10_qbz5n2kfra8p0\\localcache\\local-packages\\python310\\site-packages (from jsonschema>=4.18.0->jupyterlab-server<3,>=2.27.1->notebook->LearningNashQLearning==0.11) (23.2.0)\n",
      "Requirement already satisfied: jsonschema-specifications>=2023.03.6 in c:\\users\\federico rocca\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.10_qbz5n2kfra8p0\\localcache\\local-packages\\python310\\site-packages (from jsonschema>=4.18.0->jupyterlab-server<3,>=2.27.1->notebook->LearningNashQLearning==0.11) (2023.12.1)\n",
      "Requirement already satisfied: referencing>=0.28.4 in c:\\users\\federico rocca\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.10_qbz5n2kfra8p0\\localcache\\local-packages\\python310\\site-packages (from jsonschema>=4.18.0->jupyterlab-server<3,>=2.27.1->notebook->LearningNashQLearning==0.11) (0.35.1)\n",
      "Requirement already satisfied: rpds-py>=0.7.1 in c:\\users\\federico rocca\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.10_qbz5n2kfra8p0\\localcache\\local-packages\\python310\\site-packages (from jsonschema>=4.18.0->jupyterlab-server<3,>=2.27.1->notebook->LearningNashQLearning==0.11) (0.18.1)\n",
      "Requirement already satisfied: platformdirs>=2.5 in c:\\users\\federico rocca\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.10_qbz5n2kfra8p0\\localcache\\local-packages\\python310\\site-packages (from jupyter-core!=5.0.*,>=4.12->jupyter-server<3,>=2.4.0->notebook->LearningNashQLearning==0.11) (3.1.1)\n",
      "Requirement already satisfied: pywin32>=300 in c:\\users\\federico rocca\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.10_qbz5n2kfra8p0\\localcache\\local-packages\\python310\\site-packages (from jupyter-core!=5.0.*,>=4.12->jupyter-server<3,>=2.4.0->notebook->LearningNashQLearning==0.11) (305)\n",
      "Requirement already satisfied: python-json-logger>=2.0.4 in c:\\users\\federico rocca\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.10_qbz5n2kfra8p0\\localcache\\local-packages\\python310\\site-packages (from jupyter-events>=0.9.0->jupyter-server<3,>=2.4.0->notebook->LearningNashQLearning==0.11) (2.0.7)\n",
      "Requirement already satisfied: pyyaml>=5.3 in c:\\users\\federico rocca\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.10_qbz5n2kfra8p0\\localcache\\local-packages\\python310\\site-packages (from jupyter-events>=0.9.0->jupyter-server<3,>=2.4.0->notebook->LearningNashQLearning==0.11) (6.0)\n",
      "Requirement already satisfied: rfc3339-validator in c:\\users\\federico rocca\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.10_qbz5n2kfra8p0\\localcache\\local-packages\\python310\\site-packages (from jupyter-events>=0.9.0->jupyter-server<3,>=2.4.0->notebook->LearningNashQLearning==0.11) (0.1.4)\n",
      "Requirement already satisfied: rfc3986-validator>=0.1.1 in c:\\users\\federico rocca\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.10_qbz5n2kfra8p0\\localcache\\local-packages\\python310\\site-packages (from jupyter-events>=0.9.0->jupyter-server<3,>=2.4.0->notebook->LearningNashQLearning==0.11) (0.1.1)\n",
      "Requirement already satisfied: beautifulsoup4 in c:\\users\\federico rocca\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.10_qbz5n2kfra8p0\\localcache\\local-packages\\python310\\site-packages (from nbconvert>=6.4.4->jupyter-server<3,>=2.4.0->notebook->LearningNashQLearning==0.11) (4.12.3)\n",
      "Requirement already satisfied: bleach!=5.0.0 in c:\\users\\federico rocca\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.10_qbz5n2kfra8p0\\localcache\\local-packages\\python310\\site-packages (from nbconvert>=6.4.4->jupyter-server<3,>=2.4.0->notebook->LearningNashQLearning==0.11) (6.1.0)\n",
      "Requirement already satisfied: defusedxml in c:\\users\\federico rocca\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.10_qbz5n2kfra8p0\\localcache\\local-packages\\python310\\site-packages (from nbconvert>=6.4.4->jupyter-server<3,>=2.4.0->notebook->LearningNashQLearning==0.11) (0.7.1)\n",
      "Requirement already satisfied: jupyterlab-pygments in c:\\users\\federico rocca\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.10_qbz5n2kfra8p0\\localcache\\local-packages\\python310\\site-packages (from nbconvert>=6.4.4->jupyter-server<3,>=2.4.0->notebook->LearningNashQLearning==0.11) (0.3.0)\n",
      "Requirement already satisfied: mistune<4,>=2.0.3 in c:\\users\\federico rocca\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.10_qbz5n2kfra8p0\\localcache\\local-packages\\python310\\site-packages (from nbconvert>=6.4.4->jupyter-server<3,>=2.4.0->notebook->LearningNashQLearning==0.11) (3.0.2)\n",
      "Requirement already satisfied: nbclient>=0.5.0 in c:\\users\\federico rocca\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.10_qbz5n2kfra8p0\\localcache\\local-packages\\python310\\site-packages (from nbconvert>=6.4.4->jupyter-server<3,>=2.4.0->notebook->LearningNashQLearning==0.11) (0.10.0)\n",
      "Requirement already satisfied: pandocfilters>=1.4.1 in c:\\users\\federico rocca\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.10_qbz5n2kfra8p0\\localcache\\local-packages\\python310\\site-packages (from nbconvert>=6.4.4->jupyter-server<3,>=2.4.0->notebook->LearningNashQLearning==0.11) (1.5.1)\n",
      "Requirement already satisfied: tinycss2 in c:\\users\\federico rocca\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.10_qbz5n2kfra8p0\\localcache\\local-packages\\python310\\site-packages (from nbconvert>=6.4.4->jupyter-server<3,>=2.4.0->notebook->LearningNashQLearning==0.11) (1.3.0)\n",
      "Requirement already satisfied: fastjsonschema>=2.15 in c:\\users\\federico rocca\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.10_qbz5n2kfra8p0\\localcache\\local-packages\\python310\\site-packages (from nbformat>=5.3.0->jupyter-server<3,>=2.4.0->notebook->LearningNashQLearning==0.11) (2.19.1)\n",
      "Requirement already satisfied: wcwidth in c:\\users\\federico rocca\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.10_qbz5n2kfra8p0\\localcache\\local-packages\\python310\\site-packages (from prompt-toolkit!=3.0.37,<3.1.0,>=3.0.30->ipython<9->ipympl->LearningNashQLearning==0.11) (0.2.6)\n",
      "Requirement already satisfied: charset-normalizer<4,>=2 in c:\\users\\federico rocca\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.10_qbz5n2kfra8p0\\localcache\\local-packages\\python310\\site-packages (from requests>=2.31->jupyterlab-server<3,>=2.27.1->notebook->LearningNashQLearning==0.11) (3.1.0)\n",
      "Requirement already satisfied: urllib3<3,>=1.21.1 in c:\\users\\federico rocca\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.10_qbz5n2kfra8p0\\localcache\\local-packages\\python310\\site-packages (from requests>=2.31->jupyterlab-server<3,>=2.27.1->notebook->LearningNashQLearning==0.11) (1.26.15)\n",
      "Requirement already satisfied: executing>=1.2.0 in c:\\users\\federico rocca\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.10_qbz5n2kfra8p0\\localcache\\local-packages\\python310\\site-packages (from stack-data->ipython<9->ipympl->LearningNashQLearning==0.11) (1.2.0)\n",
      "Requirement already satisfied: asttokens>=2.1.0 in c:\\users\\federico rocca\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.10_qbz5n2kfra8p0\\localcache\\local-packages\\python310\\site-packages (from stack-data->ipython<9->ipympl->LearningNashQLearning==0.11) (2.2.1)\n",
      "Requirement already satisfied: pure-eval in c:\\users\\federico rocca\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.10_qbz5n2kfra8p0\\localcache\\local-packages\\python310\\site-packages (from stack-data->ipython<9->ipympl->LearningNashQLearning==0.11) (0.2.2)\n",
      "Requirement already satisfied: webencodings in c:\\users\\federico rocca\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.10_qbz5n2kfra8p0\\localcache\\local-packages\\python310\\site-packages (from bleach!=5.0.0->nbconvert>=6.4.4->jupyter-server<3,>=2.4.0->notebook->LearningNashQLearning==0.11) (0.5.1)\n",
      "Requirement already satisfied: fqdn in c:\\users\\federico rocca\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.10_qbz5n2kfra8p0\\localcache\\local-packages\\python310\\site-packages (from jsonschema[format-nongpl]>=4.18.0->jupyter-events>=0.9.0->jupyter-server<3,>=2.4.0->notebook->LearningNashQLearning==0.11) (1.5.1)\n",
      "Requirement already satisfied: isoduration in c:\\users\\federico rocca\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.10_qbz5n2kfra8p0\\localcache\\local-packages\\python310\\site-packages (from jsonschema[format-nongpl]>=4.18.0->jupyter-events>=0.9.0->jupyter-server<3,>=2.4.0->notebook->LearningNashQLearning==0.11) (20.11.0)\n",
      "Requirement already satisfied: jsonpointer>1.13 in c:\\users\\federico rocca\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.10_qbz5n2kfra8p0\\localcache\\local-packages\\python310\\site-packages (from jsonschema[format-nongpl]>=4.18.0->jupyter-events>=0.9.0->jupyter-server<3,>=2.4.0->notebook->LearningNashQLearning==0.11) (2.4)\n",
      "Requirement already satisfied: uri-template in c:\\users\\federico rocca\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.10_qbz5n2kfra8p0\\localcache\\local-packages\\python310\\site-packages (from jsonschema[format-nongpl]>=4.18.0->jupyter-events>=0.9.0->jupyter-server<3,>=2.4.0->notebook->LearningNashQLearning==0.11) (1.3.0)\n",
      "Requirement already satisfied: webcolors>=1.11 in c:\\users\\federico rocca\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.10_qbz5n2kfra8p0\\localcache\\local-packages\\python310\\site-packages (from jsonschema[format-nongpl]>=4.18.0->jupyter-events>=0.9.0->jupyter-server<3,>=2.4.0->notebook->LearningNashQLearning==0.11) (1.13)\n",
      "Requirement already satisfied: cffi>=1.0.1 in c:\\users\\federico rocca\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.10_qbz5n2kfra8p0\\localcache\\local-packages\\python310\\site-packages (from argon2-cffi-bindings->argon2-cffi>=21.1->jupyter-server<3,>=2.4.0->notebook->LearningNashQLearning==0.11) (1.16.0)\n",
      "Requirement already satisfied: soupsieve>1.2 in c:\\users\\federico rocca\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.10_qbz5n2kfra8p0\\localcache\\local-packages\\python310\\site-packages (from beautifulsoup4->nbconvert>=6.4.4->jupyter-server<3,>=2.4.0->notebook->LearningNashQLearning==0.11) (2.5)\n",
      "Requirement already satisfied: pycparser in c:\\users\\federico rocca\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.10_qbz5n2kfra8p0\\localcache\\local-packages\\python310\\site-packages (from cffi>=1.0.1->argon2-cffi-bindings->argon2-cffi>=21.1->jupyter-server<3,>=2.4.0->notebook->LearningNashQLearning==0.11) (2.22)\n",
      "Requirement already satisfied: arrow>=0.15.0 in c:\\users\\federico rocca\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.10_qbz5n2kfra8p0\\localcache\\local-packages\\python310\\site-packages (from isoduration->jsonschema[format-nongpl]>=4.18.0->jupyter-events>=0.9.0->jupyter-server<3,>=2.4.0->notebook->LearningNashQLearning==0.11) (1.3.0)\n",
      "Requirement already satisfied: types-python-dateutil>=2.8.10 in c:\\users\\federico rocca\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.10_qbz5n2kfra8p0\\localcache\\local-packages\\python310\\site-packages (from arrow>=0.15.0->isoduration->jsonschema[format-nongpl]>=4.18.0->jupyter-events>=0.9.0->jupyter-server<3,>=2.4.0->notebook->LearningNashQLearning==0.11) (2.9.0.20240316)\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "`sys_prefix` level settings are read-only, using `user` level for migration to `lockedExtensions`\n"
     ]
    }
   ],
   "source": [
    "# If you encounter issues in the installation of the requirements,\n",
    "# you may need to substitute the pip command with pip3\n",
    "\n",
    "!pip install LearningNashQLearning==0.12\n",
    "!jupyter labextension enable widgetsnbextension"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The autoreload extension is already loaded. To reload it, use:\n",
      "  %reload_ext autoreload\n"
     ]
    }
   ],
   "source": [
    "from LearningNashQLearning.Model.Environment import Environment\n",
    "from LearningNashQLearning.Model.NashQLearning import NashQLearning\n",
    "from LearningNashQLearning.View.PresetGames import PresetGames\n",
    "from LearningNashQLearning.View.GameEditor import EnvironmentWidget\n",
    "from LearningNashQLearning.View.EnvGraphDisplay import EnvGraphDisplay\n",
    "from LearningNashQLearning.View.FinalDisplay import FinalDisplay\n",
    "\n",
    "\n",
    "import ipywidgets as widgets\n",
    "from IPython.display import display\n",
    "\n",
    "%matplotlib widget\n",
    "\n",
    "# autoreload   \n",
    "%load_ext autoreload\n",
    "%autoreload 2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Enviroment Editor ##\n",
    "In this widget you can create your own environment, you can do it in the following interface.\n",
    "\n",
    "First of all, you need to chose the number of **players** (that is the number of agents), limited to 4 for reasons of complexity of representation, and the number of **games** (states that can be reached by the agents).  \n",
    "\n",
    "When defining the number of states, the user should keep in mind its definition as: ***S*** = *s<sub>1</sub> x s<sub>2</sub> x ... x s<sub>i</sub>*, with *i* = 1, ..., n as the number of agents and *s<sub>i</sub>* as the state space for the *i-th* agent. This means that the state space is defined as the Cartesian product of the individual state spaces for every agent. Therefore, every state represents one of the combinations of positions of the agents, and the transitions between states always involve all the agents together.  \n",
    "\n",
    "After setting the number of **players** and **games**, it is possible to define a global number of possible **actions**, equal for every **players** in every **game**, or different number of **actions** for each **player** in each **game**.  \n",
    "\n",
    "Then, in every **game** defined, for every **action profile** *A* in the set of action profiles ***A*** = *a<sub>1</sub> x a<sub>2</sub> x ... x a<sub>i</sub>*, with *a<sub>i</sub>* set of possible actions for the *i-th* **player**, it is possible to set the **probability** of the transition towards all the **games** in the environment, along with the associated **payoff**.  \n",
    "\n",
    "The graph below the settings interface shows the current state space and the possible transitions between states, with the associated **probability** or **reward**, depending on the option chosen.\n",
    "\n",
    "> **❗Note❗**: Don't worry if this sounds overwhealmig, you can always use the presets to get started."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "92a90b50f5f04f2b8a8958f6868eca38",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "VBox(children=(VBox(children=(Dropdown(description='Labels:', options=('Transition Probabilities', 'Payoffs'),…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\Federico Rocca\\AppData\\Local\\Packages\\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\\LocalCache\\local-packages\\Python310\\site-packages\\netgraph\\_node_layout.py:949: UserWarning: Maximum number of iterations reached. Aborting further node layout optimisations.\n",
      "  warnings.warn(\"Maximum number of iterations reached. Aborting further node layout optimisations.\")\n"
     ]
    }
   ],
   "source": [
    "env = Environment()\n",
    "envGraph = EnvGraphDisplay(env, timeBuffer=0.5)\n",
    "envWidget = EnvironmentWidget(env)\n",
    "vBox = widgets.VBox([ envGraph.get_widget(), envWidget.getWidget()])\n",
    "display(vBox)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. Presets\n",
    "Here you can load one of the presets that we have prepared for you. These are designed to show you the capabilities of the Nash-Q learning algorithm in different scenarios.\n",
    "\n",
    "> Remember that in the next widget you can always view and edit the environment to your liking.\n",
    "\n",
    "The presets are:\n",
    "\n",
    "0. **Empty preset**: This only selects the minimum number of players (2) and 2 games without any transitions or payoffs. View this preset as a blank canvas for your first simple envs.\n",
    "\n",
    "1. **Basic 2-2**: Defines an environment with 2 players and 2 games, with deterministic transitions and equal payoffs for each player on every transition.\n",
    "In this setup the most desirable game to be in is  game 1, where it is best for them to stay because of the payoffs. This means that the only Nash-Q equilibrium is playing the action profile (0, 0) in both the games.\n",
    "\n",
    "2. **Basic 3-2**: Defines an environment with 3 players and 2 games with the same properties of the last one, here the *Nash-Q equilibrium* consists of playing the **action profile** (0, 0, 0) in each game.\n",
    "\n",
    "3. **Basic 4-2**: Once again same settings but this time with 4 players. The *Nash-Q equilibrium* is the action profile (0, 0, 0, 0)\n",
    "\n",
    "4. **Stochastic \"Prisoner's Dilemma\"**: A more structured preset that tries to reinterpret the famous \"Prisoner's Dilemma\" in a stochastic game setting. More in details there are 2 players that can either be free (F) or imprisoned (I), this implies the existence of 4 games:\n",
    "\n",
    "    **0 - FF** - In this game the players are both free. Unfortunatly their nature doesn't allow them to be free for long. Sooner or later they always end up committing a crime and years of friendship compels them to commit crimes together. The same feelings are inevitably tested though, because they inevitably end up caught.\n",
    "    Each of them is then presented with a choice: they can either stay silent (S) or testify against the other (T). If both player confess they will get 2 years of prison each, if one confesses and the other stays silent the one that confessed will be free and the other will get 3 years of prison, if they both stay silent they will get 1 year of prison each.\n",
    "    The payoffs are set to reflect these conditions.\n",
    "    The actions SS and TT lead to the game 1 (II), while the actions TS and ST lead to the game 2 (FI) and 3 (IF) respectively.\n",
    "    The payoffs are set in such a way that there is no difference in testifing or not if the other testifies, but remaining silent when the other testifies causes a pretty big negative reward. In this game the Nash-Q equilibrium is testifing for both players (C, C).\n",
    "    \n",
    "    **1 - II** - In this game both players are imprisoned but not all hopes are lost. Since they are both in prison they can work together in trying to escape through the sewers. Unfortunatly only one can fit and then the allarm will go off. They then decide to play a game of rock, paper, scissors to decide who will be released. The winner will be released and the other will have his sentence increased for tring to escape. If they tie they will both return to their cell but still get a smaller increase in their sentence.\n",
    "    This means that they can play 3 actions: rock (R), paper (P), and scissors (S). The one that wins gets a positive reward because he gets released, while the one that loses gets a bigger negative reward because his sentence is increased; in the case of a tie, they both get a small negative reward, because they need to stay in prison with their sentence.  \n",
    "\n",
    "    **2 FI** - In game 2 one of the criminal has testified, while the other has remained silent, so the first is free and the second is in prison. The free one has the choice between *(H)* helping the other criminal escape, or *(I)* ignoring him, while the prisoner can choose between *(E)* escaping or *(R)* remaining in prison. If the first tries to help and the other wants to escape *(H, E)* they both get a small positive reward for tasting freedom once again, while if the second decides to remain in prison *(H, R)*, he gets a small negative reward while the other gets a negative reward because he gets caught in the helping; if the first decides not to help and the second tries to escape *(I, E)*, the second gets a negative reward because he can't actually escape without help, and his sentence is increased, while the first gets a neutral reward; if the second remains in prison *(I, R)*, the second gets a small negative reward while the first still gets a neutral reward.  \n",
    "    \n",
    "    **3 IF** - Game 3 is the same as number 2, but with reversed roles.\n",
    "\n",
    "5. **\"Little(grid)world\"**: a small version of a grid-world game where the grid is composed of four tiles in an upside-down T shape (3 horizontal tiles and one on top of the middle one). There are two agents on opposite sides of the board, their aim is to reach the other’s player position. The two agents can perform three actions each: they can either go up (a 0), go down (a 2) or move towards the opposite side of the board to their starting position (a 1).  When hitting a wall, the agent gets a unitary negative reward, when the two agents collide, they both get a double negative reward (-2). In the case of one agent reaching its final position, it gets a strong positive reward (+5), independently if a wall or another agent had been hit.\n",
    "> **❗Note❗**: The graph representation of the game number 5 might be quite hard to comprehend and slow to render (the number of possible states for the system is 16 and the possible actions for each state are 9). For graphic representation refer to the report published on github (https://github.com/MultiagentSystemsProject-Polimi2024/LearningNashQLearning.git). "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "0c35df4d5a0a4df187382e660379822f",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Dropdown(description='Preset:', options=(0, 1, 2, 3, 4, 5), value=0)"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "preset = PresetGames(env)\n",
    "display(preset.getWidget())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. Training of the agents ##\n",
    "In the following section, it will be possible to set the parameters for the trainig phase.  \n",
    "The paramethers that can be personalized are the following:\n",
    "- *Episodes*: number of training episodes\n",
    "- *Gamma*: **discount factor**, that defines the reduction in the values of the rewards from actrions taken in the future\n",
    "- *Epsilon*: **exploration-exploitation** paramether, defined as the probability with which the algorithm choses a random action (exploration) over the best possible action (exploitation)\n",
    "- *Pure epsilon epidoses*: number of episodes after which *epsilon* starts decreasing, opting more and more for *exploitation* over *exploration*\n",
    "- *Alfa*: **learning rate**, determines the weight of the updates on the values already known, as the speed at which the agent learns\n",
    "- *Pure training episodes*: number of trainings after which *alpha* starts decreasing, reducing the weight of the new updates\n",
    "- *Reset on goal state*: sets the environment in such a way that after reaching the goal state, the agents are brought back to the starting state, in order to learn again from the beginning\n",
    "- *Start state*: defines from which state the agents will start\n",
    "- *Goal state*: defines the goal state\n",
    "> **❗Note❗**: When using preset number 5, *\"Little(grid)world\"*, the *Start State* should be set to 2 and the *Goal State* should be set to 8; that is bececause, given the nature of states as combinations of the agents' positions, state 0 corresponds to both the agents in the left-most cell, while state 8 corresponds to the switched final position. It is also suggested to tick the *Reset on goal state* checkbox have the agents' be brought back to the initial position after reaching the goal, before continuing the learning.\n",
    "\n",
    "After that, the training phase can be run by pressing the *Train* button.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "13d7262a3e5c4226a5c35579b8e00876",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "GridBox(children=(IntText(value=1, description='Steps:'), FloatText(value=0.8, description='Gamma:'), FloatTex…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "nashQLearning = NashQLearning(env)\n",
    "display(nashQLearning.getDisplayable())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4. Results ##\n",
    "The last section shows the results from the training.\n",
    "\n",
    "First of all you can define a *window size*, as a percentage of the number of episodes, that will be used to smooth the rewards, in order to make them more readable in the graph. Lower values of *window size* allow you to better see the variations in the rewards, while higher values of this paramether make the graph more smooth, reducing the variations.\n",
    "\n",
    "The reward graph displays the value of the reward received by every player, and the sum of their reward, for every training episode; by using the slider, by typing a value in, or by navigating using the arrows, it is possible to select a specific episode, and see its rewards signaled by the red vertical line. It is also possible to automatically advance in the episodes, by pressing the play button, whith the desired speed, as the number of episodes advanced for each second.\n",
    "\n",
    "When a training episode is selected, all the information relative to that are displayed below:\n",
    "- the state-space graph shows the states, highlighting the current state and the transition that has been taken in the state; depending on your selection in the dropdown menues, it can show the **Q-tables** or the **policy** that are associated to the states in the episode chosen, and relative to the selected agent. On the edge of the graph relative to the current action and transition, it is also possible to see the differences from the previous values in the *Q-Table*.\n",
    "- below that, the interface shows the *current game*, the *current action profile*, meaning the actions taken by every agent in the state, the *current payoff*, as the reward received by every agent when taking the chosen action in the current state, and finally the *Current Policy* and the *Q-Tables*, for each player in each game, relative to the chosen episode. You should note that the *Q-Tables* will be shown only when the number of players is less than 3, because it wouldn't be possible to display a structure with more than 2 dimentions in the interface.\n",
    "\n",
    "It is also possible to tick the *Display Grid* checkbox in order to activate a grid representation of the state-space, aviable when the number of agents is <= 2 and the number of states is <= 16; it is suggested when using preset number 5, *\"Little(grid)world\"*, given its complicated graph representation. The state will be interpreted as a number in base 4 (the number of possible positions for the agents) and each digit will represent the position of the corresponding agent. The grid will show the position of the agents in the state, and the possible transitions between states as arrows"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "finalDisplay = FinalDisplay(nashQLearning, env)\n",
    "display(finalDisplay.get_widget())"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.11"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}