diff --git a/README.md b/README.md index 3d985389f..da09d1903 100644 --- a/README.md +++ b/README.md @@ -16,7 +16,8 @@ Triage is designed to: ## Quick Links -- [Dirty Duck Tutorial](https://dssg.github.io/triage/dirtyduck/) - Are you completely new to Triage? Go through the tutorial here with sample data +- [Tutorial on Google Colab](https://colab.research.google.com/github/dssg/triage/blob/master/example/colab/colab_triage.ipynb) - Are you completely new to Triage? Run through a quick tutorial hosted on google colab (no setup necessary) to see what triage can do! +- [Dirty Duck Tutorial](https://dssg.github.io/triage/dirtyduck/) - Want a more in-depth walk through of triage's functionality and concepts? Go through the dirty duck tutorial here with sample data - [QuickStart Guide](https://dssg.github.io/triage/quickstart/) - Try Triage out with your own project and data - [Triage Documentation Site](https://dssg.github.io/triage/) - Used Triage before and want more reference documentation? - [Development](https://github.com/dssg/triage#development) - Contribute to Triage development. diff --git a/docs/mkdocs.yml b/docs/mkdocs.yml index 7721f4d58..8e4b13f08 100644 --- a/docs/mkdocs.yml +++ b/docs/mkdocs.yml @@ -81,6 +81,7 @@ plugins: nav: - Home: index.md + - Online Tutorial (Google Colab): https://colab.research.google.com/github/dssg/triage/blob/master/example/colab/colab_triage.ipynb - Get started with your own project: - Quickstart guide: quickstart.md - Suggested workflow: triage_project_workflow.md diff --git a/docs/sources/index.md b/docs/sources/index.md index a2c1f04c4..a0d8d68c0 100644 --- a/docs/sources/index.md +++ b/docs/sources/index.md @@ -1,122 +1,26 @@ -Triage -====== +# Triage -Data Science Toolkit for Social Good and Public Policy Problems +[![Build Status](https://travis-ci.org/dssg/triage.svg?branch=master)](https://travis-ci.org/dssg/triage) +[![codecov](https://codecov.io/gh/dssg/triage/branch/master/graph/badge.svg)](https://codecov.io/gh/dssg/triage) +[![codeclimate](https://codeclimate.com/github/dssg/triage.png)](https://codeclimate.com/github/dssg/triage) -[![image](https://travis-ci.com/dssg/triage.svg?branch=master)](https://travis-ci.org/dssg/triage) -[![image](https://codecov.io/gh/dssg/triage/branch/master/graph/badge.svg)](https://codecov.io/gh/dssg/triage) -[![image](https://codeclimate.com/github/dssg/triage.png)](https://codeclimate.com/github/dssg/triage) -Building data science systems requires answering many design questions, turning them into modeling choices, which in turn run machine learning models. Questions such as cohort selection, unit of analysis determination, outcome determination, feature (explanantory variables) generation, model/classifier training, evaluation, selection, and list generation are often complicated and hard to choose apriori. In addition, once these choices are made, they have to be combined in different ways throughout the course of a project. +## What is Triage? -Triage is designed to: +Triage is an open source machine learning toolkit to help data scientists, machine learning developers, and analysts quickly prototype, build and evaluate end-to-end predictive risk modeling systems for public policy and social good problem. -- Guide users (data scientists, analysts, researchers) through these design choices by highlighting critical operational use questions. -- Provide an integrated interface to components that are needed throughout a data science project workflow. +While many tools (sklearn, keras, pytorch, etc.) exist to build ML models, an end-to-end project requires a lot more than just building models. Developing data science systems requires making many design decisions that need to match with how the system is going to be used. These choices then get turned into modeling choices and code. Triage lets you focus on the problem you’re solving and guides you through design choices you need to make at each step of the machine learning pipeline. -## Quick Links +## How to get started with Triage? -- [Dirty Duck Tutorial](https://dssg.github.io/triage/dirtyduck/) - Are you completely new to Triage? Go through the tutorial here with sample data -- [QuickStart Guide](https://dssg.github.io/triage/quickstart/) - Try Triage out with your own project and data -- [Triage Documentation Site](https://dssg.github.io/triage/) - Used Triage before and want more reference documentation? -- [Development](https://github.com/dssg/triage#development) - Contribute to Triage development. +### [Go through a quick online tutorial with sample data (no setup required)](https://colab.research.google.com/github/dssg/triage/blob/master/example/colab/colab_triage.ipynb) -## Installation +### [Go through a more in-depth tutorial with sample data](dirtyduck/index.md) -To install Triage, you need: +### [Get started with your own project and data](quickstart.md) -- Python 3.8+ -- A PostgreSQL 9.6+ database with your source data (events, - geographical data, etc) loaded. - - **NOTE**: If your database is PostgreSQL 11+ you will get some - speed improvements. We recommend to update to a recent - version of PostgreSQL. -- Ample space on an available disk, (or for example in Amazon Web - Services's S3), to store the needed matrices and models for your - experiments -We recommend starting with a new python virtual environment (with Python 3.6 or greater) and pip installing triage there. -```bash -$ virtualenv triage-env -$ . triage-env/bin/activate -(triage-env) $ pip install triage -``` +## Background -## Data -Triage needs data in a postgres database and a configuration file that has credentials for the database. The Triage CLI defaults database connection information to a file stored in 'database.yaml' (example in [example/database.yaml](https://github.com/dssg/triage/blob/master/example/database.yaml)). +Triage was initially developed at the University of Chicago's [Center For Data Science and Public Policy](http://dsapp.uchicago.edu) and is now being maintained and enhanced at Carnegie Mellon University. -If you don't want to install Postgres yourself, try `triage db up` to create a vanilla Postgres 12 database using docker. For more details on this command, check out [Triage Database Provisioner](db.md) - -## Configure Triage for your project - -Triage is configured with a config.yaml file that has parameters defined for each component. You can see some [sample configuration with explanations](https://github.com/dssg/triage/blob/master/example/config/experiment.yaml) to see what configuration looks like. - -## Using Triage - -1. Via CLI: -```bash - -triage experiment example/config/experiment.yaml -``` -2. Import as a python package: -```python -from triage.experiments import SingleThreadedExperiment - -experiment = SingleThreadedExperiment( - config=experiment_config, # a dictionary - db_engine=create_engine(...), # http://docs.sqlalchemy.org/en/latest/core/engines.html - project_path='/path/to/directory/to/save/data' # could be an S3 path too: 's3://mybucket/myprefix/' -) -experiment.run() -``` - -There are a plethora of options available for experiment running, affecting things like parallelization, storage, and more. These options are detailed in the [Running an Experiment](https://dssg.github.io/triage/experiments/running/) page. - -## Development - -Triag was initially developed at [University of Chicago's Center For Data Science and Public Policy](http://dsapp.uchicago.edu) and is now being maintained at Carnegie Mellon University. - -To build this package (without installation), its dependencies may -alternatively be installed from the terminal using `pip`: - - pip install -r requirement/main.txt - -### Testing - -To add test (and development) dependencies, use **test.txt**: - - pip install -r requirement/test.txt [-r requirement/dev.txt] - -Then, to run tests: - - pytest - -### Development Environment - -To quickly bootstrap a development environment, having cloned the -repository, invoke the executable `develop` script from your system -shell: - - ./develop - -A "wizard" will suggest set-up steps and optionally execute these, for -example: - - (install) begin - - (pyenv) installed - - (python-3.9.10) installed - - (virtualenv) installed - - (activation) installed - - (libs) install? - 1) yes, install {pip install -r requirement/main.txt -r requirement/test.txt -r requirement/dev.txt} - 2) no, ignore - #? 1 - -### Contributing - -If you'd like to contribute to Triage development, see the [CONTRIBUTING.md](https://github.com/dssg/triage/blob/master/CONTRIBUTING.md) document. diff --git a/docs/update_docs.py b/docs/update_docs.py index 2895ea8f2..51e0a9aaa 100644 --- a/docs/update_docs.py +++ b/docs/update_docs.py @@ -30,5 +30,5 @@ def copy_templates(): if __name__ == "__main__": #copy_templates() - update_index_md() + #update_index_md() #generate_api_docs() diff --git a/example/colab/colab_triage.ipynb b/example/colab/colab_triage.ipynb new file mode 100644 index 000000000..1bdc3829c --- /dev/null +++ b/example/colab/colab_triage.ipynb @@ -0,0 +1,5993 @@ +{ + "nbformat": 4, + "nbformat_minor": 0, + "metadata": { + "colab": { + "name": "colab_triage.ipynb", + "provenance": [], + "collapsed_sections": [], + "include_colab_link": true + }, + "kernelspec": { + "name": "python3", + "display_name": "Python 3" + }, + "language_info": { + "name": "python" + } + }, + "cells": [ + { + "cell_type": "markdown", + "metadata": { + "id": "view-in-github", + "colab_type": "text" + }, + "source": [ + "\"Open" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "4ix00QRsvd45" + }, + "source": [ + "# Colab Triage Tutorial\n", + "\n", + "## Problem Overview\n", + "\n", + "This notebook provides a quick, interactive tutorial for [triage](http://www.datasciencepublicpolicy.org/our-work/tools-guides/triage/), a python machine learning pipeline for social good problems, using a sample of the data provided by [DonorsChoose](https://www.donorschoose.org/) to the [2014 KDD Cup](https://www.kaggle.com/c/kdd-cup-2014-predicting-excitement-at-donors-choose/data). Public schools in the United States face large disparities in funding, often resulting in teachers and staff members filling these gaps by purchasing classroom supplies out of their own pockets. DonorsChoose is an online crowdfunding platform that tries to help fill this gap by allowing teachers to seek funding for projects and resources from the community (projects can include classroom basics like books and markers, larger items like lab equipment or musical instruments, specific experiences like field trips or guest speakers). Projects on DonorsChoose expire after 4 months, and if the target funding level isn't reached, the project receives no funding. Since its launch in 2000, the platform has helped fund over 2 million projects at schools across the US, but about 1/3 of the projects that are posted nevertheless fail to meet their goal and go unfunded.\n", + "\n", + "For the purposes of this tutorial, we'll imagine that DonorsChoose has hired a digital content expert who will review projects and help teachers improve their postings and increase their chances of reaching their funding threshold. Because this individualized review is a labor-intensive process, the digital content expert has time to review only 10% of the projects posted to the platform on a given day. \n", + "\n", + "### The Modeling Problem\n", + "\n", + "You are a data scientist working with DonorsChoose, and your task is to help this content expert focus their limited resources on projects that most need the help. As such, you want to build a model to identify projects that are least likely to be fully funded before they expire and pass them off to the digital content expert for review.\n", + "\n", + "In building that model, our unit of analysis (what triage calls a **cohort**) will be new projects right at the time they're posted, while the **label** we're seeing to predict is whether or not the project reaches its funding goal in the subsequent 4 months (before it expires), making our task a binary classification problem. In order to make this prediction, we'll develop **features** that include information we know about the project when it's posted as well as historical performance of other projects posted by this teacher, school, etc.\n", + "\n", + "### Outline of the Tutorial\n", + "\n", + "The remainder of this tutorial will focus on how to use `triage` to solve this problem. Starting from scratch, we'll:\n", + "- **Install our tools**, including triage and a postgres server.\n", + "- **Explore the data** to get familiar with its structure.\n", + "- **Formulate the project** to make sure the models we build meet the needs of the context (and see how to configure `triage` along the way).\n", + "- **Build models**, using `triage` to run the modeling pipeline.\n", + "- **Look at the results** to ensure they make sense.\n", + "- **Select the model to deploy** using the `audition` component of `triage`.\n", + "- **Audit our models for bias** using `aequitas`.\n", + "- **Lear about next steps** and where to go from here.\n", + "\n" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "PtAMABPn971u" + }, + "source": [ + "## Getting Set Up\n", + "\n", + "We'll need a few dependencies to run triage in a colab notebook:\n", + "- A local postgresql server (we'll use version 11)\n", + "- A simplified dataset loaded into this database (we'll use data from DonorsChoose)\n", + "- Triage and its dependencies (we'll use the current version in pypi)" + ] + }, + { + "cell_type": "code", + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/" + }, + "id": "-htIBoS7N4gK", + "outputId": "4eff8fd0-3ab1-43c6-b9eb-e8efaf7ab9ce" + }, + "source": [ + "# Install and start postgresql-11 server\n", + "!sudo apt-get -y -qq update\n", + "!wget --quiet -O - https://www.postgresql.org/media/keys/ACCC4CF8.asc | sudo apt-key add -\n", + "!echo \"deb http://apt.postgresql.org/pub/repos/apt/ `lsb_release -cs`-pgdg main\" |sudo tee /etc/apt/sources.list.d/pgdg.list\n", + "!sudo apt-get -y -qq update\n", + "!sudo apt-get -y -qq install postgresql-11 postgresql-client-11\n", + "!sudo service postgresql start\n", + "\n", + "# Setup a password `postgres` for username `postgres`\n", + "!sudo -u postgres psql -U postgres -c \"ALTER USER postgres PASSWORD 'postgres';\"\n", + "\n", + "# Setup a database with name `donors_choose` to be used\n", + "!sudo -u postgres psql -U postgres -c 'DROP DATABASE IF EXISTS donors_choose;'\n", + "\n", + "!sudo -u postgres psql -U postgres -c 'CREATE DATABASE donors_choose;'\n", + "\n", + "# Environment variables for connecting to the database\n", + "%env DEMO_DATABASE_NAME=donors_choose\n", + "%env DEMO_DATABASE_HOST=localhost\n", + "%env DEMO_DATABASE_PORT=5432\n", + "%env DEMO_DATABASE_USER=postgres\n", + "%env DEMO_DATABASE_PASS=postgres" + ], + "execution_count": 2, + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": [ + "OK\n", + "deb http://apt.postgresql.org/pub/repos/apt/ bionic-pgdg main\n", + "debconf: unable to initialize frontend: Dialog\n", + "debconf: (No usable dialog-like program is installed, so the dialog based frontend cannot be used. at /usr/share/perl5/Debconf/FrontEnd/Dialog.pm line 76, <> line 16.)\n", + "debconf: falling back to frontend: Readline\n", + "debconf: unable to initialize frontend: Readline\n", + "debconf: (This frontend requires a controlling tty.)\n", + "debconf: falling back to frontend: Teletype\n", + "dpkg-preconfigure: unable to re-open stdin: \n", + "Selecting previously unselected package cron.\n", + "(Reading database ... 155639 files and directories currently installed.)\n", + "Preparing to unpack .../00-cron_3.0pl1-128.1ubuntu1.2_amd64.deb ...\n", + "Unpacking cron (3.0pl1-128.1ubuntu1.2) ...\n", + "Selecting previously unselected package logrotate.\n", + "Preparing to unpack .../01-logrotate_3.11.0-0.1ubuntu1_amd64.deb ...\n", + "Unpacking logrotate (3.11.0-0.1ubuntu1) ...\n", + "Selecting previously unselected package netbase.\n", + "Preparing to unpack .../02-netbase_5.4_all.deb ...\n", + "Unpacking netbase (5.4) ...\n", + "Selecting previously unselected package libcommon-sense-perl.\n", + "Preparing to unpack .../03-libcommon-sense-perl_3.74-2build2_amd64.deb ...\n", + "Unpacking libcommon-sense-perl (3.74-2build2) ...\n", + "Selecting previously unselected package libjson-perl.\n", + "Preparing to unpack .../04-libjson-perl_2.97001-1_all.deb ...\n", + "Unpacking libjson-perl (2.97001-1) ...\n", + "Selecting previously unselected package libtypes-serialiser-perl.\n", + "Preparing to unpack .../05-libtypes-serialiser-perl_1.0-1_all.deb ...\n", + "Unpacking libtypes-serialiser-perl (1.0-1) ...\n", + "Selecting previously unselected package libjson-xs-perl.\n", + "Preparing to unpack .../06-libjson-xs-perl_3.040-1_amd64.deb ...\n", + "Unpacking libjson-xs-perl (3.040-1) ...\n", + "Preparing to unpack .../07-libpq-dev_14.4-1.pgdg18.04+1_amd64.deb ...\n", + "Unpacking libpq-dev (14.4-1.pgdg18.04+1) over (10.21-0ubuntu0.18.04.1) ...\n", + "Preparing to unpack .../08-libpq5_14.4-1.pgdg18.04+1_amd64.deb ...\n", + "Unpacking libpq5:amd64 (14.4-1.pgdg18.04+1) over (10.21-0ubuntu0.18.04.1) ...\n", + "Selecting previously unselected package pgdg-keyring.\n", + "Preparing to unpack .../09-pgdg-keyring_2018.2_all.deb ...\n", + "Unpacking pgdg-keyring (2018.2) ...\n", + "Selecting previously unselected package postgresql-client-common.\n", + "Preparing to unpack .../10-postgresql-client-common_241.pgdg18.04+1_all.deb ...\n", + "Unpacking postgresql-client-common (241.pgdg18.04+1) ...\n", + "Selecting previously unselected package postgresql-client-11.\n", + "Preparing to unpack .../11-postgresql-client-11_11.16-1.pgdg18.04+1_amd64.deb ...\n", + "Unpacking postgresql-client-11 (11.16-1.pgdg18.04+1) ...\n", + "Selecting previously unselected package ssl-cert.\n", + "Preparing to unpack .../12-ssl-cert_1.0.39_all.deb ...\n", + "Unpacking ssl-cert (1.0.39) ...\n", + "Selecting previously unselected package postgresql-common.\n", + "Preparing to unpack .../13-postgresql-common_241.pgdg18.04+1_all.deb ...\n", + "Adding 'diversion of /usr/bin/pg_config to /usr/bin/pg_config.libpq-dev by postgresql-common'\n", + "Unpacking postgresql-common (241.pgdg18.04+1) ...\n", + "Selecting previously unselected package postgresql-11.\n", + "Preparing to unpack .../14-postgresql-11_11.16-1.pgdg18.04+1_amd64.deb ...\n", + "Unpacking postgresql-11 (11.16-1.pgdg18.04+1) ...\n", + "Selecting previously unselected package sysstat.\n", + "Preparing to unpack .../15-sysstat_11.6.1-1ubuntu0.1_amd64.deb ...\n", + "Unpacking sysstat (11.6.1-1ubuntu0.1) ...\n", + "Setting up libcommon-sense-perl (3.74-2build2) ...\n", + "Setting up sysstat (11.6.1-1ubuntu0.1) ...\n", + "debconf: unable to initialize frontend: Dialog\n", + "debconf: (No usable dialog-like program is installed, so the dialog based frontend cannot be used. at /usr/share/perl5/Debconf/FrontEnd/Dialog.pm line 76.)\n", + "debconf: falling back to frontend: Readline\n", + "\n", + "Creating config file /etc/default/sysstat with new version\n", + "update-alternatives: using /usr/bin/sar.sysstat to provide /usr/bin/sar (sar) in auto mode\n", + "Created symlink /etc/systemd/system/multi-user.target.wants/sysstat.service → /lib/systemd/system/sysstat.service.\n", + "Setting up ssl-cert (1.0.39) ...\n", + "debconf: unable to initialize frontend: Dialog\n", + "debconf: (No usable dialog-like program is installed, so the dialog based frontend cannot be used. at /usr/share/perl5/Debconf/FrontEnd/Dialog.pm line 76.)\n", + "debconf: falling back to frontend: Readline\n", + "Setting up libtypes-serialiser-perl (1.0-1) ...\n", + "Setting up libpq5:amd64 (14.4-1.pgdg18.04+1) ...\n", + "Setting up pgdg-keyring (2018.2) ...\n", + "Removing apt.postgresql.org key from trusted.gpg: OK\n", + "Setting up libjson-perl (2.97001-1) ...\n", + "Setting up cron (3.0pl1-128.1ubuntu1.2) ...\n", + "Adding group `crontab' (GID 110) ...\n", + "Done.\n", + "Created symlink /etc/systemd/system/multi-user.target.wants/cron.service → /lib/systemd/system/cron.service.\n", + "update-rc.d: warning: start and stop actions are no longer supported; falling back to defaults\n", + "invoke-rc.d: could not determine current runlevel\n", + "invoke-rc.d: policy-rc.d denied execution of start.\n", + "Setting up logrotate (3.11.0-0.1ubuntu1) ...\n", + "Setting up netbase (5.4) ...\n", + "Setting up libpq-dev (14.4-1.pgdg18.04+1) ...\n", + "Setting up libjson-xs-perl (3.040-1) ...\n", + "Setting up postgresql-client-common (241.pgdg18.04+1) ...\n", + "Setting up postgresql-common (241.pgdg18.04+1) ...\n", + "debconf: unable to initialize frontend: Dialog\n", + "debconf: (No usable dialog-like program is installed, so the dialog based frontend cannot be used. at /usr/share/perl5/Debconf/FrontEnd/Dialog.pm line 76.)\n", + "debconf: falling back to frontend: Readline\n", + "Adding user postgres to group ssl-cert\n", + "\n", + "Creating config file /etc/postgresql-common/createcluster.conf with new version\n", + "Building PostgreSQL dictionaries from installed myspell/hunspell packages...\n", + "Removing obsolete dictionary files:\n", + "Created symlink /etc/systemd/system/multi-user.target.wants/postgresql.service → /lib/systemd/system/postgresql.service.\n", + "Setting up postgresql-client-11 (11.16-1.pgdg18.04+1) ...\n", + "update-alternatives: using /usr/share/postgresql/11/man/man1/psql.1.gz to provide /usr/share/man/man1/psql.1.gz (psql.1.gz) in auto mode\n", + "Setting up postgresql-11 (11.16-1.pgdg18.04+1) ...\n", + "debconf: unable to initialize frontend: Dialog\n", + "debconf: (No usable dialog-like program is installed, so the dialog based frontend cannot be used. at /usr/share/perl5/Debconf/FrontEnd/Dialog.pm line 76.)\n", + "debconf: falling back to frontend: Readline\n", + "Creating new PostgreSQL cluster 11/main ...\n", + "/usr/lib/postgresql/11/bin/initdb -D /var/lib/postgresql/11/main --auth-local peer --auth-host md5\n", + "The files belonging to this database system will be owned by user \"postgres\".\n", + "This user must also own the server process.\n", + "\n", + "The database cluster will be initialized with locale \"en_US.UTF-8\".\n", + "The default database encoding has accordingly been set to \"UTF8\".\n", + "The default text search configuration will be set to \"english\".\n", + "\n", + "Data page checksums are disabled.\n", + "\n", + "fixing permissions on existing directory /var/lib/postgresql/11/main ... ok\n", + "creating subdirectories ... ok\n", + "selecting default max_connections ... 100\n", + "selecting default shared_buffers ... 128MB\n", + "selecting default timezone ... Etc/UTC\n", + "selecting dynamic shared memory implementation ... posix\n", + "creating configuration files ... ok\n", + "running bootstrap script ... ok\n", + "performing post-bootstrap initialization ... ok\n", + "syncing data to disk ... ok\n", + "\n", + "Success. You can now start the database server using:\n", + "\n", + " pg_ctlcluster 11 main start\n", + "\n", + "update-alternatives: using /usr/share/postgresql/11/man/man1/postmaster.1.gz to provide /usr/share/man/man1/postmaster.1.gz (postmaster.1.gz) in auto mode\n", + "invoke-rc.d: could not determine current runlevel\n", + "invoke-rc.d: policy-rc.d denied execution of start.\n", + "Processing triggers for systemd (237-3ubuntu10.53) ...\n", + "Processing triggers for man-db (2.8.3-2ubuntu0.1) ...\n", + "Processing triggers for libc-bin (2.27-3ubuntu1.3) ...\n", + "/sbin/ldconfig.real: /usr/local/lib/python3.7/dist-packages/ideep4py/lib/libmkldnn.so.0 is not a symbolic link\n", + "\n", + " * Starting PostgreSQL 11 database server\n", + " ...done.\n", + "ALTER ROLE\n", + "NOTICE: database \"donors_choose\" does not exist, skipping\n", + "DROP DATABASE\n", + "CREATE DATABASE\n", + "env: DEMO_DATABASE_NAME=donors_choose\n", + "env: DEMO_DATABASE_HOST=localhost\n", + "env: DEMO_DATABASE_PORT=5432\n", + "env: DEMO_DATABASE_USER=postgres\n", + "env: DEMO_DATABASE_PASS=postgres\n" + ] + } + ] + }, + { + "cell_type": "code", + "metadata": { + "id": "3mWNhJ2rOVtS" + }, + "source": [ + "# Download sampled DonorsChoose data and load it into our postgres server\n", + "!curl -s -OL https://dsapp-public-data-migrated.s3.us-west-2.amazonaws.com/donors_sampled_20210920_v3.dmp\n", + "!PGPASSWORD=$DEMO_DATABASE_PASS pg_restore -h $DEMO_DATABASE_HOST -p $DEMO_DATABASE_PORT -d $DEMO_DATABASE_NAME -U $DEMO_DATABASE_USER -O -j 8 donors_sampled_20210920_v3.dmp" + ], + "execution_count": 3, + "outputs": [] + }, + { + "cell_type": "code", + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/", + "height": 1000 + }, + "id": "t5E-9VRjRlSk", + "outputId": "8c470c70-6b25-4cca-c102-7d88907281c3" + }, + "source": [ + "# Install triage and its dependencies\n", + "!pip install triage" + ], + "execution_count": 4, + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": [ + "Looking in indexes: https://pypi.org/simple, https://us-python.pkg.dev/colab-wheels/public/simple/\n", + "Collecting triage\n", + " Downloading triage-5.1.1-py2.py3-none-any.whl (250 kB)\n", + "\u001b[K |████████████████████████████████| 250 kB 13.6 MB/s \n", + "\u001b[?25hCollecting matplotlib==3.3.4\n", + " Downloading matplotlib-3.3.4-cp37-cp37m-manylinux1_x86_64.whl (11.5 MB)\n", + "\u001b[K |████████████████████████████████| 11.5 MB 77.9 MB/s \n", + "\u001b[?25hCollecting pebble==4.5.3\n", + " Downloading Pebble-4.5.3-py2.py3-none-any.whl (24 kB)\n", + "Collecting signalled-timeout==1.0.0\n", + " Downloading signalled-timeout-1.0.0.tar.gz (2.7 kB)\n", + "Collecting seaborn==0.10.1\n", + " Downloading seaborn-0.10.1-py3-none-any.whl (215 kB)\n", + "\u001b[K |████████████████████████████████| 215 kB 66.9 MB/s \n", + "\u001b[?25hCollecting pandas==1.0.5\n", + " Downloading pandas-1.0.5-cp37-cp37m-manylinux1_x86_64.whl (10.1 MB)\n", + "\u001b[K |████████████████████████████████| 10.1 MB 45.5 MB/s \n", + "\u001b[?25hCollecting s3fs==0.4.2\n", + " Downloading s3fs-0.4.2-py3-none-any.whl (19 kB)\n", + "Collecting sqlalchemy-postgres-copy==0.5.0\n", + " Downloading sqlalchemy_postgres_copy-0.5.0-py2.py3-none-any.whl (6.6 kB)\n", + "Requirement already satisfied: click==7.1.2 in /usr/local/lib/python3.7/dist-packages (from triage) (7.1.2)\n", + "Collecting python-dateutil==2.8.1\n", + " Downloading python_dateutil-2.8.1-py2.py3-none-any.whl (227 kB)\n", + "\u001b[K |████████████████████████████████| 227 kB 72.6 MB/s \n", + "\u001b[?25hCollecting boto3==1.14.45\n", + " Downloading boto3-1.14.45-py2.py3-none-any.whl (129 kB)\n", + "\u001b[K |████████████████████████████████| 129 kB 70.7 MB/s \n", + "\u001b[?25hCollecting Dickens==1.0.1\n", + " Downloading Dickens-1.0.1.tar.gz (2.3 kB)\n", + "Collecting argcmdr==0.7.0\n", + " Downloading argcmdr-0.7.0-py3-none-any.whl (33 kB)\n", + "Collecting scipy==1.5.0\n", + " Downloading scipy-1.5.0-cp37-cp37m-manylinux1_x86_64.whl (25.9 MB)\n", + "\u001b[K |████████████████████████████████| 25.9 MB 81.5 MB/s \n", + "\u001b[?25hCollecting numpy==1.21.1\n", + " Downloading numpy-1.21.1-cp37-cp37m-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (15.7 MB)\n", + "\u001b[K |████████████████████████████████| 15.7 MB 36.1 MB/s \n", + "\u001b[?25hCollecting graphviz==0.14\n", + " Downloading graphviz-0.14-py2.py3-none-any.whl (18 kB)\n", + "Collecting requests==2.24.0\n", + " Downloading requests-2.24.0-py2.py3-none-any.whl (61 kB)\n", + "\u001b[K |████████████████████████████████| 61 kB 509 kB/s \n", + "\u001b[?25hCollecting retrying==1.3.3\n", + " Downloading retrying-1.3.3.tar.gz (10 kB)\n", + "Collecting aequitas==0.42.0\n", + " Downloading aequitas-0.42.0-py3-none-any.whl (2.2 MB)\n", + "\u001b[K |████████████████████████████████| 2.2 MB 47.9 MB/s \n", + "\u001b[?25hCollecting alembic==1.4.2\n", + " Downloading alembic-1.4.2.tar.gz (1.1 MB)\n", + "\u001b[K |████████████████████████████████| 1.1 MB 67.3 MB/s \n", + "\u001b[?25h Installing build dependencies ... \u001b[?25l\u001b[?25hdone\n", + " Getting requirements to build wheel ... \u001b[?25l\u001b[?25hdone\n", + " Preparing wheel metadata ... \u001b[?25l\u001b[?25hdone\n", + "Collecting verboselogs==1.7\n", + " Downloading verboselogs-1.7-py2.py3-none-any.whl (11 kB)\n", + "Requirement already satisfied: sqlparse==0.4.2 in /usr/local/lib/python3.7/dist-packages (from triage) (0.4.2)\n", + "Collecting PyYAML==5.4.1\n", + " Downloading PyYAML-5.4.1-cp37-cp37m-manylinux1_x86_64.whl (636 kB)\n", + "\u001b[K |████████████████████████████████| 636 kB 56.6 MB/s \n", + "\u001b[?25hCollecting psycopg2-binary==2.8.5\n", + " Downloading psycopg2_binary-2.8.5-cp37-cp37m-manylinux1_x86_64.whl (2.9 MB)\n", + "\u001b[K |████████████████████████████████| 2.9 MB 50.2 MB/s \n", + "\u001b[?25hCollecting SQLAlchemy==1.3.18\n", + " Downloading SQLAlchemy-1.3.18-cp37-cp37m-manylinux2010_x86_64.whl (1.3 MB)\n", + "\u001b[K |████████████████████████████████| 1.3 MB 49.3 MB/s \n", + "\u001b[?25hCollecting adjustText==0.7.3\n", + " Downloading adjustText-0.7.3.tar.gz (7.5 kB)\n", + "Collecting wrapt==1.13.3\n", + " Downloading wrapt-1.13.3-cp37-cp37m-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl (79 kB)\n", + "\u001b[K |████████████████████████████████| 79 kB 9.2 MB/s \n", + "\u001b[?25hCollecting scikit-learn==0.23.1\n", + " Downloading scikit_learn-0.23.1-cp37-cp37m-manylinux1_x86_64.whl (6.8 MB)\n", + "\u001b[K |████████████████████████████████| 6.8 MB 52.4 MB/s \n", + "\u001b[?25hCollecting ohio==0.5.0\n", + " Downloading ohio-0.5.0-py3-none-any.whl (26 kB)\n", + "Collecting coloredlogs==14.0\n", + " Downloading coloredlogs-14.0-py2.py3-none-any.whl (43 kB)\n", + "\u001b[K |████████████████████████████████| 43 kB 2.4 MB/s \n", + "\u001b[?25hCollecting inflection==0.5.0\n", + " Downloading inflection-0.5.0-py2.py3-none-any.whl (5.8 kB)\n", + "Collecting xhtml2pdf==0.2.2\n", + " Downloading xhtml2pdf-0.2.2.tar.gz (97 kB)\n", + "\u001b[K |████████████████████████████████| 97 kB 7.7 MB/s \n", + "\u001b[?25hCollecting tabulate==0.8.2\n", + " Downloading tabulate-0.8.2.tar.gz (45 kB)\n", + "\u001b[K |████████████████████████████████| 45 kB 4.6 MB/s \n", + "\u001b[?25hCollecting markdown2==2.3.5\n", + " Downloading markdown2-2.3.5.zip (161 kB)\n", + "\u001b[K |████████████████████████████████| 161 kB 23.0 MB/s \n", + "\u001b[?25hCollecting altair==4.1.0\n", + " Downloading altair-4.1.0-py3-none-any.whl (727 kB)\n", + "\u001b[K |████████████████████████████████| 727 kB 68.1 MB/s \n", + "\u001b[?25hCollecting millify==0.1.1\n", + " Downloading millify-0.1.1.tar.gz (1.2 kB)\n", + "Collecting Flask-Bootstrap==3.3.7.1\n", + " Downloading Flask-Bootstrap-3.3.7.1.tar.gz (456 kB)\n", + "\u001b[K |████████████████████████████████| 456 kB 69.4 MB/s \n", + "\u001b[?25hCollecting Flask==0.12.2\n", + " Downloading Flask-0.12.2-py2.py3-none-any.whl (83 kB)\n", + "\u001b[K |████████████████████████████████| 83 kB 1.6 MB/s \n", + "\u001b[?25hCollecting Mako\n", + " Downloading Mako-1.2.0-py3-none-any.whl (78 kB)\n", + "\u001b[K |████████████████████████████████| 78 kB 8.4 MB/s \n", + "\u001b[?25hCollecting python-editor>=0.3\n", + " Downloading python_editor-1.0.4-py3-none-any.whl (4.9 kB)\n", + "Requirement already satisfied: jsonschema in /usr/local/lib/python3.7/dist-packages (from altair==4.1.0->aequitas==0.42.0->triage) (4.3.3)\n", + "Requirement already satisfied: entrypoints in /usr/local/lib/python3.7/dist-packages (from altair==4.1.0->aequitas==0.42.0->triage) (0.4)\n", + "Requirement already satisfied: toolz in /usr/local/lib/python3.7/dist-packages (from altair==4.1.0->aequitas==0.42.0->triage) (0.11.2)\n", + "Requirement already satisfied: jinja2 in /usr/local/lib/python3.7/dist-packages (from altair==4.1.0->aequitas==0.42.0->triage) (2.11.3)\n", + "Collecting argcomplete==1.9.4\n", + " Downloading argcomplete-1.9.4-py2.py3-none-any.whl (36 kB)\n", + "Collecting plumbum==1.6.4\n", + " Downloading plumbum-1.6.4-py2.py3-none-any.whl (110 kB)\n", + "\u001b[K |████████████████████████████████| 110 kB 73.9 MB/s \n", + "\u001b[?25hCollecting botocore<1.18.0,>=1.17.45\n", + " Downloading botocore-1.17.63-py2.py3-none-any.whl (6.6 MB)\n", + "\u001b[K |████████████████████████████████| 6.6 MB 53.5 MB/s \n", + "\u001b[?25hCollecting jmespath<1.0.0,>=0.7.1\n", + " Downloading jmespath-0.10.0-py2.py3-none-any.whl (24 kB)\n", + "Collecting s3transfer<0.4.0,>=0.3.0\n", + " Downloading s3transfer-0.3.7-py2.py3-none-any.whl (73 kB)\n", + "\u001b[K |████████████████████████████████| 73 kB 2.4 MB/s \n", + "\u001b[?25hCollecting humanfriendly>=7.1\n", + " Downloading humanfriendly-10.0-py2.py3-none-any.whl (86 kB)\n", + "\u001b[K |████████████████████████████████| 86 kB 7.5 MB/s \n", + "\u001b[?25hRequirement already satisfied: Werkzeug>=0.7 in /usr/local/lib/python3.7/dist-packages (from Flask==0.12.2->aequitas==0.42.0->triage) (1.0.1)\n", + "Requirement already satisfied: itsdangerous>=0.21 in /usr/local/lib/python3.7/dist-packages (from Flask==0.12.2->aequitas==0.42.0->triage) (1.1.0)\n", + "Collecting dominate\n", + " Downloading dominate-2.6.0-py2.py3-none-any.whl (29 kB)\n", + "Collecting visitor\n", + " Downloading visitor-0.1.3.tar.gz (3.3 kB)\n", + "Requirement already satisfied: cycler>=0.10 in /usr/local/lib/python3.7/dist-packages (from matplotlib==3.3.4->triage) (0.11.0)\n", + "Requirement already satisfied: kiwisolver>=1.0.1 in /usr/local/lib/python3.7/dist-packages (from matplotlib==3.3.4->triage) (1.4.3)\n", + "Requirement already satisfied: pyparsing!=2.0.4,!=2.1.2,!=2.1.6,>=2.0.3 in /usr/local/lib/python3.7/dist-packages (from matplotlib==3.3.4->triage) (3.0.9)\n", + "Requirement already satisfied: pillow>=6.2.0 in /usr/local/lib/python3.7/dist-packages (from matplotlib==3.3.4->triage) (7.1.2)\n", + "Requirement already satisfied: pytz>=2017.2 in /usr/local/lib/python3.7/dist-packages (from pandas==1.0.5->triage) (2022.1)\n", + "Requirement already satisfied: six>=1.5 in /usr/local/lib/python3.7/dist-packages (from python-dateutil==2.8.1->triage) (1.15.0)\n", + "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.7/dist-packages (from requests==2.24.0->triage) (2022.6.15)\n", + "Requirement already satisfied: chardet<4,>=3.0.2 in /usr/local/lib/python3.7/dist-packages (from requests==2.24.0->triage) (3.0.4)\n", + "Requirement already satisfied: idna<3,>=2.5 in /usr/local/lib/python3.7/dist-packages (from requests==2.24.0->triage) (2.10)\n", + "Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /usr/local/lib/python3.7/dist-packages (from requests==2.24.0->triage) (1.24.3)\n", + "Collecting fsspec>=0.6.0\n", + " Downloading fsspec-2022.5.0-py3-none-any.whl (140 kB)\n", + "\u001b[K |████████████████████████████████| 140 kB 84.6 MB/s \n", + "\u001b[?25hRequirement already satisfied: joblib>=0.11 in /usr/local/lib/python3.7/dist-packages (from scikit-learn==0.23.1->triage) (1.1.0)\n", + "Requirement already satisfied: threadpoolctl>=2.0.0 in /usr/local/lib/python3.7/dist-packages (from scikit-learn==0.23.1->triage) (3.1.0)\n", + "Requirement already satisfied: psycopg2 in /usr/local/lib/python3.7/dist-packages (from sqlalchemy-postgres-copy==0.5.0->triage) (2.7.6.1)\n", + "Requirement already satisfied: html5lib>=1.0 in /usr/local/lib/python3.7/dist-packages (from xhtml2pdf==0.2.2->aequitas==0.42.0->triage) (1.0.1)\n", + "Requirement already satisfied: httplib2 in /usr/local/lib/python3.7/dist-packages (from xhtml2pdf==0.2.2->aequitas==0.42.0->triage) (0.17.4)\n", + "Collecting pyPdf2\n", + " Downloading PyPDF2-2.3.1-py3-none-any.whl (198 kB)\n", + "\u001b[K |████████████████████████████████| 198 kB 68.3 MB/s \n", + "\u001b[?25hCollecting reportlab>=3.0\n", + " Downloading reportlab-3.6.10-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.8 MB)\n", + "\u001b[K |████████████████████████████████| 2.8 MB 55.7 MB/s \n", + "\u001b[?25hCollecting docutils<0.16,>=0.10\n", + " Downloading docutils-0.15.2-py3-none-any.whl (547 kB)\n", + "\u001b[K |████████████████████████████████| 547 kB 62.2 MB/s \n", + "\u001b[?25hRequirement already satisfied: webencodings in /usr/local/lib/python3.7/dist-packages (from html5lib>=1.0->xhtml2pdf==0.2.2->aequitas==0.42.0->triage) (0.5.1)\n", + "Requirement already satisfied: MarkupSafe>=0.23 in /usr/local/lib/python3.7/dist-packages (from jinja2->altair==4.1.0->aequitas==0.42.0->triage) (2.0.1)\n", + "Requirement already satisfied: typing-extensions in /usr/local/lib/python3.7/dist-packages (from kiwisolver>=1.0.1->matplotlib==3.3.4->triage) (4.1.1)\n", + "Collecting pillow>=6.2.0\n", + " Downloading Pillow-9.1.1-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (3.1 MB)\n", + "\u001b[K |████████████████████████████████| 3.1 MB 51.7 MB/s \n", + "\u001b[?25hRequirement already satisfied: pyrsistent!=0.17.0,!=0.17.1,!=0.17.2,>=0.14.0 in /usr/local/lib/python3.7/dist-packages (from jsonschema->altair==4.1.0->aequitas==0.42.0->triage) (0.18.1)\n", + "Requirement already satisfied: importlib-metadata in /usr/local/lib/python3.7/dist-packages (from jsonschema->altair==4.1.0->aequitas==0.42.0->triage) (4.11.4)\n", + "Requirement already satisfied: importlib-resources>=1.4.0 in /usr/local/lib/python3.7/dist-packages (from jsonschema->altair==4.1.0->aequitas==0.42.0->triage) (5.7.1)\n", + "Requirement already satisfied: attrs>=17.4.0 in /usr/local/lib/python3.7/dist-packages (from jsonschema->altair==4.1.0->aequitas==0.42.0->triage) (21.4.0)\n", + "Requirement already satisfied: zipp>=3.1.0 in /usr/local/lib/python3.7/dist-packages (from importlib-resources>=1.4.0->jsonschema->altair==4.1.0->aequitas==0.42.0->triage) (3.8.0)\n", + "Building wheels for collected packages: adjustText, alembic, Dickens, Flask-Bootstrap, markdown2, millify, retrying, signalled-timeout, tabulate, xhtml2pdf, visitor\n", + " Building wheel for adjustText (setup.py) ... \u001b[?25l\u001b[?25hdone\n", + " Created wheel for adjustText: filename=adjustText-0.7.3-py3-none-any.whl size=7097 sha256=5cb8e926cf54a794e98db5ddc8dfead14dbe5b87cf2363c1724c124d55691b6f\n", + " Stored in directory: /root/.cache/pip/wheels/2f/98/32/afbf902d8f040fadfdf0a44357e4ab750afe165d873bf5893d\n", + " Building wheel for alembic (PEP 517) ... \u001b[?25l\u001b[?25hdone\n", + " Created wheel for alembic: filename=alembic-1.4.2-py2.py3-none-any.whl size=159554 sha256=b0276e52e77a3cd7cefb29a7e05f54c5d6d443c041942bc41839adf15c3df576\n", + " Stored in directory: /root/.cache/pip/wheels/4e/b5/00/f93fe1c90b3d501774e91e2e99987f49d16019e40e4bd3afc3\n", + " Building wheel for Dickens (setup.py) ... \u001b[?25l\u001b[?25hdone\n", + " Created wheel for Dickens: filename=Dickens-1.0.1-py3-none-any.whl size=2643 sha256=e8ab520fb261b3233ee613553c6eb81dbbd1a18c9649c77be85311f3d33f2199\n", + " Stored in directory: /root/.cache/pip/wheels/11/7b/87/87c72b3ffee9c8830070dfc690b0df03833753e2197c7ed230\n", + " Building wheel for Flask-Bootstrap (setup.py) ... \u001b[?25l\u001b[?25hdone\n", + " Created wheel for Flask-Bootstrap: filename=Flask_Bootstrap-3.3.7.1-py3-none-any.whl size=460123 sha256=4ee5321f47b128334bf1b80349b57ac5ce742fdc98f5c22b8dfa34bafbc3d40d\n", + " Stored in directory: /root/.cache/pip/wheels/67/a2/d6/50d039c9b59b4caca6d7b53839c8100354a52ab7553d2456eb\n", + " Building wheel for markdown2 (setup.py) ... \u001b[?25l\u001b[?25hdone\n", + " Created wheel for markdown2: filename=markdown2-2.3.5-py3-none-any.whl size=33327 sha256=06fc0f9ede786a1d83e67a52414c4024cdb0ba538a6335fe825ab07451b1a51d\n", + " Stored in directory: /root/.cache/pip/wheels/46/b9/ae/4050b5eeeedc7cba8ed5a0203189c89c0fa980f683822bfa31\n", + " Building wheel for millify (setup.py) ... \u001b[?25l\u001b[?25hdone\n", + " Created wheel for millify: filename=millify-0.1.1-py3-none-any.whl size=1866 sha256=841923968d9c2e34ca7de09da6d07b7e3abbbe03a70ffe19fe4c292a50dde626\n", + " Stored in directory: /root/.cache/pip/wheels/38/26/25/c2a8bb99a5cf348903e6ac35a29878e221cc9daeb698545148\n", + " Building wheel for retrying (setup.py) ... \u001b[?25l\u001b[?25hdone\n", + " Created wheel for retrying: filename=retrying-1.3.3-py3-none-any.whl size=11447 sha256=e3f02b7975c11140b34230dc83347c67f5e23103d78063728519c9a440026da4\n", + " Stored in directory: /root/.cache/pip/wheels/f9/8d/8d/f6af3f7f9eea3553bc2fe6d53e4b287dad18b06a861ac56ddf\n", + " Building wheel for signalled-timeout (setup.py) ... \u001b[?25l\u001b[?25hdone\n", + " Created wheel for signalled-timeout: filename=signalled_timeout-1.0.0-py3-none-any.whl size=2973 sha256=d947f52f2f3cc5b0b58980c0b9353c9375f3dcf9400f1684e9f31a329ec79596\n", + " Stored in directory: /root/.cache/pip/wheels/b8/67/0e/f8daac45e46330192ff71cc9c65c86e817df05f7a4a79531d4\n", + " Building wheel for tabulate (setup.py) ... \u001b[?25l\u001b[?25hdone\n", + " Created wheel for tabulate: filename=tabulate-0.8.2-py3-none-any.whl size=23550 sha256=a85c2762c9ae5625348cc2b8227f13d4081de9935a418930d16bbc2d3d4af1e3\n", + " Stored in directory: /root/.cache/pip/wheels/33/63/72/4156fe55e8e06830d7aed3d20a6d1aacc753536843ab7330f6\n", + " Building wheel for xhtml2pdf (setup.py) ... \u001b[?25l\u001b[?25hdone\n", + " Created wheel for xhtml2pdf: filename=xhtml2pdf-0.2.2-py3-none-any.whl size=230264 sha256=d712ab3897a5e39f980482c164d8440dc63e81f29174a84a082250c9a9fb3531\n", + " Stored in directory: /root/.cache/pip/wheels/65/e6/3a/9851102d40dd8e643a4ff3ce5d69988f95d1d9b7448e37a916\n", + " Building wheel for visitor (setup.py) ... \u001b[?25l\u001b[?25hdone\n", + " Created wheel for visitor: filename=visitor-0.1.3-py3-none-any.whl size=3946 sha256=d46157218c08ad97d2597bddbe54e35495289d0bb4b232396c1d9ad4309c2080\n", + " Stored in directory: /root/.cache/pip/wheels/64/34/11/053f47218984c9a31a00f911ed98dda036b867481dcc527a12\n", + "Successfully built adjustText alembic Dickens Flask-Bootstrap markdown2 millify retrying signalled-timeout tabulate xhtml2pdf visitor\n", + "Installing collected packages: python-dateutil, pillow, numpy, jmespath, docutils, visitor, scipy, reportlab, pyPdf2, pandas, matplotlib, Flask, dominate, botocore, xhtml2pdf, tabulate, SQLAlchemy, seaborn, s3transfer, PyYAML, python-editor, plumbum, ohio, millify, markdown2, Mako, humanfriendly, fsspec, Flask-Bootstrap, Dickens, argcomplete, altair, wrapt, verboselogs, sqlalchemy-postgres-copy, signalled-timeout, scikit-learn, s3fs, retrying, requests, psycopg2-binary, pebble, inflection, graphviz, coloredlogs, boto3, argcmdr, alembic, aequitas, adjustText, triage\n", + " Attempting uninstall: python-dateutil\n", + " Found existing installation: python-dateutil 2.8.2\n", + " Uninstalling python-dateutil-2.8.2:\n", + " Successfully uninstalled python-dateutil-2.8.2\n", + " Attempting uninstall: pillow\n", + " Found existing installation: Pillow 7.1.2\n", + " Uninstalling Pillow-7.1.2:\n", + " Successfully uninstalled Pillow-7.1.2\n", + " Attempting uninstall: numpy\n", + " Found existing installation: numpy 1.21.6\n", + " Uninstalling numpy-1.21.6:\n", + " Successfully uninstalled numpy-1.21.6\n", + " Attempting uninstall: docutils\n", + " Found existing installation: docutils 0.17.1\n", + " Uninstalling docutils-0.17.1:\n", + " Successfully uninstalled docutils-0.17.1\n", + " Attempting uninstall: scipy\n", + " Found existing installation: scipy 1.4.1\n", + " Uninstalling scipy-1.4.1:\n", + " Successfully uninstalled scipy-1.4.1\n", + " Attempting uninstall: pandas\n", + " Found existing installation: pandas 1.3.5\n", + " Uninstalling pandas-1.3.5:\n", + " Successfully uninstalled pandas-1.3.5\n", + " Attempting uninstall: matplotlib\n", + " Found existing installation: matplotlib 3.2.2\n", + " Uninstalling matplotlib-3.2.2:\n", + " Successfully uninstalled matplotlib-3.2.2\n", + " Attempting uninstall: Flask\n", + " Found existing installation: Flask 1.1.4\n", + " Uninstalling Flask-1.1.4:\n", + " Successfully uninstalled Flask-1.1.4\n", + " Attempting uninstall: tabulate\n", + " Found existing installation: tabulate 0.8.9\n", + " Uninstalling tabulate-0.8.9:\n", + " Successfully uninstalled tabulate-0.8.9\n", + " Attempting uninstall: SQLAlchemy\n", + " Found existing installation: SQLAlchemy 1.4.37\n", + " Uninstalling SQLAlchemy-1.4.37:\n", + " Successfully uninstalled SQLAlchemy-1.4.37\n", + " Attempting uninstall: seaborn\n", + " Found existing installation: seaborn 0.11.2\n", + " Uninstalling seaborn-0.11.2:\n", + " Successfully uninstalled seaborn-0.11.2\n", + " Attempting uninstall: PyYAML\n", + " Found existing installation: PyYAML 3.13\n", + " Uninstalling PyYAML-3.13:\n", + " Successfully uninstalled PyYAML-3.13\n", + " Attempting uninstall: altair\n", + " Found existing installation: altair 4.2.0\n", + " Uninstalling altair-4.2.0:\n", + " Successfully uninstalled altair-4.2.0\n", + " Attempting uninstall: wrapt\n", + " Found existing installation: wrapt 1.14.1\n", + " Uninstalling wrapt-1.14.1:\n", + " Successfully uninstalled wrapt-1.14.1\n", + " Attempting uninstall: scikit-learn\n", + " Found existing installation: scikit-learn 1.0.2\n", + " Uninstalling scikit-learn-1.0.2:\n", + " Successfully uninstalled scikit-learn-1.0.2\n", + " Attempting uninstall: requests\n", + " Found existing installation: requests 2.23.0\n", + " Uninstalling requests-2.23.0:\n", + " Successfully uninstalled requests-2.23.0\n", + " Attempting uninstall: graphviz\n", + " Found existing installation: graphviz 0.10.1\n", + " Uninstalling graphviz-0.10.1:\n", + " Successfully uninstalled graphviz-0.10.1\n", + "\u001b[31mERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.\n", + "yellowbrick 1.4 requires scikit-learn>=1.0.0, but you have scikit-learn 0.23.1 which is incompatible.\n", + "xarray 0.20.2 requires pandas>=1.1, but you have pandas 1.0.5 which is incompatible.\n", + "imbalanced-learn 0.8.1 requires scikit-learn>=0.24, but you have scikit-learn 0.23.1 which is incompatible.\n", + "google-colab 1.0.0 requires pandas>=1.1.0; python_version >= \"3.0\", but you have pandas 1.0.5 which is incompatible.\n", + "google-colab 1.0.0 requires requests~=2.23.0, but you have requests 2.24.0 which is incompatible.\n", + "datascience 0.10.6 requires folium==0.2.1, but you have folium 0.8.3 which is incompatible.\n", + "albumentations 0.1.12 requires imgaug<0.2.7,>=0.2.5, but you have imgaug 0.2.9 which is incompatible.\u001b[0m\n", + "Successfully installed Dickens-1.0.1 Flask-0.12.2 Flask-Bootstrap-3.3.7.1 Mako-1.2.0 PyYAML-5.4.1 SQLAlchemy-1.3.18 adjustText-0.7.3 aequitas-0.42.0 alembic-1.4.2 altair-4.1.0 argcmdr-0.7.0 argcomplete-1.9.4 boto3-1.14.45 botocore-1.17.63 coloredlogs-14.0 docutils-0.15.2 dominate-2.6.0 fsspec-2022.5.0 graphviz-0.14 humanfriendly-10.0 inflection-0.5.0 jmespath-0.10.0 markdown2-2.3.5 matplotlib-3.3.4 millify-0.1.1 numpy-1.21.1 ohio-0.5.0 pandas-1.0.5 pebble-4.5.3 pillow-9.1.1 plumbum-1.6.4 psycopg2-binary-2.8.5 pyPdf2-2.3.1 python-dateutil-2.8.1 python-editor-1.0.4 reportlab-3.6.10 requests-2.24.0 retrying-1.3.3 s3fs-0.4.2 s3transfer-0.3.7 scikit-learn-0.23.1 scipy-1.5.0 seaborn-0.10.1 signalled-timeout-1.0.0 sqlalchemy-postgres-copy-0.5.0 tabulate-0.8.2 triage-5.1.1 verboselogs-1.7 visitor-0.1.3 wrapt-1.13.3 xhtml2pdf-0.2.2\n" + ] + }, + { + "output_type": "display_data", + "data": { + "application/vnd.colab-display-data+json": { + "pip_warning": { + "packages": [ + "PIL", + "dateutil", + "matplotlib", + "mpl_toolkits", + "numpy" + ] + } + } + }, + "metadata": {} + } + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "8mQ1nY6lksXD" + }, + "source": [ + "🛑   **NOTE: Before continuing, your colab runtime may need to be restarted for the installed packages to take effect. If a \"Restart Runtime\" button appeared at the bottom of the output above, be sure to click it before moving on to the next section!**" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "reskRriKlcpO" + }, + "source": [ + "## A Quick Look at the DonorsChoose Data\n", + "\n", + "Before getting into triage, let's just take a quick look at the data we'll be using here. To get started, we'll need to connect to the database we just created..." + ] + }, + { + "cell_type": "code", + "metadata": { + "id": "mvDxGoeCmSKQ" + }, + "source": [ + "from sqlalchemy.engine.url import URL\n", + "from triage.util.db import create_engine\n", + "import pandas as pd\n", + "\n", + "db_url = URL(\n", + " 'postgres',\n", + " host='localhost',\n", + " username='postgres',\n", + " database='donors_choose',\n", + " password='postgres',\n", + " port=5432,\n", + " )\n", + "\n", + "db_engine = create_engine(db_url)" + ], + "execution_count": 1, + "outputs": [] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "xfgHR1NfnMI4" + }, + "source": [ + "The DonorsChoose dataset contains four main tables we'll need here:\n", + "- **Projects** contains information about each project as well as some details about the teacher posting it and their school and district\n", + "- **Essays** contains the detailed descriptions that the teacher post describing their project and needs\n", + "- **Resources** contains detailed information about the specific number, type, and cost of resources being asked for in the project\n", + "- **Donations** contains information about the donations received by each project on a transactional level, as well as some details about the donor\n", + "\n", + "Let's take a look at the projects:" + ] + }, + { + "cell_type": "code", + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/", + "height": 81 + }, + "id": "dhElc5PMprk0", + "outputId": "a5a874bf-c637-45ea-d1e7-f027dcf04f51" + }, + "source": [ + "pd.read_sql('SELECT COUNT(*) FROM data.projects', db_engine)" + ], + "execution_count": 2, + "outputs": [ + { + "output_type": "execute_result", + "data": { + "text/plain": [ + " count\n", + "0 16480" + ], + "text/html": [ + "\n", + "
\n", + "
\n", + "
\n", + "\n", + "\n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + "
count
016480
\n", + "
\n", + " \n", + " \n", + " \n", + "\n", + " \n", + "
\n", + "
\n", + " " + ] + }, + "metadata": {}, + "execution_count": 2 + } + ] + }, + { + "cell_type": "code", + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/", + "height": 300 + }, + "id": "HkrUfqnmqs0m", + "outputId": "54845ddb-8552-4c08-ba21-060d1c65f852" + }, + "source": [ + "pd.read_sql('SELECT * FROM data.projects LIMIT 5', db_engine)" + ], + "execution_count": 3, + "outputs": [ + { + "output_type": "execute_result", + "data": { + "text/plain": [ + " schoolid projectid_str \\\n", + "0 29a2da790e38b6c8a1c70aced6b9c765 30c034618e67d00c641f9b5b7775c0f4 \n", + "1 06ef48acbdf9b013d4bc4bfc8d328cc1 94199c544d9d2926c1820e5c6cde1eb6 \n", + "2 06ef48acbdf9b013d4bc4bfc8d328cc1 61f42f174afef1ed2419ab948a647137 \n", + "3 06ef48acbdf9b013d4bc4bfc8d328cc1 c966f5226f42aaaf6b115d7dbaefdea3 \n", + "4 06ef48acbdf9b013d4bc4bfc8d328cc1 69e9c24111daa1e6ba64c6d5538e4df1 \n", + "\n", + " teacher_acctid school_ncesid school_latitude \\\n", + "0 0903da60e148adc6280d55f5d94791a5 192013001182 41.428391 \n", + "1 fb340c7ac3b22a5984c6a82602e4a510 450111000143 32.233070 \n", + "2 fb340c7ac3b22a5984c6a82602e4a510 450111000143 32.233070 \n", + "3 e0b5a72f41a376b28db9c2e838a24de5 450111000143 32.233070 \n", + "4 32e86c49396707f71fdc0398ab2b844b 450111000143 32.233070 \n", + "\n", + " school_longitude school_city school_state school_zip school_metro ... \\\n", + "0 -91.049135 Muscatine IA 52761 None ... \n", + "1 -80.855905 Bluffton SC 29910 rural ... \n", + "2 -80.855905 Bluffton SC 29910 rural ... \n", + "3 -80.855905 Bluffton SC 29910 rural ... \n", + "4 -80.855905 Bluffton SC 29910 rural ... \n", + "\n", + " poverty_level grade_level fulfillment_labor_materials total_asking_price \\\n", + "0 high poverty Grades 6-8 35.0 511.32 \n", + "1 high poverty Grades 3-5 35.0 167.43 \n", + "2 high poverty Grades 3-5 35.0 167.43 \n", + "3 high poverty Grades 3-5 35.0 162.14 \n", + "4 high poverty Grades 3-5 35.0 381.27 \n", + "\n", + " total_price_including_optional_support students_reached \\\n", + "0 601.55 100 \n", + "1 192.45 40 \n", + "2 196.98 45 \n", + "3 190.75 25 \n", + "4 448.55 25 \n", + "\n", + " eligible_double_your_impact_match eligible_almost_home_match date_posted \\\n", + "0 False False 2012-08-06 \n", + "1 False False 2010-07-10 \n", + "2 False False 2011-06-09 \n", + "3 False False 2011-06-13 \n", + "4 False False 2012-07-15 \n", + "\n", + " entity_id \n", + "0 234148 \n", + "1 453579 \n", + "2 353855 \n", + "3 353178 \n", + "4 239363 \n", + "\n", + "[5 rows x 36 columns]" + ], + "text/html": [ + "\n", + "
\n", + "
\n", + "
\n", + "\n", + "\n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + "
schoolidprojectid_strteacher_acctidschool_ncesidschool_latitudeschool_longitudeschool_cityschool_stateschool_zipschool_metro...poverty_levelgrade_levelfulfillment_labor_materialstotal_asking_pricetotal_price_including_optional_supportstudents_reachedeligible_double_your_impact_matcheligible_almost_home_matchdate_postedentity_id
029a2da790e38b6c8a1c70aced6b9c76530c034618e67d00c641f9b5b7775c0f40903da60e148adc6280d55f5d94791a519201300118241.428391-91.049135MuscatineIA52761None...high povertyGrades 6-835.0511.32601.55100FalseFalse2012-08-06234148
106ef48acbdf9b013d4bc4bfc8d328cc194199c544d9d2926c1820e5c6cde1eb6fb340c7ac3b22a5984c6a82602e4a51045011100014332.233070-80.855905BlufftonSC29910rural...high povertyGrades 3-535.0167.43192.4540FalseFalse2010-07-10453579
206ef48acbdf9b013d4bc4bfc8d328cc161f42f174afef1ed2419ab948a647137fb340c7ac3b22a5984c6a82602e4a51045011100014332.233070-80.855905BlufftonSC29910rural...high povertyGrades 3-535.0167.43196.9845FalseFalse2011-06-09353855
306ef48acbdf9b013d4bc4bfc8d328cc1c966f5226f42aaaf6b115d7dbaefdea3e0b5a72f41a376b28db9c2e838a24de545011100014332.233070-80.855905BlufftonSC29910rural...high povertyGrades 3-535.0162.14190.7525FalseFalse2011-06-13353178
406ef48acbdf9b013d4bc4bfc8d328cc169e9c24111daa1e6ba64c6d5538e4df132e86c49396707f71fdc0398ab2b844b45011100014332.233070-80.855905BlufftonSC29910rural...high povertyGrades 3-535.0381.27448.5525FalseFalse2012-07-15239363
\n", + "

5 rows × 36 columns

\n", + "
\n", + " \n", + " \n", + " \n", + "\n", + " \n", + "
\n", + "
\n", + " " + ] + }, + "metadata": {}, + "execution_count": 3 + } + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "Knvi1ZB8rANB" + }, + "source": [ + "Note that the `projectid_str` column can be used to link out to the other tables. For instance, let's look at what we can find out about project `30c034618e67d00c641f9b5b7775c0f4`:" + ] + }, + { + "cell_type": "code", + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/", + "height": 229 + }, + "id": "eNl81xoErL7g", + "outputId": "900016cd-eb31-4276-f198-2fd3c5a60629" + }, + "source": [ + "pd.read_sql(\"SELECT * FROM data.essays WHERE projectid_str='30c034618e67d00c641f9b5b7775c0f4'\", db_engine)" + ], + "execution_count": 4, + "outputs": [ + { + "output_type": "execute_result", + "data": { + "text/plain": [ + " projectid_str teacher_acctid \\\n", + "0 30c034618e67d00c641f9b5b7775c0f4 0903da60e148adc6280d55f5d94791a5 \n", + "\n", + " title short_description \\\n", + "0 Illuminate Your Future Can you imagine what the future will bring wit... \n", + "\n", + " need_statement \\\n", + "0 My students need LED bulbs and electronic comp... \n", + "\n", + " essay entity_id date_posted \n", + "0 Can you imagine what the future will bring wit... 234148 2012-08-06 " + ], + "text/html": [ + "\n", + "
\n", + "
\n", + "
\n", + "\n", + "\n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + "
projectid_strteacher_acctidtitleshort_descriptionneed_statementessayentity_iddate_posted
030c034618e67d00c641f9b5b7775c0f40903da60e148adc6280d55f5d94791a5Illuminate Your FutureCan you imagine what the future will bring wit...My students need LED bulbs and electronic comp...Can you imagine what the future will bring wit...2341482012-08-06
\n", + "
\n", + " \n", + " \n", + " \n", + "\n", + " \n", + "
\n", + "
\n", + " " + ] + }, + "metadata": {}, + "execution_count": 4 + } + ] + }, + { + "cell_type": "code", + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/", + "height": 81 + }, + "id": "PdLzQlGXraVX", + "outputId": "fd60bfa0-01ce-4875-c652-14d91b817979" + }, + "source": [ + "pd.read_sql(\"\"\"\n", + " SELECT project_resource_type,\n", + " COUNT(*) AS num_distinct_resources, \n", + " SUM(item_quantity) AS num_total_resources,\n", + " AVG(item_unit_price) AS avg_price,\n", + " SUM(item_unit_price * item_quantity) AS total_cost\n", + " FROM data.resources \n", + " WHERE projectid_str='30c034618e67d00c641f9b5b7775c0f4'\n", + " GROUP BY 1;\n", + " \"\"\", db_engine)" + ], + "execution_count": 5, + "outputs": [ + { + "output_type": "execute_result", + "data": { + "text/plain": [ + " project_resource_type num_distinct_resources num_total_resources \\\n", + "0 Supplies 7 113.0 \n", + "\n", + " avg_price total_cost \n", + "0 13.004286 400.19 " + ], + "text/html": [ + "\n", + "
\n", + "
\n", + "
\n", + "\n", + "\n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + "
project_resource_typenum_distinct_resourcesnum_total_resourcesavg_pricetotal_cost
0Supplies7113.013.004286400.19
\n", + "
\n", + " \n", + " \n", + " \n", + "\n", + " \n", + "
\n", + "
\n", + " " + ] + }, + "metadata": {}, + "execution_count": 5 + } + ] + }, + { + "cell_type": "code", + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/", + "height": 81 + }, + "id": "lAhWr6FJ9656", + "outputId": "07aab232-34e1-4554-ed15-85e7d27b14bf" + }, + "source": [ + "pd.read_sql(\"\"\"\n", + " SELECT \n", + " COUNT(*) AS num_donations,\n", + " SUM(donation_to_project) AS total_donation,\n", + " SUM(CASE WHEN is_teacher_acct THEN 1 ELSE 0 END) AS num_teacher_donation\n", + " FROM data.donations \n", + " WHERE projectid_str='30c034618e67d00c641f9b5b7775c0f4'\n", + " ;\n", + " \"\"\", db_engine)" + ], + "execution_count": 6, + "outputs": [ + { + "output_type": "execute_result", + "data": { + "text/plain": [ + " num_donations total_donation num_teacher_donation\n", + "0 1 511.32 0" + ], + "text/html": [ + "\n", + "
\n", + "
\n", + "
\n", + "\n", + "\n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + "
num_donationstotal_donationnum_teacher_donation
01511.320
\n", + "
\n", + " \n", + " \n", + " \n", + "\n", + " \n", + "
\n", + "
\n", + " " + ] + }, + "metadata": {}, + "execution_count": 6 + } + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "gIj8GvQF_1cV" + }, + "source": [ + "## Formulating the project\n", + "\n", + "Now that we're familiar with the available data, let's turn to the prediction problem at hand. Because reviewing and offering suggestions to posted projects will be time and resource-intensive, we might assume that DonorsChoose can only help a fraction of all projects that get posted, let's suppose 10%. Then, we might formulate our problem along the lines of:\n", + "\n", + "**Each day, for all the projects posted on that day, can we identify the 10% of projects with the highest risk of not being fully funded within 4 months to prioritize for review by digital content experts.**\n", + "\n", + "With this formulation in mind, we can define a cohort and label for our analysis. `triage` will allow us to define these directly as a SQL query, so let's start there..." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "6A7_a1SADxE9" + }, + "source": [ + "### Defining the Cohort\n", + "\n", + "Because most models to inform important decisions will need to generalize into the future, `triage` focuses on respecting the temporal nature of the data (discussed in more detail below). The `cohort` is the set of relevant entities for model training/prediction at a given point in time, which `triage` referrs to as an `as_of_date`.\n", + "\n", + "🚧   NOTE: In `triage`, an `as_of_date` is taken to be midnight at the **beginning** of that date.\n", + "\n", + "Here, the cohort is relatively straightforward: we simply want to identify all of the projects that were posted, right on the day of posting. Although we were looking at the identifier `projectid_str` above, `triage` looks for a column called `entity_id` to uniquely identify entities to its models. We've already added this column to this dataset, so we'll use that below.\n", + "\n", + "🚧   NOTE: `triage` expects entities in the data to be identified by an **integer column** called `entity_id`.\n", + "\n", + "With those details in mind, let's look at an example of how we might define the cohort from our data for this project:" + ] + }, + { + "cell_type": "code", + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/", + "height": 143 + }, + "id": "Fq5W82jLLwQF", + "outputId": "1b1bb3b0-d5a3-48db-cb5b-5d6d43671c54" + }, + "source": [ + "example_as_of_date = '2012-08-07'\n", + "\n", + "pd.read_sql(\"\"\"\n", + " SELECT distinct(entity_id)\n", + " FROM data.projects\n", + " WHERE date_posted = '{as_of_date}'::date - interval '1day'\n", + " ;\n", + " \"\"\".format(as_of_date=example_as_of_date), db_engine)" + ], + "execution_count": 7, + "outputs": [ + { + "output_type": "execute_result", + "data": { + "text/plain": [ + " entity_id\n", + "0 234035\n", + "1 234148\n", + "2 234234" + ], + "text/html": [ + "\n", + "
\n", + "
\n", + "
\n", + "\n", + "\n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + "
entity_id
0234035
1234148
2234234
\n", + "
\n", + " \n", + " \n", + " \n", + "\n", + " \n", + "
\n", + "
\n", + " " + ] + }, + "metadata": {}, + "execution_count": 7 + } + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "qiDh3DdOMrGp" + }, + "source": [ + "In `triage` we'll be able to use `{as_of_date}` as a placeholder for time just as we're doing here.\n", + "\n", + "Also note that because the `as_of_date` is taken to be midnight, we're looking at the projects posted the previous day (hence subtracting the 1 day interval in the query).\n", + "\n", + "For `triage`, we use a yaml format for configuration (described further below) and we'll be able to provide this query directly:\n", + "```\n", + "cohort_config:\n", + " query: |\n", + " SELECT distinct(entity_id)\n", + " FROM data.projects\n", + " WHERE date_posted = '{as_of_date}'::date - interval '1day'\n", + "```" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "Tv-YbkWuOFnI" + }, + "source": [ + "### Defining the Label\n", + "\n", + "For modeling, we also need to consider the outcome we care about. Returning to our formulation, we described trying to identify projects which will not be fully funded within the four months they are active on the platform.\n", + "\n", + "As with the cohort, notice that labels are calculated relative to a given point in time (the `as_of_date` described above) and over a specific time horizon (here, 4 months from posting). In triage, this time horizon is referred to as a `label_timespan` and is also available as a parameter to your label definition, again specified as a query:" + ] + }, + { + "cell_type": "code", + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/", + "height": 143 + }, + "id": "Odpo6nluPGLk", + "outputId": "783747f1-7e10-4fc1-9683-ec76d818715e" + }, + "source": [ + "example_as_of_date = '2012-08-07'\n", + "example_label_timespan = '4month'\n", + "\n", + "pd.read_sql(\"\"\"\n", + " WITH cohort_query AS (\n", + " SELECT distinct(entity_id)\n", + " FROM data.projects\n", + " WHERE date_posted = '{as_of_date}'::date - interval '1day'\n", + " )\n", + " , cohort_donations AS (\n", + " SELECT \n", + " c.entity_id, \n", + " COALESCE(SUM(d.donation_to_project), 0) AS total_donation\n", + " FROM cohort_query c\n", + " LEFT JOIN data.donations d \n", + " ON c.entity_id = d.entity_id\n", + " AND d.donation_timestamp \n", + " BETWEEN '{as_of_date}'::date - interval '1day'\n", + " AND '{as_of_date}'::date + interval '{label_timespan}'\n", + " GROUP BY 1\n", + " )\n", + " SELECT c.entity_id,\n", + " CASE \n", + " WHEN COALESCE(d.total_donation, 0) >= p.total_asking_price THEN 0\n", + " ELSE 1\n", + " END AS outcome \n", + " FROM cohort_query c\n", + " JOIN data.projects p USING(entity_id)\n", + " LEFT JOIN cohort_donations d using(entity_id)\n", + " ;\n", + " \"\"\".format(as_of_date=example_as_of_date, label_timespan=example_label_timespan), db_engine)" + ], + "execution_count": 8, + "outputs": [ + { + "output_type": "execute_result", + "data": { + "text/plain": [ + " entity_id outcome\n", + "0 234035 1\n", + "1 234148 0\n", + "2 234234 0" + ], + "text/html": [ + "\n", + "
\n", + "
\n", + "
\n", + "\n", + "\n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + "
entity_idoutcome
02340351
12341480
22342340
\n", + "
\n", + " \n", + " \n", + " \n", + "\n", + " \n", + "
\n", + "
\n", + " " + ] + }, + "metadata": {}, + "execution_count": 8 + } + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "MLeE7N-NSWam" + }, + "source": [ + "A little more complicated than our cohort query, but still reasonably straightforward: we start with the cohort defined above, then find all the donations to those projects within the label timespan (e.g., the following 4 months after posting), and finally compare that to the total price of the project to create a binary classification label for whether or not the project was fully funded.\n", + "\n", + "Notice here that because we will intervene on projects at risk for **NOT** being fully funded, we define this as our class 1 label while those that do reach their funding goal are given class 0.\n", + "\n", + "As with the cohort, we'll be able to specify this label query directly to triage in our yaml configuation:\n", + "```\n", + "label_config:\n", + " query: |\n", + " WITH cohort_query AS (\n", + " SELECT distinct(entity_id)\n", + " FROM data.projects\n", + " WHERE date_posted = '{as_of_date}'::date - interval '1day'\n", + " )\n", + " , cohort_donations AS (\n", + " SELECT \n", + " c.entity_id, \n", + " COALESCE(SUM(d.donation_to_project), 0) AS total_donation\n", + " FROM cohort_query c\n", + " LEFT JOIN data.donations d \n", + " ON c.entity_id = d.entity_id\n", + " AND d.donation_timestamp \n", + " BETWEEN '{as_of_date}'::date - interval '1day'\n", + " AND '{as_of_date}'::date + interval '{label_timespan}'\n", + " GROUP BY 1\n", + " )\n", + " SELECT c.entity_id,\n", + " CASE \n", + " WHEN COALESCE(d.total_donation, 0) >= p.total_asking_price THEN 0\n", + " ELSE 1\n", + " END AS outcome \n", + " FROM cohort_query c\n", + " JOIN data.projects p USING(entity_id)\n", + " LEFT JOIN cohort_donations d using(entity_id)\n", + "\n", + " name: 'fully_funded'\n", + "```\n", + "\n", + "For more details these two pieces of the modeling pipeline, see the [cohort and label deep dive in the triage docs](https://dssg.github.io/triage/experiments/cohort-labels/). " + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "3RTN8p6VWQ8z" + }, + "source": [ + "### Dealing with Time\n", + "\n", + "As noted above, `triage` is designed for problems where the desire to generalize to future data and therefore is careful to respect the temporal nature of the problem. This is particularly salient in two places: defining the validation strategy for model evaluation and ensuring that features only make use of information available at the time of analysis/prediction.\n", + "\n", + "For validation, the idea is generally simple: models should be trained on historical data and validated on future data. As such, `triage` constructs validation splits that reflect this process by using a certain point in time as the cut-off between training and validation and then moving this cut-off back through the data to generate multiple splits. The implementation is a bit more complicated and relies on several parameters, the details of which we won't go deep into here, but you can find a much deeper discussion in the [longer \"dirty duck\" tutorial](https://dssg.github.io/triage/dirtyduck/triage_intro/) as well as in the [experiment config docs](https://dssg.github.io/triage/experiments/experiment-config/).\n", + "\n", + "![temporal figure](https://dssg.github.io/triage/experiments/temporal_config_graph.png)\n", + "\n", + "In short, these parameters are (illustrated across three training/validation splits in the figure above):\n", + "- feature start/end times: what range of history is feature information available for?\n", + "- label start/end times: what range of history is outcome (label) data available for?\n", + "- model update frequency: what is the interval between refreshes of the model?\n", + "- test durations: over what time period will the model be in use for making predictions?\n", + "- max training history: how much historical data should be used for model training (that is, for rows/examples)?\n", + "- training/test as_of_date frequencies: within a training or validation (test) set, how frequently should cohorts be sampled?\n", + "- training/test label timespans: over what time horizon are labels (outcomes) collected?\n", + "\n", + "As with the cohorts and labels, these parameters are specified to `triage` via its yaml configuration file. Here's what this will look like for our setting:\n", + "```\n", + "temporal_config:\n", + "\n", + " # first date our feature data is good\n", + " feature_start_time: '2000-01-01'\n", + " feature_end_time: '2013-06-01'\n", + "\n", + " # first date our label data is good\n", + " # donorschoose: as far back as we have good donation data\n", + " label_start_time: '2011-09-02'\n", + " label_end_time: '2013-06-01'\n", + "\n", + " model_update_frequency: '4month'\n", + "\n", + " # length of time defining a test set\n", + " test_durations: ['3month']\n", + " # defines how far back a training set reaches\n", + " max_training_histories: ['1y']\n", + "\n", + " # we sample every day, since new projects are posted\n", + " # every day\n", + " training_as_of_date_frequencies: ['1day']\n", + " test_as_of_date_frequencies: ['1day']\n", + " \n", + " # when posted project timeout\n", + " label_timespans: ['3month']\n", + "```" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "kQtPjBJUaaVJ" + }, + "source": [ + "### Model Evaluation Metrics\n", + "\n", + "The temporal configuration described above will create several training and validation splits that can be used to estimate the generalization performance of your models and select a model specification to use going forward. In order to do so, of course, you need to choose an appropriate metric (or metrics) by which to evaluate your models. `triage` can use any of the metrics specified by `sklearn` and in general you'll want to focus on those that best reflect the goals, constraints, and deployment scenario of your project. For instance, in our example project, DonorsChoose can help only 10% of the projects posted to the site, so a metric like precision in the top 10% would reflect how efficiently these limited resources are being allocated to projects that would not be fully funded without additional support.\n", + "\n", + "Although we might want to focus on `precision@10%` as our primary metric, often it can be helpful to look at both precision and recall at a range of thresholds (both percentiles and absolute numbers) both for the purposes of debugging and understanding how sensitive your results are to the available resources, describing a \"menu\" of policy choices.\n", + "\n", + "The `scoring` section of the yaml configuration file allows you specify separate evaluation metrics for both the training and validation set results, indicating both the type of metric (e.g., `precision`, `recall`, etc) and, where needed, the thresholds at which to calculate them. Here's what that looks like for our example project:\n", + "\n", + "```\n", + "scoring:\n", + " testing_metric_groups:\n", + " -\n", + " metrics: [precision@, recall@]\n", + " thresholds:\n", + " percentiles: [1, 2, 3, 4, 5, 6, 7, 8, 9, \n", + " 10, 11, 12, 13, 14, 15, 16, 17, 18, 19,\n", + " 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, \n", + " 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, \n", + " 40, 41, 42, 43, 44, 45, 46, 47, 48, 49,\n", + " 50, 51, 52, 53, 54, 55, 56, 57, 58, 59,\n", + " 60, 61, 62, 63, 64, 65, 66, 67, 68, 69,\n", + " 70, 71, 72, 73, 74, 75, 76, 77, 78, 79,\n", + " 80, 81, 82, 83, 84, 85, 86, 87, 88, 89,\n", + " 90, 91, 92, 93, 94, 95, 96, 97, 98, 99,\n", + " 100]\n", + " top_n: [25, 50, 100]\n", + "\n", + " training_metric_groups:\n", + " -\n", + " metrics: [precision@, recall@]\n", + " thresholds:\n", + " percentiles: [1, 2, 3, 4, 5, 6, 7, 8, 9, \n", + " 10, 11, 12, 13, 14, 15, 16, 17, 18, 19,\n", + " 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, \n", + " 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, \n", + " 40, 41, 42, 43, 44, 45, 46, 47, 48, 49,\n", + " 50, 51, 52, 53, 54, 55, 56, 57, 58, 59,\n", + " 60, 61, 62, 63, 64, 65, 66, 67, 68, 69,\n", + " 70, 71, 72, 73, 74, 75, 76, 77, 78, 79,\n", + " 80, 81, 82, 83, 84, 85, 86, 87, 88, 89,\n", + " 90, 91, 92, 93, 94, 95, 96, 97, 98, 99,\n", + " 100]\n", + " top_n: [25, 50, 100]\n", + "```" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "4cY_UC4Taey9" + }, + "source": [ + "### Defining Features\n", + "\n", + "Feature generation is typically the most important aspect of how well your machine learning models will work, so `triage` provides considerable flexibility for feature definition. However, this also means that this section of the configuration file can be particularly complicated and may require some experimentation to get familiar with. A few resources may be helpful for a deeper look at how features work in `triage`:\n", + "- [Feature Definition in the Quickstart Guide](https://dssg.github.io/triage/triage_project_workflow/#define-some-additional-features)\n", + "- [Feature Generation in the triage Documentation](https://dssg.github.io/triage/experiments/experiment-config/#feature-generation)\n", + "- [Features in the Example Configuration File](https://github.com/dssg/triage/blob/master/example/config/experiment.yaml#L102)\n", + "\n", + "Features in `triage` are defined in blocks, grouping together features drawn from the same data source and allowing several related features to be constructed in a very compact format. Each of these blocks is a list item under the `feature_aggregations` section of your yaml configuration file and contains the following information:\n", + "- A `prefix` that is used to identify the group of features.\n", + "- A `from_obj` that specifies the underlying information used to construct the features in this group. This can be either a table or a query in itself (in the later case, be sure to give it an alias) and must contain both an `entity_id` column as well as a date column indicating when the information was known, identified to the feature config as the `knowledge_date_column`.\n", + "- Information about how missing values should be imputed (see the documentation for details and available options here).\n", + "- Definitions of the feature quantities/columns themselves, specified either as `aggregates` or `categoricals`, including the `metrics` for aggregations over time (e.g., `sum`, `max`, `avg`, etc).\n", + "- Time ranges over which to calculate feature information, called `intervals` (e.g., last 6 months, last 5 years, etc.)\n", + "- A level of aggregation for feature information (`groups`) -- this will almost always be just `entity_id`.\n", + "\n", + "🚧   NOTE: All features in `triage` are temporal aggregates. Just as `triage` is designed to carefully account for time in temporal cross-validation, it also does so in feature construction focusing on what information was known at training or validation time. Even features you might generally consider \"static\" need to be associated with a knowledge date for these purposes as well as an aggregation metric. This is also true for categoricals, which are first one-hot encoded from each instance then aggregated over the given time interval with the specified metric. For instance, if a patient has had several hospital stays with different primary diagnosis codes at each stay, a categorical feature using a `sum` aggregation would yield a count of how many stays had a given diagnosis while a `max` aggregation would provide an indicator of whether a given diagnosis was ever present. \n", + "\n", + "For aggregations of numeric features, the resulting feature names will have the format: \n", + "`{prefix}_entity_id_{interval}_{quantity}_{metric}`\n", + "\n", + "For categoricals, the feature names will include each categorical value after one-hot encoding:\n", + "`{prefix}_entity_id_{interval}_{quantity}_{value}_{metric}`\n", + "\n", + "🚧   WARNING: Because `triage`'s features are stored in a `postgres` database, this naming convention can sometimes run afoul of the database's 63 character limit for column names, leading to truncation. When this happens, you might encounter errors indicating a given feature column appears to be missing. This can be common with categoricals with particularly long values, so recoding can be useful in those cases (as can choosing shorter prefix names).\n", + "\n", + "For illustrative purposes here, we'll start with a single feature group including one categorical and continuous aggregate feature: the primary resource type for the project and the amount being asked for. Because these are both specified once at project posting time, we simply aggegate them over all time (that is, using `all` for our `interval`). Here's how we specify this in our feature configuration:\n", + "\n", + "```\n", + "feature_aggregations:\n", + " -\n", + " prefix: 'project_features'\n", + " from_obj: 'data.projects'\n", + " knowledge_date_column: 'date_posted'\n", + "\n", + " aggregates_imputation:\n", + " all:\n", + " type: 'zero'\n", + "\n", + " categoricals_imputation:\n", + " all:\n", + " type: 'null_category' \n", + "\n", + " categoricals:\n", + " -\n", + " column: 'resource_type'\n", + " metrics:\n", + " - 'max' \n", + " choice_query: 'select distinct resource_type from data.projects'\n", + " \n", + " aggregates:\n", + " -\n", + " quantity: 'total_asking_price'\n", + " metrics:\n", + " - 'sum'\n", + " \n", + " # Since our time-aggregate features are precomputed, feature interval is \n", + " # irrelvant. We keep 'all' as a default.\n", + " intervals: ['all'] \n", + " groups: ['entity_id']\n", + "```" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "0YNaIVySaj2Z" + }, + "source": [ + "### Model and Hyperparameter Grid\n", + "\n", + "You specify the types of models you want to explore, along with their hyperparameters, in the `grid_config` section of your yaml configuration file. Because there's generally no way to know a priori what model specification will work best for a given problem, `triage` makes it easy to run and explore an extensive grid by providing lists of values for each hyperparameter and training models for the full cross-product of these values.\n", + "\n", + "Currently, `triage` can work with any classifiction method with an `sklearn`-style interface. In addition to machine learning algorithms found in standard packages, `triage` includes a couple of built-in methods you might find useful:\n", + "- `ScaledLogisticRegression` wraps the `sklearn` logistic regression with a min-max scaler to ensure that the input features are on the same scale for regularization. It accepts the same hyperparameters as the underlying `sklearn` method.\n", + "- `BaselineRankMultiFeature` is a simple baseline method that ranks examples by one or more features, replicating a comonsense approach that could be taken without making use of machine learning. This method takes a single hyperparameter, `rules`, specified as a list of dictionaries with the keys `feature` and `low_value_high_score` to specify the directin of the ranking. Examples are sorted first by the first feature in this list, then the next, and so on.\n", + "- `SimpleThresholder` is another basic baseline method, allowing you to specify a heuristic, rule-based approach to classifying examples. It uses two hyperparameters: a list of `rules` (e.g., `feature_1 > 3`) and a `logical_operator` (e.g., `and` or `or`) to specify how the rules are combined.\n", + "\n", + "To specify a model type in your grid config, you use the model's class path as a key and each hyperparameter as a key another level down. For example:\n", + "```\n", + "'module.submodule.ClassName':\n", + " param_1: [1,3,5,10,20]\n", + " param_2: [100, 500, 1000]\n", + "```\n", + "\n", + "For our purposes here, we'll start with a very small grid that can run quickly in a colab notebook. Here's how that will look:\n", + "```\n", + "grid_config:\n", + " 'sklearn.ensemble.RandomForestClassifier':\n", + " n_estimators: [150]\n", + " max_depth: [50]\n", + " min_samples_split: [25]\n", + " \n", + " 'sklearn.tree.DecisionTreeClassifier':\n", + " max_depth: [3]\n", + " max_features: [null]\n", + " min_samples_split: [25]\n", + " \n", + " 'triage.component.catwalk.estimators.classifiers.ScaledLogisticRegression':\n", + " C: [0.1]\n", + " penalty: ['l1']\n", + " \n", + " 'triage.component.catwalk.baselines.rankers.BaselineRankMultiFeature':\n", + " rules:\n", + " - [{feature: 'project_features_entity_id_all_total_asking_price_sum', low_value_high_score: False}]\n", + "```" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "-lOxjerFm79c" + }, + "source": [ + "### Auditing Models for Bias\n", + "\n", + "The final section of the configuration file specifies how you want to evaluate your models for bias and fairness using the [aequitas](http://www.datasciencepublicpolicy.org/our-work/tools-guides/aequitas/) toolkit. In order to do so, you need to tell `triage` what attributes are relavent for bias audits, a table or SQL query to specify these attributes (the `from_obj`), and the reference group to calculate disparities relative to (the value for this group will serve as the denominator for disparity calculations. Like the evaluation metrics described above, you'll also need to specify the set of thresholds against which you want to calculate fairness metrics. Note that `aequitas` will calculate the full range of confusion matrix-derived disparity metrics for all of your models, allowing you to explore how your models perform under different conceptualizations of fairness.\n", + "\n", + "To illustrate the use of a bias audit in our example project, we'll look at the `teacher_prefix` attribute as a proxy for the sex of the teacher, using `Mr.` as a reference group. Note that the `from_obj_table` will be joined using an `entity_id` and `as_of_date`, so you must specify a `knowledge_date_column` in the config, as some attributes (or your knowledge of them) might change over time. `aequitas` will use the most recent value of the attribute it finds for a given entity prior to the specified `as_of_date`. Here's how we turn that into a section in our configuration yaml:\n", + "\n", + "```\n", + "bias_audit_config:\n", + " from_obj_table: 'data.projects'\n", + " attribute_columns:\n", + " - 'teacher_prefix'\n", + " knowledge_date_column: 'date_posted'\n", + " entity_id_column: 'entity_id'\n", + " ref_groups_method: 'predefined'\n", + " ref_groups:\n", + " 'teacher_prefix': 'Mr.'\n", + " thresholds:\n", + " percentiles: [5, 10, 15, 20, 25, 50, 100]\n", + " top_n: [25, 50, 100]\n", + "```" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "7ZSSdXx-NcTE" + }, + "source": [ + "## Running Triage\n", + "\n", + "Now that we've walked through the various aspects of configuring triage, we're ready to run our model grid! In order to do so, we need three pieces:\n", + "- Our configuration file, pulling together the elements described above into a single yaml file we'll call `experiment_config.yaml` (in `triage`, an \"experiment\" is a run with a set of parameters and model types).\n", + "- Credentials for connecting to your database, stored in a configuration file called `database.yaml` (alternatively, you can specify them through environment variables)\n", + "- Code to run your `triage` experiment. This can be done via either a command line tool or python interface, the latter of which provides more flexibility so we'll focus on that approach here with a short python script called `run.py`.\n", + "\n", + "The following three sections provides the contents of each of these three files for our DonorsChoose project. In a real project, of course, these would be stored as separate files on your system, but here we include them inline. The `run.py` sets up logging, connects to the database, loads your configuration file, and creates and runs a `MultiCoreExperiment` object from `triage`." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "XX-inX6o7QBE" + }, + "source": [ + "### experiment_config.yaml" + ] + }, + { + "cell_type": "code", + "metadata": { + "id": "RdDjQCovS1GG" + }, + "source": [ + "config_yaml = \"\"\"\n", + "config_version: 'v7'\n", + "\n", + "model_comment: 'triage demo'\n", + "\n", + "random_seed: 1995\n", + "\n", + "temporal_config:\n", + "\n", + " # first date our feature data is good\n", + " feature_start_time: '2000-01-01'\n", + " feature_end_time: '2013-06-01'\n", + "\n", + " # first date our label data is good\n", + " # donorschoose: as far back as we have good donation data\n", + " label_start_time: '2011-09-02'\n", + " label_end_time: '2013-06-01'\n", + "\n", + " model_update_frequency: '4month'\n", + "\n", + " # length of time defining a test set\n", + " test_durations: ['3month']\n", + " # defines how far back a training set reaches\n", + " max_training_histories: ['1y']\n", + "\n", + " # we sample every day, since new projects are posted\n", + " # every day\n", + " training_as_of_date_frequencies: ['1day']\n", + " test_as_of_date_frequencies: ['1day']\n", + " \n", + " # when posted project timeout\n", + " label_timespans: ['3month']\n", + " \n", + "\n", + "cohort_config:\n", + " query: |\n", + " SELECT distinct(entity_id)\n", + " FROM data.projects\n", + " WHERE date_posted = '{as_of_date}'::date - interval '1day'\n", + "\n", + "label_config:\n", + " query: |\n", + " WITH cohort_query AS (\n", + " SELECT distinct(entity_id)\n", + " FROM data.projects\n", + " WHERE date_posted = '{as_of_date}'::date - interval '1day'\n", + " )\n", + " , cohort_donations AS (\n", + " SELECT \n", + " c.entity_id, \n", + " COALESCE(SUM(d.donation_to_project), 0) AS total_donation\n", + " FROM cohort_query c\n", + " LEFT JOIN data.donations d \n", + " ON c.entity_id = d.entity_id\n", + " AND d.donation_timestamp \n", + " BETWEEN '{as_of_date}'::date - interval '1day'\n", + " AND '{as_of_date}'::date + interval '{label_timespan}'\n", + " GROUP BY 1\n", + " )\n", + " SELECT c.entity_id,\n", + " CASE \n", + " WHEN COALESCE(d.total_donation, 0) >= p.total_asking_price THEN 0\n", + " ELSE 1\n", + " END AS outcome \n", + " FROM cohort_query c\n", + " JOIN data.projects p USING(entity_id)\n", + " LEFT JOIN cohort_donations d using(entity_id)\n", + "\n", + " name: 'fully_funded'\n", + "\n", + "\n", + "feature_aggregations:\n", + " -\n", + " prefix: 'project_features'\n", + " from_obj: 'data.projects'\n", + " knowledge_date_column: 'date_posted'\n", + "\n", + " aggregates_imputation:\n", + " all:\n", + " type: 'zero'\n", + "\n", + " categoricals_imputation:\n", + " all:\n", + " type: 'null_category' \n", + "\n", + " categoricals:\n", + " -\n", + " column: 'resource_type'\n", + " metrics:\n", + " - 'max' \n", + " choice_query: 'select distinct resource_type from data.projects'\n", + " \n", + " aggregates:\n", + " -\n", + " quantity: 'total_asking_price'\n", + " metrics:\n", + " - 'sum'\n", + " \n", + " # Since our time-aggregate features are precomputed, feature interval is \n", + " # irrelvant. We keep 'all' as a default.\n", + " intervals: ['all'] \n", + " groups: ['entity_id']\n", + "\n", + "grid_config:\n", + " 'sklearn.ensemble.RandomForestClassifier':\n", + " n_estimators: [150]\n", + " max_depth: [50]\n", + " min_samples_split: [25]\n", + " \n", + " 'sklearn.tree.DecisionTreeClassifier':\n", + " max_depth: [3]\n", + " max_features: [null]\n", + " min_samples_split: [25]\n", + " \n", + " 'triage.component.catwalk.estimators.classifiers.ScaledLogisticRegression':\n", + " C: [0.1]\n", + " penalty: ['l1']\n", + " \n", + " 'triage.component.catwalk.baselines.rankers.BaselineRankMultiFeature':\n", + " rules:\n", + " - [{feature: 'project_features_entity_id_all_total_asking_price_sum', low_value_high_score: False}]\n", + "\n", + "\n", + "scoring:\n", + " testing_metric_groups:\n", + " -\n", + " metrics: [precision@, recall@]\n", + " thresholds:\n", + " percentiles: [1, 2, 3, 4, 5, 6, 7, 8, 9, \n", + " 10, 11, 12, 13, 14, 15, 16, 17, 18, 19,\n", + " 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, \n", + " 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, \n", + " 40, 41, 42, 43, 44, 45, 46, 47, 48, 49,\n", + " 50, 51, 52, 53, 54, 55, 56, 57, 58, 59,\n", + " 60, 61, 62, 63, 64, 65, 66, 67, 68, 69,\n", + " 70, 71, 72, 73, 74, 75, 76, 77, 78, 79,\n", + " 80, 81, 82, 83, 84, 85, 86, 87, 88, 89,\n", + " 90, 91, 92, 93, 94, 95, 96, 97, 98, 99,\n", + " 100]\n", + " top_n: [25, 50, 100]\n", + "\n", + " training_metric_groups:\n", + " -\n", + " metrics: [precision@, recall@]\n", + " thresholds:\n", + " percentiles: [1, 2, 3, 4, 5, 6, 7, 8, 9, \n", + " 10, 11, 12, 13, 14, 15, 16, 17, 18, 19,\n", + " 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, \n", + " 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, \n", + " 40, 41, 42, 43, 44, 45, 46, 47, 48, 49,\n", + " 50, 51, 52, 53, 54, 55, 56, 57, 58, 59,\n", + " 60, 61, 62, 63, 64, 65, 66, 67, 68, 69,\n", + " 70, 71, 72, 73, 74, 75, 76, 77, 78, 79,\n", + " 80, 81, 82, 83, 84, 85, 86, 87, 88, 89,\n", + " 90, 91, 92, 93, 94, 95, 96, 97, 98, 99,\n", + " 100]\n", + " top_n: [25, 50, 100]\n", + " \n", + "bias_audit_config:\n", + " from_obj_table: 'data.projects'\n", + " attribute_columns:\n", + " - 'teacher_prefix'\n", + " knowledge_date_column: 'date_posted'\n", + " entity_id_column: 'entity_id'\n", + " ref_groups_method: 'predefined'\n", + " ref_groups:\n", + " 'teacher_prefix': 'Mr.'\n", + " thresholds:\n", + " percentiles: [5, 10, 15, 20, 25, 50, 100]\n", + " top_n: [25, 50, 100]\n", + "\n", + "individual_importance:\n", + " methods: [] # empty list means don't calculate individual importances\n", + " n_ranks: 1 \n", + "\"\"\"" + ], + "execution_count": 9, + "outputs": [] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "qcqeRvUT7V1D" + }, + "source": [ + "### database.yaml" + ] + }, + { + "cell_type": "code", + "metadata": { + "id": "3i88ZFQupfp6" + }, + "source": [ + "database_yaml = \"\"\"\n", + "host: localhost\n", + "user: postgres\n", + "db: donors_choose\n", + "pass: postgres\n", + "port: 5432\n", + "role: postgres\n", + "\"\"\"" + ], + "execution_count": 10, + "outputs": [] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "wHR9wnAw7d__" + }, + "source": [ + "### run.py" + ] + }, + { + "cell_type": "code", + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/" + }, + "id": "jYzBKFG3qDhQ", + "outputId": "31af303e-74df-4ce9-8332-09a96593ba07" + }, + "source": [ + "import yaml\n", + "\n", + "from sqlalchemy.engine.url import URL\n", + "from triage.util.db import create_engine\n", + "from triage.experiments import MultiCoreExperiment\n", + "import logging\n", + "\n", + "import os\n", + "\n", + "from sqlalchemy.event import listens_for\n", + "from sqlalchemy.pool import Pool\n", + "\n", + "def run_triage():\n", + "\n", + " # andrew_id = os.getenv('USER')\n", + " # user_path = os.path.join('/data/users/', andrew_id)\n", + " user_path = '/content'\n", + "\n", + " # add logging to a file (it will also go to stdout via triage logging config)\n", + " log_filename = os.path.join(user_path, 'triage.log')\n", + " logger = logging.getLogger('')\n", + " hdlr = logging.FileHandler(log_filename)\n", + " hdlr.setLevel(15) # verbose level\n", + " hdlr.setFormatter(logging.Formatter('%(name)-30s %(asctime)s %(levelname)10s %(process)6d %(filename)-24s %(lineno)4d: %(message)s', '%d/%m/%Y %I:%M:%S %p'))\n", + " logger.addHandler(hdlr)\n", + "\n", + " # creating database engine\n", + " # dbfile = os.path.join(user_path, 'database.yaml')\n", + "\n", + " # with open(dbfile, 'r') as dbf:\n", + " # dbconfig = yaml.safe_load(dbf)\n", + "\n", + " dbconfig = yaml.safe_load(database_yaml)\n", + " print(dbconfig['role'])\n", + "\n", + " # assume group role to ensure shared permissions\n", + " @listens_for(Pool, \"connect\")\n", + " def assume_role(dbapi_con, connection_record):\n", + " logging.debug(f\"setting role {dbconfig['role']};\")\n", + " dbapi_con.cursor().execute(f\"set role {dbconfig['role']};\")\n", + " # logging.debug(f\"setting role postres;\")\n", + " # dbapi_con.cursor().execute(f\"set role postgres;\")\n", + "\n", + " db_url = URL(\n", + " 'postgres',\n", + " host=dbconfig['host'],\n", + " username=dbconfig['user'],\n", + " database=dbconfig['db'],\n", + " password=dbconfig['pass'],\n", + " port=dbconfig['port'],\n", + " )\n", + "\n", + " db_engine = create_engine(db_url)\n", + "\n", + " triage_output_path = os.path.join(user_path, 'triage_output')\n", + " os.makedirs(triage_output_path, exist_ok=True)\n", + "\n", + " # loading config file\n", + " # with open('%s_triage_config.yaml' % andrew_id, 'r') as fin:\n", + " # config = yaml.safe_load(fin)\n", + "\n", + " config = yaml.safe_load(config_yaml)\n", + "\n", + " # creating experiment object\n", + " experiment = MultiCoreExperiment(\n", + " config = config,\n", + " db_engine = db_engine,\n", + " project_path = triage_output_path,\n", + " n_processes=2,\n", + " n_bigtrain_processes=1,\n", + " n_db_processes=2,\n", + " replace=True,\n", + " save_predictions=True\n", + " )\n", + "\n", + " # experiment.validate()\n", + " experiment.run()" + ], + "execution_count": 11, + "outputs": [ + { + "output_type": "stream", + "name": "stderr", + "text": [ + "/usr/local/lib/python3.7/dist-packages/statsmodels/tools/_testing.py:19: FutureWarning: pandas.util.testing is deprecated. Use the functions in the public API at pandas.testing instead.\n", + " import pandas.util.testing as tm\n" + ] + } + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "FyqBcKHk7lTC" + }, + "source": [ + "### Let's run triage!\n", + "\n", + "With these three files in place, we can simply run our model grid by calling `run_triage()`. Doing so will train and validate the four model specifications described above across three temporal validation splits. The run will output a log of its progress and store results into the postgres database." + ] + }, + { + "cell_type": "code", + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/" + }, + "id": "ZUAcMwv2qzLe", + "outputId": "957d5b08-2230-4b7d-a7f1-4b7399d5d1c5" + }, + "source": [ + "run_triage()" + ], + "execution_count": 12, + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": [ + "postgres\n", + "\u001b[32m2022-06-21 18:43:50\u001b[0m - \u001b[1;30mVERBOSE\u001b[0m \u001b[34mMatrices and trained models will be saved in /content/triage_output\u001b[0m\n", + "\u001b[32m2022-06-21 18:43:50\u001b[0m - \u001b[1;30m NOTICE\u001b[0m \u001b[35mReplace flag is set to true. Matrices, models, evaluations and predictions (if they exist) will be replaced\u001b[0m\n", + "\u001b[32m2022-06-21 18:43:50\u001b[0m - \u001b[1;30m INFO\u001b[0m No results_schema_versions table exists, which means that this installation is fresh. Upgrading db.\n", + "\u001b[32m2022-06-21 18:43:50\u001b[0m - \u001b[1;30m INFO\u001b[0m Context impl PostgresqlImpl.\n", + "\u001b[32m2022-06-21 18:43:50\u001b[0m - \u001b[1;30m INFO\u001b[0m Will assume transactional DDL.\n", + "\u001b[32m2022-06-21 18:43:50\u001b[0m - \u001b[1;30m INFO\u001b[0m Running upgrade -> 8b3f167d0418, empty message\n", + "\u001b[32m2022-06-21 18:43:50\u001b[0m - \u001b[1;30m INFO\u001b[0m Running upgrade 8b3f167d0418 -> 0d44655e35fd, empty message\n", + "\u001b[32m2022-06-21 18:43:50\u001b[0m - \u001b[1;30m INFO\u001b[0m Running upgrade 0d44655e35fd -> 264245ddfce2, empty message\n", + "\u001b[32m2022-06-21 18:43:50\u001b[0m - \u001b[1;30m INFO\u001b[0m Running upgrade 264245ddfce2 -> 72ac5cbdca05, Change importance to float\n", + "\u001b[32m2022-06-21 18:43:50\u001b[0m - \u001b[1;30m INFO\u001b[0m Running upgrade 72ac5cbdca05 -> 7d57d1cf3429, empty message\n", + "\u001b[32m2022-06-21 18:43:50\u001b[0m - \u001b[1;30m INFO\u001b[0m Running upgrade 7d57d1cf3429 -> 89a8ce240bae, Split results into model_metadata, test_results, and train_resultss\n", + "\u001b[32m2022-06-21 18:43:50\u001b[0m - \u001b[1;30m INFO\u001b[0m Running upgrade 89a8ce240bae -> 2446a931de7a, Changing column names and removing redundancies in table names\n", + "\u001b[32m2022-06-21 18:43:50\u001b[0m - \u001b[1;30m INFO\u001b[0m Running upgrade 2446a931de7a -> d0ac573eaf1a, model_group_stored_procedure\n", + "\u001b[32m2022-06-21 18:43:50\u001b[0m - \u001b[1;30m INFO\u001b[0m Running upgrade d0ac573eaf1a -> 38f37d013686, Associate experiments with models and matrices\n", + "\u001b[32m2022-06-21 18:43:50\u001b[0m - \u001b[1;30m INFO\u001b[0m Running upgrade 38f37d013686 -> 0bca1ba9706e, add_matrix_uuid_to_eval\n", + "\u001b[32m2022-06-21 18:43:50\u001b[0m - \u001b[1;30m INFO\u001b[0m Running upgrade 0bca1ba9706e -> 50e1f1bc2cac, empty message\n", + "\u001b[32m2022-06-21 18:43:50\u001b[0m - \u001b[1;30m INFO\u001b[0m Running upgrade 50e1f1bc2cac -> cfd5c3386014, add_experiment_runs\n", + "\u001b[32m2022-06-21 18:43:50\u001b[0m - \u001b[1;30m INFO\u001b[0m Running upgrade cfd5c3386014 -> 97cf99b7348f, evaluation_randomness\n", + "\u001b[32m2022-06-21 18:43:50\u001b[0m - \u001b[1;30m INFO\u001b[0m Running upgrade 97cf99b7348f -> 609c7cc51794, rankify_predictions\n", + "\u001b[32m2022-06-21 18:43:50\u001b[0m - \u001b[1;30m INFO\u001b[0m Running upgrade 609c7cc51794 -> b4d7569d31cb, aequitas\n", + "\u001b[32m2022-06-21 18:43:50\u001b[0m - \u001b[1;30m INFO\u001b[0m Running upgrade b4d7569d31cb -> 8cef808549dd, empty message\n", + "\u001b[32m2022-06-21 18:43:50\u001b[0m - \u001b[1;30m INFO\u001b[0m Running upgrade 8cef808549dd -> a20104116533, empty message\n", + "\u001b[32m2022-06-21 18:43:50\u001b[0m - \u001b[1;30m INFO\u001b[0m Running upgrade a20104116533 -> fa1760d35710, empty message\n", + "\u001b[32m2022-06-21 18:43:50\u001b[0m - \u001b[1;30m INFO\u001b[0m Running upgrade fa1760d35710 -> 9bbfdcf8bab0, empty message\n", + "\u001b[32m2022-06-21 18:43:50\u001b[0m - \u001b[1;30m INFO\u001b[0m Running upgrade 9bbfdcf8bab0 -> 4ae804cc0977, empty message\n", + "\u001b[32m2022-06-21 18:43:50\u001b[0m - \u001b[1;30m INFO\u001b[0m Running upgrade 4ae804cc0977 -> a98acf92fd48, add nuke triage function\n", + "\u001b[32m2022-06-21 18:43:50\u001b[0m - \u001b[1;30m INFO\u001b[0m Running upgrade a98acf92fd48 -> 45219f25072b, hash-partitioning predictions tables\n", + "\u001b[32m2022-06-21 18:43:50\u001b[0m - \u001b[1;30m INFO\u001b[0m PostgreSQL 11 or greater found (PostgreSQL 11): Using hash partitioning\n", + "\u001b[32m2022-06-21 18:43:51\u001b[0m - \u001b[1;30m INFO\u001b[0m Running upgrade 45219f25072b -> 1b990cbc04e4, empty message\n", + "\u001b[32m2022-06-21 18:43:51\u001b[0m - \u001b[1;30m INFO\u001b[0m Running upgrade 1b990cbc04e4 -> 264786a9fe85, add label_value to prodcution table\n", + "\u001b[32m2022-06-21 18:43:51\u001b[0m - \u001b[1;30m INFO\u001b[0m Running upgrade 264786a9fe85 -> ce5b50ffa8e2, Break ties in list predictions\n", + "\u001b[32m2022-06-21 18:43:51\u001b[0m - \u001b[1;30m INFO\u001b[0m Running upgrade ce5b50ffa8e2 -> 670289044eb2, Add production prediction metadata\n", + "\u001b[32m2022-06-21 18:43:51\u001b[0m - \u001b[1;30m INFO\u001b[0m Running upgrade 670289044eb2 -> cdd0dc9d9870, rename production schema and list_predcitons to triage_predcition and predictions \n", + "\u001b[32m2022-06-21 18:43:51\u001b[0m - \u001b[1;30m INFO\u001b[0m Running upgrade 45219f25072b -> b097e47ba829, empty message\n", + "\u001b[32m2022-06-21 18:43:51\u001b[0m - \u001b[1;30m INFO\u001b[0m Running upgrade b097e47ba829, cdd0dc9d9870 -> 079a74c15e8b, merge b097e47ba829 with cdd0dc9d9870\n", + "\u001b[32m2022-06-21 18:43:51\u001b[0m - \u001b[1;30m INFO\u001b[0m Running upgrade 079a74c15e8b -> 5dd2ba8222b1, add run_type\n", + "\u001b[32m2022-06-21 18:43:51\u001b[0m - \u001b[1;30mVERBOSE\u001b[0m \u001b[34mUsing random seed [1995] for running the experiment\u001b[0m\n", + "\u001b[32m2022-06-21 18:43:51\u001b[0m - \u001b[1;30m NOTICE\u001b[0m \u001b[35mscoring.subsets missing in the configuration file or unrecognized. No subsets will be generated\u001b[0m\n", + "\u001b[32m2022-06-21 18:43:52\u001b[0m - \u001b[1;30mSUCCESS\u001b[0m \u001b[1;32mExperiment validation ran to completion with no errors\u001b[0m\n", + "\u001b[32m2022-06-21 18:43:52\u001b[0m - \u001b[1;30mVERBOSE\u001b[0m \u001b[34mComputed and stored temporal split definitions\u001b[0m\n", + "\u001b[32m2022-06-21 18:43:52\u001b[0m - \u001b[1;30m INFO\u001b[0m Setting up cohort\n", + "\u001b[32m2022-06-21 18:43:57\u001b[0m - \u001b[1;30mSUCCESS\u001b[0m \u001b[1;32mCohort set up in the table cohort_all_entities_005a7918d4c2be39b7e923a84f33ded2 successfully\u001b[0m\n", + "\u001b[32m2022-06-21 18:43:57\u001b[0m - \u001b[1;30m INFO\u001b[0m Setting up labels\n", + "\u001b[32m2022-06-21 18:44:10\u001b[0m - \u001b[1;30mSUCCESS\u001b[0m \u001b[1;32mLabels set up in the table labels_fully_funded_1feff6f9a63afed112773010f3dc4254 successfully \u001b[0m\n", + "\u001b[32m2022-06-21 18:44:10\u001b[0m - \u001b[1;30m INFO\u001b[0m Creating features tables (before imputation) \n", + "\u001b[32m2022-06-21 18:44:10\u001b[0m - \u001b[1;30m INFO\u001b[0m Creating collate aggregations\n", + "\u001b[32m2022-06-21 18:44:10\u001b[0m - \u001b[1;30mVERBOSE\u001b[0m \u001b[34mStarting Feature aggregation\u001b[0m\n", + "\u001b[32m2022-06-21 18:44:10\u001b[0m - \u001b[1;30m NOTICE\u001b[0m \u001b[35mImputed feature table project_features_aggregation_imputed did not exist, need to build features\u001b[0m\n", + "\u001b[32m2022-06-21 18:44:10\u001b[0m - \u001b[1;30m INFO\u001b[0m Processing query tasks with 2 processes\n", + "\u001b[32m2022-06-21 18:44:10\u001b[0m - \u001b[1;30m INFO\u001b[0m Processing features for project_features_entity_id\n", + "\u001b[32m2022-06-21 18:44:10\u001b[0m - \u001b[1;30m INFO\u001b[0m Beginning insert batch\n", + "\u001b[32m2022-06-21 18:44:10\u001b[0m - \u001b[1;30m INFO\u001b[0m Beginning insert batch\n", + "\u001b[32m2022-06-21 18:44:11\u001b[0m - \u001b[1;30m INFO\u001b[0m Beginning insert batch\n", + "\u001b[32m2022-06-21 18:44:11\u001b[0m - \u001b[1;30m INFO\u001b[0m Beginning insert batch\n", + "\u001b[32m2022-06-21 18:44:11\u001b[0m - \u001b[1;30m INFO\u001b[0m Beginning insert batch\n", + "\u001b[32m2022-06-21 18:44:11\u001b[0m - \u001b[1;30m INFO\u001b[0m Beginning insert batch\n", + "\u001b[32m2022-06-21 18:44:11\u001b[0m - \u001b[1;30m INFO\u001b[0m Beginning insert batch\n", + "\u001b[32m2022-06-21 18:44:11\u001b[0m - \u001b[1;30m INFO\u001b[0m Beginning insert batch\n", + "\u001b[32m2022-06-21 18:44:11\u001b[0m - \u001b[1;30m INFO\u001b[0m Beginning insert batch\n", + "\u001b[32m2022-06-21 18:44:11\u001b[0m - \u001b[1;30m INFO\u001b[0m Beginning insert batch\n", + "\u001b[32m2022-06-21 18:44:11\u001b[0m - \u001b[1;30m INFO\u001b[0m Beginning insert batch\n", + "\u001b[32m2022-06-21 18:44:11\u001b[0m - \u001b[1;30m INFO\u001b[0m Beginning insert batch\n", + "\u001b[32m2022-06-21 18:44:11\u001b[0m - \u001b[1;30m INFO\u001b[0m Beginning insert batch\n", + "\u001b[32m2022-06-21 18:44:11\u001b[0m - \u001b[1;30m INFO\u001b[0m Beginning insert batch\n", + "\u001b[32m2022-06-21 18:44:11\u001b[0m - \u001b[1;30m INFO\u001b[0m Beginning insert batch\n", + "\u001b[32m2022-06-21 18:44:11\u001b[0m - \u001b[1;30m INFO\u001b[0m Beginning insert batch\n", + "\u001b[32m2022-06-21 18:44:11\u001b[0m - \u001b[1;30m INFO\u001b[0m Beginning insert batch\n", + "\u001b[32m2022-06-21 18:44:11\u001b[0m - \u001b[1;30m INFO\u001b[0m Beginning insert batch\n", + "\u001b[32m2022-06-21 18:44:12\u001b[0m - \u001b[1;30m INFO\u001b[0m Beginning insert batch\n", + "\u001b[32m2022-06-21 18:44:12\u001b[0m - \u001b[1;30m INFO\u001b[0m Beginning insert batch\n", + "\u001b[32m2022-06-21 18:44:12\u001b[0m - \u001b[1;30m INFO\u001b[0m Beginning insert batch\n", + "\u001b[32m2022-06-21 18:44:12\u001b[0m - \u001b[1;30m INFO\u001b[0m Done. successes: 21, failures: 0\n", + "\u001b[32m2022-06-21 18:44:12\u001b[0m - \u001b[1;30m INFO\u001b[0m project_features_entity_id completed\n", + "\u001b[32m2022-06-21 18:44:12\u001b[0m - \u001b[1;30m INFO\u001b[0m Processing features for project_features_aggregation\n", + "\u001b[32m2022-06-21 18:44:24\u001b[0m - \u001b[1;30m INFO\u001b[0m Done. successes: 0, failures: 0\n", + "\u001b[32m2022-06-21 18:44:29\u001b[0m - \u001b[1;30m INFO\u001b[0m project_features_aggregation completed\n", + "\u001b[32m2022-06-21 18:44:29\u001b[0m - \u001b[1;30mSUCCESS\u001b[0m \u001b[1;32mFeatures (before imputation) were stored in the tables \"features\".\"project_features_aggregation\" successfully\u001b[0m\n", + "\u001b[32m2022-06-21 18:44:29\u001b[0m - \u001b[1;30m INFO\u001b[0m Imputing missing values in features\n", + "\u001b[32m2022-06-21 18:44:29\u001b[0m - \u001b[1;30mVERBOSE\u001b[0m \u001b[34mStarting Feature imputation\u001b[0m\n", + "\u001b[32m2022-06-21 18:44:29\u001b[0m - \u001b[1;30m INFO\u001b[0m Processing query tasks with 2 processes\n", + "\u001b[32m2022-06-21 18:44:29\u001b[0m - \u001b[1;30m INFO\u001b[0m Processing features for project_features_aggregation_imputed\n", + "\u001b[32m2022-06-21 18:44:29\u001b[0m - \u001b[1;30m INFO\u001b[0m Done. successes: 0, failures: 0\n", + "\u001b[32m2022-06-21 18:44:29\u001b[0m - \u001b[1;30m INFO\u001b[0m project_features_aggregation_imputed completed\n", + "\u001b[32m2022-06-21 18:44:29\u001b[0m - \u001b[1;30mSUCCESS\u001b[0m \u001b[1;32mImputed features were stored in the tables \"features\".\"project_features_aggregation_imputed\" successfully\u001b[0m\n", + "\u001b[32m2022-06-21 18:44:29\u001b[0m - \u001b[1;30mVERBOSE\u001b[0m \u001b[34mFound 1 total feature subsets\u001b[0m\n", + "\u001b[32m2022-06-21 18:44:29\u001b[0m - \u001b[1;30m INFO\u001b[0m Building matrices\n", + "\u001b[32m2022-06-21 18:44:29\u001b[0m - \u001b[1;30mVERBOSE\u001b[0m \u001b[34mIt is necessary to build 6 matrices\u001b[0m\n", + "\u001b[32m2022-06-21 18:44:29\u001b[0m - \u001b[1;30m INFO\u001b[0m Starting parallel matrix building: 6 matrices, 2 processes\n", + "\u001b[32m2022-06-21 18:44:29\u001b[0m - \u001b[1;30m INFO\u001b[0m Matrix 6a751eabcf4722abda70f77c0d9d712d saved in /content/triage_output/matrices/6a751eabcf4722abda70f77c0d9d712d.csv.gz\n", + "\u001b[32m2022-06-21 18:44:29\u001b[0m - \u001b[1;30m INFO\u001b[0m Matrix 495afa5517735f1e336108cd7911b8aa saved in /content/triage_output/matrices/495afa5517735f1e336108cd7911b8aa.csv.gz\n", + "\u001b[32m2022-06-21 18:44:29\u001b[0m - \u001b[1;30m INFO\u001b[0m Matrix 051df0ba6431460b81bd18a25fea0d99 saved in /content/triage_output/matrices/051df0ba6431460b81bd18a25fea0d99.csv.gz\n", + "\u001b[32m2022-06-21 18:44:30\u001b[0m - \u001b[1;30m INFO\u001b[0m Matrix 67a0cc5dc9ab89cdf0b1ae3a7883b145 saved in /content/triage_output/matrices/67a0cc5dc9ab89cdf0b1ae3a7883b145.csv.gz\n", + "\u001b[32m2022-06-21 18:44:30\u001b[0m - \u001b[1;30m INFO\u001b[0m Matrix 10b581471b80ac2c5ca865e56be6cfe7 saved in /content/triage_output/matrices/10b581471b80ac2c5ca865e56be6cfe7.csv.gz\n", + "\u001b[32m2022-06-21 18:44:30\u001b[0m - \u001b[1;30m INFO\u001b[0m Matrix 363cae6e28d220afc10d2be99b01a09a saved in /content/triage_output/matrices/363cae6e28d220afc10d2be99b01a09a.csv.gz\n", + "\u001b[32m2022-06-21 18:44:30\u001b[0m - \u001b[1;30m INFO\u001b[0m Done. successes: 6, failures: 0\n", + "\u001b[32m2022-06-21 18:44:30\u001b[0m - \u001b[1;30mSUCCESS\u001b[0m \u001b[1;32mMatrices were stored in /content/triage_output/matrices successfully\u001b[0m\n", + "\u001b[32m2022-06-21 18:44:30\u001b[0m - \u001b[1;30m INFO\u001b[0m Starting parallel subset creation: 0 subsets, 2 processes\n", + "\u001b[32m2022-06-21 18:44:30\u001b[0m - \u001b[1;30m INFO\u001b[0m Done. successes: 0, failures: 0\n", + "\u001b[32m2022-06-21 18:44:32\u001b[0m - \u001b[1;30mSUCCESS\u001b[0m \u001b[1;32mProtected groups stored in the table protected_groups_cf250342c293e834c6fa34f241461aef successfully\u001b[0m\n", + "\u001b[32m2022-06-21 18:44:32\u001b[0m - \u001b[1;30mVERBOSE\u001b[0m \u001b[34mSplit train/test tasks into three task batches. - each batch has models from all splits\u001b[0m\n", + "\u001b[32m2022-06-21 18:44:32\u001b[0m - \u001b[1;30mVERBOSE\u001b[0m \u001b[34mBatch 1: Baselines or simple classifiers (e.g. DecisionTree, SLR) (9 tasks total)\u001b[0m\n", + "\u001b[32m2022-06-21 18:44:32\u001b[0m - \u001b[1;30mVERBOSE\u001b[0m \u001b[34mBatch 2: Heavyweight classifiers. (3 tasks total)\u001b[0m\n", + "\u001b[32m2022-06-21 18:44:32\u001b[0m - \u001b[1;30mVERBOSE\u001b[0m \u001b[34mBatch 3: All classifiers not found in one of the other batches. (0 tasks total)\u001b[0m\n", + "\u001b[32m2022-06-21 18:44:32\u001b[0m - \u001b[1;30m INFO\u001b[0m 4 models groups will be trained, tested and evaluated\n", + "\u001b[32m2022-06-21 18:44:32\u001b[0m - \u001b[1;30m INFO\u001b[0m Training, testing and evaluating models\n", + "\u001b[32m2022-06-21 18:44:32\u001b[0m - \u001b[1;30mVERBOSE\u001b[0m \u001b[34m3 train/test tasks found.\u001b[0m\n", + "\u001b[32m2022-06-21 18:44:32\u001b[0m - \u001b[1;30m INFO\u001b[0m Starting parallelizable batch train/testing with 9 tasks, 2 processes\n", + "\u001b[32m2022-06-21 18:44:32\u001b[0m - \u001b[1;30mVERBOSE\u001b[0m \u001b[34mTraining sklearn.tree.DecisionTreeClassifier({'max_depth': 3, 'max_features': None, 'min_samples_split': 25}) [de38e91e633948e5a5b4198f520ed837] on train matrix 495afa5517735f1e336108cd7911b8aa\u001b[0m\n", + "\u001b[32m2022-06-21 18:44:32\u001b[0m - \u001b[1;30mVERBOSE\u001b[0m \u001b[34mTraining triage.component.catwalk.estimators.classifiers.ScaledLogisticRegression({'C': 0.1, 'penalty': 'l1'}) [e1e70aac9d362ffc6416b38afab7dbe2] on train matrix 495afa5517735f1e336108cd7911b8aa\u001b[0m\n", + "\u001b[32m2022-06-21 18:44:33\u001b[0m - \u001b[1;30m NOTICE\u001b[0m \u001b[35mYou got feature values that are out of the range: (0, 1). The feature values will cutoff to fit in the range (0, 1).\u001b[0m\n", + "\u001b[32m2022-06-21 18:44:33\u001b[0m - \u001b[1;30m NOTICE\u001b[0m \u001b[35mModel 1, not found from previous runs. Adding the new model\u001b[0m\n", + "\u001b[32m2022-06-21 18:44:33\u001b[0m - \u001b[1;30m NOTICE\u001b[0m \u001b[35mModel 2, not found from previous runs. Adding the new model\u001b[0m\n", + "\u001b[32m2022-06-21 18:44:33\u001b[0m - \u001b[1;30mWARNING\u001b[0m \u001b[33mThe selected algorithm, doesn't support a standard way of calculate the importance of each feature used. Falling back to ad-hoc methods (e.g. in LogisticRegression we will return Odd Ratios instead coefficients)\u001b[0m\n", + "\u001b[32m2022-06-21 18:44:33\u001b[0m - \u001b[1;30mSUCCESS\u001b[0m \u001b[1;32mTrained model id 1: sklearn.tree.DecisionTreeClassifier({'max_depth': 3, 'max_features': None, 'min_samples_split': 25}) [de38e91e633948e5a5b4198f520ed837] on train matrix 495afa5517735f1e336108cd7911b8aa. \u001b[0m\n", + "\u001b[32m2022-06-21 18:44:33\u001b[0m - \u001b[1;30mSUCCESS\u001b[0m \u001b[1;32mTrained model id 2: triage.component.catwalk.estimators.classifiers.ScaledLogisticRegression({'C': 0.1, 'penalty': 'l1'}) [e1e70aac9d362ffc6416b38afab7dbe2] on train matrix 495afa5517735f1e336108cd7911b8aa. \u001b[0m\n", + "\u001b[32m2022-06-21 18:44:33\u001b[0m - \u001b[1;30m NOTICE\u001b[0m \u001b[35mYou got feature values that are out of the range: (0, 1). The feature values will cutoff to fit in the range (0, 1).\u001b[0m\n", + "\u001b[32m2022-06-21 18:44:40\u001b[0m - \u001b[1;30m INFO\u001b[0m NumExpr defaulting to 2 threads.\n", + "\u001b[32m2022-06-21 18:44:40\u001b[0m - \u001b[1;30m INFO\u001b[0m getcrosstabs: attribute columns to perform crosstabs:teacher_prefix\n", + "\u001b[32m2022-06-21 18:44:41\u001b[0m - \u001b[1;30m INFO\u001b[0m NumExpr defaulting to 2 threads.\n", + "\u001b[32m2022-06-21 18:44:41\u001b[0m - \u001b[1;30m INFO\u001b[0m getcrosstabs: attribute columns to perform crosstabs:teacher_prefix\n", + "get_disparity_predefined_group()\n", + "\u001b[32m2022-06-21 18:44:41\u001b[0m - \u001b[1;30m INFO\u001b[0m get_group_value_fairness...\n", + "\u001b[32m2022-06-21 18:44:41\u001b[0m - \u001b[1;30m INFO\u001b[0m getcrosstabs: attribute columns to perform crosstabs:teacher_prefix\n", + "get_disparity_predefined_group()\n", + "\u001b[32m2022-06-21 18:44:42\u001b[0m - \u001b[1;30m INFO\u001b[0m get_group_value_fairness...\n", + "\u001b[32m2022-06-21 18:44:42\u001b[0m - \u001b[1;30m INFO\u001b[0m getcrosstabs: attribute columns to perform crosstabs:teacher_prefix\n", + "get_disparity_predefined_group()\n", + "\u001b[32m2022-06-21 18:44:42\u001b[0m - \u001b[1;30m INFO\u001b[0m get_group_value_fairness...\n", + "\u001b[32m2022-06-21 18:44:42\u001b[0m - \u001b[1;30m INFO\u001b[0m Model 1 evaluation on test matrix 6a751eabcf4722abda70f77c0d9d712d completed.\n", + "get_disparity_predefined_group()\n", + "\u001b[32m2022-06-21 18:44:43\u001b[0m - \u001b[1;30m INFO\u001b[0m get_group_value_fairness...\n", + "\u001b[32m2022-06-21 18:44:43\u001b[0m - \u001b[1;30m INFO\u001b[0m Model 2 evaluation on test matrix 6a751eabcf4722abda70f77c0d9d712d completed.\n", + "\u001b[32m2022-06-21 18:44:43\u001b[0m - \u001b[1;30m NOTICE\u001b[0m \u001b[35mYou got feature values that are out of the range: (0, 1). The feature values will cutoff to fit in the range (0, 1).\u001b[0m\n", + "\u001b[32m2022-06-21 18:44:52\u001b[0m - \u001b[1;30m INFO\u001b[0m getcrosstabs: attribute columns to perform crosstabs:teacher_prefix\n", + "\u001b[32m2022-06-21 18:44:53\u001b[0m - \u001b[1;30m INFO\u001b[0m getcrosstabs: attribute columns to perform crosstabs:teacher_prefix\n", + "get_disparity_predefined_group()\n", + "\u001b[32m2022-06-21 18:44:53\u001b[0m - \u001b[1;30m INFO\u001b[0m get_group_value_fairness...\n", + "\u001b[32m2022-06-21 18:44:53\u001b[0m - \u001b[1;30m INFO\u001b[0m getcrosstabs: attribute columns to perform crosstabs:teacher_prefix\n", + "get_disparity_predefined_group()\n", + "\u001b[32m2022-06-21 18:44:53\u001b[0m - \u001b[1;30m INFO\u001b[0m get_group_value_fairness...\n", + "\u001b[32m2022-06-21 18:44:54\u001b[0m - \u001b[1;30m INFO\u001b[0m getcrosstabs: attribute columns to perform crosstabs:teacher_prefix\n", + "get_disparity_predefined_group()\n", + "\u001b[32m2022-06-21 18:44:54\u001b[0m - \u001b[1;30m INFO\u001b[0m get_group_value_fairness...\n", + "\u001b[32m2022-06-21 18:44:54\u001b[0m - \u001b[1;30m INFO\u001b[0m Model 1 evaluation on train matrix 495afa5517735f1e336108cd7911b8aa completed.\n", + "\u001b[32m2022-06-21 18:44:54\u001b[0m - \u001b[1;30mVERBOSE\u001b[0m \u001b[34mTraining triage.component.catwalk.baselines.rankers.BaselineRankMultiFeature({'rules': [{'feature': 'project_features_entity_id_all_total_asking_price_sum', 'low_value_high_score': False}]}) [942fda8a13abbf322a3a78c8cdb7ba1a] on train matrix 495afa5517735f1e336108cd7911b8aa\u001b[0m\n", + "\u001b[32m2022-06-21 18:44:54\u001b[0m - \u001b[1;30m NOTICE\u001b[0m \u001b[35mModel 3, not found from previous runs. Adding the new model\u001b[0m\n", + "\u001b[32m2022-06-21 18:44:54\u001b[0m - \u001b[1;30mSUCCESS\u001b[0m \u001b[1;32mTrained model id 3: triage.component.catwalk.baselines.rankers.BaselineRankMultiFeature({'rules': [{'feature': 'project_features_entity_id_all_total_asking_price_sum', 'low_value_high_score': False}]}) [942fda8a13abbf322a3a78c8cdb7ba1a] on train matrix 495afa5517735f1e336108cd7911b8aa. \u001b[0m\n", + "get_disparity_predefined_group()\n", + "\u001b[32m2022-06-21 18:44:54\u001b[0m - \u001b[1;30m INFO\u001b[0m get_group_value_fairness...\n", + "\u001b[32m2022-06-21 18:44:55\u001b[0m - \u001b[1;30m INFO\u001b[0m Model 2 evaluation on train matrix 495afa5517735f1e336108cd7911b8aa completed.\n", + "\u001b[32m2022-06-21 18:44:55\u001b[0m - \u001b[1;30mVERBOSE\u001b[0m \u001b[34mTraining sklearn.tree.DecisionTreeClassifier({'max_depth': 3, 'max_features': None, 'min_samples_split': 25}) [468fc2d1aca51d1150deeb77b030c679] on train matrix 67a0cc5dc9ab89cdf0b1ae3a7883b145\u001b[0m\n", + "\u001b[32m2022-06-21 18:44:55\u001b[0m - \u001b[1;30m NOTICE\u001b[0m \u001b[35mModel 4, not found from previous runs. Adding the new model\u001b[0m\n", + "\u001b[32m2022-06-21 18:44:55\u001b[0m - \u001b[1;30mSUCCESS\u001b[0m \u001b[1;32mTrained model id 4: sklearn.tree.DecisionTreeClassifier({'max_depth': 3, 'max_features': None, 'min_samples_split': 25}) [468fc2d1aca51d1150deeb77b030c679] on train matrix 67a0cc5dc9ab89cdf0b1ae3a7883b145. \u001b[0m\n", + "\u001b[32m2022-06-21 18:44:55\u001b[0m - \u001b[1;30m INFO\u001b[0m NumExpr defaulting to 2 threads.\n", + "\u001b[32m2022-06-21 18:44:55\u001b[0m - \u001b[1;30m INFO\u001b[0m getcrosstabs: attribute columns to perform crosstabs:teacher_prefix\n", + "get_disparity_predefined_group()\n", + "\u001b[32m2022-06-21 18:44:56\u001b[0m - \u001b[1;30m INFO\u001b[0m get_group_value_fairness...\n", + "\u001b[32m2022-06-21 18:44:56\u001b[0m - \u001b[1;30m INFO\u001b[0m getcrosstabs: attribute columns to perform crosstabs:teacher_prefix\n", + "get_disparity_predefined_group()\n", + "\u001b[32m2022-06-21 18:44:57\u001b[0m - \u001b[1;30m INFO\u001b[0m get_group_value_fairness...\n", + "\u001b[32m2022-06-21 18:44:57\u001b[0m - \u001b[1;30m INFO\u001b[0m Model 3 evaluation on test matrix 6a751eabcf4722abda70f77c0d9d712d completed.\n", + "\u001b[32m2022-06-21 18:44:59\u001b[0m - \u001b[1;30m INFO\u001b[0m getcrosstabs: attribute columns to perform crosstabs:teacher_prefix\n", + "get_disparity_predefined_group()\n", + "\u001b[32m2022-06-21 18:45:00\u001b[0m - \u001b[1;30m INFO\u001b[0m get_group_value_fairness...\n", + "\u001b[32m2022-06-21 18:45:00\u001b[0m - \u001b[1;30m INFO\u001b[0m getcrosstabs: attribute columns to perform crosstabs:teacher_prefix\n", + "get_disparity_predefined_group()\n", + "\u001b[32m2022-06-21 18:45:01\u001b[0m - \u001b[1;30m INFO\u001b[0m get_group_value_fairness...\n", + "\u001b[32m2022-06-21 18:45:01\u001b[0m - \u001b[1;30m INFO\u001b[0m Model 3 evaluation on train matrix 495afa5517735f1e336108cd7911b8aa completed.\n", + "\u001b[32m2022-06-21 18:45:01\u001b[0m - \u001b[1;30mVERBOSE\u001b[0m \u001b[34mTraining triage.component.catwalk.estimators.classifiers.ScaledLogisticRegression({'C': 0.1, 'penalty': 'l1'}) [c2f98311493fd3b59c40358764835f19] on train matrix 67a0cc5dc9ab89cdf0b1ae3a7883b145\u001b[0m\n", + "\u001b[32m2022-06-21 18:45:01\u001b[0m - \u001b[1;30m NOTICE\u001b[0m \u001b[35mModel 5, not found from previous runs. Adding the new model\u001b[0m\n", + "\u001b[32m2022-06-21 18:45:01\u001b[0m - \u001b[1;30mWARNING\u001b[0m \u001b[33mThe selected algorithm, doesn't support a standard way of calculate the importance of each feature used. Falling back to ad-hoc methods (e.g. in LogisticRegression we will return Odd Ratios instead coefficients)\u001b[0m\n", + "\u001b[32m2022-06-21 18:45:01\u001b[0m - \u001b[1;30mSUCCESS\u001b[0m \u001b[1;32mTrained model id 5: triage.component.catwalk.estimators.classifiers.ScaledLogisticRegression({'C': 0.1, 'penalty': 'l1'}) [c2f98311493fd3b59c40358764835f19] on train matrix 67a0cc5dc9ab89cdf0b1ae3a7883b145. \u001b[0m\n", + "\u001b[32m2022-06-21 18:45:01\u001b[0m - \u001b[1;30m NOTICE\u001b[0m \u001b[35mYou got feature values that are out of the range: (0, 1). The feature values will cutoff to fit in the range (0, 1).\u001b[0m\n", + "\u001b[32m2022-06-21 18:45:05\u001b[0m - \u001b[1;30m INFO\u001b[0m NumExpr defaulting to 2 threads.\n", + "\u001b[32m2022-06-21 18:45:05\u001b[0m - \u001b[1;30m INFO\u001b[0m getcrosstabs: attribute columns to perform crosstabs:teacher_prefix\n", + "get_disparity_predefined_group()\n", + "\u001b[32m2022-06-21 18:45:06\u001b[0m - \u001b[1;30m INFO\u001b[0m get_group_value_fairness...\n", + "\u001b[32m2022-06-21 18:45:06\u001b[0m - \u001b[1;30m INFO\u001b[0m getcrosstabs: attribute columns to perform crosstabs:teacher_prefix\n", + "get_disparity_predefined_group()\n", + "\u001b[32m2022-06-21 18:45:07\u001b[0m - \u001b[1;30m INFO\u001b[0m get_group_value_fairness...\n", + "\u001b[32m2022-06-21 18:45:07\u001b[0m - \u001b[1;30m INFO\u001b[0m Model 4 evaluation on test matrix 051df0ba6431460b81bd18a25fea0d99 completed.\n", + "\u001b[32m2022-06-21 18:45:12\u001b[0m - \u001b[1;30m INFO\u001b[0m NumExpr defaulting to 2 threads.\n", + "\u001b[32m2022-06-21 18:45:12\u001b[0m - \u001b[1;30m INFO\u001b[0m getcrosstabs: attribute columns to perform crosstabs:teacher_prefix\n", + "get_disparity_predefined_group()\n", + "\u001b[32m2022-06-21 18:45:13\u001b[0m - \u001b[1;30m INFO\u001b[0m get_group_value_fairness...\n", + "\u001b[32m2022-06-21 18:45:13\u001b[0m - \u001b[1;30m INFO\u001b[0m getcrosstabs: attribute columns to perform crosstabs:teacher_prefix\n", + "get_disparity_predefined_group()\n", + "\u001b[32m2022-06-21 18:45:14\u001b[0m - \u001b[1;30m INFO\u001b[0m get_group_value_fairness...\n", + "\u001b[32m2022-06-21 18:45:14\u001b[0m - \u001b[1;30m INFO\u001b[0m Model 5 evaluation on test matrix 051df0ba6431460b81bd18a25fea0d99 completed.\n", + "\u001b[32m2022-06-21 18:45:20\u001b[0m - \u001b[1;30m INFO\u001b[0m getcrosstabs: attribute columns to perform crosstabs:teacher_prefix\n", + "get_disparity_predefined_group()\n", + "\u001b[32m2022-06-21 18:45:21\u001b[0m - \u001b[1;30m INFO\u001b[0m get_group_value_fairness...\n", + "\u001b[32m2022-06-21 18:45:21\u001b[0m - \u001b[1;30m INFO\u001b[0m getcrosstabs: attribute columns to perform crosstabs:teacher_prefix\n", + "get_disparity_predefined_group()\n", + "\u001b[32m2022-06-21 18:45:22\u001b[0m - \u001b[1;30m INFO\u001b[0m get_group_value_fairness...\n", + "\u001b[32m2022-06-21 18:45:22\u001b[0m - \u001b[1;30m INFO\u001b[0m Model 4 evaluation on train matrix 67a0cc5dc9ab89cdf0b1ae3a7883b145 completed.\n", + "\u001b[32m2022-06-21 18:45:22\u001b[0m - \u001b[1;30mVERBOSE\u001b[0m \u001b[34mTraining triage.component.catwalk.baselines.rankers.BaselineRankMultiFeature({'rules': [{'feature': 'project_features_entity_id_all_total_asking_price_sum', 'low_value_high_score': False}]}) [4c850b235b18c32436c2c9158d47316e] on train matrix 67a0cc5dc9ab89cdf0b1ae3a7883b145\u001b[0m\n", + "\u001b[32m2022-06-21 18:45:22\u001b[0m - \u001b[1;30m NOTICE\u001b[0m \u001b[35mModel 6, not found from previous runs. Adding the new model\u001b[0m\n", + "\u001b[32m2022-06-21 18:45:22\u001b[0m - \u001b[1;30mSUCCESS\u001b[0m \u001b[1;32mTrained model id 6: triage.component.catwalk.baselines.rankers.BaselineRankMultiFeature({'rules': [{'feature': 'project_features_entity_id_all_total_asking_price_sum', 'low_value_high_score': False}]}) [4c850b235b18c32436c2c9158d47316e] on train matrix 67a0cc5dc9ab89cdf0b1ae3a7883b145. \u001b[0m\n", + "\u001b[32m2022-06-21 18:45:24\u001b[0m - \u001b[1;30m INFO\u001b[0m NumExpr defaulting to 2 threads.\n", + "\u001b[32m2022-06-21 18:45:24\u001b[0m - \u001b[1;30m INFO\u001b[0m getcrosstabs: attribute columns to perform crosstabs:teacher_prefix\n", + "get_disparity_predefined_group()\n", + "\u001b[32m2022-06-21 18:45:24\u001b[0m - \u001b[1;30m INFO\u001b[0m get_group_value_fairness...\n", + "\u001b[32m2022-06-21 18:45:25\u001b[0m - \u001b[1;30m INFO\u001b[0m getcrosstabs: attribute columns to perform crosstabs:teacher_prefix\n", + "get_disparity_predefined_group()\n", + "\u001b[32m2022-06-21 18:45:25\u001b[0m - \u001b[1;30m INFO\u001b[0m get_group_value_fairness...\n", + "\u001b[32m2022-06-21 18:45:26\u001b[0m - \u001b[1;30m INFO\u001b[0m Model 6 evaluation on test matrix 051df0ba6431460b81bd18a25fea0d99 completed.\n", + "\u001b[32m2022-06-21 18:45:27\u001b[0m - \u001b[1;30m INFO\u001b[0m getcrosstabs: attribute columns to perform crosstabs:teacher_prefix\n", + "get_disparity_predefined_group()\n", + "\u001b[32m2022-06-21 18:45:27\u001b[0m - \u001b[1;30m INFO\u001b[0m get_group_value_fairness...\n", + "\u001b[32m2022-06-21 18:45:28\u001b[0m - \u001b[1;30m INFO\u001b[0m getcrosstabs: attribute columns to perform crosstabs:teacher_prefix\n", + "\u001b[32m2022-06-21 18:45:28\u001b[0m - \u001b[1;30m INFO\u001b[0m getcrosstabs: attribute columns to perform crosstabs:teacher_prefix\n", + "get_disparity_predefined_group()\n", + "get_disparity_predefined_group()\n", + "\u001b[32m2022-06-21 18:45:28\u001b[0m - \u001b[1;30m INFO\u001b[0m get_group_value_fairness...\n", + "\u001b[32m2022-06-21 18:45:28\u001b[0m - \u001b[1;30m INFO\u001b[0m get_group_value_fairness...\n", + "\u001b[32m2022-06-21 18:45:29\u001b[0m - \u001b[1;30m INFO\u001b[0m Model 5 evaluation on train matrix 67a0cc5dc9ab89cdf0b1ae3a7883b145 completed.\n", + "\u001b[32m2022-06-21 18:45:29\u001b[0m - \u001b[1;30m INFO\u001b[0m getcrosstabs: attribute columns to perform crosstabs:teacher_prefix\n", + "\u001b[32m2022-06-21 18:45:29\u001b[0m - \u001b[1;30mVERBOSE\u001b[0m \u001b[34mTraining sklearn.tree.DecisionTreeClassifier({'max_depth': 3, 'max_features': None, 'min_samples_split': 25}) [6e904c2e00f3ae0b31de9b98ca0c1584] on train matrix 363cae6e28d220afc10d2be99b01a09a\u001b[0m\n", + "\u001b[32m2022-06-21 18:45:29\u001b[0m - \u001b[1;30m NOTICE\u001b[0m \u001b[35mModel 7, not found from previous runs. Adding the new model\u001b[0m\n", + "\u001b[32m2022-06-21 18:45:29\u001b[0m - \u001b[1;30mSUCCESS\u001b[0m \u001b[1;32mTrained model id 7: sklearn.tree.DecisionTreeClassifier({'max_depth': 3, 'max_features': None, 'min_samples_split': 25}) [6e904c2e00f3ae0b31de9b98ca0c1584] on train matrix 363cae6e28d220afc10d2be99b01a09a. \u001b[0m\n", + "get_disparity_predefined_group()\n", + "\u001b[32m2022-06-21 18:45:30\u001b[0m - \u001b[1;30m INFO\u001b[0m get_group_value_fairness...\n", + "\u001b[32m2022-06-21 18:45:30\u001b[0m - \u001b[1;30m INFO\u001b[0m Model 6 evaluation on train matrix 67a0cc5dc9ab89cdf0b1ae3a7883b145 completed.\n", + "\u001b[32m2022-06-21 18:45:30\u001b[0m - \u001b[1;30mVERBOSE\u001b[0m \u001b[34mTraining triage.component.catwalk.estimators.classifiers.ScaledLogisticRegression({'C': 0.1, 'penalty': 'l1'}) [ee410a6a06ba9e5b6eea7f1794ed1b39] on train matrix 363cae6e28d220afc10d2be99b01a09a\u001b[0m\n", + "\u001b[32m2022-06-21 18:45:30\u001b[0m - \u001b[1;30m NOTICE\u001b[0m \u001b[35mModel 8, not found from previous runs. Adding the new model\u001b[0m\n", + "\u001b[32m2022-06-21 18:45:30\u001b[0m - \u001b[1;30mWARNING\u001b[0m \u001b[33mThe selected algorithm, doesn't support a standard way of calculate the importance of each feature used. Falling back to ad-hoc methods (e.g. in LogisticRegression we will return Odd Ratios instead coefficients)\u001b[0m\n", + "\u001b[32m2022-06-21 18:45:30\u001b[0m - \u001b[1;30mSUCCESS\u001b[0m \u001b[1;32mTrained model id 8: triage.component.catwalk.estimators.classifiers.ScaledLogisticRegression({'C': 0.1, 'penalty': 'l1'}) [ee410a6a06ba9e5b6eea7f1794ed1b39] on train matrix 363cae6e28d220afc10d2be99b01a09a. \u001b[0m\n", + "\u001b[32m2022-06-21 18:45:37\u001b[0m - \u001b[1;30m INFO\u001b[0m NumExpr defaulting to 2 threads.\n", + "\u001b[32m2022-06-21 18:45:37\u001b[0m - \u001b[1;30m INFO\u001b[0m getcrosstabs: attribute columns to perform crosstabs:teacher_prefix\n", + "get_disparity_predefined_group()\n", + "\u001b[32m2022-06-21 18:45:38\u001b[0m - \u001b[1;30m INFO\u001b[0m get_group_value_fairness...\n", + "\u001b[32m2022-06-21 18:45:38\u001b[0m - \u001b[1;30m INFO\u001b[0m getcrosstabs: attribute columns to perform crosstabs:teacher_prefix\n", + "\u001b[32m2022-06-21 18:45:39\u001b[0m - \u001b[1;30m INFO\u001b[0m NumExpr defaulting to 2 threads.\n", + "\u001b[32m2022-06-21 18:45:39\u001b[0m - \u001b[1;30m INFO\u001b[0m getcrosstabs: attribute columns to perform crosstabs:teacher_prefix\n", + "get_disparity_predefined_group()\n", + "\u001b[32m2022-06-21 18:45:39\u001b[0m - \u001b[1;30m INFO\u001b[0m get_group_value_fairness...\n", + "\u001b[32m2022-06-21 18:45:39\u001b[0m - \u001b[1;30m INFO\u001b[0m Model 7 evaluation on test matrix 10b581471b80ac2c5ca865e56be6cfe7 completed.\n", + "get_disparity_predefined_group()\n", + "\u001b[32m2022-06-21 18:45:40\u001b[0m - \u001b[1;30m INFO\u001b[0m get_group_value_fairness...\n", + "\u001b[32m2022-06-21 18:45:40\u001b[0m - \u001b[1;30m INFO\u001b[0m getcrosstabs: attribute columns to perform crosstabs:teacher_prefix\n", + "get_disparity_predefined_group()\n", + "\u001b[32m2022-06-21 18:45:41\u001b[0m - \u001b[1;30m INFO\u001b[0m get_group_value_fairness...\n", + "\u001b[32m2022-06-21 18:45:41\u001b[0m - \u001b[1;30m INFO\u001b[0m Model 8 evaluation on test matrix 10b581471b80ac2c5ca865e56be6cfe7 completed.\n", + "\u001b[32m2022-06-21 18:45:54\u001b[0m - \u001b[1;30m INFO\u001b[0m getcrosstabs: attribute columns to perform crosstabs:teacher_prefix\n", + "get_disparity_predefined_group()\n", + "\u001b[32m2022-06-21 18:45:54\u001b[0m - \u001b[1;30m INFO\u001b[0m get_group_value_fairness...\n", + "\u001b[32m2022-06-21 18:45:55\u001b[0m - \u001b[1;30m INFO\u001b[0m getcrosstabs: attribute columns to perform crosstabs:teacher_prefix\n", + "get_disparity_predefined_group()\n", + "\u001b[32m2022-06-21 18:45:55\u001b[0m - \u001b[1;30m INFO\u001b[0m get_group_value_fairness...\n", + "\u001b[32m2022-06-21 18:45:56\u001b[0m - \u001b[1;30m INFO\u001b[0m Model 7 evaluation on train matrix 363cae6e28d220afc10d2be99b01a09a completed.\n", + "\u001b[32m2022-06-21 18:45:56\u001b[0m - \u001b[1;30m INFO\u001b[0m getcrosstabs: attribute columns to perform crosstabs:teacher_prefix\n", + "\u001b[32m2022-06-21 18:45:56\u001b[0m - \u001b[1;30mVERBOSE\u001b[0m \u001b[34mTraining triage.component.catwalk.baselines.rankers.BaselineRankMultiFeature({'rules': [{'feature': 'project_features_entity_id_all_total_asking_price_sum', 'low_value_high_score': False}]}) [cec877d278f1ea40c1677dbf2354a9b8] on train matrix 363cae6e28d220afc10d2be99b01a09a\u001b[0m\n", + "\u001b[32m2022-06-21 18:45:56\u001b[0m - \u001b[1;30m NOTICE\u001b[0m \u001b[35mModel 9, not found from previous runs. Adding the new model\u001b[0m\n", + "\u001b[32m2022-06-21 18:45:56\u001b[0m - \u001b[1;30mSUCCESS\u001b[0m \u001b[1;32mTrained model id 9: triage.component.catwalk.baselines.rankers.BaselineRankMultiFeature({'rules': [{'feature': 'project_features_entity_id_all_total_asking_price_sum', 'low_value_high_score': False}]}) [cec877d278f1ea40c1677dbf2354a9b8] on train matrix 363cae6e28d220afc10d2be99b01a09a. \u001b[0m\n", + "get_disparity_predefined_group()\n", + "\u001b[32m2022-06-21 18:45:57\u001b[0m - \u001b[1;30m INFO\u001b[0m get_group_value_fairness...\n", + "\u001b[32m2022-06-21 18:45:57\u001b[0m - \u001b[1;30m INFO\u001b[0m getcrosstabs: attribute columns to perform crosstabs:teacher_prefix\n", + "\u001b[32m2022-06-21 18:45:57\u001b[0m - \u001b[1;30m INFO\u001b[0m NumExpr defaulting to 2 threads.\n", + "\u001b[32m2022-06-21 18:45:57\u001b[0m - \u001b[1;30m INFO\u001b[0m getcrosstabs: attribute columns to perform crosstabs:teacher_prefix\n", + "get_disparity_predefined_group()\n", + "\u001b[32m2022-06-21 18:45:58\u001b[0m - \u001b[1;30m INFO\u001b[0m get_group_value_fairness...\n", + "\u001b[32m2022-06-21 18:45:58\u001b[0m - \u001b[1;30m INFO\u001b[0m Model 8 evaluation on train matrix 363cae6e28d220afc10d2be99b01a09a completed.\n", + "get_disparity_predefined_group()\n", + "\u001b[32m2022-06-21 18:45:58\u001b[0m - \u001b[1;30m INFO\u001b[0m get_group_value_fairness...\n", + "\u001b[32m2022-06-21 18:45:58\u001b[0m - \u001b[1;30m INFO\u001b[0m getcrosstabs: attribute columns to perform crosstabs:teacher_prefix\n", + "get_disparity_predefined_group()\n", + "\u001b[32m2022-06-21 18:45:59\u001b[0m - \u001b[1;30m INFO\u001b[0m get_group_value_fairness...\n", + "\u001b[32m2022-06-21 18:45:59\u001b[0m - \u001b[1;30m INFO\u001b[0m Model 9 evaluation on test matrix 10b581471b80ac2c5ca865e56be6cfe7 completed.\n", + "\u001b[32m2022-06-21 18:46:00\u001b[0m - \u001b[1;30m INFO\u001b[0m getcrosstabs: attribute columns to perform crosstabs:teacher_prefix\n", + "get_disparity_predefined_group()\n", + "\u001b[32m2022-06-21 18:46:01\u001b[0m - \u001b[1;30m INFO\u001b[0m get_group_value_fairness...\n", + "\u001b[32m2022-06-21 18:46:01\u001b[0m - \u001b[1;30m INFO\u001b[0m getcrosstabs: attribute columns to perform crosstabs:teacher_prefix\n", + "get_disparity_predefined_group()\n", + "\u001b[32m2022-06-21 18:46:02\u001b[0m - \u001b[1;30m INFO\u001b[0m get_group_value_fairness...\n", + "\u001b[32m2022-06-21 18:46:02\u001b[0m - \u001b[1;30m INFO\u001b[0m Model 9 evaluation on train matrix 363cae6e28d220afc10d2be99b01a09a completed.\n", + "\u001b[32m2022-06-21 18:46:02\u001b[0m - \u001b[1;30m INFO\u001b[0m Done. successes: 9, failures: 0\n", + "\u001b[32m2022-06-21 18:46:02\u001b[0m - \u001b[1;30m INFO\u001b[0m Starting parallelizable batch train/testing with 3 tasks, 1 processes\n", + "\u001b[32m2022-06-21 18:46:02\u001b[0m - \u001b[1;30mVERBOSE\u001b[0m \u001b[34mTraining sklearn.ensemble.RandomForestClassifier({'max_depth': 50, 'min_samples_split': 25, 'n_estimators': 150}) [b86731b91ced3187be3a876b23303cb8] on train matrix 495afa5517735f1e336108cd7911b8aa\u001b[0m\n", + "\u001b[32m2022-06-21 18:46:02\u001b[0m - \u001b[1;30m NOTICE\u001b[0m \u001b[35mModel 10, not found from previous runs. Adding the new model\u001b[0m\n", + "\u001b[32m2022-06-21 18:46:02\u001b[0m - \u001b[1;30mSUCCESS\u001b[0m \u001b[1;32mTrained model id 10: sklearn.ensemble.RandomForestClassifier({'max_depth': 50, 'min_samples_split': 25, 'n_estimators': 150}) [b86731b91ced3187be3a876b23303cb8] on train matrix 495afa5517735f1e336108cd7911b8aa. \u001b[0m\n", + "\u001b[32m2022-06-21 18:46:04\u001b[0m - \u001b[1;30m INFO\u001b[0m NumExpr defaulting to 2 threads.\n", + "\u001b[32m2022-06-21 18:46:04\u001b[0m - \u001b[1;30m INFO\u001b[0m getcrosstabs: attribute columns to perform crosstabs:teacher_prefix\n", + "get_disparity_predefined_group()\n", + "\u001b[32m2022-06-21 18:46:04\u001b[0m - \u001b[1;30m INFO\u001b[0m get_group_value_fairness...\n", + "\u001b[32m2022-06-21 18:46:04\u001b[0m - \u001b[1;30m INFO\u001b[0m getcrosstabs: attribute columns to perform crosstabs:teacher_prefix\n", + "get_disparity_predefined_group()\n", + "\u001b[32m2022-06-21 18:46:05\u001b[0m - \u001b[1;30m INFO\u001b[0m get_group_value_fairness...\n", + "\u001b[32m2022-06-21 18:46:05\u001b[0m - \u001b[1;30m INFO\u001b[0m Model 10 evaluation on test matrix 6a751eabcf4722abda70f77c0d9d712d completed.\n", + "\u001b[32m2022-06-21 18:46:06\u001b[0m - \u001b[1;30m INFO\u001b[0m getcrosstabs: attribute columns to perform crosstabs:teacher_prefix\n", + "get_disparity_predefined_group()\n", + "\u001b[32m2022-06-21 18:46:06\u001b[0m - \u001b[1;30m INFO\u001b[0m get_group_value_fairness...\n", + "\u001b[32m2022-06-21 18:46:07\u001b[0m - \u001b[1;30m INFO\u001b[0m getcrosstabs: attribute columns to perform crosstabs:teacher_prefix\n", + "get_disparity_predefined_group()\n", + "\u001b[32m2022-06-21 18:46:07\u001b[0m - \u001b[1;30m INFO\u001b[0m get_group_value_fairness...\n", + "\u001b[32m2022-06-21 18:46:07\u001b[0m - \u001b[1;30m INFO\u001b[0m Model 10 evaluation on train matrix 495afa5517735f1e336108cd7911b8aa completed.\n", + "\u001b[32m2022-06-21 18:46:07\u001b[0m - \u001b[1;30mVERBOSE\u001b[0m \u001b[34mTraining sklearn.ensemble.RandomForestClassifier({'max_depth': 50, 'min_samples_split': 25, 'n_estimators': 150}) [9c5d0d402e699c5622ae2af9660248de] on train matrix 67a0cc5dc9ab89cdf0b1ae3a7883b145\u001b[0m\n", + "\u001b[32m2022-06-21 18:46:08\u001b[0m - \u001b[1;30m NOTICE\u001b[0m \u001b[35mModel 11, not found from previous runs. Adding the new model\u001b[0m\n", + "\u001b[32m2022-06-21 18:46:08\u001b[0m - \u001b[1;30mSUCCESS\u001b[0m \u001b[1;32mTrained model id 11: sklearn.ensemble.RandomForestClassifier({'max_depth': 50, 'min_samples_split': 25, 'n_estimators': 150}) [9c5d0d402e699c5622ae2af9660248de] on train matrix 67a0cc5dc9ab89cdf0b1ae3a7883b145. \u001b[0m\n", + "\u001b[32m2022-06-21 18:46:09\u001b[0m - \u001b[1;30m INFO\u001b[0m NumExpr defaulting to 2 threads.\n", + "\u001b[32m2022-06-21 18:46:09\u001b[0m - \u001b[1;30m INFO\u001b[0m getcrosstabs: attribute columns to perform crosstabs:teacher_prefix\n", + "get_disparity_predefined_group()\n", + "\u001b[32m2022-06-21 18:46:10\u001b[0m - \u001b[1;30m INFO\u001b[0m get_group_value_fairness...\n", + "\u001b[32m2022-06-21 18:46:10\u001b[0m - \u001b[1;30m INFO\u001b[0m getcrosstabs: attribute columns to perform crosstabs:teacher_prefix\n", + "get_disparity_predefined_group()\n", + "\u001b[32m2022-06-21 18:46:11\u001b[0m - \u001b[1;30m INFO\u001b[0m get_group_value_fairness...\n", + "\u001b[32m2022-06-21 18:46:11\u001b[0m - \u001b[1;30m INFO\u001b[0m Model 11 evaluation on test matrix 051df0ba6431460b81bd18a25fea0d99 completed.\n", + "\u001b[32m2022-06-21 18:46:12\u001b[0m - \u001b[1;30m INFO\u001b[0m getcrosstabs: attribute columns to perform crosstabs:teacher_prefix\n", + "get_disparity_predefined_group()\n", + "\u001b[32m2022-06-21 18:46:12\u001b[0m - \u001b[1;30m INFO\u001b[0m get_group_value_fairness...\n", + "\u001b[32m2022-06-21 18:46:13\u001b[0m - \u001b[1;30m INFO\u001b[0m getcrosstabs: attribute columns to perform crosstabs:teacher_prefix\n", + "get_disparity_predefined_group()\n", + "\u001b[32m2022-06-21 18:46:13\u001b[0m - \u001b[1;30m INFO\u001b[0m get_group_value_fairness...\n", + "\u001b[32m2022-06-21 18:46:13\u001b[0m - \u001b[1;30m INFO\u001b[0m Model 11 evaluation on train matrix 67a0cc5dc9ab89cdf0b1ae3a7883b145 completed.\n", + "\u001b[32m2022-06-21 18:46:13\u001b[0m - \u001b[1;30mVERBOSE\u001b[0m \u001b[34mTraining sklearn.ensemble.RandomForestClassifier({'max_depth': 50, 'min_samples_split': 25, 'n_estimators': 150}) [e7e6927dd1de32f2249b42bb7acf8b51] on train matrix 363cae6e28d220afc10d2be99b01a09a\u001b[0m\n", + "\u001b[32m2022-06-21 18:46:14\u001b[0m - \u001b[1;30m NOTICE\u001b[0m \u001b[35mModel 12, not found from previous runs. Adding the new model\u001b[0m\n", + "\u001b[32m2022-06-21 18:46:14\u001b[0m - \u001b[1;30mSUCCESS\u001b[0m \u001b[1;32mTrained model id 12: sklearn.ensemble.RandomForestClassifier({'max_depth': 50, 'min_samples_split': 25, 'n_estimators': 150}) [e7e6927dd1de32f2249b42bb7acf8b51] on train matrix 363cae6e28d220afc10d2be99b01a09a. \u001b[0m\n", + "\u001b[32m2022-06-21 18:46:16\u001b[0m - \u001b[1;30m INFO\u001b[0m NumExpr defaulting to 2 threads.\n", + "\u001b[32m2022-06-21 18:46:16\u001b[0m - \u001b[1;30m INFO\u001b[0m getcrosstabs: attribute columns to perform crosstabs:teacher_prefix\n", + "get_disparity_predefined_group()\n", + "\u001b[32m2022-06-21 18:46:16\u001b[0m - \u001b[1;30m INFO\u001b[0m get_group_value_fairness...\n", + "\u001b[32m2022-06-21 18:46:16\u001b[0m - \u001b[1;30m INFO\u001b[0m getcrosstabs: attribute columns to perform crosstabs:teacher_prefix\n", + "get_disparity_predefined_group()\n", + "\u001b[32m2022-06-21 18:46:17\u001b[0m - \u001b[1;30m INFO\u001b[0m get_group_value_fairness...\n", + "\u001b[32m2022-06-21 18:46:17\u001b[0m - \u001b[1;30m INFO\u001b[0m Model 12 evaluation on test matrix 10b581471b80ac2c5ca865e56be6cfe7 completed.\n", + "\u001b[32m2022-06-21 18:46:18\u001b[0m - \u001b[1;30m INFO\u001b[0m getcrosstabs: attribute columns to perform crosstabs:teacher_prefix\n", + "get_disparity_predefined_group()\n", + "\u001b[32m2022-06-21 18:46:19\u001b[0m - \u001b[1;30m INFO\u001b[0m get_group_value_fairness...\n", + "\u001b[32m2022-06-21 18:46:19\u001b[0m - \u001b[1;30m INFO\u001b[0m getcrosstabs: attribute columns to perform crosstabs:teacher_prefix\n", + "get_disparity_predefined_group()\n", + "\u001b[32m2022-06-21 18:46:20\u001b[0m - \u001b[1;30m INFO\u001b[0m get_group_value_fairness...\n", + "\u001b[32m2022-06-21 18:46:20\u001b[0m - \u001b[1;30m INFO\u001b[0m Model 12 evaluation on train matrix 363cae6e28d220afc10d2be99b01a09a completed.\n", + "\u001b[32m2022-06-21 18:46:20\u001b[0m - \u001b[1;30m INFO\u001b[0m Done. successes: 3, failures: 0\n", + "\u001b[32m2022-06-21 18:46:20\u001b[0m - \u001b[1;30m INFO\u001b[0m Starting parallelizable batch train/testing with 0 tasks, 2 processes\n", + "\u001b[32m2022-06-21 18:46:20\u001b[0m - \u001b[1;30m INFO\u001b[0m Done. successes: 0, failures: 0\n", + "\u001b[32m2022-06-21 18:46:20\u001b[0m - \u001b[1;30mSUCCESS\u001b[0m \u001b[1;32mTraining, testing and evaluatiog models completed\u001b[0m\n", + "\u001b[32m2022-06-21 18:46:20\u001b[0m - \u001b[1;30mSUCCESS\u001b[0m \u001b[1;32mAll matrices that were supposed to be build were built. Awesome!\u001b[0m\n", + "\u001b[32m2022-06-21 18:46:20\u001b[0m - \u001b[1;30mSUCCESS\u001b[0m \u001b[1;32mAll models that were supposed to be trained were trained. Awesome!\u001b[0m\n" + ] + } + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "sPXmtJ667osT" + }, + "source": [ + "## Checking the results\n", + "\n", + "Running `triage` will generate two types of outputs: objects stored to disk and results stored in the database.\n", + "\n", + "Two types of objects will be stored to disk in the `project_path` specified in creating the experiment object:\n", + "- The matrices used for model training and validation, stored as CSV files and associated metadata in yaml format.\n", + "- The trained model objects themselves, stored as `joblib` pickles, which can be loaded and applied to new data.\n", + "\n", + "In the database, `triage` will store results and metadata in several tables. Below is a very brief tour of the most important of these tables.\n", + "\n", + "In the **triage_metadata** schema, you'll find information about your run and the models that were created:\n", + "- `triage_metadata.triage_runs`: metadata about every time `triage` is run, identified by a `run_id`\n", + "- `triage_metadata.experiments`: configuration information for an experiment, identified by an `experiment_hash`. Note that a config file can be run multiple times, so a specific experiment might be associated with multiple `triage_runs` records. The `experiment_hash` can be linked to the `run_hash` in the `triage_runs` table where `run_type='experiment'`\n", + "- `triage_metadata.model_groups`: in `triage` a `model_group` represents a full specification of a model type, set of hyperparameters, set of features, and training set parameters\n", + "- `triage_metadata.models`: a `model` represents the application of a `model_group` to a given training set, yielding a set of trained parameters (such as the coefficients of a logistic regression, the splits of a decision tree, etc). The models are identified by both a `model_id` and `model_hash` and can be linked to their `model_group` via the `model_group_id`\n", + "- `triage_metadata.experiment_models`: the association between models and experiments (linking an `experiment_hash` to a `model_hash`)\n", + "\n", + "In the **test_results** schema, you'll find information about the validation performance of the models:\n", + "- `test_results.evaluations`: performance of each model on the metrics specified in the `scoring` section of your configuration file\n", + "- `test_results.predictions`: individual entity-level predicted scores from each model\n", + "- `test_results.prediction_metadata`: metadata associated with the predictions\n", + "- `test_results.aequitas`: performance of each model on the fairness metrics using the parameters specified in your `bias_audit_config`\n", + "\n", + "In the **train_results** schema, you'll find model performance on the training set, as well as feature importances:\n", + "- `train_results.evaluations`: similar to `test_results.evaluations` but for the training set (often may be overfit, but can be useful for debugging)\n", + "- `train_results.predictions`: similar to `test_results.predictions` but for the training set\n", + "- `train_results.prediction_metadata`: metadata associated with the predictions\n", + "- `train_results.feature_importances`: overall feature importances from model training, usining the built-in method for the classifier (if one exists)\n", + "\n", + "Finally, a few intermediate tables can be particularly useful for debugging:\n", + "- Tables containing your `cohort` and `label` will be generated in the `public` schema and identified by an associated hash that can be found in your logs.\n", + "- The `features` schema contains two types of useful tables: tables containing calculated features for each feature group and \"matrix\" tables that provide the mapping from each training/validation matrix to `(entity_id, as_of_date)` pairs. Note, however, that these tables may be overwritten if a new run is performed with different feature logic, cohort, or underlying data and should not be assumed to be persistant across runs.\n", + "\n", + "Let's take a quick look at some of these outputs to confirm that our models ran as expected. First, we'll need a database connection..." + ] + }, + { + "cell_type": "code", + "metadata": { + "id": "CpfPG-nyq1Nk" + }, + "source": [ + "import yaml\n", + "from sqlalchemy.engine.url import URL\n", + "from triage.util.db import create_engine\n", + "import pandas as pd\n", + "\n", + "dbconfig = yaml.safe_load(database_yaml)\n", + "db_url = URL(\n", + " 'postgres',\n", + " host=dbconfig['host'],\n", + " username=dbconfig['user'],\n", + " database=dbconfig['db'],\n", + " password=dbconfig['pass'],\n", + " port=dbconfig['port'],\n", + " )\n", + "\n", + "db_engine = create_engine(db_url)" + ], + "execution_count": 13, + "outputs": [] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "ajIxNg-S57e6" + }, + "source": [ + "`triage_metadata.model_groups` should contain four records (for each model type/hyperparameter combination specified in our grid), while `triage_metadata.models` should have twelve (each model group trained on the three validation splits):" + ] + }, + { + "cell_type": "code", + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/", + "height": 358 + }, + "id": "8Xjhe5S86V2H", + "outputId": "b33a0fb8-130d-4758-e43d-b6ac7431f592" + }, + "source": [ + "pd.read_sql('SELECT * FROM triage_metadata.model_groups;', db_engine)" + ], + "execution_count": 14, + "outputs": [ + { + "output_type": "execute_result", + "data": { + "text/plain": [ + " model_group_id model_type \\\n", + "0 1 sklearn.ensemble.RandomForestClassifier \n", + "1 2 sklearn.tree.DecisionTreeClassifier \n", + "2 3 triage.component.catwalk.estimators.classifier... \n", + "3 4 triage.component.catwalk.baselines.rankers.Bas... \n", + "\n", + " hyperparameters \\\n", + "0 {'max_depth': 50, 'n_estimators': 150, 'min_sa... \n", + "1 {'max_depth': 3, 'max_features': None, 'min_sa... \n", + "2 {'C': 0.1, 'penalty': 'l1'} \n", + "3 {'rules': [{'feature': 'project_features_entit... \n", + "\n", + " feature_list \\\n", + "0 [project_features_entity_id_all_resource_type_... \n", + "1 [project_features_entity_id_all_resource_type_... \n", + "2 [project_features_entity_id_all_resource_type_... \n", + "3 [project_features_entity_id_all_resource_type_... \n", + "\n", + " model_config \n", + "0 {'state': 'active', 'label_name': 'fully_funde... \n", + "1 {'state': 'active', 'label_name': 'fully_funde... \n", + "2 {'state': 'active', 'label_name': 'fully_funde... \n", + "3 {'state': 'active', 'label_name': 'fully_funde... " + ], + "text/html": [ + "\n", + "
\n", + "
\n", + "
\n", + "\n", + "\n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + "
model_group_idmodel_typehyperparametersfeature_listmodel_config
01sklearn.ensemble.RandomForestClassifier{'max_depth': 50, 'n_estimators': 150, 'min_sa...[project_features_entity_id_all_resource_type_...{'state': 'active', 'label_name': 'fully_funde...
12sklearn.tree.DecisionTreeClassifier{'max_depth': 3, 'max_features': None, 'min_sa...[project_features_entity_id_all_resource_type_...{'state': 'active', 'label_name': 'fully_funde...
23triage.component.catwalk.estimators.classifier...{'C': 0.1, 'penalty': 'l1'}[project_features_entity_id_all_resource_type_...{'state': 'active', 'label_name': 'fully_funde...
34triage.component.catwalk.baselines.rankers.Bas...{'rules': [{'feature': 'project_features_entit...[project_features_entity_id_all_resource_type_...{'state': 'active', 'label_name': 'fully_funde...
\n", + "
\n", + " \n", + " \n", + " \n", + "\n", + " \n", + "
\n", + "
\n", + " " + ] + }, + "metadata": {}, + "execution_count": 14 + } + ] + }, + { + "cell_type": "code", + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/", + "height": 802 + }, + "id": "dZnlWPbMrFN-", + "outputId": "1a2d0fca-b224-4e9b-e9f1-f730ff42f078" + }, + "source": [ + "pd.read_sql('SELECT * FROM triage_metadata.models;', db_engine)" + ], + "execution_count": 15, + "outputs": [ + { + "output_type": "execute_result", + "data": { + "text/plain": [ + " model_id model_group_id model_hash \\\n", + "0 1 2 de38e91e633948e5a5b4198f520ed837 \n", + "1 2 3 e1e70aac9d362ffc6416b38afab7dbe2 \n", + "2 3 4 942fda8a13abbf322a3a78c8cdb7ba1a \n", + "3 4 2 468fc2d1aca51d1150deeb77b030c679 \n", + "4 5 3 c2f98311493fd3b59c40358764835f19 \n", + "5 6 4 4c850b235b18c32436c2c9158d47316e \n", + "6 7 2 6e904c2e00f3ae0b31de9b98ca0c1584 \n", + "7 8 3 ee410a6a06ba9e5b6eea7f1794ed1b39 \n", + "8 9 4 cec877d278f1ea40c1677dbf2354a9b8 \n", + "9 10 1 b86731b91ced3187be3a876b23303cb8 \n", + "10 11 1 9c5d0d402e699c5622ae2af9660248de \n", + "11 12 1 e7e6927dd1de32f2249b42bb7acf8b51 \n", + "\n", + " run_time batch_run_time \\\n", + "0 2022-06-21 18:44:33.010279 2022-06-21 18:44:32.593620 \n", + "1 2022-06-21 18:44:33.007427 2022-06-21 18:44:32.593620 \n", + "2 2022-06-21 18:44:54.566299 2022-06-21 18:44:32.593620 \n", + "3 2022-06-21 18:44:55.279857 2022-06-21 18:44:32.633577 \n", + "4 2022-06-21 18:45:01.720629 2022-06-21 18:44:32.633577 \n", + "5 2022-06-21 18:45:22.552256 2022-06-21 18:44:32.633577 \n", + "6 2022-06-21 18:45:29.175130 2022-06-21 18:44:32.781213 \n", + "7 2022-06-21 18:45:30.387069 2022-06-21 18:44:32.781213 \n", + "8 2022-06-21 18:45:56.272109 2022-06-21 18:44:32.781213 \n", + "9 2022-06-21 18:46:02.390343 2022-06-21 18:44:32.593620 \n", + "10 2022-06-21 18:46:07.813865 2022-06-21 18:44:32.633577 \n", + "11 2022-06-21 18:46:13.927388 2022-06-21 18:44:32.781213 \n", + "\n", + " model_type \\\n", + "0 sklearn.tree.DecisionTreeClassifier \n", + "1 triage.component.catwalk.estimators.classifier... \n", + "2 triage.component.catwalk.baselines.rankers.Bas... \n", + "3 sklearn.tree.DecisionTreeClassifier \n", + "4 triage.component.catwalk.estimators.classifier... \n", + "5 triage.component.catwalk.baselines.rankers.Bas... \n", + "6 sklearn.tree.DecisionTreeClassifier \n", + "7 triage.component.catwalk.estimators.classifier... \n", + "8 triage.component.catwalk.baselines.rankers.Bas... \n", + "9 sklearn.ensemble.RandomForestClassifier \n", + "10 sklearn.ensemble.RandomForestClassifier \n", + "11 sklearn.ensemble.RandomForestClassifier \n", + "\n", + " hyperparameters model_comment \\\n", + "0 {'max_depth': 3, 'max_features': None, 'min_sa... triage demo \n", + "1 {'C': 0.1, 'penalty': 'l1'} triage demo \n", + "2 {'rules': [{'feature': 'project_features_entit... triage demo \n", + "3 {'max_depth': 3, 'max_features': None, 'min_sa... triage demo \n", + "4 {'C': 0.1, 'penalty': 'l1'} triage demo \n", + "5 {'rules': [{'feature': 'project_features_entit... triage demo \n", + "6 {'max_depth': 3, 'max_features': None, 'min_sa... triage demo \n", + "7 {'C': 0.1, 'penalty': 'l1'} triage demo \n", + "8 {'rules': [{'feature': 'project_features_entit... triage demo \n", + "9 {'max_depth': 50, 'n_estimators': 150, 'min_sa... triage demo \n", + "10 {'max_depth': 50, 'n_estimators': 150, 'min_sa... triage demo \n", + "11 {'max_depth': 50, 'n_estimators': 150, 'min_sa... triage demo \n", + "\n", + " batch_comment config train_end_time test \\\n", + "0 None None 2012-04-01 False \n", + "1 None None 2012-04-01 False \n", + "2 None None 2012-04-01 False \n", + "3 None None 2012-08-01 False \n", + "4 None None 2012-08-01 False \n", + "5 None None 2012-08-01 False \n", + "6 None None 2012-12-01 False \n", + "7 None None 2012-12-01 False \n", + "8 None None 2012-12-01 False \n", + "9 None None 2012-04-01 False \n", + "10 None None 2012-08-01 False \n", + "11 None None 2012-12-01 False \n", + "\n", + " train_matrix_uuid training_label_timespan model_size \\\n", + "0 495afa5517735f1e336108cd7911b8aa 90 days 0.0625 \n", + "1 495afa5517735f1e336108cd7911b8aa 90 days 0.0625 \n", + "2 495afa5517735f1e336108cd7911b8aa 90 days 0.0625 \n", + "3 67a0cc5dc9ab89cdf0b1ae3a7883b145 90 days 0.0625 \n", + "4 67a0cc5dc9ab89cdf0b1ae3a7883b145 90 days 0.0625 \n", + "5 67a0cc5dc9ab89cdf0b1ae3a7883b145 90 days 0.0625 \n", + "6 363cae6e28d220afc10d2be99b01a09a 90 days 0.0625 \n", + "7 363cae6e28d220afc10d2be99b01a09a 90 days 0.0625 \n", + "8 363cae6e28d220afc10d2be99b01a09a 90 days 0.0625 \n", + "9 495afa5517735f1e336108cd7911b8aa 90 days 0.0625 \n", + "10 67a0cc5dc9ab89cdf0b1ae3a7883b145 90 days 0.0625 \n", + "11 363cae6e28d220afc10d2be99b01a09a 90 days 0.0625 \n", + "\n", + " random_seed built_in_triage_run \n", + "0 908907174 1 \n", + "1 1656233507 1 \n", + "2 1259133573 1 \n", + "3 1106414652 1 \n", + "4 897217774 1 \n", + "5 1431439151 1 \n", + "6 1463730397 1 \n", + "7 1879462244 1 \n", + "8 827031307 1 \n", + "9 1443952767 1 \n", + "10 1252071336 1 \n", + "11 590967750 1 " + ], + "text/html": [ + "\n", + "
\n", + "
\n", + "
\n", + "\n", + "\n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + "
model_idmodel_group_idmodel_hashrun_timebatch_run_timemodel_typehyperparametersmodel_commentbatch_commentconfigtrain_end_timetesttrain_matrix_uuidtraining_label_timespanmodel_sizerandom_seedbuilt_in_triage_run
012de38e91e633948e5a5b4198f520ed8372022-06-21 18:44:33.0102792022-06-21 18:44:32.593620sklearn.tree.DecisionTreeClassifier{'max_depth': 3, 'max_features': None, 'min_sa...triage demoNoneNone2012-04-01False495afa5517735f1e336108cd7911b8aa90 days0.06259089071741
123e1e70aac9d362ffc6416b38afab7dbe22022-06-21 18:44:33.0074272022-06-21 18:44:32.593620triage.component.catwalk.estimators.classifier...{'C': 0.1, 'penalty': 'l1'}triage demoNoneNone2012-04-01False495afa5517735f1e336108cd7911b8aa90 days0.062516562335071
234942fda8a13abbf322a3a78c8cdb7ba1a2022-06-21 18:44:54.5662992022-06-21 18:44:32.593620triage.component.catwalk.baselines.rankers.Bas...{'rules': [{'feature': 'project_features_entit...triage demoNoneNone2012-04-01False495afa5517735f1e336108cd7911b8aa90 days0.062512591335731
342468fc2d1aca51d1150deeb77b030c6792022-06-21 18:44:55.2798572022-06-21 18:44:32.633577sklearn.tree.DecisionTreeClassifier{'max_depth': 3, 'max_features': None, 'min_sa...triage demoNoneNone2012-08-01False67a0cc5dc9ab89cdf0b1ae3a7883b14590 days0.062511064146521
453c2f98311493fd3b59c40358764835f192022-06-21 18:45:01.7206292022-06-21 18:44:32.633577triage.component.catwalk.estimators.classifier...{'C': 0.1, 'penalty': 'l1'}triage demoNoneNone2012-08-01False67a0cc5dc9ab89cdf0b1ae3a7883b14590 days0.06258972177741
5644c850b235b18c32436c2c9158d47316e2022-06-21 18:45:22.5522562022-06-21 18:44:32.633577triage.component.catwalk.baselines.rankers.Bas...{'rules': [{'feature': 'project_features_entit...triage demoNoneNone2012-08-01False67a0cc5dc9ab89cdf0b1ae3a7883b14590 days0.062514314391511
6726e904c2e00f3ae0b31de9b98ca0c15842022-06-21 18:45:29.1751302022-06-21 18:44:32.781213sklearn.tree.DecisionTreeClassifier{'max_depth': 3, 'max_features': None, 'min_sa...triage demoNoneNone2012-12-01False363cae6e28d220afc10d2be99b01a09a90 days0.062514637303971
783ee410a6a06ba9e5b6eea7f1794ed1b392022-06-21 18:45:30.3870692022-06-21 18:44:32.781213triage.component.catwalk.estimators.classifier...{'C': 0.1, 'penalty': 'l1'}triage demoNoneNone2012-12-01False363cae6e28d220afc10d2be99b01a09a90 days0.062518794622441
894cec877d278f1ea40c1677dbf2354a9b82022-06-21 18:45:56.2721092022-06-21 18:44:32.781213triage.component.catwalk.baselines.rankers.Bas...{'rules': [{'feature': 'project_features_entit...triage demoNoneNone2012-12-01False363cae6e28d220afc10d2be99b01a09a90 days0.06258270313071
9101b86731b91ced3187be3a876b23303cb82022-06-21 18:46:02.3903432022-06-21 18:44:32.593620sklearn.ensemble.RandomForestClassifier{'max_depth': 50, 'n_estimators': 150, 'min_sa...triage demoNoneNone2012-04-01False495afa5517735f1e336108cd7911b8aa90 days0.062514439527671
101119c5d0d402e699c5622ae2af9660248de2022-06-21 18:46:07.8138652022-06-21 18:44:32.633577sklearn.ensemble.RandomForestClassifier{'max_depth': 50, 'n_estimators': 150, 'min_sa...triage demoNoneNone2012-08-01False67a0cc5dc9ab89cdf0b1ae3a7883b14590 days0.062512520713361
11121e7e6927dd1de32f2249b42bb7acf8b512022-06-21 18:46:13.9273882022-06-21 18:44:32.781213sklearn.ensemble.RandomForestClassifier{'max_depth': 50, 'n_estimators': 150, 'min_sa...triage demoNoneNone2012-12-01False363cae6e28d220afc10d2be99b01a09a90 days0.06255909677501
\n", + "
\n", + " \n", + " \n", + " \n", + "\n", + " \n", + "
\n", + "
\n", + " " + ] + }, + "metadata": {}, + "execution_count": 15 + } + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "zyC6b5e07wBl" + }, + "source": [ + "You can find the predictions from each model for each project in `test_results.predictions`. Let's take a quick look..." + ] + }, + { + "cell_type": "code", + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/", + "height": 270 + }, + "id": "1H_tiuoUrYzS", + "outputId": "096ae198-a966-4f5c-fd15-d1205707fa78" + }, + "source": [ + "pd.read_sql('SELECT * FROM test_results.predictions LIMIT 5;', db_engine)" + ], + "execution_count": 16, + "outputs": [ + { + "output_type": "execute_result", + "data": { + "text/plain": [ + " model_id entity_id as_of_date score label_value rank_abs_no_ties \\\n", + "0 3 254981 2012-04-12 1.00000 0 1 \n", + "1 3 249195 2012-05-12 0.99740 1 2 \n", + "2 3 255422 2012-04-09 0.99481 0 3 \n", + "3 3 253874 2012-04-18 0.99221 0 4 \n", + "4 3 250376 2012-05-09 0.98961 1 5 \n", + "\n", + " rank_abs_with_ties rank_pct_no_ties rank_pct_with_ties \\\n", + "0 1 0.00258 0.00259 \n", + "1 2 0.00515 0.00518 \n", + "2 3 0.00773 0.00777 \n", + "3 4 0.01031 0.01036 \n", + "4 5 0.01289 0.01295 \n", + "\n", + " matrix_uuid test_label_timespan \n", + "0 6a751eabcf4722abda70f77c0d9d712d 90 days \n", + "1 6a751eabcf4722abda70f77c0d9d712d 90 days \n", + "2 6a751eabcf4722abda70f77c0d9d712d 90 days \n", + "3 6a751eabcf4722abda70f77c0d9d712d 90 days \n", + "4 6a751eabcf4722abda70f77c0d9d712d 90 days " + ], + "text/html": [ + "\n", + "
\n", + "
\n", + "
\n", + "\n", + "\n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + "
model_identity_idas_of_datescorelabel_valuerank_abs_no_tiesrank_abs_with_tiesrank_pct_no_tiesrank_pct_with_tiesmatrix_uuidtest_label_timespan
032549812012-04-121.000000110.002580.002596a751eabcf4722abda70f77c0d9d712d90 days
132491952012-05-120.997401220.005150.005186a751eabcf4722abda70f77c0d9d712d90 days
232554222012-04-090.994810330.007730.007776a751eabcf4722abda70f77c0d9d712d90 days
332538742012-04-180.992210440.010310.010366a751eabcf4722abda70f77c0d9d712d90 days
432503762012-05-090.989611550.012890.012956a751eabcf4722abda70f77c0d9d712d90 days
\n", + "
\n", + " \n", + " \n", + " \n", + "\n", + " \n", + "
\n", + "
\n", + " " + ] + }, + "metadata": {}, + "execution_count": 16 + } + ] + }, + { + "cell_type": "code", + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/", + "height": 426 + }, + "id": "y6W6q6C98kPu", + "outputId": "22ad2fb5-c6f2-4d9f-c3db-bec72782e14b" + }, + "source": [ + "pd.read_sql(\"\"\"\n", + " SELECT \n", + " model_id, \n", + " COUNT(*) AS num_preds, \n", + " COUNT(DISTINCT entity_id) AS distinct_entities, \n", + " AVG(label_value) AS non_funded_rate\n", + " FROM test_results.predictions \n", + " GROUP BY 1\n", + " ORDER BY 1;\n", + " \"\"\", db_engine)" + ], + "execution_count": 17, + "outputs": [ + { + "output_type": "execute_result", + "data": { + "text/plain": [ + " model_id num_preds distinct_entities non_funded_rate\n", + "0 1 388 388 0.394330\n", + "1 2 388 388 0.394330\n", + "2 3 388 388 0.394330\n", + "3 4 1124 1124 0.378114\n", + "4 5 1124 1124 0.378114\n", + "5 6 1124 1124 0.378114\n", + "6 7 635 635 0.420472\n", + "7 8 635 635 0.420472\n", + "8 9 635 635 0.420472\n", + "9 10 388 388 0.394330\n", + "10 11 1124 1124 0.378114\n", + "11 12 635 635 0.420472" + ], + "text/html": [ + "\n", + "
\n", + "
\n", + "
\n", + "\n", + "\n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + "
model_idnum_predsdistinct_entitiesnon_funded_rate
013883880.394330
123883880.394330
233883880.394330
34112411240.378114
45112411240.378114
56112411240.378114
676356350.420472
786356350.420472
896356350.420472
9103883880.394330
1011112411240.378114
11126356350.420472
\n", + "
\n", + " \n", + " \n", + " \n", + "\n", + " \n", + "
\n", + "
\n", + " " + ] + }, + "metadata": {}, + "execution_count": 17 + } + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "w6aGkcdb9Fmr" + }, + "source": [ + "Note that the `label_value` column here is the actual label (e.g., if the project actually failed to be fully funded in four months).\n", + "\n", + "If we want to see how the model performed on your evaluation metrics, you can find these results calculated in `test_results.evaluations`. Let's look at how our models did in term of precision at the top 10% since that reflects our deployment setting:" + ] + }, + { + "cell_type": "code", + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/", + "height": 490 + }, + "id": "8PqCVkEh-P_Q", + "outputId": "98f07b2a-9cb0-4d2d-f329-36873e01a175" + }, + "source": [ + "pd.read_sql(\"\"\"\n", + " SELECT *\n", + " FROM test_results.evaluations \n", + " WHERE metric='precision@' AND parameter='10_pct'\n", + " ORDER BY model_id;\n", + " \"\"\", db_engine)" + ], + "execution_count": 18, + "outputs": [ + { + "output_type": "execute_result", + "data": { + "text/plain": [ + " model_id evaluation_start_time evaluation_end_time as_of_date_frequency \\\n", + "0 1 2012-04-01 2012-06-30 1 days \n", + "1 2 2012-04-01 2012-06-30 1 days \n", + "2 3 2012-04-01 2012-06-30 1 days \n", + "3 4 2012-08-01 2012-10-31 1 days \n", + "4 5 2012-08-01 2012-10-31 1 days \n", + "5 6 2012-08-01 2012-10-31 1 days \n", + "6 7 2012-12-01 2013-02-28 1 days \n", + "7 8 2012-12-01 2013-02-28 1 days \n", + "8 9 2012-12-01 2013-02-28 1 days \n", + "9 10 2012-04-01 2012-06-30 1 days \n", + "10 11 2012-08-01 2012-10-31 1 days \n", + "11 12 2012-12-01 2013-02-28 1 days \n", + "\n", + " metric parameter num_labeled_examples num_labeled_above_threshold \\\n", + "0 precision@ 10_pct 388 38 \n", + "1 precision@ 10_pct 388 38 \n", + "2 precision@ 10_pct 388 38 \n", + "3 precision@ 10_pct 1124 112 \n", + "4 precision@ 10_pct 1124 112 \n", + "5 precision@ 10_pct 1124 112 \n", + "6 precision@ 10_pct 635 63 \n", + "7 precision@ 10_pct 635 63 \n", + "8 precision@ 10_pct 635 63 \n", + "9 precision@ 10_pct 388 38 \n", + "10 precision@ 10_pct 1124 112 \n", + "11 precision@ 10_pct 635 63 \n", + "\n", + " num_positive_labels sort_seed matrix_uuid \\\n", + "0 153 None 6a751eabcf4722abda70f77c0d9d712d \n", + "1 153 None 6a751eabcf4722abda70f77c0d9d712d \n", + "2 153 None 6a751eabcf4722abda70f77c0d9d712d \n", + "3 425 None 051df0ba6431460b81bd18a25fea0d99 \n", + "4 425 None 051df0ba6431460b81bd18a25fea0d99 \n", + "5 425 None 051df0ba6431460b81bd18a25fea0d99 \n", + "6 267 None 10b581471b80ac2c5ca865e56be6cfe7 \n", + "7 267 None 10b581471b80ac2c5ca865e56be6cfe7 \n", + "8 267 None 10b581471b80ac2c5ca865e56be6cfe7 \n", + "9 153 None 6a751eabcf4722abda70f77c0d9d712d \n", + "10 425 None 051df0ba6431460b81bd18a25fea0d99 \n", + "11 267 None 10b581471b80ac2c5ca865e56be6cfe7 \n", + "\n", + " subset_hash best_value worst_value stochastic_value num_sort_trials \\\n", + "0 1.000000 0.026316 0.496491 30 \n", + "1 1.000000 0.000000 0.435965 30 \n", + "2 0.342105 0.342105 0.342105 0 \n", + "3 1.000000 0.000000 0.448512 30 \n", + "4 1.000000 0.000000 0.447024 30 \n", + "5 0.383929 0.383929 0.383929 0 \n", + "6 1.000000 0.000000 0.501587 30 \n", + "7 1.000000 0.000000 0.470370 30 \n", + "8 0.603175 0.603175 0.603175 0 \n", + "9 0.394737 0.394737 0.394737 0 \n", + "10 0.428571 0.428571 0.428571 0 \n", + "11 0.444444 0.444444 0.444444 0 \n", + "\n", + " standard_deviation \n", + "0 0.052963 \n", + "1 0.070370 \n", + "2 0.000000 \n", + "3 0.036301 \n", + "4 0.040472 \n", + "5 0.000000 \n", + "6 0.060922 \n", + "7 0.059462 \n", + "8 0.000000 \n", + "9 0.000000 \n", + "10 0.000000 \n", + "11 0.000000 " + ], + "text/html": [ + "\n", + "
\n", + "
\n", + "
\n", + "\n", + "\n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + "
model_idevaluation_start_timeevaluation_end_timeas_of_date_frequencymetricparameternum_labeled_examplesnum_labeled_above_thresholdnum_positive_labelssort_seedmatrix_uuidsubset_hashbest_valueworst_valuestochastic_valuenum_sort_trialsstandard_deviation
012012-04-012012-06-301 daysprecision@10_pct38838153None6a751eabcf4722abda70f77c0d9d712d1.0000000.0263160.496491300.052963
122012-04-012012-06-301 daysprecision@10_pct38838153None6a751eabcf4722abda70f77c0d9d712d1.0000000.0000000.435965300.070370
232012-04-012012-06-301 daysprecision@10_pct38838153None6a751eabcf4722abda70f77c0d9d712d0.3421050.3421050.34210500.000000
342012-08-012012-10-311 daysprecision@10_pct1124112425None051df0ba6431460b81bd18a25fea0d991.0000000.0000000.448512300.036301
452012-08-012012-10-311 daysprecision@10_pct1124112425None051df0ba6431460b81bd18a25fea0d991.0000000.0000000.447024300.040472
562012-08-012012-10-311 daysprecision@10_pct1124112425None051df0ba6431460b81bd18a25fea0d990.3839290.3839290.38392900.000000
672012-12-012013-02-281 daysprecision@10_pct63563267None10b581471b80ac2c5ca865e56be6cfe71.0000000.0000000.501587300.060922
782012-12-012013-02-281 daysprecision@10_pct63563267None10b581471b80ac2c5ca865e56be6cfe71.0000000.0000000.470370300.059462
892012-12-012013-02-281 daysprecision@10_pct63563267None10b581471b80ac2c5ca865e56be6cfe70.6031750.6031750.60317500.000000
9102012-04-012012-06-301 daysprecision@10_pct38838153None6a751eabcf4722abda70f77c0d9d712d0.3947370.3947370.39473700.000000
10112012-08-012012-10-311 daysprecision@10_pct1124112425None051df0ba6431460b81bd18a25fea0d990.4285710.4285710.42857100.000000
11122012-12-012013-02-281 daysprecision@10_pct63563267None10b581471b80ac2c5ca865e56be6cfe70.4444440.4444440.44444400.000000
\n", + "
\n", + " \n", + " \n", + " \n", + "\n", + " \n", + "
\n", + "
\n", + " " + ] + }, + "metadata": {}, + "execution_count": 18 + } + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "z_vwKbQo727Q" + }, + "source": [ + "Finally, if you need to work with the training/validation matrices generated by triage or the model objects themselves, you can find them in your project path (here, `triage_output`). Let's take a quick look..." + ] + }, + { + "cell_type": "code", + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/" + }, + "id": "Tyx1QXJ3sAtQ", + "outputId": "17ef6319-142f-40df-ff76-e04497b8286f" + }, + "source": [ + "!ls triage_output/" + ], + "execution_count": 19, + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": [ + "matrices trained_models\n" + ] + } + ] + }, + { + "cell_type": "code", + "metadata": { + "id": "YnM6vADdvx-N", + "colab": { + "base_uri": "https://localhost:8080/" + }, + "outputId": "aeeebacd-3c3d-4abd-f54a-6919f8a0dba9" + }, + "source": [ + "!ls -la triage_output/matrices/\n" + ], + "execution_count": 20, + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": [ + "total 128\n", + "drwxr-xr-x 2 root root 4096 Jun 21 18:44 .\n", + "drwxr-xr-x 4 root root 4096 Jun 21 18:44 ..\n", + "-rw-r--r-- 1 root root 10686 Jun 21 18:44 051df0ba6431460b81bd18a25fea0d99.csv.gz\n", + "-rw-r--r-- 1 root root 3391 Jun 21 18:44 051df0ba6431460b81bd18a25fea0d99.yaml\n", + "-rw-r--r-- 1 root root 6201 Jun 21 18:44 10b581471b80ac2c5ca865e56be6cfe7.csv.gz\n", + "-rw-r--r-- 1 root root 3343 Jun 21 18:44 10b581471b80ac2c5ca865e56be6cfe7.yaml\n", + "-rw-r--r-- 1 root root 23598 Jun 21 18:44 363cae6e28d220afc10d2be99b01a09a.csv.gz\n", + "-rw-r--r-- 1 root root 9979 Jun 21 18:44 363cae6e28d220afc10d2be99b01a09a.yaml\n", + "-rw-r--r-- 1 root root 9055 Jun 21 18:44 495afa5517735f1e336108cd7911b8aa.csv.gz\n", + "-rw-r--r-- 1 root root 4123 Jun 21 18:44 495afa5517735f1e336108cd7911b8aa.yaml\n", + "-rw-r--r-- 1 root root 17532 Jun 21 18:44 67a0cc5dc9ab89cdf0b1ae3a7883b145.csv.gz\n", + "-rw-r--r-- 1 root root 7027 Jun 21 18:44 67a0cc5dc9ab89cdf0b1ae3a7883b145.yaml\n", + "-rw-r--r-- 1 root root 3990 Jun 21 18:44 6a751eabcf4722abda70f77c0d9d712d.csv.gz\n", + "-rw-r--r-- 1 root root 3367 Jun 21 18:44 6a751eabcf4722abda70f77c0d9d712d.yaml\n" + ] + } + ] + }, + { + "cell_type": "code", + "metadata": { + "id": "PlJ017aVMhdS" + }, + "source": [ + "# clean up the database connection\n", + "db_engine.dispose()" + ], + "execution_count": 21, + "outputs": [] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "KFingR1B32rm" + }, + "source": [ + "## Model Selection\n", + "\n", + "`triage` includes a component called `audition` that can help you visualize your model results over time and narrow down your best-performing models. Here we'll provide a quick introduction, but you can find more depth in the [audition tutorial](https://github.com/dssg/triage/blob/master/src/triage/component/audition/Audition_Tutorial.ipynb) as well as the [audition documentation](https://dssg.github.io/triage/audition/audition_intro/) and [model selection concepts overview](https://dssg.github.io/triage/audition/model_selection/). In general, `audition` is best run using a notebook to iteratively explore and narrow down your models." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "Csg-HDeiPDma" + }, + "source": [ + "### Audition Parameters\n", + "\n", + "To run `audition`, you'll need to specify a few parameters:\n", + "\n", + "`metric` and `parameter` together specify the evaluation metric of interest for your project. Note that these need to be calculated as part of the scoring section in your `triage` config and should match the values in the columns of the same name in `test_results.evaluations`\n", + "\n", + "The `run_hash` is an identifier for the run with your complete model grid that you want to evaluate -- the easiest way to find this is from the `triage_metadata.triage_runs` table. This will likely be the `run_hash` associated with the most recent record in that table, but you should be able to figure out which run you want to use from there." + ] + }, + { + "cell_type": "code", + "metadata": { + "id": "RW1vSrhQP4ww" + }, + "source": [ + "from triage.component.audition import Auditioner\n", + "from triage.component.audition.pre_audition import PreAudition\n", + "from triage.component.audition.rules_maker import SimpleRuleMaker, RandomGroupRuleMaker, create_selection_grid\n", + "\n", + "from matplotlib import pyplot as plt\n", + "plt.style.use('ggplot')\n", + "%matplotlib inline\n", + "\n", + "import yaml\n", + "from sqlalchemy.engine.url import URL\n", + "from triage.util.db import create_engine\n", + "\n", + "import logging\n", + "logging.basicConfig(level=logging.WARNING)\n", + "\n", + "import pandas as pd\n", + "pd.set_option('precision', 4)" + ], + "execution_count": 22, + "outputs": [] + }, + { + "cell_type": "code", + "metadata": { + "id": "Jwg62Co837C4" + }, + "source": [ + "metric = 'precision@'\n", + "parameter = '10_pct'\n", + "run_hash = '8440d33cdc0f4cb808d573a05b065d17'" + ], + "execution_count": 23, + "outputs": [] + }, + { + "cell_type": "code", + "metadata": { + "id": "HyeFeaOEPLzk" + }, + "source": [ + "dbconfig = yaml.safe_load(database_yaml)\n", + "db_url = URL(\n", + " 'postgres',\n", + " host=dbconfig['host'],\n", + " username=dbconfig['user'],\n", + " database=dbconfig['db'],\n", + " password=dbconfig['pass'],\n", + " port=dbconfig['port'],\n", + " )\n", + "\n", + "conn = create_engine(db_url)" + ], + "execution_count": 24, + "outputs": [] + }, + { + "cell_type": "code", + "metadata": { + "id": "Sq9qO4-_QPJl" + }, + "source": [ + "# table where audition results will be stored\n", + "best_dist_table = 'audition_best_dist'" + ], + "execution_count": 25, + "outputs": [] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "ZVRftIpfQZxs" + }, + "source": [ + "### Pre-Audition: Models and Temporal Splits\n", + "\n", + "Because you may have run several experiments as you iterate, explore, and debug, `audition` needs to know which set of model groups and temporal validation splits to focus on for model selection. While you can specify these directly, `triage` also provides some `pre-audition` tools to help define these.\n", + "\n", + "For example, `get_model_groups_from_experiment()` and `get_train_end_times()` (note that this will return the `train_end_times` associated with the set of model groups returned by one of the `get_model_groups` methods, so those should be run first). Note that the `baseline_model_types` parameter in the constructor is optional and can be used to identify model groups as baselines rather than candidates for model selection." + ] + }, + { + "cell_type": "code", + "metadata": { + "id": "5NmsPZT7Rx_J" + }, + "source": [ + "pre_aud = PreAudition(\n", + " conn, \n", + " baseline_model_types=[\n", + " 'sklearn.dummy.DummyClassifier',\n", + " 'triage.component.catwalk.baselines.rankers.BaselineRankMultiFeature',\n", + " 'triage.component.catwalk.baselines.thresholders.SimpleThresholder'\n", + " ]\n", + ")\n", + "\n", + "# select model groups by experiment hash id\n", + "model_groups = pre_aud.get_model_groups_from_experiment(run_hash)\n", + "\n", + "# Note that this will find train_end_times associated with the model groups defined above\n", + "end_times = pre_aud.get_train_end_times(after='1900-01-01')" + ], + "execution_count": 26, + "outputs": [] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "rLpWTrG_SM88" + }, + "source": [ + "`get_model_groups_from_experiment()` returns a dictionary with keys `model_groups` and `baseline_model_groups`.\n", + "\n", + "How many of each did we get?" + ] + }, + { + "cell_type": "code", + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/" + }, + "id": "zFqr_IEBSbui", + "outputId": "a1010b01-12f8-47bd-d240-731df28226d1" + }, + "source": [ + "# Number of non-baseline model groups:\n", + "print(len(model_groups['model_groups']))" + ], + "execution_count": 27, + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": [ + "3\n" + ] + } + ] + }, + { + "cell_type": "code", + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/" + }, + "id": "HLuZIZr9SfiY", + "outputId": "78a69415-dd7b-487b-e434-187121ef4c94" + }, + "source": [ + "# Number of baseline model groups:\n", + "print(len(model_groups['baseline_model_groups']))" + ], + "execution_count": 28, + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": [ + "1\n" + ] + } + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "AOYgNVIGSkl2" + }, + "source": [ + "`get_train_end_times()` returns a list of `train_end_times`:" + ] + }, + { + "cell_type": "code", + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/" + }, + "id": "YSVYkQNzSqJR", + "outputId": "73f7804e-4651-4ee1-f90b-bd1acb480927" + }, + "source": [ + "end_times" + ], + "execution_count": 29, + "outputs": [ + { + "output_type": "execute_result", + "data": { + "text/plain": [ + "[Timestamp('2012-04-01 00:00:00'),\n", + " Timestamp('2012-08-01 00:00:00'),\n", + " Timestamp('2012-12-01 00:00:00')]" + ] + }, + "metadata": {}, + "execution_count": 29 + } + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "fHeaYey0S3mr" + }, + "source": [ + "### Setting Up Your Auditioner\n", + "\n", + "`Auditioner` is the main API to do the rules selection and model groups selection. It filters model groups using a two-step process.\n", + "\n", + "- Broad thresholds to filter out truly bad models\n", + "- A selection rule grid to find the best model groups over time for each of a variety of methods\n", + "\n", + "Note that model groups that don't have a full set of `train_end_time` splits associated with them will be excluded from the analysis, so **it's important to ensure that all model groups have been completed across all train/test splits**\n", + "\n", + "When we set up our auditioner object, we need to give it a database connection, the model groups to consider (and optionally baseline model groups), train_end_times, and tell it how we're going to filter the models. Note that the `initial_metric_filters` parameter specified below tells `Auditioner` what metric and parameter we'll be using and starts off without any initial filtering constraints (which is what you'll typically want):" + ] + }, + { + "cell_type": "code", + "metadata": { + "id": "HRjTq_cITqv1" + }, + "source": [ + "aud = Auditioner(\n", + " db_engine = conn,\n", + " model_group_ids = model_groups['model_groups'],\n", + " train_end_times = end_times,\n", + " initial_metric_filters = [{'metric': metric, 'parameter': parameter, 'max_from_best': 1.0, 'threshold_value': 0.0}],\n", + " distance_table = best_dist_table,\n", + " baseline_model_group_ids = model_groups['baseline_model_groups'] # optional\n", + ")" + ], + "execution_count": 30, + "outputs": [] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "vnYukR-2WJYI" + }, + "source": [ + "### Using Audition for Model Selection\n", + "\n", + "We can use the `plot_model_groups` method to visualize the performance of our model groups over temporal split (note that the plot may take a few minutes to generate). When this method is called, it applies the metric filters specificied to get rid of really bad model groups with respect the metric of interest. A model group is discarded if:\n", + "\n", + "- It’s never close to the “best” model (based on the `max_from_best` filter) or\n", + "- If it’s metric is below a certain number (based on the `threshold_value` filter) at least once\n", + "\n", + "As a starting point, we don't filter out any models, but can iteratively narrow our grid by refining these filters. Let's take a look at our models:" + ] + }, + { + "cell_type": "code", + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/", + "height": 811 + }, + "id": "x-FZ-8JVYbvl", + "outputId": "12912ea9-b213-46f3-fa67-cef7ab7cb43f" + }, + "source": [ + "aud.plot_model_groups()" + ], + "execution_count": 31, + "outputs": [ + { + "output_type": "display_data", + "data": { + "text/plain": [ + "
" + ], + "image/png": "iVBORw0KGgoAAAANSUhEUgAAAtMAAAGICAYAAACDVWyYAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8QVMy6AAAACXBIWXMAAAsTAAALEwEAmpwYAAB10ElEQVR4nO3dd5xTVf7/8dedyQwzFLGMoJQVlGGkCCqiqCuyLq5lRVTwAJZdbKgoil3Xspafbe0F2+qqK7aDuoK7+rWsdbErdkSw0qWItGHS7u+PexMyYUqSSTKBeT8fjzxmcnNz7zn3JjOffPK55ziu6yIiIiIiIukrau4GiIiIiIhsrBRMi4iIiIhkSMG0iIiIiEiGFEyLiIiIiGRIwbSIiIiISIYUTIuIiIiIZEjBtDSZ4zhjHcdx67kNzVMbNncc53LHcXat47HXHcd5PR/tyITjOMMcx/nccZx1/jHbvLnblMw/tmmPo+k4Tje/T2ObuP9T/e0cUMdjtzqOU+M4Tp+m7EPq5r+/j69nues4To/maFcyx3F+cBxnch73d5jjOGfna3/Z5B+rh9JYPyvv43xzHKeL4zh3OI7zjuM4a/0+dKtn3TLHcW5wHGeh4zjV/nMG57nJie2p93+aFJ5AczdANilHAvOSln2Vp31vDvzV3//HSY+Nz1Mb0uY4TgB4FHgbOA0IAquatVGF6R5gDHCf4zh9XdddBeA4zp7ABOBy13W/bM4GbsLG4v2v+Eczt6PQHAYMBW5u5nZk4nBgZRrrLwT2BL7NTXNqcxynE977+o/ADoAD/AA8C0xyXXd+ipvqARjgI+At4A8NrPuAv7/zgO/w/h6/6DjOnq7rfpJ2J5puc+r/nyYFRsG0ZNMnruvOSWVFx3Faua5bk+sGAbium6+APhOdgXaAdV33zeZuTKFyXdd1HOdE4FPgOuA0x3Fa4f0D/MJf1uLk830kzcNxHAcocV03mK1tuq47I831a4B3s7X/hjiOcxJwO/AOcBdeQsYBKoHRwHjHcca5rmtT2Nybrut29Ld7IvUE047j9AeOAo53XfdBf9kbwJfAlcChTeqUbPpc19VNtybd8DJXLtCjkccHA1OAFXiBN8BA4Cm8T9/VwCzgGqC8ju0cDkwHVuNlVd7H+yPXzd9+8m2s/7zXgdeTtlUF/MtvSzXeP4oDk9a53N9OJfAff78/ApcBRSkcl22BfwJLgRrgM+CYOrafeHu9ge095B+n3fAy2bHj9Uf/8bPxsjcrganA1knP3wy4E1jgt2cWcBbgJK23C14WZx0wH7gUuML7c1FrvQBwEfC1v70FwE1AWcI6sXMzNmHZQOBlYJnfh++Au1J8rV0IRIF9gKuBMLBbCs+LtWM8XibxZ2At8G+gW9K6PwCTgZOAOf5x+Bj4XSP7GODv47cJyyb4y/5fwrJKf9kfE5btDrziv8bWAP8Fdq/n/O+ZcP5v8x87CpjB+vfG58DJSc/f19/uKn8fLwJ9G+nT6/W9Rln/vh6E9+3KSv81cHvia8Bf9wr/GK7Eez+8CgxKWmeIv71D8V6nS/3bZGDzFM5xyuctlWMBHOAf51/94zoLuCzhXCQflx+y/Po7Hu+9FQIO9x/rD0wDfvHP/3Rgn3r697Lf9jV4H0JPSNrHQwn3twEeZv3fhoV+2zrU9z72lx/jb3udf64eAbatpz+jgZl+ez4k4X2SsO7p/jk5vIFjeaS/jUNT+ZuR8LwT/T50q+OxS/G+FWxdx+u2BmiVxn5cvL9NF7P+/9qbwM51rJvR/zTdCu/W7A3QbeO/sf6fahVegBW7FSc9Phf4G95Xowf6j40ALgEO8f8BjAcWAU8k7SMWlPzLf84BeIHcGUAr/4+SixeID/JvW/vPfZ2EIBXoBCzBC+KOAYYB/wdEgIMS1rvc3+YXwDl+u2/zlx3XyDFpA3zj72cccBBewOEC4/x1ugAj/WVX+W3u3cA2H/L/4H6F94/2QNYHvTcBz+F9TXm8v55NeG6Rv+4avy9/SOjLNQnrVeD9o54JjML7Knu6f+7cpPY84W/vMv/YTMD7cPJ0wjrdEv8JAG2B5f7xHoYXQI0F7kvxtRbAC5B+wgsy/pbi82LtmJtwnI7DCxq+wcv8xdb9Ae+fYOIxeMc/zlUN7KPIP3aXJSz7F17Q9HbCspP9trfz7/fD+4f7kf96GAF84C/rn3T+V+F9oJvgH7s9gN/ifcC41T8Pf8B7X1yQ8Nw/4n3wmAoM929v++3t2kCfevvH+1PWv696J72vZ+Nl74biBSUR4Iqk7dwPHAv8Du+9/gRe8LJTwjpD/O19D9zh92OCfxweTuEcp3TeUjkWwPZ4QdSjeO+z/fzzdr3/+A54H7B/Tjguu2Tx9Tcf7+/OGOD3/v52xXu//c9/nRyMF1jXAAMSnj/c798beAHsUOBM4MqkfTyUcP9lvx1H4yU9jsQrrepW1/vYXzbOX/aE35YT/ePxDdA2aV8/4r2mR/rnfwbe34rNE9ar8vvy+4be//7Po/D+tm6Wyvvff05DwfQTwKw6lhv/OX3S2E/sPE/3X4Oj8D6ILQO2TFgv4/9puhXerdkboNvGf2P9P9Xk2/+SHr+lke04eMHSMXjBwVb+8s3wgohnGnhu7I/9iXU89jq1g+kb/X82PRKWFft/8D5OWHY5dQTOeFm/lxrpy+n+c4ckLX/F/4cT+6DRI/mfVAPbfMhfd3DCsn7+slmxbfrLb8YL2GL7OaSu/eAFOTVAhX//arwgp2vCOm3wsk5uwrJ9/O39KWl7R/vLd046L2P9+7v59/s14fV2AOv/YW3wDUYjr4+vSPhWAdjbX56ctUs+Bu3wPgQ80sh+pgKv+b8X+c+5yT8Xbf3lTwDvJjznKTYMLDbzn/tMwrLY+R+etM9zgeWNtGsO8N+kZZv55/XWRp77Ov57OWn5WL89yYHzv4FvGtheMd77fBZ+Zt1fPsTf3sNJ69+JFxA7jbQzpfOWyrFg/YfceoM1/3zMy9Hrby2wTdI2/ov3QaE06VjOBJ7178dqiz+kgW/P2DCYXg2ckUL7xybsd3HstZ6w3m/99c5I2tcvwBYJy2J/B45KWPYg8GDC/Z3wPuTU4GXoDyQhGMb7oHRyfW2uow8NBdMvkfCeTFg+1H/OBtn/Bvbj+q+lNknHLwRclfB6y/h/mm6Fd9NoHpJNh+N9hR+7nZD0+L+Sn+A4zmaO41zvOM63eH80Q3hfFcZq5AD2wsto3peldg7G+8MZr+92XTcCPA7s7DjOZknr/yfp/hfAb1LYx3zXdV9PWj4Z2Bov45eJNW7t2uqv/Z+v+H1IXB7AKzWJtScKPFZHe0rxSgfwf77ruu7c2Aqu667By6YlOhAvcHnKcZxA7Ib3Tym2v7rMxgsc73Uc5xjHcbrW29P6nYb3T2Zb0j+OT7muG43dcV13OutLJxIlH4NVeK+D5PWSvQrs6ThOGbAz3kVEf8N7be/jr/M74LWE5wwG/u267oqE/a3Eyzrum7T9EF6wmugDYAvHcSY7jnNI8mgwjuNU4mU2H006V2vxApKmjliQ/P74nKT3h+M4Qx3Hec1xnGV4H2RDQE+8bGQq22sFdEyhLQ2etzSOxSd+G59wHGek4zgdUth3KtJ5/S2K3XEcpxzvtTAFiCa028H7gB5rdxWwHXB/4n5S8AFwnuM4ZzqOs5Nfp92QKqADXuY+znXd/+FloZNft++4rvtLwv3P/Z+Jr5NheAE1juO0xiu9WY6Xxb8GuDdpm/+pYz+F4nn/7yYAruv+gFdKGDvP2f6fJs1MwbRk0xeu636YcJuV9PjCOp7zIHAKXp3l/nhB+Gn+Y2X+z638n8kjhWRqy3rasgjvn9MWScuXJ92vSWhbJvuIPZ6JFYl33PUXJf2StF5seaydW+JlL5MvYkpuz7Z4Gadkycs64AXha/CCjtjtZ//xraiD67q/4gWTC/AuLvrJcZwvHMcZUdf6yRzHGYP3T3csXobxfj+oSFV9feuc4XrJXsML/PbC6+enrusuxvtq/nf+8H0d8ILumIZeK8mvxSVJH5pwXfcNvK/lu+J9YF3iOM4rjuP081eJBYIPUPtchfC+sajzXKWhrvdHq9gdf2iv5/GynyfgfV09EK90pK73UV3bo551kzV23lI6Fv4H7QPw/kc+AixyHOddx3GaGryl+rpKfj1siZcNvrSOdp+O92GqiMz/Vo7C+/B2Pt61HfMdx7nM32ZdYn8v6nvdJv99q3VO3fUXzZYBOI6zJV7bY0H2If5jxnXdV1zX/Sfwl6Rt/oxXlpYNv7Dhew3W9yP5NdmYxs5ztv+nSTPTaB6ST27iHT97NxxvWLPbEpbvlPS8pf7PznhZ4aZajnfBTbJt/DYmB6aZ7qOurNs2CY/n03JgS8dxSpMC6uT2LKTuDGDysmV4X73vU8e64AXLdXK9YaZG+EHwbnh1gtZxnP6u69Z7fh3HqcCr857muu4/Hcf5Hq8u9FxSH82jvr59kuJ6jQ3J9Tne63U/vAs5Y0Hzq3j1l3PxPuhMT3hOQ6/H5NeiW8d6uK77FN63BG3xyiWuB/7PcZwueOcKvOP8Sh1Pz9ooEfUYgZeNPsJ13VBsoeM4W5D04TALGjtvKR8L13VfA17zR43ZG68u/D+O43RzXXdpHc9tSvs+SVqWfJ5X4H2zNAnvouYNuK4bdRwn8W9lylzX/RkviXGa4zhVwJ/xLr5bAtxdx1Nify/qe91+lM7+WR+LrPN/dsMrFVqbsE7y8HCdWf+/oam+BA53HKd10j57470mUhqlKkFjr8Ns/0+TZqbMtDSnVnjZllDS8rFJ99/Gy2qNa2BbsUxHeQr7fQMYlDh4v+M4xXjZmRn+V+xN9QbQxXGcvZOWH4WXUcn3cH1v4L3fj0xafjTeP4t3/Pvv4B2bePmF4zht8LLBif4PL3PUPunbiNit3mA6xnXdsOu67+Jl24qAXo085Ta818x4//lv4X31+1f/6/tUjEzMtvnnpwvr+x+TfAza4X3dnLxeLa7rung1xvvjfdBIDKZ3wSuFej/pH/YbwMH+PhL3N8zfVspc113tuu6/8Y7LtngZsFl4dat96jlXnzWy2RpSe1/VpzXeRYnxANFxnP1ovFQqE42dt7SPheu6Na7rvopXrtMG6O4/lMlxSfX1l9yGNXgXEPfHu65jg7b7q37j9+/EFEo16tvXLNd1/4L3Qa5vPavNwsu0jk5c6DjOXnhlJq+nudvYh/PYBEA/A12TMuPdEvZT7O/75TT3U5/ngBIS/j76H/ZH4V0fk+7wkwf7fzdj2+qG941M7Dxn+3+aNDNlpqXZuK77q+M47wLnOI6zEO/T+vEkZVVc113lOM5FwB2O4zyNV6e3Cq8mdZ3runfg/WFfBox2HOczvPKD713XXcaGbsEL2F92HOeveCNfjMer4fxjlrr3EN4V9M84jhMbIulovCDr5OSv6vPgBbxSg3scx9kaLxMTuwL/2oRM2y14x+Ilx3Eux/uDfh7eiApxruu+7jjO43jZ0JvxhnSK4v3DOxhvJIlvkhvhOM4heP9AnsUbtaEN3tXrq2ggoHAc52C8DyKnuLUnbLgAL+i8z3Gc/fxgtiHtgGcdx7kXr3b9Wrw67uRs3+KkY3CB39arGtk+eKUek/ACyLf8ZTPw+vg7vAxnoqvwvtb+r+M41+MFnRfgBaHJ627AcZwr8bJer+F9I9AF75h+4rruEn+d04CpjuOUAhbvvdYRrxzlJ9d1G5p45Cu8sX1H4U3asaqOEq6G/B8wEXjIcZwH8d5nl9J4lj8TDZ4313XdVI6F4zin4NUhP4/3bUIFXjZ7AesziV/hfdtzKt4Ff+tc142VKdQn1ddfXc7GG2LtRcdxHsD7FqkCb5SPYtd1L/T7NxF4BnjVcZx78LLLvfCGuftr8kYdx2mPl6V/lPVD8Q3HK3t4KXl98K4xcRznMrxrHybjXXvRGe8C5tmkOcGPv7038L7F+BzvNTMJ+JvjONfhnZ9r/dUr8f5OrSKpZrsujuOM9H8d4P88yHGcJXglU2/4+5/hOM6TwK2O45Tg/W06Fe+D09Hp9MVXjfc6vAEvAXAF3v+ZW/z9Zft/mjS3+q5M1E23VG+kPs70Bo/jBV8v4P0h+Rnvyv0/UvdIGCOB9/D+UK30fz8k4fHD8P7Bhah95fnr1D3O9LN447Cuo+FxpgNJyx+igTFlE9bbFq/ess5xpv110h3NY4PRA0gax7i+Y876caYX4mWjv6HucaZ3JbVxpovwPjDExpn91f/9b3gZ69j5TTwXVcCTeP+s1uH9o38e2KOBfrfDGwrvjeS2+o8f6u/jpAa2EWtHbJzfJXgXnf0H6J607g94wcGJeMFjDV4wvF+K74de/r7eTVo+lTpe1/5je5DiONN1PPePeBdrLfTbOhevJrhT0np74l28+It/7H/AG1lkz0b6s41/jlb57X+9ofc1/vsmadkE/5xX413sNpQNR9kZ4m9vaD2v5W6NtDPl89bYsfAfn+ofy9i4y1OoPcReG7yLln/x21fv34RMXn8NvLaewPtbWYP3IX0acHDSevvhfbha7d8+JWFUIhJG88AL9u7F+4AdG+/4A2qPtBFr/9ik/cTGma7BC/zqHWe6nr9blyfcPwjvb0h3//4Rfltc/xydmfD7Q6Qw9njCfuq6vZ60Xrl/bhb5+3iPOt6rKe7varwa73n+tt6i7nGmM/qfplvh3Rz/hImIbLL8r1m/xwu4729k3R/whoI7Jg9NkxYgnddfS+Y4zj/xJjA62HXd7/wscSXeyEi/Oo7TGy87W93ghpqR4zgucLXrupc0d1skf1QzLSIiIoXgJLySmU8dx7kGb6zpuXjDAe6Klwn/wnGc7g1sQyTvFEyLiIhIs3O9iz2PAf6EN4b0B3jlDyvxrqnojzdR1Pf5bpvjOMVOwtjkddwUT7VgKvMQERGRguN4E2h1wbu4+QfXddc18pRctuV1Gp4k5mHXdcfmpzVSaBRMi4iIiDTAH3+7XQOrLHW9mQ6lBVIwLSIiIiKSoY19nGl9EhARERGRfNlgQqSNPZhmwYJGJ1rLiYqKCpYuzdZMpoWvpfUX1OeWQn1uGdTnTV9L6y+oz/nWqVOnOpfr6lMRERERkQwpmBYRERERyZCCaRERERGRDCmYFhERERHJkIJpEREREZEMKZgWEREREcmQgmkRERERkQwpmBYRERERyZCCaRERERGRDCmYFhERERHJkIJpEREREZEMKZgWEREREcmQgmkRERERkQwpmBYRERERyZCCaRERERGRDCmYFhERERHJkIJpEREREZEMKZgWEREREcmQgmkRERERkQwpmBYRERERyZCCaRERERGRDCmYFhERERHJkIJpEREREZEMKZgWEREREclQIB87Mcb8AzgE+Nla27eOxx3gNuBgYC0w1lr7cT7aJiIiIiKSqXxlph8CDmzg8YOASv82Drg7D20SEREREWmSvGSmrbVvGmO6NbDKcOCf1loXeNcYs7kxZltr7cJ8tE8yFI0QWPEduJEmbyqyeClu9bqU1/+p+lcWBVc3eb8NKSsvZ111dU73UWjU55ZBfW4ZWlqfW1p/oeX2ud+eprmbUUtegukUdAbmJtyf5y/bIJg2xozDy15jraWioiIvDUwWCASabd/Noc7+rpxP0ZrZTd62G42y8sMv0nrOZ5F5rCoKNXnfDXMAN8f7KDTqc8ugPrcMLa3PLa2/0BL77OCw37Dxzd2MWgolmE6ZtfY+4D7/rrt06dJmaUdFRQXNte/mUFd/i39dRKvVq6nu/gfc0rYZb9tdt47Q5lGK+/Sm6DddU3rONzOm0a3dthzcfZeM99uYrbbYgmW//JKz7Rci9bllUJ9bhpbW55bWX2i5fW6u+KtTp051Li+UYHo+kBhFdfGXSQFzIkEA3EAZFDXhpRQFioqgrBynpFVKTwkC5a3aUlbWJvP9NqK8zWaUVQdztv1CpD63DOpzy9DS+tzS+gstt89rqgsrmVkowfQ04HRjzBPAHsCvqpfeCET9MouikiZtxg1523FKUttOTThMFJeyQNP2KyIiItJU+Roa73FgCFBhjJkH/BUoAbDW3gM8jzcs3hy8ofGOy0e7pGmcSBDXKYai4qZtyA+mCaT2clwbrgGgrLi0afsVERERaaJ8jeYxppHHXeC0fLRFsseJhqC46dlhNxT2tleaWnBcHfK+0ioLKJgWERGR5qUZECVzkRBuE0s8AAinl5leF/HWL89CIC8iIiLSFAqmJWPZykzHyzxSrJmuDnuZ6VaqmRYREZFmpmBaMuZEgrhFTS+1cENhcByc4tRqr9f5mezyFINvERERkVxRMC2Zi4Zws5GZDodSzkrD+mC6dSC1YfREREREckXBtGTMiYQgGyNqhMJpBdM1fs20hsYTERGR5qZgWjLmRINZuQDRDQVx0giM10WCFFNEaXGhDJMuIiIiLZWCaclMNAyum6XRPMJQknpgXBMOUZKN/YqIiIg0kYJpyYgTm/0wK+NMh1Ke/RC8zHQrDYsnIiIiBUDBtGTGr1t2s1IzHYI0yjyC0RAlCqZFRESkACiYlozEMtPNUuYRCdNK9dIiIiJSABRMS2Yi3sQpNDGYdqNRiERwUpz9ECAUCVGqmmkREREpAAqmJSPxzHRTyzzCYe9nSerbqYmEKMtGeYmIiIhIEymYlow4kSxdgBifSjyNzHQ0pKnERUREpCAomJbM+GUeTa2Zdv1gOtVxpoORMBGiKvMQERGRgqBgWjLiREPgOFDUxAsBY5np0tSC49hU4uUBlXmIiIhI81MwLRlxoqHsjOQRC6ZTvABxbbgG0FTiIiIiUhgUTEtmIqGsjDHt+hcgpjppS7UffCuYFhERkUKgYFoy4kSCTR4WD0g7M13jl3loNA8REREpBAqmJSNONISbjanEw2HASTmYro7EaqaVmRYREZHmp2BaMhMJ4hZlaSrxkgCO46S0+rqwN4pIeRrjUouIiIjkioJpyYgTDTV9jGnwguk0sszr/CH5WgdaNX3fIiIiIk2kYFoyE8nOaB5uOJzyxYfgDY1XhEOrNKYfFxEREckVBdOSPjeK40ayk5kOhtKa/bAmEqKkqWNbi4iIiGSJgmlJn38RYFZqpsPplXnUREKUaiQPERERKRAKpiVtTtSfSjxLo3k4aWSmg5EQpcpMi4iISIFQMC1pc/zMNNnIEIdCkE7NdCSozLSIiIgUDAXTkr5orMyjaZlp13UhzQsQQ9EwrYqVmRYREZHCoGBa0ub4w9M1eTSPNGc/BK/Mo1U2Zl4UERERyQIF05I2Jxor82hiUBsOez/TyEwHoyHKAirzEBERkcKgYFrSFxvNo4m1y66fmU61zCMcjRB2I7TKxpB8IiIiIlmgYFrSFs9M57nMo9pfv1UaQ+mJiIiI5JKCaUmbEwk2OSsNpF3msS7sB9PKTIuIiEiBUDAt6YuGmp6VJv0yj2r/wsfyQKsm71tEREQkGxRMS9qcSCgrE7bEM9Mpl3l4wXSZyjxERESkQCiYlvRFgk0fFg/W10ynWuYRy0yrzENEREQKhIJpSZsTDWVl9kM3FILiAI7jpLT+On8UkfISDY0nIiIihUHBtKTNiYayl5kuTWMqcf8CxHKVeYiIiEiBUDAt6ctSzbQbCuOkMfthLJguU5mHiIiIFAgF05Ie1/XKPIqyMTReCNLIMgejQUqcAEVFetmKiIhIYVBUIumJxmY/zEaZRxhKUs9M14RDlCorLSIiIgVEwbSkxYlNJZ6lcaZTHWMavAsQS7MxWYyIiIhIliiYlrQ4UW94umyM5kE4lPKweADBSIhWRalnskVERERyTcG0pCeLmWmvzCOdzHSQVspMi4iISAFRMC1pcfyaaZpYu+yGw4Cb1mgeoWiYVsXKTIuIiEjhUDAt6fFnIXSbOppHmrMfgl/moTGmRUREpIAomJa0OFkazcP1g+lUM9PRaJSQG6Y0G0PyiYiIiGSJgmlJS2w0D5paMx0Oez9TzEzHpxJXZlpEREQKiIJpSU80iFsUAMdp2nbSLPNYG/LKS1TmISIiIoVEwbSkxYmEsjIsXrplHppKXERERAqRgmlJixMNZW9YPIDS1ALz6niZR6um71tEREQkS/I2zpgx5kDgNqAYuN9ae13S478BHgY299e50Fr7fL7aJymKBLMTTPuZZlLMTFeHawAoU5mHiIiIFJC8ZKaNMcXAJOAgoDcwxhjTO2m1SwBrrd0FGA3clY+2SXqcaKjJY0yDX+ZRXIxTlNpLMFbmUa5JW0RERKSA5KvMY3dgjrX2O2ttEHgCGJ60jgts5v/eHliQp7ZJGpxICDcbAW0onHJWGqDGL/NQZlpEREQKSb7KPDoDcxPuzwP2SFrncuAlY8wEoA0wND9Nk7REQ00fFg+8Mo80JmypiWWm03iOiIiISK4V0tzMY4CHrLU3GWP2BB4xxvS11kYTVzLGjAPGAVhrqaioaIamQiAQaLZ9N4dYf4vmtsLdqgNuE/u+pnVrKC2lTYrbCSwqpW15G7bp0LFJ+01HSzvHoD63FOpzy9DS+tzS+gvqc6HIVzA9H+iacL+LvyzRCcCBANbad4wxZUAF8HPiStba+4D7/Lvu0qVLc9LgxlRUVNBc+24OFRUVLP15Ia1XryK4cg3hkqb1PbR0GZSWUJ3iMVz26y9Eg5G8HvOWdo5BfW4p1OeWoaX1uaX1F9TnfOvUqVOdy/MVTH8AVBpjuuMF0aOBo5LW+Qn4PfCQMaYXUAYsyVP7JAVOxJs4JSsXIIZDFLVpnfL66yIhSjXGtIiIiBSYvFyAaK0NA6cDLwIzvUX2S2PMlcaYQ/3VzgFOMsZ8CjwOjLXWuvlon6TIvwgwO+NMp1czHYyGaKWRPERERKTA5K1m2h8z+vmkZZcl/P4VsHe+2iPpc6J+MJ2V0TxCkMbIHMFIiNaBsqbvV0RERCSLNAOipCwWTDd1NA83EgHXxSlJ/bNcMBpWmYeIiIgUHAXTkjq/ZtptalAbSm/2Q4CaSJBWCqZFRESkwCiYlpQ58ZrpJpZ5hMPez5LUtxOKhjRhi4iIiBQcBdOSMieandE83KC3nVTLPNaFQrigzLSIiIgUHAXTkrpICNcpBqeJL5t4Zjq14LjaLy8pD7Rq2n5FREREskzBtKTMiYayM8Z0yAumnRSD6bUhL5hWZlpEREQKjYJpSV0klKUxpv1ykRQvQFwX9mq1yxVMi4iISIFRMC0pc6JByMYY02mWeazzyzxa6QJEERERKTAKpiVlTpYy024oDI6DU1yc0vqxzHTrEtVMi4iISGFRMC2pi4aaPsY0QDi9qcSrw7ELEJWZFhERkcKiYFpS5kSyVOYRSi+YjmWmNc60iIiIFBoF05IyJ5qtMo8QThqBcTAaopgiSotTnzFRREREJB8UTEtqomFw3eyM5hEOp13mUaqRPERERKQAKZiW1ERisx82vczDDYVSnv0QIBgJUZqNIF5EREQkyxRMS2oiXt1yVi5ADIUgzTKPEmWmRUREpAApmJbUhGsAsljmkXpmuiYSopXqpUVERKQApRShGGN6A8ustYuNMW2B84AocIO1dm0uGygFIuplppta5uFGoxCJpDyVOHhlHu1L2zRpvyIiIiK5kGpm+nFgc//3G4HBwCDg3hy0SQqRXzPd5Mx0bPbDdMo8ImHVTIuIiEhBSvW7827W2lnGGAc4AugNVAPf56xlUljCsQsQmxjUBv3tpHMBYjSoqcRFRESkIKWamV5njGkH7A78ZK1dCtQAZTlrmRQUJ5qdzLTrZ6ZTLfOoCYeJ4lKWjcliRERERLIs1fTgY8CrQDvgTn/Zrigz3XJEQuA4UNTECwFDfu11isH0Oj8j3kqjeYiIiEgBSikzba09C7gYONVaGwumo8BZuWqYFJhIELcoS1OJAwRSC8qr/VptTSUuIiIihSjlNKO19qWk+x9mvzlSsMLBrIwxnW6ZR7UffCuYFhERkUJUbzBtjHkLcBvbgLV2cFZbJIUpGoRsjKiRZplHtV/mUR5QzbSIiIgUnoYy0/fnrRVS+MJB3CxNJQ4OToplHuvCfmZaNdMiIiJSgOqNaKy1D+ezIVLYnEiw2WY/BGhT0qrp+xYRERHJslRnQHSAE4ExQIW1tp8xZjCwjbXW5rKBUiCiIShu2/TthEJpTdgSK/MoU5mHiIiIFKBUx5m+EjgBuA/4jb9sHnBBLholBShLmWk3FEprKvGaSIgiHFqlWBYiIiIikk+pBtNjgUOstU+w/qLE74Htc9EoKTBuFKKRrNRME0q/zKOkqWNbi4iIiORIqsF0MbDa/z0WTLdNWCabMr9uOSujeYTTK/OoiYQo1eyHIiIiUqBSDaafB242xrSCeA31VcBzuWqYFI74VOJZGmfaSSszHaRUmWkREREpUKkG02cD2wK/Au3xMtLboZrpFsGJZaazkSEOhlIeYxqUmRYREZHCllLKz1q7EjjcGNMR7wLEudbaRTltmRQOf0rvpl6A6EajEAmndQFiMBKiTUlZk/YrIiIikisNzYBYV9Z6iX+LP26tjeamaYXJCa6iZOlMnDXtKf311+ZuToOWLlvI94u+a/J2nGgNZU6Y1W2XQ6A8vjziuixeGSTlF4DrUrxmGaGfw4TdZSk9ZU24mm2Kt0q/0SIiIiJ50FBmOkwK04njXZzYYjiRIMVrf8YprqZ47armbk6DvvvpExasXkpZFsokSlqVU+0sA2f9Z6y1NREWrw4SKHJS31CxQ7VTQ2TN0pRWb1VcSrfNOqTbXBEREZG8aCiY7p7w+x+BkcC1wI+sr5d+OndNK0zR8q2o7nEIbSoqqF6aWkDYXKoX/UKbNp3Zd8iJTd5WRUUFS5P6+9Mv6/h4/mr2r9yCNq1a1GcqEREREaDh6cR/jP1ujDkb2M1au8Jf9I0x5kPgQ+DunLZQMhaK1FASyN003KGo98VFoDiNzLSIiIjIJiTV0TzaA62TlrX2l0uBCkVClORwGu5wxAumSxRMi4iISAuV6gC+DwOvGGNuBeYCXYEz/OVSoMKRICXFOcxMR1wCjkORo2BaREREWqZUg+nzgTnAKKATsBC4E/h7jtolWRCOhAiU5C4zHYpElZUWERGRFi3VcaajwD3+TTYC4XCQSDRMaSB3YzSHoq6CaREREWnRUp6n2RhzHHAs0BmYDzxirX0wVw2TpgkH1wEQyGUwHXEpKU617F5ERERk05NSJGSMuRi4EHgCr1b6CeB8f7kUoHCwGoCSVjkOptMZY1pERERkE5NqZvpEYEjScHkvAm8CV+eiYdI0wVANQM6HxmurMg8RERFpwVL9jr4N/jTiCZYB5XWsKwUgFFwLQKA0d6corAsQRUREpIVLNTP9f8CjxpgLgZ/wZkC8GngxVw2TpgmHvJrpkpLcZaaDEV2AKCIiIi1bqpnp04FVwGfAauATYA0wITfNkqYK+WUepaXJc+1kRzjq4oJqpkVERKRFS3VovJXAn4wxY4EKYKk/XJ4UqFhmOldlHqGId/o1moeIiIi0ZOkMjdca6AG0BXoYYwCw1r6dm6ZJUwRD63BwKCnNzWgeIU0lLiIiIpJaMG2M+RPejIdBoDrhIRf4TQ7aJU0UjgQJFJfkbPvxYFplHiIiItKCpZqZ/hswwlr7cqY7MsYcCNwGFAP3W2uvq2MdA1yOF6R/aq09KtP9tXSh0DpKinM4lXjUC6YDykyLiIhIC5ZqwWsQeD3TnRhjioFJwEFAb2CMMaZ30jqVwEXA3tbaPsDETPcnEAoHKSnO3UgeYZV5iIiIiKQcTF8K3GyMqchwP7sDc6y131lrg3gzKA5PWuckYJK19hcAa+3PGe5LgHA4SCCQyzIP7wLEUl2AKCIiIi1YqmUe3wBXAuNjFx4CDuBaa4tTeH5nYG7C/XnAHknr9AQwxkzHKwW53Fr7fym2T5KEojW0Lmmfs+0H/cx0QDXTIiIi0oKlGkw/AvwTeJLaFyBmuy2VwBCgC/CmMWYna+2KxJWMMeOAcQDWWioqMk2WN00gEGi2faeiuKSILTbfKmttTO7v/OpfaV9dTMcOW2dl+4Wo0M9xLqjPLYP63DK0tD63tP6C+lwoUg2mtwIus9a6Ge5nPtA14X4Xf1miecB71toQ8L0x5hu84PqDxJWstfcB9/l33aVLl2bYpKapqKigufaditWrV1HTJpS1Nib3d8ny1axbGyzoY9BUhX6Oc0F9bhnU55ahpfW5pfUX1Od869SpU53LUw2mHwSOxctOZ+IDoNIY0x0viB4NJI/U8SwwBnjQr83uCXyX4f5atEgkTDgaoiSQuwsQQ5Eopbr4UERERFq4VIPp3YHTjTEXA4sTH7DWDm7sydbasDHmdOBFvHrof1hrvzTGXAl8aK2d5j/2B2PMV0AEOM9auyyNvogv7E8lXlKSmwlbwBsaT8PiiYiISEuXajD9d/+WMWvt88DzScsuS/jdBc72b9IEoZq1AARKcpmZdjUsnoiIiLR4KQXT1tqHc90QyZ5QaB0ApTnMTIcjLm1KUxnIRURERGTTpUGCN0FhP5gO5DCYDkZcDYsnIiIiLZ6C6U1QLJguKSnP3T6iKvMQERERUTC9CQoG/WC6NDfBdCTqEnEVTIuIiIjUG0wbY95N+P2v+WmOZEOsZrqkVW6C6XDUG268RGUeIiIi0sI1lJnuaYyJFd2ek4/GSHaEw97QeLmqmQ5GogCUFOuLDREREWnZGhrNYyrwjTHmB6DcGPNmXSulMs605FcwtI5AcQlFRbkJdkMRLzOtcaZFRESkpas3mLbWHmeM+S3QDRgIPJCvRknThMM1BIpLc7d9P5jWDIgiIiLS0jU4zrS19n/A/4wxpRpreuMRCgcpKS7J3fb9mmkNjSciIiItXaqTtvzDGDME+BPQGZgPPGKtfS2HbZMMhcLrKCnK7eyHgEbzEBERkRYvpaJaY8yJgAUWAc8AC4HHjTEn5bBtkqFwJJTjqcR1AaKIiIgIpJiZBs4H9rfWfhpbYIx5Enga+HsuGiaZC0WCtMthzXQo4lKEyjxEREREUk0tbgV8lbRsFrBldpsj2RCK1FCSw6nEQ5r9UERERARIPZj+H3CzMaY1gDGmDXAD8HauGiaZiUajhCMhSgK5rZkOqMRDREREJOVg+hSgP/CrMWYxsMK/f3KO2iUZCsdmPwzkdmg8zX4oIiIikvpoHguBwcaYLkAnYIG1dl5OWyYZiQXTuZr9ELwZEFXmISIiIpL6BYgA+AG0gugCFgpWA1BaWp6zfYSjLmUlKvMQERERUUS0iYkF04Ec10xrWDwRERERBdObnFCwBoBADjPTIdVMi4iIiAAKpjc5obBXM52rMo+o6xJ2NTSeiIiICKRRM22MaQ9UAW0Tl1trX812oyRzsTKPkla5CaZjU4kHFEyLiIiIpBZMG2PGApOA1cDahIdcYPvsN0syFQ75ZR45Gs0j7AfTpQqmRURERFLOTF8NjLTWvpDLxkjThSI1FBcFKC5Oa6CW1Lcf9TPTRaoQEhEREUk1IgoAL+WyIZId4VCQQHFJzrYfikQBVDMtIiIiQurB9PXAJcYYpSMLXDC8jtLi3A6LByrzEBEREYHUyzzOArYBzjfGLEt8wFr7m6y3SjIWjgQJFOduKvH1ZR4KpkVERERSDaaPyWkrJGtCoRpKcziVeCwzrTIPERERkRSDaWvtG7luiGRHKFJD29ab53D7ykyLiIiIxKQ6NF4JcAlwLNAJWAA8AlxtrQ3mrnmSrnA0lNOpxMPRKCVFDo6jYFpEREQk1TKPvwG7A6cAPwLbAZcCm+HVU0uBCEWCOQ2mQxHNfigiIiISk2owfSTQ31obu/hwljHmY+BTFEwXjHAoiOu6lARydwFiMOJSUqxBXUREREQg9aHx6ktFKkVZQEI1awAoKcnNVOLgzYBYonppERERESD1zPQU4DljzBXAT3hlHpcANlcNk/SFQusAKMnhaB7BSJQ2pcU5276IiIjIxiTVYPp8vOB5EusvQHwc+H85apdkIBSsBqCkNHfBdDiqmmkRERGRmFSHxgsCl/k3KVDxYDrH40wrmBYRERHx1BtMG2MGW2vf9H/fr771rLWv5qJhkr5QuAaAktLc1Ey7rkso6hIo0gWIIiIiItBwZvouoK//+wP1rOMC22e1RZKxcNCrmQ7kKDMd9qcSL1VmWkRERARoIJi21vZN+L17fpojTRHPTLfKTWY6GJv9UMG0iIiICJDi0HjGmKn1LH8mu82RpgiG1lHkFBHI0TjT4Ygy0yIiIiKJUi1+/V09y4dkqR2SBeFwDYHi3E3YEvLLPFQzLSIiIuJpcDQPY8yV/q+lCb/HbI83tbgUiFC4hpJcBtORKIBG8xARERHxNTY0Xlf/Z1HC7+BdeDgXuDwHbZIMhcLBnE4lHvLLPDQDooiIiIinwWDaWnscgDHmbWvt3/PTJMlUOFJDoLhVzrYfK/NQZlpERETEk2rxa40xpl/iAmNMf2PMsTlok2QoFAnlNDMd1mgeIiIiIrWkGkxfhVfWkWgumk68oHg107nLTAcjLoEihyJHwbSIiIgIpB5MbwasTFr2K7B5VlsjTRKOBAmU5C6YDkeiqpcWERERSZBqMP0VMCJp2eHAzOw2RzIVDgeJulFKA7mtmVa9tIiIiMh6jY3mEXMB8LwxZhTwLdAD+D1wcK4aJumJTyUeyM1U4uCN5lFSrDGmRURERGJSioystf8DdgI+ANoA7wN9rbXTc9g2SUMoWA1ASascB9Mq8xARERGJSzUzjbX2R+C6HLZFmiAeTOe4zKOtyjxERERE4lIOpo0xhwL7AhVAPKKy1v4pB+2SNIVCfplHaXnu9hGJUqpgWkRERCQupWDaGPNX4BTgCeBI4F7gKODJVHdkjDkQuA0oBu631taZ5TbGjACeAgZaaz9MdfstXdgPpktzGky7GmNaREREJEGqV5MdD+xvrT0LCPo/hwHdUnmyMaYYmAQcBPQGxhhjetexXjvgTOC9FNslvqB/AWJJSW5qpkORKC6aSlxEREQkUarB9ObW2i/834PGmBJr7ft4ZR+p2B2YY639zlobxMtwD69jvauA64F1KW5XfOFwDZC7Mo9QJDaVuEbzEBEREYlJtWb6W2NMH2vtl8AXwKnGmF+AX1J8fmdqz6A4D9gjcQVjzK5AV2vtf4wx59W3IWPMOGAcgLWWioqKFJuQXYFAoNn2XZe5ZSWUlZWxbacuOdn+6mCUtm3b0qFiSyq2bJ2TfRSaQjvH+aA+twzqc8vQ0vrc0voL6nOhSDWYvgTYyv/9QuAxoC0wPhuNMMYUATcDYxtb11p7H3Cff9ddunRpNpqQtoqKCppr33VZ/stSomE3Z21yW7Vj9erVrP61iKXRtTnZR6EptHOcD+pzy6A+twwtrc8trb+gPudbp06d6lzeaDDtB7rrgHcB/PKOHmnufz7QNeF+F39ZTDugL/C6MQZgG2CaMeZQXYSYmnC4hkBRSc62H4pEAXQBooiIiEiCRoNpa23UGDPVWtuuCfv5AKg0xnTHC6JH440GEtvHr3hD7gFgjHkdOFeBdOpC4SAlxTkcY9qvmS5VzbSIiIhIXKqR0ZvGmEGZ7sRaGwZOB14EZnqL7JfGmCv98aulicLhIIFA7jLTQT8zXaLMtIiIiEhcqjXTPwIvGGOm4l1I6MYesNZelsoGrLXPA88nLavzudbaISm2S3yhSA1tSjfP4fb9Mg8NjSciIiISl2owXQ486/+eOFyEu+Gq0hxCkSAlgdLcbT/sUuw4FCuYFhEREYmrN5g2xpxurb3Tv3u1tXZOntokGQhFggQCuauZDkaiKvEQERERSdJQzfTVCb9/nOuGSOYikTCRaJjSQG5mPwQ/mFZWWkRERKSWhso8vjPG3AR8CZQYY46vayVr7T9y0jJJWTjkTRgZKMnlaB5RDYsnIiIikqShYHoUcD4wBigBjq1jHRdQMN3MQjXVQK6DaVdlHiIiIiJJ6g2mrbXfACcCGGP+a639fd5aJWkJ+Znp0pIclnmEoxpjWkRERCRJStGRAunCFqrxpvcuKS3P3T4iUQ2LJyIiIpJEqcZNQDhcA0AgxxcglqrMQ0RERKQWBdObgFDIC6ZLy1rnZPuRqEvURRcgioiIiCRRML0JCAW9Mo9AjmqmQxFvbh4NjSciIiJSm4LpTUAoHARyGExHvanES3QBooiIiEgtio42AaFwDYHiEoqKcnM645lplXmIiIiI1KJgehMQDtcQKC7N3fYVTIuIiIjUqaFJW6QOK4Mr+WLFF7Rb145VK1fFl7vRCJE530IkXGt914VFK4OEo27O2lQTWkUg0Jo3vl2Rk+0HIy5OabmGxhMRERFJomA6XQ6UFJVQWlRKSVFJfHG0OoizcjVOeRsoWX9Yw1FYE47QKlCUswv4SgKtaLf59jkbuq602GHrLVvTtlUkJ9sXERER2VgpmE7TZiWbMbBiIBUVFSwtXRpfHmUp4ehqAjsNoqiiIr585bowy+asYLcu7eiyee6m+861iootWbp0aeMrioiIiLQgqpnOlrBf3lFSUnuxX96hCU9ERERENj0KprPEDYUAcAK1k/2xkTA04YmIiIjIpkfBdLbUk5nWsHIiIiIimy4F09niZ6bZIDPtT3iSozGgRURERKT5KMLLEjcUguIATlLQHIoqMy0iIiKyqVIwnS3hcK0h8eKLIy5FQLHGaBYRERHZ5CiYzhI3GMJJqpcGb8KTkmIdZhEREZFNkaK8bAmHILBhMB2OuirxEBEREdlEKZjOllDdZR6hSFTBtIiIiMgmSsF0lrjhEE4dmWmvzEPBtIiIiMimSMF0toRCUFpHmUfEpUQXH4qIiIhskjasS5DMhMIbjDEN3tB4AV2AKCIiBSYSibBu3ToAHGfjT/osXryYmpqa5m5GXqnP2eO63lDGZWVlFBcXp/VcBdNZ4IbDgFvnaB7hiEupyjxERKSARCIRqquradOmzSYRSAMEAoG0g6CNnfqcXa7rsmbNGsrLy9Pah1Km2RCb/TApmI5EXcKuS0BlHiIiUkDWrVu3SQXSItngOA5t2rSJf2OTKgXTWeD6wbSTVOYR1uyHIiJSoBRIi2wok/eFgulsCIe9n0mZ6VDEC6ZLVTMtIiIFRIG0SP3SfX8oysuGeso8gpEoAAFlpkVEREQ2SQqms6DeMg8/M62h8URERArPTTfdxIQJE5q7GQVjjz324M0338zJtt977z322Wef+P05c+aw//7707NnTx544AEuuOACbrnllpzsO9c0mkc2hOop81DNtIiISNr22GMPli5dSlFREW3atGHIkCFcffXVtGnTprmblpK5c+cyaNAgWrduHV+23Xbb8corr+StDZ07d+Z///sf3bt3jy9btWoVN9xwAy+88AIrVqxg6623ZujQoUycOJEtt9wyp+3ZY489eOutt+L37777bvbaay9efvnlnO43H5SZzoawX+aRlJmO1UwrmBYREUnPgw8+yOzZs3nppZf44osvuOOOO5q7SWmbOXMms2fPZvbs2RkF0uHYNVlZEAwGGTVqFN988w2PPvoos2bNYtq0aWyxxRbMmDEja/tJ1bx586iqqmrydrJ5jDKlYDoL3FAIiotxksYkDPk10yVFOswiIiKZ6NChA0OGDOHLL78E4M4772SvvfaiZ8+eDBkyhBdeeCG+7pNPPslhhx3GlVdeSe/evRk0aBCvvvpq/PGffvqJESNG0LNnT0aPHs3y5ctr7eull17id7/7Hb169WLkyJHMnj07/tgee+zB3XffzdChQ+nRowfnnHMOS5Ys4ZhjjqFnz56MGjWKFStWNNqfRYsWMXbsWPr06cPee+/No48+Gn/spptu4qSTTmLChAlUVVVhrWXlypWcc8457LLLLgwYMIDrr7+eSCQCwPfff8+IESPYcccd6du3L6eccgoARxxxBAD7778/lZWVTJ06laeeeor58+fzwAMP0LNnT4qKiqioqOCss87i97///QbtnDFjBsOGDaNXr17ssssuXHzxxQSDQcAbj/mvf/0r/fr1o6qqit///vd8/fXXAPz3v/9lyJAh9OzZkwEDBnDPPfcA8PbbbzNgwAAAjjzySN5++20uueQSKisr+fbbb5k4cSLXX399fP8vv/wy+++/P7169eLQQw/lq6++qnUuJk2axNChQ6msrGz2gFplHtnQwOyHDroAUURECtutb85j9pLqnO6jcutyJg7ukvbzFixYwGuvvcbee+8NeOUSzzzzDB06dOC5555jwoQJTJ8+nc6dOwNeEHjkkUfy+eefM3nyZM4991w++ugjHMfhtNNOY8CAATz22GPMmDGDP/3pTxxwwAEAfPvtt4wfP55//OMf7Lnnnvz9739n7NixvPbaa5SWlgLwn//8h8cff5xwOMwBBxzAF198wU033USPHj049thj+cc//sHZZ5/dYH/Gjx9PVVUVH330EXPmzGHMmDFst912/Pa3vwW8gP7ee+/ltttuo6amhtNPP52tttqK6dOns3btWv785z/TqVMnjj32WK677joGDx7MlClTCAaDfPbZZwA888wzdO7cmZdffjle5nHqqacyZMiQlEtliouLufzyy+nfvz8LFy7kmGOO4eGHH+akk07ijTfe4L333uOtt95is802Y86cOWy22WYAnHvuudxzzz3ssccerFixgrlz526w7SlTpjBy5EiOOOIIjjrqqA0e/+KLLzjnnHN46KGH6N+/P08//TTHHXccb775JgE/3nr22Wd5+OGH2XLLLePLmotSptkQDm1QLw3eBYgq8RAREUnfCSecQM+ePRk4cCBbbbUV55xzDgDDhg1jm222oaioiOHDh9O9e3c++eST+PO6dOnC0UcfTXFxMcYYFi9ezJIlS5g/fz6ffvop559/Pq1atWLQoEHsv//+8edNmzaN3//+9wwePJiSkhJOOeUU1q1bx4cffhhf5/jjj2frrbdm2223ZY899mCXXXahb9++lJWVcdBBB/HFF1/U6sNOO+1Er1696NWrF/fccw/z58/ngw8+4OKLL6asrIy+ffty1FFH8dRTT8WfM2DAAA488ECKiopYvXo1r776KldccQWtW7emoqKCk046ialTpwJQUlLC/PnzWbRoEWVlZey+++71Hs9ffvmFjh07pnz8+/Xrx4ABAwgEAnTt2pVjjjmGd999F/BmIVy9ejVz5szBdV0qKyvj2w4EAnzzzTesWrWKzTffnJ122inlfcZMnjyZY445hl133TV+HktLS/n444/j65xwwgl07tyZ8vLytLefbcpMZ4EbCm8wkgdAMOJSojGmRUSkwGWSMc61Bx54gMGDB/POO+9w+umns3z5ctq3b8+UKVO47777mDdvHgBr1qypVa6x9dZbx3+PBVqxddq3b1/rosAuXbqwYMECABYvXkyXLuuPQ1FREdtuuy2LFi2KL6uoqIj/XlZWVmtfZWVlrFmzplYfPv/881pZ048//pjNN9+ctm3bxpd17tyZTz/9NH6/U6dO8d/nzZtHKBRi1113jS+LRqPxdS677DKuvfZaDjnkENq3b8/JJ5/M6NGj6zyeW2yxBYsXL67zsbp8++23XHHFFXz22WdUV1cTDofp168fAL/97W857rjjuPjii5k3bx4HH3wwl156Ke3atePvf/87t912G9deey29evXioosuYrfddkt5vwDz589nypQpPPjgg/FlwWCwVvsTj1NzU6SXDaEGMtMaFk9ERCRje+65J0ceeSRXXXUV8+bN4/zzz+fqq6/miy++YObMmVRVVeG6bqPb6dixI7/++itr166NL5s/f36tx2MBOnh1wQsXLmSbbbbJWl86duzIihUrWL16da02JO4jccKQTp060apVKz7//HNmzpzJzJkzmTVrFq+99hrg1ZPfcMMNfPzxx1x//fX85S9/4fvvv69z3/vssw9vvPFGrf435KKLLqJHjx7873//Y9asWVx44YW1jvMJJ5zA//3f//H666/z3XffcffddwOw88478+CDD/Lpp59ywAEHxOu407HttttyxhlnxPs8c+ZMvv32Ww477LC0t5UPCqazwA2HcOoIpkNRlXmIiIg01UknncSbb77Jr7/+iuM48WHcnnzySWbNmpXSNrp06UK/fv248cYbCQaDvP/++7WGZRs2bBj//e9/eeuttwiFQtx7772UlpamnVVtSOfOndltt9249tprWbduHV999RVPPPFE/ILBZB07dmTw4MFceeWVrFq1img0yg8//MA777wDeKUpscx6+/btcRyHIn/Qg6233pqffvopvq0RI0bQqVMnTjrpJObMmUM0GmX58uXcfvvt/Pe//91g32vWrKFdu3a0adOGOXPm8M9//jP+2CeffMLHH39MKBSidevWtGrViqKiIoLBIM888wwrV66kpKSEdu3axduTjqOPPppHHnmEjz/+GNd1Wbt2La+88kqtDyGFRMF0NoTCdWamQ5EoAWWmRUREmmSrrbZi5MiR3HrrrYwbN47hw4fTv39/Zs6cycCBA1PezqRJk5gxYwZ9+vTh5ptvZuTIkfHHevTowR133MGll17KTjvtxMsvv8xDDz0Uv/gwWyZNmsTcuXMZMGAAJ554Iueccw6DBw+ud/3bbruNYDDIkCFD6N27N+PGjePnn38GvKB22LBhVFZWctxxx3HFFVew3XbbAXD22WczceJEevXqxbRp02jVqhVPPPEEO+ywA6NHj6aqqopDDjmE5cuXs8suu2yw30svvZR//etf9OzZk/POO49DDz00/tiqVas4//zz6d27N7vvvjtbbLEFp556KgBPP/00gwYNoqqqikceeYQ777wz7WPUv39/brjhBi655BJ69+7N3nvvjbU27e3ki5PKVyMFzI19Isu3iooKli5dCkDwP/+haPsdCPTasdY6L369nA5tS9ilS7vmaGJWJfa3pVCfWwb1uWVQn2tbu3ZtrdrhTUEgEGj2IdLyTX3OjfreH36d9gZZUmWmm8iNRMB1cUrqHhovoAsQRURERDZZivSaKlT37IdR1yUcdSlVzbSIiIjIJkvBdFPFvmooqV1TFfanEteELSIiIiKbLgXTTeT6menkMo+QH0xraDwRERGRTZeC6aaqp8wjFPWDaWWmRURERDZZCqabyA15ZR5O0tA58cy0LkAUERER2WTlbTpxY8yBwG1AMXC/tfa6pMfPBk4EwsAS4Hhr7Y/5al/GQkHvZ1JmOhyNAirzEBEREdmU5SVtaowpBiYBBwG9gTHGmN5Jq80AdrPW9gOeAv6Wj7Y1WfwCxNqTtgR1AaKIiIjIJi9fNQi7A3Ostd9Za4PAE8DwxBWsta9Za2MTxr8LdMlT25rEDYXBcXCKi2stj43moaHxRERECsMxxxyT0kx6lZWV/Phj4X85noq3336bAQMG5Gz7F1xwAbfcckv8/sMPP0z//v2prKxk+fLlm9SxrE++yjw6A3MT7s8D9mhg/ROAF+p6wBgzDhgHYK2loqIiW21MSyAQoKKiguo2rQlvsSXtktqxsOZX2q522KbD1jjOxh9Qx/rbkqjPLYP63DKoz7UtXryYQCBvlZ5p22233ViyZAnFxcUUFxfTs2dPjDEce+yxFBXVnwdsrE9PPPFESvv//vvv02pvXQYPHszcuV7os27dOkpKSij2E29nnnkmEydObPI+wOvzxx9/zI033sgHH3xAUVER3bp1Y+zYsYwZM4bi4mIcx8nZ+b7pppviv4dCIa688kqef/55+vTpA2TnWCbL9Wu3VatWaf29KLh3kjHmGGA3YN+6HrfW3gfc5991m2t62Ng0reGlS4nWrKMmqR1Ll6+mZm0Ny5Yta5b2ZZum4m0Z1OeWQX1uGRrqc01NTTywK0Su6/Lggw8yePBgVq5cybvvvstll13Ghx9+WCsLmqjQptZ+9dVX47+PHDmSI444gqOOOiq+LNbWcDiccXAYCAR49913GTNmDBMnTuTWW29liy224PPPP2fSpEkceeSRRCIRXNfNy7FZuHAh69atY4cddmjy/uo7Lvk4zzU1NXW+d/zpxDeQrzKP+UDXhPtd/GW1GGOGAhcDh1pra/LUtiZxQyGcQMkGy0MRV8PiiYiINNFmm23GH/7wB+6++26mTJnC119/TU1NDVdeeSUDBw6kf//+XHDBBVRXV8ef8+KLL7L//vtTVVXFXnvtxWuvvQZ4Qe1jjz0GeBnTESNGsOOOO9K3b19OOeWU+PM7d+4cz6iuXLmSM844g5122ondd9+dW2+9lag/yMCTTz7JYYcdxpVXXknv3r0ZNGhQrSC6LnPnzqVz5848/vjjDBw4EGMM4GXN9913X3r37s1RRx3FvHnz4s+ZM2cOo0ePpk+fPuyzzz5MmzYt/tj/+3//jyOPPJLTTjuNLbfcEsdx6NevH/fee2+d+7/zzjvZa6+96NmzJ0OGDOGFF9YXAtR3TFzX5a9//Sv9+vWjqqqK3//+93z99dcATJw4keuvv55vv/2WwYMHA9CrVy+OPPLIDY5lQ+ctVo4yadIkdt55Z84666wGj2MhyVdm+gOg0hjTHS+IHg0clbiCMWYX4F7gQGvtz3lqV9OFw1Cy4WEMRVwCGslDREQ2Apu9fQ0ly77O6T5CW+3Iyr3+kvHzd9llF7bddlvee+89Hn/8cX788UdeeuklSkpKOO2007jpppu48MILmTFjBmeeeSb33Xcfv/3tb1m8eDFr1qzZYHs33HADgwcPZsqUKQSDQT777LM693vJJZewatUq3nnnHX755RfGjBlDx44dGTNmDAAzZszgyCOP5PPPP2fy5Mmce+65fPTRR42WeL7zzju88cYbOI7Diy++yB133MFDDz1E9+7dufPOOxk/fjzTpk1j7dq1jB49mvPOO4/Jkyfz9ddfM3r0aHbccUe6devGRx99xHnnnZfycdxuu+145pln6NChA8899xwTJkxg+vTpdOzYsd5j8sYbb/Dee+/x1ltvsdlmmzFnzhw222yzWtvdYYcdeO211xg0aBAzZ86sM6t8zTXXbHDebr31Vi666CIAlixZwooVK3jvvffiH1g2BnnJTFtrw8DpwIvATG+R/dIYc6Ux5lB/tRuAtsAUY8wnxphp9WyuoLihEE5J3ZnpUo0xLSIikjUdO3ZkxYoVPProo1x++eVsscUWtG3blgkTJvDss88C8PjjjzNq1CgGDx5MUVER2267LT169NhgW4FAgPnz57No0SLKysrYfffdN1gnEokwbdo0LrroItq2bUvXrl05+eSTefrpp+PrdOnShaOPPpri4mKMMSxevJglS5Y02pdzzjmH1q1bU15eziOPPMLpp59OZWUlgUCAM844gy+//JJ58+bx8ssv07VrV0aNGkUgEKBv374cfPDB/Pvf/+bXX38lGo3SsWPHlI/hsGHD2GabbSgqKmL48OF0796dTz75pMFjEggEWL16NXPmzMF1XSorK9PaJ3jZ7brO29SpU+PrFBUVcc4559CqVSvKy8vT2n5zylvNtLX2eeD5pGWXJfw+NF9tyapQCOoq84i6lJcomBYRkcLXlIxxPi1atIhIJEJ1dTUHHXRQfLnrukQiEcCr291vv/0a3dYll1zCDTfcwCGHHEL79u05+eSTGT16dK11li9fTigUokuX9QOMdenShYULF8bvb7311vHfYwFgXZnwZIn1t/PmzeOyyy7jyiuvrLXOwoULmT9/PjNmzKBXr17x5eFwmBEjRtC+fXuKiopYvHhxnR8Y6jJlyhTuu+++eBnJmjVrWL58OVD/Mfntb3/Lcccdx8UXX8y8efM4+OCDufTSS2nXrl1K+wRYtmxZg+cNYKuttqKsrCzlbRaKgrsAcaNTb5lHlM1aFe7FHSIiIhuTTz75hEWLFnHAAQcwadIkXn31Vbbddtv447EL07bddlt++OGHRrfXoUMHbrjhBgDef/99Ro8ezR577EH37t3j62y55ZaUlJQwb948evbsCcD8+fNr7TdTiWUgnTp14owzzuCII47YYL358+czaNCgOkciCQQCDBgwgOeff56999670X3OmzeP888/nyeffJIBAwZQXFzM/vvvj+t6w/k2dExOOOEETjjhBJYuXcopp5zC3Xffzfnnn59yf7fcckvKyso2OG+bAqVOm8CNRiESwamjLigUcSkNqGZaRESkKVatWsXLL7/M+PHjOeKII+jTpw9HH300l19+eXzEhYULF8YvMhwzZgzWWt566y2i0SgLFy5kzpw5G2z3ueeeY8GCBQC0b98ex3E2GHavuLiYYcOGcf3117N69WrmzZvHfffdV2fQ2xTHHnssd955J7NmzQK8ix6fe+45AIYOHcp3333HU089RSgUIhQK8cknnzB79mwALr74Yqy13H333fEM85dffsmpp566wX7Wrl2L4zhsueWWgHcBZWyfDR2TTz75hI8//phQKETr1q1p1apVg0MU1qWoqKjO8/b666+ntZ1CpGC6KeKzH5bWWuy6LqGoSyDNF5qIiIh4jjvuOHr27MnAgQO5/fbbGTduXHxYvL/85S9069aNYcOGUVVVxejRo+MB8y677MLNN9/MFVdcwY477sjIkSNrjYwR8+mnnzJs2DAqKys57rjjuOKKK9huu+02WO+qq66idevW7Lnnnhx22GEcfvjhG5SDNNVBBx3E+PHjGT9+PFVVVey3337xDwdt27blscceY+rUqey6667svPPOXH311dTUeIOeDRw4EGst06dPZ6+99qJPnz5ccMEFdZa69OzZk3HjxjF8+HD69+/PzJkzGThwYKPHZNWqVZx//vn07t2b3XffnS222KLOYL0xdZ23b7/9NsOjVjicWGp/I+XGPkHlW0VFBUt++IHQ669TvPPOFCfUU4UiUf4zczl9t2lDj4qNp4C+IRqjtWVQn1sG9bllaKjPa9eupXXr1nluUW4V2jjT+aA+50Z97w+/zn2DsgOlTpvA9U9m8mgeIX8qcQ2NJyIiIrJpUzDdFKGQ9zMpmA5HvWC6VJO2iIiIiGzSFEw3RSyYTroAMehnpjUDooiIiMimTcF0EzRW5lGiSVtERERENmmK9pqi3jIPbwrMEtVMi4iIiGzSFEw3gRsKAc4G40zHyjwCKvMQERER2aQpmG6KemY/DKtmWkRERKRFUDDdFKEQBEo2XBxxCTgORY6CaREREZFNmYLpJnBDoQ0uPgQIRV1lpUVERDYSTz75JIcddlj8fmVlJT/++GPzNSgFN910E+PHj8/pPiZOnMj1119f7+O333475557bk7bsDFQMN0UobrLPEKRqIJpERGRDO2xxx7ssMMOVFZW0rt3b4499ljmz5+ft/3Pnj27zqnF0zVx4kS6detGZWUlffr0qTXteS7ddNNNdO7cmfvvv7/W8vvvv5/OnTtz0003pb3Nt99+mwEDBtRadsYZZ3DjjTcCMHfuXDp37kxlZWX8NnTo0Mw7kbDNQp/lUcF0U4QbKPPQSB4iIiIZe/DBB5k9ezYff/wxW2+9NZdeemlzNykjp556KrNnz+bDDz9km2224ZxzzsnLfrfffnueeuqpWsumTJnC9ttvn9P9zpw5k9mzZzN79mxeeeWVnO6rMa7rEvVHWMslBdNN4IbDOHVmpl1KNca0iIhIk5WVlfHHP/6Rb775BoBXXnmFP/zhD1RVVbHbbrvVyrKuW7eOCRMm0KdPH3r16sXBBx/MkiVLAFi5ciXnnHMOu+yyCwMGDOD6668nEonUuc/OnTvz/fffA152+S9/+QvHHnssPXv25JBDDuGHH36IrztnzhxGjx5Nnz592GeffZg2bVqd2ywvL2fYsGF8+eWX8WUN9SWWlbXWMnDgQPr27cttt91W57ZDoRDjx4/npJNOIhgMArDzzjtTXV3NrFmzAJg1axY1NTXsvPPO8ecll7ck9z1m7dq1HHvssSxevDiedV60aBE33XQTEyZMqLNNiRo6Rg0dgyOOOAKAXr16UVlZyYcffsgNN9xQa5/J2euRI0dy3XXXMXz4cHr06MGPP/6Y8jnK1IaRoKQuGNpgjGnwaqbbqsxDREQ2EnfPupvvVn2X031s3257Tq06Ne3nVVdXM23aNHbddVcAWrduzW233UZVVRVff/01Y8aMoU+fPhxyyCFMmTKFlStX8uGHH1JaWsqXX35JWVkZAGeddRZbbbUV06dPZ+3atfz5z3+mU6dOHHvssY22YerUqUyePJmddtopXkd89913s3btWkaPHs15553H5MmT+frrrxk9ejQ77rgjPXv2rLWNtWvX8uyzz9KtW7f4svr6cuCBB8bX+eCDD3jzzTf57rvvOOSQQzj44IOprKysdXzGjRvHVlttxR133EFxcXH8sREjRvDUU09x8cUXM2XKFEaMGBH/UJKO1q1b88gjjzBhwgQ++uijtJ7b2DFq6Bg888wzDBo0iJkzZxLwhyF+6623Gt3n008/zeTJk9lhhx1Yu3Yt++23X0rnKFNKn2bIjUYhEq7zAsSwaqZFRESa5IQTTqBXr17suOOOvPXWW5x6qheI77XXXvTq1YuioiJ69+7N8OHDeeeddwAoKSnhl19+4fvvv6e4uJh+/frRrl07lixZwquvvsoVV1xB69atqaio4KSTTmLq1KkpteWggw5il112IRAIcPjhh8ezyy+//DJdu3Zl1KhRBAIB+vbty8EHH8y///3v+HPvvfdeevXqRc+ePfnggw+4/fbb44811JeYs846i/Lycvr06UPv3r356quv4o+tWrWKY445hm7dunHLLbfUCqTBC6afffZZQqEQU6dOjWd6c2mnnXaiV69e9OrVi3vuuafRY5TKMUiXMYaqqioCgQCvvfZao+eoqZSZzlSsGL6uzHTE1eyHIiKy0cgkY5xrDzzwAIMHDyYSifDiiy8ycuRIXnvtNebNm8c111zDrFmzCIVCBINB/vjHPwJe8LhgwQLGjx/PypUrOeKII7jggguYN28eoVAont0GiEajdOrUKaW2bL311vHfy8vLWbNmDQDz589nxowZ9OrVK/54OBxmxIgR8fsnn3wyF1xwAfPnz+foo4/m22+/pXfv3gB8/PHH9fYlpkOHDnXuG+Cjjz4iHA4zadIknDqG4+3cuTPdunXjuuuuo3v37nTu3Dml/jbF559/Hs8iA9x1110NHqNUjkG6Es9rKueoqRRMZ8iNTSWeNPthOOoSRRO2iIiIZENxcTEHH3wwF1xwAe+//z7XXHMNY8eOZfLkyZSVlXHZZZfxyy+/AF5m+uyzz+bss89m7ty5HHvsseywww7st99+tGrVaoNAr6k6derEoEGDeOKJJxpdt3Pnzlx55ZVMnDiRoUOHUl5ezumnn15vX1IxZMgQdtxxR0aNGsVTTz1VK+iPGTlyJOeccw4333zzBo+1bt2a6urq+P2ff/653n3VFaynorFj1NAxqGufqbQ58XnpnKNMqcwjQ65f4J9c5rF+9kMdWhERkaZyXZcXX3yRX3/9lcrKSlavXs3mm29OWVkZM2bM4Nlnn42vO336dGbOnEkkEqFt27YEAgGKioro2LEjgwcP5sorr2TVqlVEo1F++OGHJpcTDB06lO+++46nnnqKUChEKBTik08+Yfbs2XWuP3jwYDp27Mijjz4K0GBfUjV+/HgOO+wwRo0axfLlyzd4/NBDD+Wxxx5j2LBhGzzWu3dvvvnmG7744gvWrVvX4JB5W2+9NStWrGDlypVpta+xY9TQMdhqq60oKiqqNeZ33759effdd5k/fz4rV67kzjvvbNL+s0ERX4bqy0wHI94QLBoaT0REJHPHHXcclZWVVFVVcf3113PrrbdSVVXFNddcw4033kjPnj255ZZbagWJS5YsYdy4cVRVVTFkyBD23HPP+Nf5t912G8FgkCFDhtC7d2/GjRvXYCY2FW3btuWxxx5j6tSp7Lrrruy8885cffXV1NTU1PucU045hbvvvpuampoG+5KOs846iwMOOIBRo0ZtkNkuLy9n8ODBlJeXb/C8HXbYgYkTJzJ69Gh++9vfsvvuu9e7jx49ejB8+HD23HNPevXqxaJFi1JqW2PHqKFjUF5ezhlnnMFhhx1Gr169+Oijj9h333059NBDGTp0KAcddFCjY1lnco7S5bium7WNNQN3wYIFzbLj9jU1LHn5ZQL77ENR+/bx5cvXhnjzu1/Za7vN6NCutFnalgsVFRUsXbq0uZuRV+pzy6A+twzqc21r166ldevWeW5RbgUCgYKf3CPb1OfcqO/94ddib5AtVWY6Q7HMdHKZR9Av8wioZlpERERkk6dgOkNusJ4LEOM10wqmRURERDZ1CqYzFfaD6aTMdMivmS4p0qEVERER2dQp4suQGwxCILDBsC2hqDLTIiIiIi2FgukMuaF6phKPuBQ7DsUazUNERERkk6dgOkNuKIRTx8DvoYirYfFEREREWggF0xlyQyEIbJiZDkddSlXiISIiItIiKJjOkBsMQemGwXQwEtWweCIiIiIthILpDDVU5qGLD0VERArHk08+yWGHHZb356bqmGOOwVqb9vPeffdd9tlnnxy0qLBkenzyRcF0huq7ADEccSlRzbSIiEjG3n//fQ499FB23HFH+vTpw/Dhw/nkk0+au1kAdO7cme+//z6r25w8eTLGmLT3PWjQIN56661Gn/fkk0/StWvX+PTsQ4cO5eWXX25Sm/Mp1ePTXBRMZ8B1XahvNI+oS0mxDquIiEgmVq1axZ///GeOP/54vvzySz788EPOPvtsSktLm7tpG7UBAwYwe/ZsZs6cyZ///GfGjx/Pr7/+mvX9RCKRrG+z0Cnqy0Q4DLgq8xAREcmy7777DoDDDjuM4uJiysvL2XfffenduzcAjz76KPvuuy89e/ZkyJAhfP755wDcfvvt7LXXXvHlL7zwQr37mDNnDqNHj6ZPnz7ss88+TJs2Lf7Y8uXLGTt2LFVVVfzxj3/kxx9/TKndK1eu5IwzzmCnnXZi991359ZbbyUa9SZyi0QiXHHFFfTt25dBgwbx4IMP0rlzZ8LhMAAjR47kscceA+D7779nxIgR7LjjjvTt25dTTjkFgCOOOAKA/fffn8rKSqZOncr06dMZMGBAvA3z58/nxBNPZKeddqJPnz5cfPHFG7SzqKiIkSNHsnbt2niWu6amhiuvvJKBAwfSv39/LrjgAqqrq+PPueuuu9hll13Yddddeeyxx2plyCdOnMiFF17IscceS48ePZg+fTqLFi3ipJNOYqeddmLQoEE88MAD8W3NmDGDgw46iKqqKvr378/ll18OwLp165gwYQJ9+vShV69eHHzwwSxZsmSD4xONRrn11lvZfffd6devH2eccQYrV64EYO7cuXTu3BlrLQMHDqRv377cdtttKZ2/ptgwGpTG+S/+5Mx0JOoScTU0noiIbFyq75xE5Ntvc7qP4h12oPz00xpdb/vtt6eoqIgzzzyT4cOHs+uuu7L55psD8Nxzz3HzzTfzwAMP0L9/f3744QdK/P/F3bp145lnnqFDhw4899xzTJgwgenTp9OxY8da21+7di2jR4/mvPPOY/LkyXz99deMHj2aHXfckZ49e3LxxRfTqlUrZsyYwU8//cTRRx9N165dG233JZdcwqpVq3jnnXf45ZdfGDNmDB07dmTMmDE8+uijvPbaa7z00ku0bt2ak08+ud7t3HDDDQwePJgpU6YQDAb57LPPAHjmmWfo3LkzL7/8Mt27dwfgvffeiz8vEonw5z//mb333pv33nuPoqKi+HMTRSIRnnzySUpKSujSpQsA11xzDT/++CMvvfQSJSUlnHbaadx6661cdNFFvPbaa9x33308+eST/OY3v+H888/fYJvPPvssjzzyCA8//DA1NTUcfvjhHHDAAUyaNImFCxcyevRodthhB4YMGcJll13GCSecwMiRI1mzZg1ff/01AFOmTGHlypV8+OGHlJaW8uWXX1JWVrbBvp544gmmTJnClClTqKio4Mwzz+Tiiy/mjjvuiK/zwQcf8Oabb/Ldd99xyCGHcPDBB1NZWdnoOcyUMtMZcEPeVOLJmemwP/uhhsYTERHJTLt27Xj22WdxHIfzzjuPfv36MXbsWJYsWcLjjz/Oqaeeys4774zjOHTv3j0eEB566KFss802FBUVMXz4cLp3715nnfXLL79M165dGTVqFIFAgL59+3LwwQfz73//m0gkwvPPP895551H69at2XHHHRk5cmSjbY5EIkybNo2LLrqItm3b0rVrV04++WSefvppwPsQcMIJJ9CpUyc233xzTjut/g8VgUCA+fPns2jRIsrKyth9991TOm4zZsxg8eLFXHrppbRu3XqD53788cf06tWL7bffnquuuorbb7+diooKXNfl0Ucf5fLLL2eLLbagbdu2TJgwgalTp8bbPmrUKKqqqigvL+fss8/eYN9/+MMfGDhwIEVFRcycOZNly5Zx1llnUVpaynbbbcdRRx0V315JSQk//PADy5cvp02bNvHMeklJCb/88gvff/89xcXF9OvXj3bt2m2wr6effpqTTjqJ7bbbjjZt2nDhhRcybdq0eJYf4KyzzqK8vJw+ffrQu3dvvvrqq5SOYaaUmc5EPZnpUCQ2lbg+o4iIyMYjlYxxPlVWVnLrrbcCXknGhAkT+Otf/8qCBQvYbrvt6nyOtZa7776befPmAbBmzRqWL1++wXrz589nxowZ9OrVK74sHA4zYsQIli1bRjgcplOnTvHHunTpUisDXJfly5cTCoXigX3seQsXLgRg8eLFtbaZ+HuySy65hBtuuIFDDjmE9u3bc/LJJzN69OgG9w+wYMECunTpQqCOElSAXXfdlWeffZY1a9ZwzjnnxC/yXLZsGdXV1Rx00EHxdV3Xjdc+L168mH79+jXY9sRl8+bNY/HixbWObyQSYY899gDgxhtv5MYbb2Tw4MH85je/4ayzzmL//fdnxIgRLFiwgPHjx7Ny5UqOOOIILrjggvg3DzGLFy/e4DiHw+F4SQhAhw4d4r+Xl5ezZs2ahg9eEymYzkTQy0wnB9PBiFcbpXGmRUREsqNHjx4YY5g8eTKdOnWqs4Z53rx5nHPOOTz55JMMGDCA4uJi9t9/f2/AgCSdOnVi0KBBPPHEExs8FolECAQCLFiwgB49egBe8N2YLbfckpKSEubNm0fPnj3jz9t2220BL7iLBdbgBb716dChAzfccAPgjWoyevRo9thjj3hpR306derE/PnzCYfD9QbUAG3atOHaa69lr732YvTo0fTu3ZuysjJeffXVeHuT29NY2x1nfdzTqVMnunbtyvTp0+vc//bbb89dd91FNBrl+eef5+STT+aLL76gdevWnH322Zx99tnMnTuXY489lh122IExY8bUen7Hjh3jH5jAO86BQICtt966VjvzSSnUDLhhv8wjKZgOxzLTqpkWERHJyJw5c7jnnnviQdv8+fN59tln2XXXXRkzZgz33HMPn332Ga7r8v333zNv3jzWrl2L4zhsueWWgDcU3KxZs+rc/tChQ/nuu+946qmnCIVChEIhPvnkE2bPnk1xcTEHHXQQN910E9XV1XzzzTdMmTJlg22EQiHWrVsXvwEMGzaM66+/ntWrVzNv3jzuu++++EWDw4YN44EHHmDhwoX8+uuv3HXXXfX2/7nnnov3vX379jiOQ1GRF65tvfXW/PTTT3U+b5dddqFDhw5cc801rF27lnXr1vHBBx/Uue4WW2zBmDFjuOWWWygqKuLoo4/m8ssvZ+nSpQAsXLiQ119/Pd52ay2zZ8+muro6/o1BfXbZZRfatm3LpEmTqK6uJhKJ8PXXX8dLbp5++mmWLVtGUVERm222GeAF49OnT2fmzJlEIhHatm1LIBCI9zvR4Ycfzt///nd++ukn1qxZw3XXXcehhx7a4AeIXFMwnYlQ3ZnpUDRW5qFgWkREJBNt2rRhxowZDBs2jB49enDooYdSVVXFZZddxrBhwzjjjDM47bTT6NmzJyeccAIrVqygZ8+enHLKKQwfPpz+/fszc+ZMBg4cWOf227Zty2OPPcbUqVPZdddd2Xnnnbn66qupqakB4Oqrr2bNmjXsvPPOnHXWWYwaNWqDbfzud79jhx12iN+efPJJrrrqKlq3bs2ee+7JYYcdxuGHHx4vzzj66KMZPHgwQ4cO5YADDmC//fYjEAhQXFy8wbY//fRThg0bRmVlJccddxxXXHFFvLTl7LPPZuLEifTq1avWCCQAxcXFPPzww/zwww8MHDiQ3XbbbYN1Ep144om8+uqrfPXVV/zlL3+hW7duDBs2jKqqKkaPHs23/gWp++23H8cffzxHHnkke++9N7vuuitAvUMVxtrx5Zdfsueee7LTTjtx7rnnxkfceP311/nd735HZWUlf/3rX7nrrrsoLy9nyZIljBs3jqqqKoYMGcKee+7JiBEjNtj+UUcdxciRIzniiCMYNGgQZWVlXHXVVfX2Mx+cur4C2Yi4DX1VkiuRb76hfMECagYPxkn41PTD8nV8smA1B1RtQXnJhm+QjVlFRUX8E2tLoT63DOpzy6A+17Z27Vpat26d5xblViAQqHURWiF79dVXufDCC3n//febtJ3m6PPs2bPZb7/9+P7775slG5yPPtf3/vBrwzfImCoznQE3HIbi4lqBNEDIr5kuqeNrCREREWmZqqur+e9//0s4HGbhwoXcfPPNHHjggc3drJS98MIL1NTUsGLFCq6++mr233//Zi2rKDSK+jIRCm9QLw1emYeDLkAUERGR2m666SZ69+7NAQccQGVlJeedd15zNyllkydPpn///uy9994UFxdz7bXXNneTCoo+VmTADQVx6qgVCkVcjTEtIiIitZSXl/P88883dzMy9uijjzZ3EwqaMtOZCNeTmY64BDTGtIiIFLiN/HopkZxK9/2hyC8ToTAENgymwxFXw+KJiMhGQQG1yIYyeV8omM6AV+ZRd820hsUTEZFCV1ZWxpo1axRQiyRwXZc1a9ZQVlaW1vNUM52Jess8orRttWkNiSciIpue4uJiysvLWbt2LVB7BruNVatWreJjRbcU6nP2xD5YlpeX1zn+d0MUTGciFKq3ZlplHiIisjEoLi6mTZs2zd2MrNFY4i1DIfY5b8G0MeZA4DagGLjfWntd0uOtgH8CA4BlwChr7Q/5al+q3EgEXLfeofFKdAGiiIiISIuRl8jPGFMMTAIOAnoDY4wxvZNWOwH4xVrbA7gFuD4fbUubP5V48tB4UdclrJppERERkRYlX2nU3YE51trvrLVB4AlgeNI6w4GH/d+fAn5vjCm8yDQWTCdlpkMRr9ZGE7aIiIiItBz5KvPoDMxNuD8P2KO+day1YWPMr8BWQEEVxvyysprvFqwh+PUylrcqX/+Af0G0aqZFREREWo6N7gJEY8w4YByAtZZOnTrldf+dOnWi78Cd87rPQpHvY10I1OeWQX1uGdTnTV9L6y+oz4UgX2Ue84GuCfe7+MvqXMcYEwDa412IWIu19j5r7W7W2t0Ap7luxpiPmnP/6q/6rD6rz+qz+qw+t9z+qs/NdttAvjLTHwCVxpjueEHzaOCopHWmAX8G3gFGAq9aazWavIiIiIgUrLxkpq21YeB04EVgprfIfmmMudIYc6i/2gPAVsaYOcDZwIX5aJuIiIiISKbyVjNtrX0eeD5p2WUJv68DjsxXe7LgvuZuQJ61tP6C+txSqM8tg/q86Wtp/QX1uSA4sekTRUREREQkPZquT0REREQkQxvd0Hj5tKlMgZ6OFPo8GLgV6AeMttY+lfdGZlkKfT4bOBEIA0uA4621P+a9oVmUQp9PAU4DIsBqYJy19qu8NzSLGutzwnoj8CaOGmit/TCPTcy6FM7zWOAG1o+udKe19v68NjKLUjnHxhgDXI43O8Cn1trki+E3Kimc41uA3/l3WwMdrLWb57WRWZZCn3+DNwnc5v46F/qlphutFPq8HfAPYGtgOXCMtXZe3huaJcaYfwCHAD9ba/vW8biDdzwOBtYCY621H+e3lespM12PTWoK9BSl2OefgLHAY/ltXW6k2OcZwG7W2n54Qdbf8tvK7Eqxz49Za3ey1u6M19+b89vK7Eqxzxhj2gFnAu/lt4XZl2qfgSettTv7t405kG60v8aYSuAiYG9rbR9gYr7bmU2p9Nlae1bs/AJ3AM/kvaFZlOLr+hK8gQ52wRs97K78tjK7UuzzjcA//f9TVwLX5reVWfcQcGADjx8EVPq3ccDdeWhTvRRM12/TmQI9dY322Vr7g7X2MyDaHA3MgVT6/Jq1dq1/9128cdI3Zqn0eWXC3TbE5/jcaKXyfga4Cu9D8bp8Ni5HUu3zpiKV/p4ETLLW/gJgrf05z23MtnTP8Rjg8by0LHdS6bMLbOb/3h5YkMf25UIqfe4NvOr//lodj29UrLVv4mXY6zMc78ODa619F9jcGLNtflq3IQXT9atrCvTO9a3jD//3K94U6BurVPq8qUm3zycAL+S0RbmXUp+NMacZY77Fy0yfkae25UqjfTbG7Ap0tdb+J58Ny6FUX9sjjDGfGWOeMsZ0rePxjUUq/e0J9DTGTDfGvOt/db4xS/nvl18G0J31AdfGKpU+Xw4cY4yZhzeK2IT8NC1nUunzp8AR/u+HA+2MMRtzPNKYgopXFEyLpMgYcwywG16N6SbPWjvJWrsDcAHe16abLGNMEV4pyznN3ZY8ew7o5n81/DLrv2nbVAXwvhYegpel/bsxZvPmbFAejQaestZGmrsheTAGeMha2wWvpvYR/z2+KTsX2NcYMwPYF+86iJZwrgvCpv7iaoqsTYG+EUmlz5ualPpsjBkKXAwcaq2tyVPbciXd8/wEcFguG5QHjfW5HdAXeN0Y8wMwCJhmjNktby3MvkbPs7V2WcLr+X68i6k3Vqm8rucB06y1IWvt98A3eMH1xiqd9/JoNv4SD0itzycAFsBa+w5QBlTkpXW5kcp7eYG19gi/Tvxif9mKvLUw/woqXtFoHvVriVOgp9LnTU2jfTbG7ALcCxy4CdRYQmp9rrTWzvbv/hGYzcatwT5ba38l4Z+tMeZ14NyNfDSPVM7zttbahf7dQ/FmqN1YpfL361m8rOWDxpgKvLKP7/LZyCxL6W+2MWZHYAu8/1Ubu1T6/BPwe+AhY0wvvGB6SV5bmV2pvJcrgOXW2ijeRbb/yHsr82sacLox5glgD+DXhL9leafMdD1a4hToqfTZGDPQr0M7ErjXGPNl87W46VI8zzcAbYEpxphPjDHTmqm5WZFin083xnxpjPkE77X95+ZpbXak2OdNSop9PsM/z5/i1cWPbZ7WNl2K/X0RWGaM+QrvIq3zrLUb7beJabyuRwNPbOTJHiDlPp8DnOS/rh/HGzZto+17in0eAswyxnwDdASubpbGZokx5nG8D39Vxph5xpgTjDGn+MO2glcL/x0wB/g7ML6ZmgpoBkQRERERkYwpMy0iIiIikiEF0yIiIiIiGVIwLSIiIiKSIQXTIiIiIiIZUjAtIiIiIpIhjTMtIgXJGPMQMM9ae4kxZh/gfmttVTM3KyPGmCrgSWAH4GJr7e153r8LVFpr5+Rzv/lijHkBb+i3BmdwNMasBvpZazfmsaVFpMAomBaRgmetfQtoNJA2xlwO9LDWHpPzRqXnfOA1a+3Ozd2QbEn8sNPcbbHWHpTiem2zuV9jzEC8aZz3BsqB74HHgDuttcFs7ktECpfKPEREcm87oN4JjowxxXlsS0EyxmxUyR1jzATgEeA5YFdga+BovHP9P2PM5s3XOhHJJ03aIiIFwZ+2/QGgEm92KxeY45d5DAEmW2u7+OtegDdj32bAArzZr0rwpph1gBrgW2ttf2PMcXiZ4S54Uwpfb62919/OEGAycAtwARAB/mKtfdB/vBz4f8BIYHPgc2B/a221MWYQcDPQG/gRONNa+3od/XoV2BcIAWG8wOsvQDVe4LUvMBxvmuC7gZ393y+y1k7zt/EQsBboDuwDfAqMwJt19c/AYmCMtXZGPcfWBc4EJvrH7EHgAn/qYYwxxwPnAdsA7wPjrLU/GmMcv49H403J/CPedNx7AZP8cxTEy7oPS2e/xpixwEn+/v7k9/0qvJnbDNAK+BdwlrW22t/ecOAKYHu8c3matfb//OnfJ1tr7zfG9MB7He3sH/P/WmtHJbSn0lo7xxjTHrgDOMg/tn8Hrklo24nAu8AJwApgvLX2BX87Q/BeM7+z1q6oo9+nAQOstcfXdT5EZNOizLSINDtjTCnwLF6mb0tgCl6wWNe6VXhT6w601rYDDgB+sNb+H3AN8KS1tq21tr//lJ+BQ/CCueOAW4wxuyZschugPdAZL3CaZIzZwn/sRmAAXvC4JV5QHjXGdAb+gxdob4n3Vf/Txpitk9trrd0PeAs43W/XN/5DR+EFju2A9/AynC8BHYAJwKN+X+NdBy4BKvA+LLwDfOzffwov6G3I4cBueMH8cOB4/3gOxwvuj8DLrr6FNwUzwB+AwUBP/xgZYJm19j7gUeBvfp82CKQb269vD7wpgWPTH1/n72tnoAfeObnMb+fuwD/xgv7N/Xb9UMf+rsI7jlvgfYC6o5523eH3aXu8DzR/wnt9JLZtFt7x/RvwgP/hAuCveMH1CmPMmf50x7OMMVcZYy4B7gIG+QG7iGziNqqv1URkkzUIL7N8q7XWBZ4yxpxdz7oRvKxlb2PMEmvtDw1t2Fr7n4S7bxhjXsLL7n7sLwsBV1prw8Dz/kVqVcaY9/ECv0HW2vn+um8DGGOOAZ631j7vL3/ZGPMhcDDQ4EVwCaZaa6f729sZaAtc52eLXzXG/BsvC3y5v/6/rLUf+ev/Cy+Y+6d//0m8DxgNud5auxxYboy51d/2/cApwLXW2pn+tq4B/mKM2c4/Nu2AHYH3Y+ukqb79Aiyw1t7h7zcCjMO7QHB5QlseAy7C+6DzD2vty/5z51O3EF7Gv5O1dh7wv+QV/LKa0cDO1tpVwCpjzE3AsXhZbYAfrbV/99d/GC9A7miMWQH8xlr7jjGmN94HrH2ApXgfat621rrGmC/xvmX5MK2jJSIbHQXTIlIIOgHz/UA65se6VvS/op+IF2T2Mca8CJxtrV1Q1/rGmIPwMok98b6Na41XrhGzzA+kY9biBbYVeKUN39ax2e2AI40xiRnZEuC1+jpYh7kJv3cC5sbKLnw/4mVmYxYn/F5dx/3GLq5L3N+P/j7B68ttfjAZ4wCdrbWvGmPuxCvp2M4Y8wxwrrV2ZSP7SmW/yY9tjXduPjLGJLYjVk/eFa/8pzHn42Wn3zfG/ALcZK39R9I6FXjnK/E1lny8F8V+sdau9dvUFu81FDv2fYHpsdFBjDHP4n2zEGtvfQG/iGxCFEyLSCFYCHQ2xjgJAfVvqDuQxVr7GPCYMWYz4F7gerysYq2LQIwxrYCn8b7Cn2qtDfkBj0PjlgLr8Iaz+zTpsbnAI9bak1LYTn0S27oA6GqMKUoIqH8DfLPh0zLWlfUXQf7G3yd4fbnaWvtoXU/yh/G73RjTAbB4ZRaXJrU/k/2StI2leB8K+iR8E5BoLt65aJC1dhFeLTbGmN8Crxhj3kwaFnAp6zPYXyW0LZXgdznrA+YvgJuNMbEa7sOAD40xZwA/W2sXprA9EdnIKZgWkULwDt7FeWcYY+4ChgG7U0em168j7gxMxwt2q1mfvVwM7J8QlJbilYQsAcJ+lvoPeEFQg/wL0f6BFywd6297d7zykMnAB8aYA4BX8LKcg/AumJyXQf/fw8uIn+9niPf2j8HADLZVn/OMMe/hZVfPZH2N9T3AVcaYT6y1X/p1vn+w1k7xh34rwuvzGrzjHQv2F+PVG2e631r84/13vJr20621P/u16X2ttS/ilV+85Je/vAZsC7Sz1n6duB1jzJHAO/55+AUvYI8m7StijLHA1caYP+HVvZ+NVyPfIGvtOmPMImPMAGvtR8aYG/DqzJfi1WqPwKunL7ThGUUkR3QBoog0O39M3iOAsXiZv1HAM/Ws3grvQrWleF/Fd8CrqQXvwkWAZcaYj/162DPwMqq/4F30Ny2Npp2LVxLygd+u64Eia+1cvIvp/oIXqM/Fy9hm9DfV7/8wvJElluLV5/4pOVBsoqnAR8AneMHeA/6+/4XXryeMMSvxPmjExm3eDG+Ui1/wyiCWATf4jz2AV7e+ws/2p7XfelwAzAHe9dvyCv744tba9/EvIAV+Bd7AyywnGwi859e+T8MbZaWuSVom4H1A+A6vrvoxILkcpD5XAfcaY9paa2+z1na21va31p6Hl1k/O81SGBHZiGloPBERyYlNeeZFY8x5eKVFF+NlyoN4FyJeg1fDP70ZmycieaTMtIiISJqstTfgZbdPwsumL8L7huRKBdIiLYtqpkVERDJgrX0Dr9xERFowlXmIiIiIiGRIZR4iIiIiIhlSMC0iIiIikiEF0yIiIiIiGVIwLSIiIiKSIQXTIiIiIiIZUjAtIiIiIpKh/w/sBYUq4ObvoAAAAABJRU5ErkJggg==\n" + }, + "metadata": { + "needs_background": "light" + } + }, + { + "output_type": "display_data", + "data": { + "text/plain": [ + "
" + ], + "image/png": "iVBORw0KGgoAAAANSUhEUgAAAtMAAAGSCAYAAAAhGE1lAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8QVMy6AAAACXBIWXMAAAsTAAALEwEAmpwYAAB/gklEQVR4nO3dd5hU5d3/8feZsr1Qlra7CAi7dAGRYgmigVgiYsEDWJ5ojNiisRtjiZpo4s9orNFYoj5iO6iPYqIxJmKJXcGOFCmyS6/bd6ec3x/nzDCzjdlhG/B5XddcO6ffM7ssn73ne+7bsG0bERERERFpOU9HN0BEREREZE+lMC0iIiIikiSFaRERERGRJClMi4iIiIgkSWFaRERERCRJCtMiIiIiIklSmBaRPZJhGJMNw7ANw5jcgmNuNAxjjxsP1DCMaYZhPG0YxlLDMMKGYbzVzL7DDcP4l2EYFYZhbDEM4zHDMLq1Y3Prt2ey+77vFf/fGIbRxX09Bzay7a3mvjcisnfaK365icg+aSFwsPs1UY+4x7QLwzB+7IbgUsMwgoZhbDYM403DMM40DMPbglOdAIwGPgRKmrlePvAWkA7MAC4EpgB/78AwOxn4LXvP/zddcF5PgzANXOA+RGQf4uvoBojIvsEwjFTbtmtb63y2bZfhhMuWHFNCM2G0tRiGkQ08AfwEeBL4FVCKE8QmADcD5xuGMcO27TUJnPIc27bD7rn/28x+VwJ+YJpt29vd/dcCb+ME8heTeDn7HMMwDMBv23ZdS46zbfvbNmqSiHRie0tPgYi0g0iZhGEYIw3DWGAYRpVhGOsMw7g5tuczpgTjJMMwHjYMYxOwIWb7HMMwvjAMo8btrX20fimCYRg+wzCuNgzjW3e/TYZh/NMwjCH1rjE55pijDMN43zCMHW6ZwxLDMG6o3/5618kxDOM+wzDWGoZR6x5zqRuo6r+e4919N7uPuYZhdKl3vhTgn0ABMNi27fNt237etu0PbNt+zbbtG4GhwFLgjURKMCJBOgHHA/+IBGn32HeAH4DpCZ4Dt+fcNgxjkmEYL8WUjNxvGEZ6vX0zDcP4o2EY37vv33rDMF4wDKOXYRg34vTiAgTcczZbZrOr74dhGL3dXv6LGzn2KsMwAoZh9IhZd5JhGB+6P6vbDcOYZxjGfvWOW+V+L39uGMZ3QB3w00bO3x9Y6S4+HHk9hmGc6W6PK/OI+bk5wTCMvxqGsdVtw12GYXgNwxhnGMZ/DcOoNAzjG8MwjmrkmocbhvEfwzDK3f1eNwxjRHPvoYi0L4VpEUnGS8C/cXo7nwauB25oZL97AQM4AzgTwDCMPwL3u8cfj9ObejTwmhFf+vAscAvwqnudc4BvgT6NNcgwjP2B+ThhZ6Z77juBzKZehPsHwD+As4A7gGk4QfhO99r13Q3YwKnATcDJ7rpY1wNdgSNs2y5t5JoGUIPzfqwE/thU+1rCDbkDgK8b2fwNMCyJ084FlgMnAX/G+R48EHPNFOAN4CLgceA44JfAVpz34BHgUXf3w3BKbJoss0nk+2Hb9nqcn53TGznFGcA/bdve5J7vPOAFnJ+bGcC5wAjgbcP59CDWEcBlON/Xo4EvGzn/Ove9APhDzOv5R1OvyXUXUInzc3kvzicVdwH/C/zNPedW4EXDMPJi3o+fAv8BKtzXeyqQDbxrGEbfXVxTRNqLbdt66KGHHgk9gBtxwuSv661/GCgHurjLk939/q/efv2BEHBDvfWHuvuf4C4f6S5f3ExbIteY7C7PcJdzdtX+mOXj3GPOrLffI0AtkFfvWk/U2+8+nGBsuMsZOMHn8Jh9LgJWu/v9H3AV8Ja7bTBQDeS24Hvw38jx9dbnu208r5Ftc4HvW3CNM91zPVhv/bXu96/YXf65u9/xCfzM+BK4bqLfj9Pc/QbH7DPaXWe6y1nADuBv9c41AKfn+ZKYdauAKqB3Am3s717nF41seyv2exPzc1O/DQvd9YfFrDvAXfezmHXLgf/UOzYH2Azclej3Uw899Gjbh3qmRSQZVr3lZ3HCS/2Pn/+v3vJUnE/EnnLLOHyGYfiAj3DC+CR3v5/gBIuHW9Cmz4EA8KxhGDMMw+iZwDGTgDBO73qsuUAKDXtR6/dAfgWkAr3c5SOBDbZtvw1gGMaJOD2sD+D0dn6PUy8NgG3bS3BqqSck0NaO0Nj32QOMd5d/Aqy3bXt+K10v0e/H/+H80XJGzD5n4ITnSFsOxgme9X/W1gDfsfNnLeJD2+n1bguv1Vv+Dqi0bfu/9dYB9AUwDKMIGEjD9lcBH9Cw/SLSQRSmRSQZG5pYLqi3fl295UjAXY4TfGMf2UB3d3t3YKtt29WJNsi27eXAUTi/154E1ru1soc3c1g39zr1bzRbH7M91tZ6y5EbKtPcr8XEl1mcg9Mr+Ufbtt+ybfsK4N1659gI5LH7tuP8AdK1kW3daNj2ROzq+9wd54+B1pLQ98O27Sqc8o3TDIcXmA3Ms227xt038rP2bxr+rI1k589aRP2f1da0rd5yHc73KyrmNUd+liLtf5SG7T+Ohu0XkQ6i0TxEJBm9gBX1lqFhsKp/s9kW9+tPaBgwYrdvBroZhpHewkC9AFhgGEYqTunIzcA/DMPob9v25kYO2epeJ6VegOsds70lfDjlHBH92dlTGrGQ+J7oApzXu1ts264yDGMVMLyRzcNwRvRoqV449daxy7Dz+7yZhp9G7I6WfD+eBH6GU4udjlNL/2TM9sjP0pnEv4aI8nrLnW388Uj7r8H5g6C+Fo00IiJtRz3TIpIMs97yLJyP3b/axXFv4HyMv59t25828oiMlPAvnBsXf5FM42zbrrVt+03g/+HcgDigiV3fxvk9eEq99afhhJUPWnjpEmBQzPJGnEAdK7psGMaPcHpbWzTEXzPmAz81DCM35hqHAf1oGOoT0dj3OYxTlgPO96m3YRjTmjlHpPc+vZl9Ilry/ViA836f4T5WEd/r/z5OYB7UxM/akgTa05iWvJ7dsQTnNQ1vov2N3SApIh1APdMikoxz3JEXPsEprfgFcKNt2zuaO8i27e8Nw7gNuM8wjME44akGp050KvCIbdsLbNteYBjGC8Cd7qgFb+KMnzwJZ+i3t+qf2x25YRLO6B9rcEonrgHW0vgIF+DUsv4XeNAdTu0b4Fj39fyhid7s5rwFPGkYxv62ba/AKUX4nTtc2gfAMcCJwEeGYRyL8xH+b21nzOwmGYbRDxjnLnYHwoZhzHCXP7Fte7X7/HacUR/mG4bxByAX5w+Kj2hYv56IYw3DuB0nNI/HGebuf23bXuZun4tTyvKMe72PcMp1jsK5Qe47nJE0AC43DOM1IGTb9qdNXC/h74dt22HDMJ7CGaHDD/zZtm07ZnuZYRhXAve753oNp6a6ADgc50bB+rXZidiA02s8yzCML3FG6Vhp2/aW5g9rGdu2bcMwLgRedkdNsXA+CegFHAL8YNv2na15TRFJUkffAamHHnrsOQ92jswwAqdnsBqnnvV3gCdmv8nuflOaOM8ZOL2xlTg92otxRsYojNnHhzN6xFKcXslNOEF5cL1rTHaXDwZexgnStTg1sPOIH/HhRmJG83DX5bjXXudeZylwKe4IHc29HnaOetE/Zt1zbjsMnJD3uLuPjVMrfrv7vARnMpZE3vczY85R/3FmvX1H4nwCUIlTSvM40L2F3+fI9Sa5r6UCp8TifiC93r5Z7mta7b5/64DngZ7udq973EacXm17F9fe5fcjZt/hMe9DcRPnOxbnZ7UM5+a9ZTjD0Q2L2WcVMLcF788JOH8kBGK/BzQ9mkf9n5vHgZJGzmsDv6+37mDg7+73ssZt67PAwR39+0APPfRwHpHhnEREdilmEg6/bdvBDm5Op2QYRgHwKc5Y3BfZth00DKM7zg1lS3BmQcyxbXtVR7VxV9xJSB4Dimznxk4REWmCaqZFRFqR7UzUMhVnKLxFbjDNwpmgpQfOR/R3Gobxcoc1UkREWo1qpkVEWplt218bhjEKuAJnRJHY2eo24ow60SozH7aUO1Zxc0Lt0hARkb2EyjxERNqYYRj5ODdEbrdt+4cObEd/nB7y5hxhN3KDp4iINE5hWkRkH+GOCnHALnZbYtt2/TGYRUSkCQrTIiIiIiJJ2tNrpvWXgIiIiIi0F6P+ij09TLN27dqOboKIyB4lLy+PzZt3ewZzEZF9Sn5+fqPrNTSeiIiIiEiSFKZFRERERJKkMC0iIiIikiSFaRERERGRJO3xNyCKiIhIywUCAerq6gAwjAYDFIjsUyJDRaekpOD3+1t0rMK0iIjIPqampgaAjIwMBWkRl23b1NbWEgqFSEtLS/g4lXmIiIjsYyJhQUFaZCfDMEhLSyMUCrXoOIVpERGRfYxCtEjTWvrvQ2FaRERERCRJCtMiIiLSqUyYMIF33nmno5shkhCFaREREdnr3XHHHVx00UUd3QzZCylMi4iIiIgkSWFaREREOp0vvviCyZMnM2zYMC699NLocH4Ab7zxBlOnTmXo0KEcf/zxfPvtt9Ft999/P2PHjqW4uJgf/ehHvPvuuyxYsIB7772X+fPnU1RUxJQpUxq9ZmlpKb/4xS8YOXIkw4cP59prrwVg1apVnHLKKQwfPpwRI0bwy1/+kh07djR7TYBwOMx9993HIYccwvDhwzn33HPZtm1bW7xd0oGMyCDVeyh77dq1Hd0GEZE9Sl5eHps3b+7oZkgHqqqqIiMjI27djBkzGux33HHHceaZZ1JdXc0ZZ5zRYPspp5zCzJkz2bp1K3PmzGmw/YwzzmD69OmUlpZSUFCQcPsmTJhAZmYmTz75JBkZGZx55pkccsghXH311Xz99deceuqpPP7444waNYoXXniBO+64g3feeYc1a9Ywa9Ys/v73v9O7d2/WrFlDKBSif//+3HHHHaxatYp777230WuGQiGOOuooDj30UK6++mo8Hg9ffvkl48ePZ+XKlaxZs4YJEyZQUVHBOeecw4gRI7j55ptZvnx5k9d85JFHeOmll3jooYfo3r07119/PRUVFfzlL39J+L2Q9tfYvw+A/Px8gAZDfahnWkRERDqdM888k4KCArp27crFF1/Myy+/DMDcuXM5/fTTOfDAA/F6vZimSUpKCgsXLsTr9VJXV8fSpUsJBAL07duX/v37J3S9RYsWsWHDBq6//noyMjJIS0tj/PjxAAwYMIBJkyaRmppK9+7dmTNnDh9++CFAs9d88sknufrqq8nPzyc1NZXLL7+cf/zjHwSDwVZ/v6TjaAZEERER4fnnn29yW3p6erPbu3Xr1uz2lvRKR7i9gAAUFhayYcMGwCnFmDdvHo899lh0e11dHRs2bODggw/mpptu4s4772Tp0qUcfvjh/Pa3v6V37967vN7atWspLCzE52sYjTZt2sQNN9zAxx9/TEVFBeFwmNzcXMAJ2k1ds6SkhF/84hd4PDv7Lr1eL5s2baJPnz4tfk+kc1LPtIiIiHQ6sWWcpaWl9OrVC4A+ffpw8cUXs3jx4ujj+++/54QTTgDgxBNP5KWXXuKjjz7CMAxuueUWYNcTceTn51NaWtpor/Ef//hHDMPg3//+N0uWLOHee+8ltky2qWvm5+czd+7cuLauWLFCQXovozAtIiIinc4TTzzB2rVr2bZtG/fccw/Tpk0D4LTTTuPJJ59k4cKF2LZNVVUV//73v6moqGD58uX897//pba2ltTUVNLS0qK9wnl5eaxZs4ZwONzo9caMGUPPnj259dZbqaqqoqamhk8++QSAiooKMjMzycnJYd26dTzwwAPR45q75hlnnMFtt91GSUkJAFu2bOH1119vs/dMOobCtIiIiHQ6J5xwAqeeeiqHHHII/fr145JLLgFg1KhR3H777Vx33XUMGzaMQw89FMuyAKfc4w9/+AMjR45kzJgxbN68mWuuuQZwbqYEGDFiBEcddVSD63m9Xp544glWrVrFuHHjOOigg5g/fz4Al112GV999RVDhgzhf/7nfzjmmGOixzV3zV/84hdMnTqV2bNnU1xczLRp01i4cGGbvWfSMTSah4jIPkajeUhToxWIiEbzEBERERFpNwrTIiIiIiJJUpgWEREREUmSwrSIiIiISJLaZdIW0zT/BhwHbLQsa0Qj2w3gbuBYoAo407Is3e4qIiIiIp1ae/VMPw4c3cz2Y4Ai9zEHeKCZfUVEREREOoV2CdOWZb0DbG1ml+nA/1qWZVuW9SHQxTRNTQ8kIiIiIp1au5R5JKAAWBOzXOKuW1d/R9M05+D0XmNZFnl5ee3SQBGRvYXP59Pvzn3chg0b8Pk6SwQQ6VxSU1Nb9Dtyj/uXZFnWQ8BD7qKtiQdERFpGk7ZIbW0tXq+3o5vR4e644w5WrVrFvffe29FN6RQmTJjA7bffzqRJk1r93B999BFXXHEF7777LuBMw37++eezevVqrr76apYuXUrv3r259NJLW/3aLVVbW9vo70h30pYGOkuYLgX6xiwXuutERERkHzNhwgQ2b96Mx+MhMzOTyZMnc8stt5CZmdnRTUvImjVrmDhxYtwsev369ePf//53u7WhoKCA//73vwwYMCC6rry8nNtvv53XXnuN7du306NHD6ZMmcIll1xCt27d2rQ9EyZMiAZpgAceeIBDDjmEN954o02v2x46y9B484H/MU3TME1zIrDDsqwGJR4iIiKyb3jsscdYtmwZ//rXv/j666/3yN7jxYsXs2zZMpYtW5ZUkA4Gg63Wlrq6OmbOnMnSpUt56qmnWLJkCfPnz6dr164sWrSo1a6TqJKSEgYPHrzb52nN9yhZ7RKmTdN8BvgAGGyaZolpmmebpnmeaZrnubu8CqwAlgMPAxe0R7tERESkc+vZsyeTJ0/mm2++AeC+++7jkEMOobi4mMmTJ/Paa69F933uuec44YQTuPnmmxk2bBgTJ07kzTffjG7/4YcfOPnkkykuLmbWrFls3Ro/NsK//vUvjjjiCIYOHcqMGTNYtmxZdNuECRN44IEHmDJlCoMGDeLyyy9n06ZNnH766RQXFzNz5ky2b9++y9ezfv16zjzzTIYPH86hhx7KU089Fd12xx13cM4553DRRRcxePBgLMuirKyMyy+/nDFjxjB27Fhuu+02QqEQACtXruTkk09myJAhjBgxgvPOc2LVSSedBMDUqVMpKiri5Zdf5vnnn6e0tJRHH32U4uJiPB4PeXl5XHrppfz4xz9u0M5FixYxbdo0hg4dypgxY7j22mupq6sDwLZtfvvb33LAAQcwePBgfvzjH/Pdd98B8J///IfJkydTXFzM2LFjefDBBwF4//33GTt2LACnnHIK77//Ptdddx1FRUV8//33XHLJJdx2223R67/xxhtMnTqVoUOHcvzxx/Ptt9/GfS/uv/9+pkyZQlFRUYcH6nYp87Asa/YuttvAhe3RFhEREYl31zslLNtU3abXKOqRziWTClt83Nq1a1mwYAGHHnoo4JRLvPjii/Ts2ZNXXnmFiy66iPfee49evXoBTgg85ZRT+Oqrr5g7dy5XXHEFn332GYZhcOGFFzJ27FiefvppFi1axP/8z/9w1FFHAfD9999zwQUX8Le//Y2DDz6Yhx9+mDPPPJMFCxaQkpICwD/+8Q+eeeYZgsEgRx11FF9//TV33HEHgwYN4owzzuBvf/sbl112WbOv54ILLmDw4MF89tlnLF++nNmzZ9OvXz8OO+wwwAn0f/3rX7n77rupra3ll7/8Jd27d+e9996jqqqKn/3sZ+Tn53PGGWdE65vnzZtHXV0dX375JQAvvvgiBQUFvPHGG9Eyj/PPP5/JkycnXCrj9Xq58cYbGTVqFOvWreP000/niSee4JxzzuHtt9/mo48+4t133yUnJ4fly5eTk5MDwBVXXMGDDz7IhAkT2L59O2vWrGlw7nnz5jFjxgxOOukkTj311Abbv/76ay6//HIef/xxRo0axQsvvMBZZ53FO++8Q2pqKgAvvfQSTzzxBN26devwm2k7S5mHiIiISNTZZ59NcXEx48aNo3v37lx++eUATJs2jd69e+PxeJg+fToDBgzg888/jx5XWFjIaaedhtfrxTRNNmzYwKZNmygtLeWLL77gqquuIjU1lYkTJzJ16tTocfPnz+fHP/4xkyZNwu/3c95551FTU8Onn34a3efnP/85PXr0oE+fPkyYMIExY8YwYsQI0tLSOOaYY/j666/jXsPIkSMZOnQoQ4cO5cEHH6S0tJRPPvmEa6+9lrS0NEaMGMGpp57K888/Hz1m7NixHH300Xg8HioqKnjzzTe56aabyMjIIC8vj3POOYeXX34ZcEbmKS0tZf369aSlpTF+/Pgm389t27ZF/+BIxAEHHMDYsWPx+Xz07duX008/nQ8//DB63YqKCpYvX45t2xQVFUXP7fP5WLp0KeXl5XTp0oWRI0cmfM2IuXPncvrpp3PggQdGv48pKSksXLhzPr+zzz6bgoIC0tPTW3z+1tZZbkAUERGRDpJMj3Fbe/TRR5k0aRIffPABv/zlL9m6dSu5ubnMmzePhx56iJKSEgAqKyvjyjV69OgRfR4JWpF9cnNz424KLCwsZO3atYAzXGBh4c73wePx0KdPH9avXx9dFztcWlpaWty10tLSqKysjHsNX331VVyv6cKFC+nSpQtZWVnRdQUFBXzxxRfR5dgRI0pKSggEAhx44IHRdeFwOLrPddddx+23385xxx1Hbm4u5557LrNmzWr0/ezatSsbNmxodFtjvv/+e2666Sa+/PJLqqurCQaDHHDAAQAcdthhnHXWWVx77bWUlJRw7LHHcv3115Odnc3DDz/M3XffzR/+8AeGDh3KNddcw0EHHZTwdQFKS0uZN28ejz32WHRdXV1dXPubGlmjI6hnWkRERDqtgw8+mFNOOYXf/e53lJSUcNVVV3HLLbfw9ddfs3jxYgYPHoxt27s8T69evdixYwdVVVXRdaWlpXHbIwEdnLrgdevW0bt371Z7Lb169WL79u1UVFTEtSH2GoZhRJ/n5+eTmprKV199xeLFi1m8eDFLlixhwYIFgFNPfvvtt7Nw4UJuu+02fvOb37By5cpGr/2jH/2It99+O+71N+eaa65h0KBB/Pe//2XJkiX8+te/jnufzz77bP75z3/y1ltvsWLFCh54wJm8evTo0Tz22GN88cUXHHXUUdE67pbo06cPF198cfQ1L168mO+//54TTjihxedqDwrTIiIi0qmdc845vPPOO+zYsQPDMKLDuD333HMsWbIkoXMUFhZywAEH8Kc//Ym6ujo+/vjjuGHZpk2bxn/+8x/effddAoEAf/3rX0lJSWlxr2pzCgoKOOigg/jDH/5ATU0N3377Lc8++2z0hsH6evXqxaRJk7j55pspLy8nHA6zatUqPvjgAwBeeeWVaM96bm4uhmHg8TjRrkePHvzwww/Rc5188snk5+dzzjnnsHz5csLhMFu3buWee+7hP//5T4NrV1ZWkp2dTWZmJsuXL+d///d/o9s+//xzFi5cSCAQICMjg9TUVDweD3V1dbz44ouUlZXh9/vJzs6OtqclTjvtNJ588kkWLlyIbdtUVVXx73//O+6PkM5EYVpEREQ6te7duzNjxgzuuusu5syZw/Tp0xk1ahSLFy9m3LhxCZ/n/vvvZ9GiRQwfPpw777yTGTNmRLcNGjSIe++9l+uvv56RI0fyxhtv8Pjjj0dvPmwt999/P2vWrGHs2LH84he/4PLLL292kpS7776buro6Jk+ezLBhw5gzZw4bN24E4IsvvmDatGkUFRVx1llncdNNN9GvXz8ALrvsMi655BKGDh3K/PnzSU1N5dlnn2XgwIHMmjWLwYMHc9xxx7F161bGjBnT4LrXX389//d//0dxcTFXXnklxx9/fHRbeXk5V111FcOGDWP8+PF07dqV888/H4AXXniBiRMnMnjwYJ588knuu+++Fr9Ho0aN4vbbb+e6665j2LBhHHrooViW1eLztBcjkY9GOjE78heZiIgkRjMgSlVVVVztsIjs1NS/D7dO26i/Xj3TIiIiIiJJUpgWEREREUmSwrSIiIiISJIUpkVEREREkqQwLSIiIiKSJIVpEREREZEkKUyLiIiIiCRJYVpEREREJEkK0yIiIrJPOP300xOaSa+oqIjVq1e3Q4va3vvvv8/YsWPb7PxXX301f/7zn6PLTzzxBKNGjaKoqIitW7fuVe9lUzQDoojIPkYzIEpnnwFxwoQJbN68Ga/Xi9frpaioiBkzZnD66afj8ewZ/YBHHHEEJSUlANTU1OD3+/F6vQBcdNFFXHzxxa12rUWLFnHnnXfy6aef4vF46N+/P//zP//DzJkzef/997nooov47LPPWu16TQkEAgwZMoT58+czfPjwNr9eW2npDIi+dmiTiIiISIs89thjTJo0ibKyMj788ENuuOEGFi1aFNcL2pktWLAg+nzGjBmcdNJJnHrqqQ32CwaD+HzJx7FPP/2U2bNnc8kll3D33XfTtWtXvvrqK+6//35mzpyZ9HmTsWnTJmpqahg8ePBun2t335f2tGf8eSciIiL7pJycHH7yk5/wwAMPMG/ePL777jtqa2u5+eabGTduHKNGjeLqq6+muro6eszrr7/O1KlTGTx4MIccckg02M6YMYOnn34agJUrV3LyySczZMgQRowYwXnnnRc9vqCggJUrVwJQVlbGxRdfzMiRIxk/fjx33XUX4XAYgOeee44TTjiBm2++mWHDhjFx4kTefPPNZl/PmjVrKCgo4JlnnmHcuHGYpgnAs88+y+GHH86wYcM49dRTo73aAMuXL2fWrFkMHz6cH/3oR8yfPz+67fe//z2nnHIKF154Id26dcMwDA444AD++te/Nnr9++67j0MOOYTi4mImT57Ma6+9Ft3W1Hti2za//e1vOeCAAxg8eDA//vGP+e677wC45JJLuO222/j++++ZNGkSAEOHDuWUU05p8F42932LlKPcf//9jB49mksvvbTZ97Ez2TMiv4iIiLSZnPdvxb/luza9RqD7EMoO+U3Sx48ZM4Y+ffrw0Ucf8cwzz7B69Wr+9a9/4ff7ufDCC7nrrru45pprWLRoEb/61a946KGHOOyww9iwYQOVlZUNznf77bczadIk5s2bR11dHV9++WWj173uuusoLy/ngw8+YNu2bcyePZtevXoxe/ZswCmxOOWUU/jqq6+YO3cuV1xxBZ999hmG0aAaIM4HH3zA22+/jWEYvP7669x77708/vjjDBgwgPvuu48LLriA+fPnU1VVxaxZs7jyyiuZO3cu3333HbNmzWLIkCH07duXzz77jCuvvDLh97Ffv368+OKL9OzZk1deeYWLLrqI9957j169ejX5nrz99tt89NFHvPvuu+Tk5LB8+XJycnLizjtw4EAWLFjAxIkTWbx4caO9yrfeemuT3zdwera3b9/ORx99FP2DZU+gnmkRERHZI/Tq1Yvt27fz1FNPceONN9K1a1eysrK46KKLePnllwF45plnmDlzJpMmTcLj8dCnTx8GDRrU4Fw+n4/S0lLWr19PWloa48ePb7BPKBRi/vz5XHPNNWRlZdG3b1/OPfdcXnjhheg+hYWFnHbaaXi9XkzTZMOGDWzatGmXr+Xyyy8nIyOD9PR0nnzySX75y19SVFSEz+fj4osv5ptvvqGkpIQ33niDvn37MnPmTHw+HyNGjODYY4/l73//O9u3byccDtOrV6+E38Np06bRu3dvPB4P06dPZ8CAAXz++efNvic+n4+KigqWL1+ObdsUFRW16Jrg9G43930D8Hg8XH755aSmppKent6i83ck9UyLiIjs43anx7g9rV+/nlAoRHV1Ncccc0x0vW3bhEIhANatW8eRRx65y3Ndd9113H777Rx33HHk5uZy7rnnMmvWrLh9tm7dSiAQoLCwMLqusLCQdevWRZd79OgRfR4JgI31hNfn3swGQElJCTfccAM333xz3D7r1q2jtLSURYsWMXTo0Oj6YDDIySefTJcuXfB4PGzYsKHRPxgaM2/ePB566KFoGUllZSVbt24Fmn5PDjvsMM466yyuvfZaSkpKOPbYY7n++uvJzs5O6JoAW7Zsafb7BtC9e3fS0tISPmdnoTAtIiIind7nn3/O+vXrOeqoo7j//vt588036dOnT4P9+vTpw6pVq3Z5vp49e3L77bcD8PHHHzNr1iwmTJjAgAEDovt069YNv99PSUkJxcXFAJSWljZ63ZaKLQPJz8/n4osv5qSTTmqwX2lpKRMnTuTZZ59t9Dxjx47l1Vdf5dBDD93lNUtKSrjqqqt47rnnGDt2LF6vl6lTpxIZ2a259+Tss8/m7LPPZvPmzZx33nk88MADXHXVVQm/3m7dupGWltbk921PpjIPERER6bTKy8t54403uOCCCzjppJMYPnw4p512GjfeeGN0iMd169bx1ltvATB79mwsy+Ldd98lHA6zbt06li9f3uC8r7zyCpHhdXNzczEMo8Gwe16vl2nTpnHbbbdRUVFBSUkJDz30UKOhd3ecccYZ3HfffSxZsgRwbnp85ZVXAJgyZQorVqzg+eefJxAIEAgE+Pzzz1m2bBkA1157LZZl8cADD0R7mL/55hvOP//8BtepqqrCMAy6desGODdQRq7Z3Hvy+eefs3DhQgKBABkZGaSmprZ4iEKPx9Ps921PpjAtIiIinc5ZZ51FcXEx48aN45577mHOnDnRYfF+85vf0L9/f6ZNm8bgwYOZNWsW33//PeDcqHjnnXdy0003MWTIEGbMmBE3MkbEF198wbRp0ygqKuKss87ipptuol+/fg32+93vfkdGRgYHH3wwJ5xwAieeeGKDcpDddcwxx3DBBRdwwQUXMHjwYI488sjoCCRZWVk8/fTTvPzyyxx44IGMHj2aW265hdraWgDGjRuHZVm89957HHLIIQwfPpyrr7660VKX4uJi5syZw/Tp0xk1ahSLFy9m3Lhxu3xPysvLueqqqxg2bBjjx4+na9eujYb1XWnu+7Yn06QtIiL7GE3aIp190haRjtTSSVvUMy0iIiIikiSFaRERERGRJClMi4iIiIgkSWFaRERERCRJCtMiIiIiIklSmBYRERERSZLCtIiIiIhIkhSmRURERESSpDAtIiIi+7TnnnuOE044IbpcVFTE6tWrO65BCbjjjju46KKL2vQal1xyCbfddluT2++55x6uuOKKNm3DnkBhWkRERDqVCRMmMHDgQIqKihg2bBhnnHEGpaWl7Xb9ZcuWNTq1eEtdcskl9O/fn6KiIoYPH86sWbNYvnx5K7SweXfccQcFBQU88sgjcesfeeQRCgoKuOOOO1p8zvfff5+xY8fGrbv44ov505/+BMCaNWsoKCigqKgo+pgyZUryLyLmnMFgcLfO09YUpkVERKTTeeyxx1i2bBkLFy6kR48eXH/99R3dpKScf/75LFu2jE8//ZTevXtz+eWXt8t1999/f55//vm4dfPmzWP//fdv0+suXryYZcuWsWzZMv7973+36bV2xbZtwuFwm19HYVpEREQ6rbS0NH7605+ydOlSAP7973/zk5/8hMGDB3PQQQfF9bLW1NRw0UUXMXz4cIYOHcqxxx7Lpk2bACgrK+Pyyy9nzJgxjB07lttuu41QKNToNQsKCli5ciXg9C7/5je/4YwzzqC4uJjjjjuOVatWRfddvnw5s2bNYvjw4fzoRz9i/vz5jZ4zPT2dadOm8c0330TXNfdaIr2ylmUxbtw4RowYwd13393ouQOBABdccAHnnHMOdXV1AIwePZrq6mqWLFkCwJIlS6itrWX06NHR4+qXt9R/7RFVVVWcccYZbNiwIdrrvH79+oRLTZp7j5p7D0466SQAhg4dSlFREZ9++mmDa9bvvZ4xYwZ//OMfmT59OoMGDWL16tUJf4+S5WvVs4mIiMge54ElD7CifEWbXmP/7P05f/D5LT6uurqa+fPnc+CBBwKQkZHB3XffzeDBg/nuu++YPXs2w4cP5+ijj2bevHmUlZXx6aefkpKSwjfffENaWhoAl156Kd27d+e9996jqqqKn/3sZ+Tn53PGGWfssg0vv/wyc+fOZeTIkdE64gceeICqqipmzZrFlVdeydy5c/nuu++YNWsWQ4YMobi4OO4cVVVVvPTSS/Tv3z+6rrnXEvHJJ5/wzjvvsGLFCo477jiOPfZYioqK4t6fOXPm0L17d+699168Xm9028knn8zzzz/Ptddey7x58zj55JOjf5S0REZGBk8++SQXXXQRn332WYuO3dV71Nx78OKLLzJx4kQWL16Mz+dE1rfffnuX13zhhReYO3cuAwcOpKqqiiOPPDKh71Gy1DMtIiIinc7ZZ5/N0KFDGTJkCO+++y7nn+8E8UMOOYShQ4fi8XgYNmwY06dP54MPPgDA7/ezbds2Vq5cidfr5YADDiA7O5tNmzbx5ptvctNNN5GRkUFeXh7nnHMOL7/8ckJtOeaYYxgzZgw+n48TTzwx2rv8xhtv0LdvX2bOnInP52PEiBEce+yx/P3vf48e+9e//pWhQ4dSXFzMJ598wj333BPd1txribj00ktJT09n+PDhDBs2jG+//Ta6rby8nNNPP53+/fvz5z//OS5IgxOmX3rpJQKBAC+//HK0p7ctjRw5kqFDhzJ06FAefPDBXb5HibwHLWWaJoMHD8bn87FgwYJdfo92l3qmRURE9nHJ9Bi3tUcffZRJkyYRCoV4/fXXmTFjBgsWLKCkpIRbb72VJUuWEAgEqKur46c//SnghMe1a9dywQUXUFZWxkknncTVV19NSUkJgUAg2rsNEA6Hyc/PT6gtPXr0iD5PT0+nsrISgNLSUhYtWsTQoUOj24PBICeffHJ0+dxzz+Xqq6+mtLSU0047je+//55hw4YBsHDhwiZfS0TPnj0bvXbk+GAwyP33349hGA3aXVBQQP/+/fnjH//IgAEDKCgoSOj17o6vvvoq2osM8Je//KXZ9yiR96ClYr+viXyPdpfCtIiIiHRaXq+XY489lquvvpqPP/6YW2+9lTPPPJO5c+eSlpbGDTfcwLZt2wCnZ/qyyy7jsssuY82aNZxxxhkMHDiQI488ktTU1AZBb3fl5+czceJEnn322V3uW1BQwM0338wll1zClClTSE9P55e//GWTryURhx9+OEOHDmXmzJk8//zzcaE/YsaMGVx++eXceeedDbZlZGRQXV0dXd64cWOT12osrCdiV+9Rc+9BY9dMpM2xx7Xke5QslXmIiIhIp2XbNq+//jo7duygqKiIiooKunTpQlpaGosWLeKll16K7vvee++xePFiQqEQWVlZ+Hw+PB4PvXr1YtKkSdx8882Ul5cTDodZtWrVbpcTTJkyhRUrVvD8888TCAQIBAJ8/vnnLFu2rNH9J02aRK9evXjqqacAmn0tibrgggs44YQTmDlzJlu3bm2w/fjjj+fpp59m2rRpDbYNGzaMpUuX8vXXX1NTU9PskHk9evRg+/btlJWVtah9u3qPmnsPunfvjsfjiRvze/jw4Xz44YeUlpZSVlbGfffdt1vXbw0K0yIiItLpnHXWWRQVFTF48GBuu+027rrrLgYPHsytt97Kn/70J4qLi/nzn/8cFxI3bdrEnDlzGDx4MJMnT+bggw+Ofpx/9913U1dXx+TJkxk2bBhz5sxptic2EVlZWTz99NO8/PLLHHjggYwePZpbbrmF2traJo8577zzeOCBB6itrW32tbTEpZdeylFHHcXMmTMb9Gynp6czadIk0tPTGxw3cOBALrnkEmbNmsVhhx3G+PHjm7zGoEGDmD59OgcffDBDhw5l/fr1CbVtV+9Rc+9Beno6F198MSeccAJDhw7ls88+Y9KkSRx//PFMmTKFY445ZpdjWSfzPWopw7btVjtZB7DXrl3b0W0QEdmj5OXlsXnz5o5uhnSgqqoqMjIyOroZIp1SU/8+3FrsBrUn6pkWEREREUmSwrSIiIiISJIUpkVEREREkqQwLSIiIiKSJIVpEREREZEkKUyLiIiIiCSp3WZANE3zaOBuwAs8YlnWH+tt3w94Auji7vNry7Jeba/2iYiIiIi0VLv0TJum6QXuB44BhgGzTdMcVm+36wDLsqwxwCzgL+3RNhERERGRZLVXmcd4YLllWSssy6oDngWm19vHBnLc57mAZmMRERGR3fbcc89xwgkntPuxiTr99NOxLKvFx3300Uf86Ec/aoMWdS7Jvj/tpb3CdAGwJma5xF0X60bgdNM0S4BXgYvap2kiIiLSmXz88cccf/zxDBkyhOHDhzN9+nQ+//zzjm4WAAUFBaxcubJVzzl37lxM02zxtSdMmMC77767y+Oee+45+vbtG52efcqUKbzxxhu71eb2lOj701HarWY6AbOBxy3LusM0zYOBJ03THGFZVjh2J9M05wBzACzLIi8vrwOaKiKy5/L5fPrduY/bsGEDPl9nigA7lZeX87Of/YzbbruN6dOnU1dXx0cffUR6enrSbfZ6vRiGkdTxjR3r8/k67P1L5tper5eDDjqIV155hXA4zNy5c7ngggv4/PPPyc3NbdX2hUIhvF5vq56zvaWmprbod2R7/SSUAn1jlgvddbHOBo4GsCzrA9M004A8YGPsTpZlPQQ85C7amzdvbpMGi4jsrfLy8tDvzn1bbW1tpw08S5cuBeD444/Htm38fj+HHXYYAMFgkKeeeoqHHnqIdevWkZ+fz7333svIkSO57777ePrpp9m8eTP5+flcffXVHHPMMYAT8GzbJhgMArB8+XKuu+46vvrqK7p168aVV17J8ccfD8DWrVu57LLL+OCDDxg0aBCHH3543LGRdsQuA5SVlXHdddexYMEC0tPTOfXUU7n44ovxeDyEQiF+//vfM2/ePLKysjj33HO57rrrWL16NT6fjxkzZnDSSSdx6qmnsnLlSq644gq++eYbfD4fhx12GA8++CAnnXQSAEcccQSGYfCnP/2JHj16cNFFF/HZZ58BUFpaym9/+1s++ugjwuEwJ5xwArfcckuD13/iiSdy5ZVXsmzZMkaPHk1tbS233XYbr7zyCnV1dRx99NHceOONpKenA/CXv/yFhx9+GMMwuOKKK7jyyiv573//y4ABA7jkkktIS0ujtLSUDz74gL/97W8UFxdz/fXX8+GHH5KZmck555zD2WefDcCiRYv4zW9+w4oVK0hLS+PEE0/kxhtvpKamhiuvvJI333yTcDjMgAEDeOKJJ+jRo0fc+xMOh7nnnnt4+umnqampYfLkyfz+978nJyeHNWvWMHHiRP785z9z++23U11dzTnnnMOvfvWrFv0M1tbWNvo7Mj8/v9H92ytMfwIUmaY5ACdEzwJOrbfPD8CPgcdN0xwKpAGb2ql9IiIi+6zq++4n9P33bXoN78CBpP/ywl3ut//+++PxePjVr37F9OnTOfDAA+nSpQsAr7zyCnfeeSePPvooo0aNYtWqVfj9fgD69evHiy++SM+ePXnllVe46KKLeO+99+jVq1fc+auqqpg1axZXXnklc+fO5bvvvmPWrFkMGTKE4uJirr32WlJTU1m0aBE//PADp512Gn379q3fzAauu+46ysvL+eCDD9i2bRuzZ8+mV69ezJ49m6eeeooFCxbwr3/9i4yMDM4999wmz3P77bczadIk5s2bR11dHV9++SUAL774IgUFBbzxxhsMGDAAgPfffz96XCgU4mc/+xmHHnooH330ER6PJ3psrFAoxHPPPYff76ewsBCAW2+9ldWrV/Ovf/0Lv9/PhRdeyF133cU111zDggULeOihh3juuefYb7/9uOqqqxqc86WXXuLJJ5/kiSeeoLa2lhNPPJGjjjqK+++/n3Xr1jFr1iwGDhzI5MmTueGGGzj77LOZMWMGlZWVfPfddwDMmzePsrIyPv30U1JSUvjmm29IS0trcC3Lspg3bx7z5s0jLy+PX/3qV1x77bXce++90X0++eQT3nnnHVasWMFxxx3HscceS1FR0S6/h8lql5ppy7KCwC+B14HFzirrG9M0bzZN83h3t8uBc0zT/AJ4BjjTsiy7PdonIiIinUN2djYvvfQShmFw5ZVXcsABB3DmmWeyadMmnnnmGc4//3xGjx6NYRgMGDAgGginTZtG79698Xg8TJ8+nQEDBjRaZ/3GG2/Qt29fZs6cic/nY8SIERx77LH8/e9/JxQK8eqrr3LllVeSkZHBkCFDmDFjxi7bHAqFmD9/Ptdccw1ZWVn07duXc889lxdeeAFw/gg4++yzyc/Pp0uXLlx4YdN/VPh8PkpLS1m/fj1paWmMHz8+ofdt0aJFbNiwgeuvv56MjIwGxy5cuJChQ4ey//7787vf/Y577rmHvLw8bNvmqaee4sYbb6Rr165kZWVx0UUX8fLLL0fbPnPmTAYPHkx6ejqXXXZZg2v/5Cc/Ydy4cXg8HhYvXsyWLVu49NJLSUlJoV+/fpx66qnR8/n9flatWsXWrVvJzMxk7Nix0fXbtm1j5cqVeL1eDjjgALKzsxtc68UXX+Scc86hX79+ZGZm8utf/5r58+fHfVJw6aWXkp6ezvDhwxk2bBjffvttQu9hstqt4McdM/rVeutuiHn+LXBoe7VHREREHIn0GLenoqIi7rrrLsApybjooov47W9/y9q1a+nXr1+jx8ybN4+HHnqIkpISACorK9m6dWuD/UpLS1m0aBFDhw6NrgsGg5x88sls2bKFYDAY93F+YWEhH330UbPt3bp1K4FAIBrsI8etW7cOcGrUY8/ZVLkAOD3ct99+O8cddxy5ubmce+65zJo1q9nrA6xdu5bCwsIm66kPPPBAXnrpJSorK7n88sujN3lu2bKF6urqaEkMgG3bhEKhaNsPOOCAZtseu66kpIQNGzbEvb+hUIgJEyYA8Kc//Yk//elPTJo0if32249LL72UqVOncvLJJ7N27VouuOACysrKOOmkk7j66qujnzxEbNiwocH7HAwG2bRpZzFDz549o8/T09OprKxs/s3bTZ3z7gMRERERYNCgQZimydy5c8nPz2f16tUN9ikpKeGqq67iueeeY+zYsXi9XqZOnYptN/yAOz8/n4kTJ/Lss8822BYKhfD5fKxdu5ZBgwYBTvjelW7duuH3+ykpKaG4uDh6XJ8+fQAn3EWCNTjBtyk9e/bk9ttvB5xRTWbNmsWECROipR1Nyc/Pp7S0lGAw2OwNipmZmfzhD3/gkEMOYdasWQwbNoy0tDTefPPNaHvrt2dXbTcMI64dffv25b333mv0+vvvvz9/+ctfCIfDvPrqq5x77rl8/fXXZGRkcNlll3HZZZexZs0azjjjDAYOHMjs2bPjju/Vq1f0DyZw3mefz0ePHj3i2tmeNJ24iIiIdBrLly/nwQcfjIa20tJSXnrpJQ488EBmz57Ngw8+yJdffolt26xcuZKSkhKqqqowDINu3boBzlBwS5YsafT8U6ZMYcWKFTz//PMEAgECgQCff/45y5Ytw+v1cswxx3DHHXdQXV3N0qVLmTdvXoNzBAIBampqog9wykxuu+02KioqKCkp4aGHHoreNDht2jQeffRR1q1bx44dO/jLX5qel+6VV16Jvvbc3FwMw8DjceJajx49+OGHHxo9bsyYMfTs2ZNbb72Vqqoqampq+OSTTxrdt2vXrsyePZs///nPeDweTjvtNG688cboTXfr1q3jrbfeirbdsiyWLVtGdXV19BODpowZM4asrCzuv/9+qqurCYVCfPfdd9GSmxdeeIEtW7bg8XjIyXGmFzEMg/fee4/FixcTCoXIysrC5/NFX3esE044gYcffpgffviByspK/vjHP3L88cd36Og0CtMiIiLSaWRmZrJo0SKmTZvGoEGDOP744xk8eDA33HAD06ZN4+KLL+bCCy+kuLiYs88+m+3bt1NcXMycOXOYPn06o0aNYvHixYwbN67R82dlZfH000/z8ssvc+CBBzJ69GhuueUWamtrAbjllluorKxk9OjRXHrppcycObPBOY444ggGDhwYfTz33HP87ne/IyMjg4MPPpgTTjiBE088MVqecdpppzFp0iSmTJnCUUcdxZFHHonP52t0RJUvvviCadOmUVRUxFlnncVNN90ULW257LLLuOSSSxg6dCjz58+PO87r9fLEE0+watUqxo0bx0EHHdRgn1i/+MUvePPNN/n222/5zW9+Q//+/Zk2bRqDBw9m1qxZfO/ekHrkkUfy85//nFNOOYVDDz2UAw88EICUlJRGzxtpxzfffMPBBx/MyJEjueKKKygrKwPgrbfe4ogjjqCoqIjf/va3/OUvfyE9PZ1NmzYxZ84cBg8ezOTJkzn44IM5+eSTG5x/1qxZ0dE9Jk6cSFpaGr/73e+afJ3twWjsI5A9iN3cRyUiItKQhsaTqqoqMjIyOroZ+6w333yTX//613z88ccd3ZQWW7ZsGUceeSQrV67stGOV766m/n24teFG/fXqmRYRERFpQ9XV1fznP/8hGAyybt067rzzTo4++uiOblbCXnvtNWpra9m+fTu33HILU6dO3WuDdDIUpkVERETa2B133MGwYcM46qijKCoq4sorr+zoJiVs7ty5jBo1ikMPPRSv18sf/vCHjm5Sp6IyDxGRfYzKPERlHiJNU5mHiIiIiEg7UZgWERHZx+zhn0qLtKmW/vtQmBYREdnHeL1eampqFKpFYti2TU1NTaNDFjZHt2KKiIjsY9LS0ggEAlRVVQHxM9iJ7Isif1impKQ0mMJ8VxSmRURE9kF+v7/FoUFEGkooTJummQOcDgwBSoGnLcta05YNExERERHp7HZZM22a5lTgEyAX+CewHXjVNM3RbdoyEREREZFOrtlxpk3THAC8DkyxLOuHmPUHAjcDxwP3A5dZllXdxm1tjMaZFhFpIY0zLSLSck2NM72rMo+rgVsty/rBNM33gD4x2yotywqbplkDXA78vrUaKyIiIiKyJ9hVmccU4P/c528AfwCGA38E/uGuvw+Y2SatExERERHpxHYVprsCZe7z84FH3XKOR4BT3fUrgMK2aZ6IiIiISOe1qzC9FhjoPl8NnGWaZhpwNrDeXV8AbGyb5omIiIiIdF67CtMvAT93n58NnANscddF1p/KzpIPEREREZF9xq5uQLwD+Ng0zX9blvUmMDF2o2mahwPn1l8vIiIiIrIvaHZoPADTNIcAzwPvAs/iTNrSBzgF5wZF07Ksr9u4nU3R0HgiIi2kofFERFquqaHxdjlpi2VZ3wFjgA+AX+CMK30e8AUwpgODtIiIiIhIh9plz3Qnp55pEZEWUs+0iEjLJTtpS5Rpmj8BRgNZsesty7phN9smIiIiIrJHSihMm6Z5H2ACC4CqNm2RiIiIiMgeItGe6VOBUZZlrWnLxoiIiIiI7El2eQOiazOwvQ3bISIiIiKyx0m0Z/oO4CnTNP8AbIjdYFnWilZvlYiIiIjIHiDRMP2A+/W4euttwNt6zRERERER2XMkFKYty0q0HEREREREZJ+R8NB4AKZp7gcUACW6GVFERERE9nWJDo3XB2cq8YOBLUB30zQ/BGZZlqVZU0RERERkn5Ro+cYDONOHd7Usqw/QFVgEPNhWDRMRERER6ewSDdOHAZdbllUJ4H69CjikrRomIiIiItLZJRqmtwHD6q0bjMaeFhEREZF9WKI3IP4/4N+maT4KrAb6AWcB17dVw0REREREOruEeqYty3oYmAnkAdPcr6dalvVQG7ZNRERERKRTM2zb7ug27A577VoNJiIi0hJ5eXls3ry5o5shIrJHyc/PBzDqr2+yzMM0zWsty7rFfX5zU/tZlnVDazRQRERERGRP01zNdGHM875t3RARERERkT2NyjxERPYxKvMQEWm5Fpd5xDJNcxiwxbKsDaZpZgFXAmHgdsuyqlqzoSIiIiIie4pEx5l+BujiPv8TMAmYCPy1DdokIiIiIrJHSHSc6f6WZS0xTdMATsKZwKUaWNlmLRMRERER6eQS7ZmuMU0zGxgP/GBZ1magFkhrs5aJiIiIiHRyifZMPw28CWQD97nrDkQ90yIiIiKyD0t0BsRLgWuB8y3LioTpMHBpWzVMRERERKSz09B4IiL7GA2NJyLScsnMgPhPy7KOdp+/CzSaui3LmtRKbRQRERER2aM0VzP9vzHPH9ndC5mmeTRwN+AFHrEs64+N7GMCN+IE9y8syzp1d68rIiIiItJW2qXMwzRNL7AUmAqUAJ8Asy3L+jZmnyLAAo60LGubaZo9LcvauItTq8xDRKSFVOYhItJyuzsD4j3As5ZlvR+z7hDAtCzrkgROMR5YblnWCvfYZ4HpwLcx+5wD3G9Z1jaABIK0iIiIiEirqqmpYfuO7WzYsYGN5RvZVLEJI9Pg8uMvb3T/RIfGmw1cUW/dZ8BLwCUJHF8ArIlZLgEm1NunGMA0zfdwSkFutCzrnwm2T0RERERkl8LhMJWVlZSXl1NRUcH2su1sqtjE4NGDqQpX8fnSz/m+9HtsbAwM0lPTKUwpbPJ8iYZpm4bD6HkbWbc7fEARMBkoBN4xTXOkZVnbY3cyTXMOMAfAsizy8vJasQkiIns/n8+n350islcLBAKUl5dTVlZGeXk5O3bsYNSoUaRmpPLJ15/w30//S7VRTY1RQ9AbJC0tDX+Nn5zMHIYVD2PU/qPI75pPQbcCslOyMYwG1R1RiYbpd4Hfm6Z5lWVZYdM0PTg3Cr6b4PGlQN+Y5UJ3XawS4CPLsgLAStM0l+KE609id7Is6yHgIXfRVt2fiEjLqGZaRPYG1dXVVFRURHuY+/btS9euXSktLWXBggUECVLrqaXWU0soJcR3fEdKVgrVNdV4+3opzCykZ3ZPemT1IMefQ25KLmne+Mm968rr2MIWIFoz3UCiYfpXwN+BdaZprgb2A9YB0xI8/hOgyDTNATghehZQf6SOl3DKSR4zTTMPp+xjRYLnFxEREZG9SDgcpqqqKhqWu3btSl5eHjt27OCf//wngUAgum/ACFCTUkMXTxc2spFQcQhvqpduad1ITU0lJy0nGphz8nLI9eeS4k1plXYmPJqH2xs9HqeHeQ3wsWVZ4UQvZJrmscBdOOUhf7Ms6xbTNG8GPrUsa75pmgZwB3A0EAJusSzr2V2cVqN5iIi0kHqmRaSzCAaDVFRUUFFRQVpaGnl5eQQCAV599VUqKysJh3dGzeHDhzN69Gh21Ozgwy8+hHQIp4YJ+AJ4U7x4PE71cbY/m2x/Nrn+XHJSnBDt9/h3u61NjebRkjDtByYC+ZZlPWeaZiaAZVmVu9265ClMi4i0kMK0iLSn2tpaKioqsG07er/Gm2++ybZt26iuro7uN2DAAA499FAA/vvf/5KekY4v04edZhNKCVHrraU8UE7IDgFgYDihOSU32uuc7cvG6/G2yevY3aHxRgLzgVqceufngMOBnwEzW62VIiIiIrJHsW2bqqoq6urq6Nq1KwCffvopGzdupLy8PFqO0bNnT37yk58AkJqaSp8+fcjOziYrK4vMrExIgx8qfmBHYAdGscGmwCbCttMz7Q17yfHmUJhZ6ARnfy5Z/iw8RmuOhZGcRGumHwBusCzrSdM0t7nr3gYebptmiYiIiEhnEQqFqKqqIjs7G4ClS5dSWlpKeXk5lZWVhEIhsrOzmT59OuCUb6SmppKXl0dWVhbZ2dnk5OQ428JBhh40lLJAGTvqdlASKKG8ttzpsgV8Hh85/hz6ZfYjJ8UJzpm+zGZH1OhIiYbp4cBc97kNTnmHaZrpbdIqEREREWlXdXV1pKQ4N+WVlpayZs2a6M1/lZWVeL1eZs2ahWEY7Nixg6qqKrp06UJhYSHZ2dnRoA0wceJE55yhOic0B3awom4FO9bvoCpYFd0vxZNCbkouPdN6Rss1MnwZ7fvCd1OiYXoVMBb4NLLCNM3xwPI2aJOIiIiItAHbtjEMg61bt8aF5fLycmprazn55JNJT09n69atlJSUkJ2dTc+ePaPlGJHjx40b1+DcNaEaNlZvZEdgR7TXuSZUE92e5k0jNyWXgoyC6M2B9Yei2xMlGqavB/5hmuaDQIppmtcA5+FMAS4iIiIinUhlZWW0DCN2LOapU6fSvXt3tm7dytdff01mZiZZWVnst99+ZGVlRUfEGDFiBCNHjmzy/FXBqmhgjnytC9dFt2f6Muma0jVappHjz2m1oeg6m5aM5jEGJzz3wxka72HLsj5rw7YlQqN5iIi0kEbzENnz1dXVsWHDhriwXF5ezkEHHURhYSHr1q3jP//5Dz6fj8zMzGjP8uDBg8nOziYUCmEYRjQ8N8W2bSqDlU5vc11ZtNc5GA5G98n2Z+8cw9nvDEXn8yTaX7vnSHo0D9M0vcBSYJhlWRe0ftNEREREJFY4HGbLli0NepYHDhzIoEGDqK6u5u233wYgJSWF7Oxs8vLyojXPPXr04OSTTyYtLa3RG/e83obDx4XtMBWBirgyjdih6DyGhxx/Dn3S+0TLNLL92XiNthmKbk+xyzBtWVbINM0QkEb0PksRERER2R2xITnytWfPngwZMoRQKMTrr78e3TdSjuHzOdEtOzubY489lqysrGiAjuXz+aL7NiYUDlEeLI8r0ygPlGM740zgNbzk+J2h6CLBOcvXOYai62wS7YO/C7BM07wVKMEd0QPAsixN+S0iIiJST2R2v0gJRkVFBRkZGYwYMQKAV199lbo6p87Y4/GQlZVFt27dAPD7/Rx55JHREF2/J9nj8UT33ZVAOEBZoCyuTKMiUBHd7vf4yfHnMCB7QHQM5wxfRqcdiq6zSTRM3+d+nVpvvY0zPbiIiIjIPqempiauZ9kwjGhYfv3119m2bVt0X7/fT2FhYXT54IMPxu/3k52dTUZGw/Dq1ui2SF2oLq5MoyxQFjcUXao3lRx/Dr3TekdvDkz3aaTj3ZHwDYidlG5AFBFpId2AKJK4yOx+kbBcU1MTDcvvvPMOP/zwQ9z+Xbt25ac//SkAq1evJhwOR2/+S0tr3WHgakI1cWUaZYGyuKHoMnwZznTbbplGrj+XVG9qq7ZhX7Jb04lHmKZZAOQDpZZlKcWKiIjIHi8UClFRURHtYS4uLsbj8fDll1/yzTffEAqFovt6vV6GDRuGx+OhX79+5OXlRcNydnZ2XJ1yv379WqV9tm1TFaraWabhfg2EA9F9Mn2ZdEvtFi3TyE7JJsWzdw5F19kkFKZN09wPeAo4GNgKdDNN8wPgdMuyVrdh+0RERER2W11dXTQs9+nTh5SUFFasWMEXX3xBZWVl3L6FhYXR+uXIUHKRsJyRkREdTq61wnKssB2mMlgZV98cOxSdgUG2P5teab3ixnD2elR121ES7Zl+AvgMONqdRjwL+J27fnIbtU1EREQkYdXV1ZSXl5OdnU16ejobN25k4cKF0dn9IqZOnUqvXr1IT0+Pm90v8jU93akhLiwsjKtxbm0hO7RzKLq6smhwDtthwBlRI9ufTX56fjQ4Z/mz9vmh6DqbRMP0WOAnlmUFACzLqjBN82pgS5u1TERERCRGOBymoqICv99Peno65eXlfPbZZ5SXl1NZWUkw6PTeHnrooQwYMCA6PFxkdr9IWM7NzQWgT58+9OnTp13aHgqHnNrmmDKNikBFdCg6n8dHjj+H/TL3i9Y4Z/oyNRTdHiDRMP0hMB54L2bdQcAHrd4iERER2WcFAgFCoRBpaWnU1dWxaNGi6M1/lZWV2LbN6NGjGTFiBF6vN9oT3adPn2hY7t69OwDdunVjypQp7f4a6sJ1lNeVx42qURncWUri9/jJ9efSI7tHNDhneDUU3Z4qodE8TNN8ADgV+AfOVOJ9gWOBp4HoLeGWZd3QNs1skkbzEBFpIY3mIR0tGAxGb9T76quvKCsri9Yz19TUUFxczPjx4wmHw7z44ovRsZZzcnLIysqiR48e5OTkdPCrcNSGauOn2q4rozpUHd2e5k2Lm2o7NyWXNG/rjuoh7WN3R/NIA150n/fEmQnx/4B0nGANMRO5iIiIiACsXLmSbdu2xY3F3KtXLyZPngzA8uXLAcjKyore+NejRw/AmZhkxowZHdX0BqqD1fHBOVBGbWhnLXaGL4PclFz2S9kvOqpGilcjauztNM60iMg+Rj3T0po2bNjA1q1bqaioiPYwp6amcvTRRwPwz3/+k61bt8bVLOfl5TFgwADAqYOOjI7RWdi2TVWwqsHkJ7FD0WX5s6KBOSclhxx/Dn6PvwNbLW2tVcaZFhERkX1LWVkZW7ZsietZrqur47jjjgNgyZIl/PDDD9GZ/Lp27Ro3zfURRxxBSkpKk/XAHR2kw3aYimBFXJlGWaCMkO2MLR0Ziq53eu9omUa2L1tD0UmUwrSIiMg+rKamhu3bt0fDciQw/+QnP8Hn87Fs2TIWL14MQHp6OtnZ2XTr1i3ao3zQQQcxfvz4Jmf3S03tPDPuhewQ5YHyuDKN8kB5g6HoCjMLo73OWf4sjaghzVKYFhER2YuFw+FoQI4Ny+PGjSMrK4uVK1fy2WefAU4vceRmv0AggM/nY/DgwQwcOJCsrKy42f0iMjIy2vslJSQYDkbHbY6UaZQHyqPbI0PR9cvsFx3DOdOXqRE1pMWaDNOmad5uWdaV7vMjLct6s/2aJSIiIomKnd0vEpaLi4vp1q0bJSUlvPPOO9F9/X4/WVlZ1NXVAdC3b1+6dOnSYHa/iKysrHZ9LcmoC9U1GMO5KlgV3Z7iSSE3JZeeaT2jo2pk+DrnHwGy52muZ3oOcKX7/CWgc4xBIyIisg+KzO4XCct9+vShZ8+ebNmyhddeey1u37S0NAoKCujWrRs9evTgkEMOaTC7X0RWVtYeEZgjakI1cWUaO+p2UBOqiW5P86aRm5JLQUZB9OZADUUnUXYYwkGMcND5aocgHNi57H6NPI9dJr/xkWWaC9NfmKb5PPAtkGqa5s2N7dQBY0vHqS7bgseXgj81HY9XVSsiIrJniszuFwnL2dnZ5OfnU1NTw0svvRSd3Q/AMAz8fn90KuwxY8bETYnt9+8cVSI9PZ3999+/I17SbqsKVsWVaeyo20FduC66PdOXSdeUrtEyjRx/joai25tEg28gJuiG6i1HvgbcYByzHN0/Ztmtj0+IYWB7/ODxYXuazpjNpc8ZOL3T/XCGAenbyD4dPq7e8hdv2blgeOneoxcDi4qxPT4WffEVeHzg8ePx+TG8Prr36E3vgkJsw8f3K1fh8aU4D38qXn8qWTldyMrpQhgPFdW1+PypeFNS8fnTMDrZ0D0iIrJnCQQC0bDs8/kiQ20xf/58ysvLiR2udtCgQeTn55OWlkZRUVFcWM7MzIyWY6SkpDB8+PAOeT2txbZtKoOVDcZwDoZ3/gGR7c+mR1qPaJlGjj8HXzMBR9pZvZ7dpIJutKfYDdAtGL7ZNrw7Q6/7sD1+8KUT9vjB441uiwbk6DExgdnjjS6T4I2nTf4UWpa1Efg9gGmaPsuyzkr4FbWjrqOnEQ7WYgcDhAI1pGZnEcrKg3AQT2o2oWAdBGsJ1lZghwLYvlp8mQHCgVrKFn/S4Hy+ggLyCvtSV1fHykUL4zd6/PTt158+BX2pqQvxzeLvMHwpGF4/htePx+unT+F+dM3rQU1diNJ166Nh3etPw+Pzk9stj7SMLAIhm7qQ7fSqpzjbRERkz1ddXU1FRQWhUIjevXsD8M4777Bx40ZqanaWI/Tu3TsapvPz86O1zNnZ2WRnZ8eVY4wdO7Z9X0QbCtthKgIVcWUa5YHy6FB0HsNDtj+bPul9omUa2f5svIaGomsVtg12w9AbH4ADMUE3pmc4NujWD8wtaUI01LpfDR+2NxXbnxkTbhuG353r/PHnMHzQgTeOJjxpi2maXYFpQAFQCvzdsqytbdi2ROzWpC3BulpCgRpCgTqCdTWEg7WkpfjJTPMTrKtlXekaQoG6mLBeS68e3ejWJZfaqnKWLP6WcCgAoTrsYAA7FKR/v0J6du9KedkOvv32mwbXHDSoiO7du7Njxw6++27xzg2GBzw+iocMI7drd7aXVbBy9Rrw+vH4UjC8PgxvCv0HDiIjM4eyymo2b93mhnWn99zj9dM1rye+lDQCIZtgGPWqi0gDmrRl94TDYaqqqqiuro7O1PfFF1+wZs0aKisrCQSciT1ycnI4/vjjAfjss88IBAJxYTkrK4uUlL27JCEUDlEeLI8r0ygPlGO7H2x7Da/Ty5yyc/KTLJ+GoouqX98b7b1tLPw2XQbRoEa4JU2I67Vt6qvf6eX1ul/rh916PcZ7qqYmbUkoTJumeTDwD+A7YDWwHzAU+KllWR+0aktbptPOgGiHQwRqnYAeCtQSCtQRqqshKzOdNL+XmqoKNq5fSyiwM6iHg3XsV5hPVkYa27duYvXK792Q7vyjsEMBhg0pJis9lY0bN7Jy5YoG1x058gAyMjJYt24dP/yweucGwwdeP6PGHEhKWgbrNm5m3YZNTo+6LwWP14/hS6F4yDC8/lS2bC+jrKIyrlfd40+lR68+2IaXumCYsOF1ymD8e/d/BiJ7G4XpXQsGg1RUVNClSxcAvv/+e1avXk15eTmVlZWEw2F8Ph+zZs0C4PPPP2fbtm1xYTk7O5ucnH3n3v1AOOAMRRdTplERqIhu93v80UlPImM4Z/gy9p6h6MKhnUE31NKyhoY3wLW8vtfTdNCN9gA3H3Rje4ojPcGy0+7OgHgXcIFlWc9GVpimORO4BxjXCu3b6xgeLynpmUBmg20hwJ8NBb0GN3psHZBRAENHNn7uKiCjKEDxQVVOSHeDeLCuBjs3h1qPjS9zK127btoZ1t3edbJ6E/Ya2J4yAOy6SoJV27HDQexQEH/vFLwG1KxaxbYN6xtcu++EiQCsW7GCTZs2Rl4teP14/WkceNA48PhY9UMJ28rKMbwpbr16Cilp6QwqHort8bJh0xZq6oJOWE9Jw+P140/LoEu3PPD4qA0EMXwp6lUXkTZTU1NDSkoKHo+H9evXs2LFiugNgNXV1QCccsoppKamUl1dTU1NDV27dmW//faL9izbto1hGIwePbpjX0w7qwvVNZhqO3YoulRvKjn+HHqn9Y72Oqf70ps5YzurX8+bTNCt31OcdH2vE27j63sb1u/WD7oNenzVm99hEg3TxYBVb93zwIOt2xxJlMfnJ9WX2+i2EJCZ05fM/RpuC+OE9W6F0K3hZmoBbJtuA2vIqa3e2aseqCUcqKU2rxuEg6T6+tOl9zbCgTrCwTpCwVq82ITTu2PYIcJ4IBQkVFtFyO1Vt31e/N2d65R/+y3l5WVx105Pz6DPAQcAsOKbr6mocHs0DB94fWTndmXIsBHg8fLd0u+pqXMCd6RXPSsnl7799sf2eFlTuh7b8Lg966l4/KmkZ2aTmZ0LHi+1gRBef5p61UX2YrZtY9s2Ho+H7du3s3LlyrjJSwKBAMceeyzdunWjoqKCdevWkZ2dTZ8+faJh2et1euZGjBjBiBEjOvgVdYyaUE1cmUZZoCxuKLoMX4Yza2BGYTQ4p3pbadZD22725rWkR3poSRMaqd+1fWm7vHkt/ka4zlPfK60v0TC9DJgFPB2z7hTg+1ZvkXQ8w8CXko4vpWEvQqTSqmtuP7o2cmhkwKL8wkPJb2R7FUA4RH7/Ywm6YT3Sq+4lTG3XLhAOkBnujbeqIhrWw8E6UtL8hFNzMewQoXCYcF0VdkyvelpFJv5Mpzdp+5efEQwG4q7dvXseeYMGAfDNp58QCoWI9KobXj95PXvTf/9B4PGy6Muv8XhTMNxedWd7L/J69SGMhzWl63aOBONLwZeSRkZ2LmkZWdiGh0AIfCnqVRdpL9XV1dEyjNjh5Q4//HAKCgqorKxk8eLF0dn98vLy4sZcHjRoEIPc3w/7Ktu2qQpV7SzTcL8Gwjt/l2b6MumW2i1appGdkk2Kx+2UiB3GLFjb+M1roUAj9b6Nj/5AONSy+t7YYcxien5tf0ZM0G3k5rVIvW+DsgjvHl3fK+0n0Z+SS4C/m6Z5MU7NdH+gCDiubZolezWPl5T0LFLSG04SEPm12bPLgEYPjYT1gX1/1Oj2Kvcu5QGFU6M3lzr16tWk+n3U5mZhhIPkDO/i9rY7Qd0OBfDnZmOnZDk16qEgwbpqCDk163YoQCjYkxTPFgKBAFsXfdbg2oWFfelaUEBtbS1ffb7IWen2qhteP337DaBXn3yqawMsWbYCw+d3A7sT1vsU9iWnSzdq6oJs3LzV7VVPweP2oGdld8Gflk7I9hDGwJeiSQhk3xEMBiktLW0QlkeMGEFRURE1NTV8+umn0RExunTpQmFhYXSq6z59+jB79uy9pz53N4VDdVTWlVNeu5Udddsoq9tOWd0OguE6sEN4bJtsTyoF3nRyPKl08aSRa/jxBWyo3YZhb254A1wLyhx21vf643t0fWmEm6jfjR/ZoZGRHlTmIDFs24Zw2P10o+FXu5lt0X1sG0Jh9w9FG/Ib6yZs+WgePwXygbXAq3v6aB4iLRYOYYeD1FRVEKytdkpcArWE6mrIzkwnMz2VQG01JT+sIhx0hmC0Q87XPr170r1rLlUVZSz59mvsUBA7VIcdcv4zGjigH3l5eQ1HenEVFTlTA2/fvo0lS5YQ7VX3+DB8KRQNHkpOl65s21HBDyVrwevD40t1bzL1s9+AgaRlZFFRVcO2HeVxveoeXwrZXbrh9acQsg3w+DQJ0l6sM96AaNs2GzdujN7gFwnLhYWFjBgxgrq6OizLqTZMS0uLlmEMGDCA/Px8wuEwtbW1DWb32+PFDWPWgvrdmOVwqJbyYBVlgQp2BCvYEaqgPFxHyL25zWt4yPakOIHZfWR7UvC64XRXN69Fe4CbKmto0POr4LsniIbRJsJmgzAaCrnLNthhbPdrU+ewY/alkX3jzh8ON9i3ybAc2bcNpkLpP2cOJDuaRyemMC17D/c/zXCgjuqqCkJ1NdGhGUOBWrrmZpOe4qOibAcb1pXEh/VgLfv370dmeipbN21g1crv3R72yGgwIUaOGEZGRibr169n9epVDS4/atRo0tLSKC0tpaRkDbjDHBkep2d95OgD8aems37jZjZu2eqG9J1hfVDxEAyvn21lFVRW1zq96pEbTH0p5Hbtjm34CGPg8alWvSN1VJiOTIUdOyV2ly5dojfvPffccwQCAQzDiJZj9O/fP1p+ERktI3Z2v05lN6Ypjh/Xt94NcC0QMLyUEWS7HWCHXcv2cB3ldoCwYYDhxefxk+PLjhmKLpdMfzaGNyUu/EZ7flXfm7RGez6bCouNBNdmw2JzxzcIs418DYexm9wWatmnDLvDMMDjcb96wWO4QwUbGO5XZ7snbhuGB6PecnRf93xG9LyemGvs/GrUW66/r9HItoIBA2A3RvMQkbZmGGD48aT6yUxtOAoMQBBI6wL99hvT6PZaIHM/GN7Y/A52mKpwkPT9qhgwssotg3FuMA0Hagn36E6dx8bn34/szP0JB+qwQ3VOaA/W4knNBMMmVFdFsHwLdjjgjBDj/uef0tUJQVUrV7Jp44Z6L83D+PHjAVi+fDlbtmwGj1OrjteHPzWDUWMOxDa8rPqhlLLKKry+FHBHg0lNz6Tf/oPA8LJpy3bqQuFoGYzXn0ZKWgYZ2TnYhhfb8GJoOKcOETu7X+wsf5EJRxYsWEBZmXPjsc/nIysri9zcnTdSH3HEEaSnp8fN7hera9fG7tRIUmebptjt+d1Z39vMeL7ucq0dpjxUw/ZQJWXBSnYEK6gMVkYv4ff4yfXn0jdmDOcMb8Oh6NqmD2/3JBUkY3tNYz+aj/3q9qA23itar+czNlhGzx/Z1nSPbfRrmzOaDJLRsNgghPrB6+zjaTJIRsJsE2HTDb7xYTZmm9fbZBiN/bo33VOkMC2yrzA84E3Bn5mCP7NLo7sEgeyug8hu5D6soPvI6wd59bbZ4TDVOOGkS2El6VUV0V71cNCZ1Ki2d0+McIDUcE+yum3Fdm8sDYcC+AzbCRbhEKHqHQR2bCEQ06seSk0hJdMZ3WVHIyPBZGRkMnKkM5bkV199RVV1Dbg96nh95HTpRtHgYeDxsmTZCieMu/XqHp+frJwu9Cnoi+3xsX7jZvB44yZESknLIC0jWzcksXN2v0hYDgQC0bD8zjvvsG7duui+qamp9OzZM7o8btw4vF5vg9n9ImL3jbOLG9XaZ5ri+iUO7TNNcURtqHbnTYE1ztfqUHV0e5o3jRx/DvkZ+c4kKL5s0jyp8QEvYENtlTPKSYOP5uuHxcaCZFMf0TcMljvP33iobRCW63+M3x7xvtmw6HHDYExY9HrB52zzNNFbGt+zujNYNtqz6k0gbO6qZ1U6BZV5iEinFw4F8RDGCAepqiinrqZyZ696sA6fYdOrR3cIB/lh1QpqqircsB4gHKojOyONAf36Ytghvvp8EbU1lW4JjBO2unXrRlFRMQCfRkd62SkvrwcDBw4E4OOPP3Z6v92gbnj89OidT99+/Qnj4evFS+KCuseXQre8nnTL60nINli/cTMed7IjZ3jGVFLTM/CnZbRbLWlzZR7hcDiuZrmiooIxY8ZgGAYff/wxS5cujds/Ozub448/HsMwWFtaStC9fyArI5VUn6eRINz+0xQndPNavZEebDzYeADvrutFmwybTe/TXD1odbiaHcFKysIVlIWdr7V2wA2ZYdLtFHLsdHLstOgjJWy0c68ojX40H/8RfdNh02imRzWpj+h3dfxe3Csq7Wd3J23BNE0/MBHItyzrOdM0MwEsy6ps/kgRkd0TuRnS9qaQ3jWDxm4xi0Su/O6NT4ZU634t6j+lwbZwsI4qbIxwkMLuEwnV1UZ71UOBOjJS/dR1yYFwkMxBPsLBAHawlnAogB2sw5uS7gSkQBWBHevBvbnU6dkLkVNRSEqwkLq6OjYuWtjg+vvttx99+uRTXV3Nl1997QQUd8hGw+tnv/4DyevZi6qaOpavXI3XFwnrTmDv1buAzNyu1AaCMTeXOr3qXn8q6RlZePypTh28YRAMBtm2dQuV5TuoKNvOoP77keLzsHzpd3zz1RcY4SAeQngI4/eA3SNEqt/L0LTN9BvoIzMthfRUP+kpPuePnGXzIRxkkFvfa1fb7j0ANoRxw2PMOvcRxgduaY6Nx+nNdUOsbfixSQW8bqj1YGPELBvYhie6P+46wjZ2kz2qdRCu3uVd/u31Eb3tgSojSJm3ljKjxnlQTcATxsDpDc3ypNPdyCbHm0lOShbZ3kz8Hn8jH6HXrzdtrh60fr1pw9DZ7Ef06hUViZNQmDZNcyQwH+f/o0LgOeBw4GfAzDZrnYhIO4jcEGmTSk7PxuvVA24A7D/R6cGOC4ZAjft8eOHkuO3hkDO1cCU2drCW/dJHRYdtDLv16r6sDGpS0qkLVpCdFyYcDDo966E6qAtg7KgkbG8iVF5GePUydyQY9+Y12ybcrz+e3FyCZWVsWbkyPgza0L9fP7Kzc9ixYwc/rCnF7/MSrHUm3TBsCPcfgJGWRkFlJbmV5aT4/fh9zsPn9WJ88BYBPOTgITsm2NbiwbYNdgbdyFcD2+N1w7vHDcxuUMNdt8uewZD7CDS+udmgV78n1P243e8ESU8zH803+vF77D7eZm5cqt/zGbMujE1luJodwTLKghXOtNuBMmf0HPwYZDsTn0Sm2k7JJduXjVf1/yKdXqI90w8AN1iW9aRpmtvcdW8DD7dNs0T2XXa9kFY/tCWybCd1PMDObXa95frbGyw3cm470WObWbZbfLzbjphlO8lrxy3vpkjhSKMDt20towanDjyfHuDFeURUQ2U1QBoD8+PrikPBAAZQWRbGDuTSO6+7E8bDQcKhoPO8OpPqgJdwVQZZ5JGZmkko1SAlPYPU9Aw8KWnUerx4c33kdvNhuyUsYa+POq87eUVjH9E3UfNpROtNm6hFTfQj+ki96R72EX3IDlEWKN85+UmgjPJAOeHoUHReJzhnFkYnP8nyZ+HRcHGylwm7/5c4v5Ztp1IJ90Mg247+urax437lht3/Q3Z+oGVHb5S13XNGzhV3HtsmDO7v/frXjzlPzDXDdvx5o8/ZWb4fdvc9pYlxphMN08OBue5zG5zyDtM097IBPfctCm17Z2jrfAynwsxwewhbuGzELrsffccvs7PXsIlzeZK8duxy9GPtyLb6ywmcy2h2ezPnjlk26i37Y5b9hkFaM8dnGgaZhkGPHj063TjTe6pgOBjtZY5MtV0eKI9u93l85Phz6JfZLzocXaYvU2USe5FoYHP/D2hJYGws6IXd/wsiQTJ6jZj9WhIYwzHXrB8YI0EyNjDWD7SR1wFNB8+488QE387GAxiGEfer13B/N3uivzYNd796+zZz3kTD9CpgLPBpZIVpmuOB5S1/Ka0rvHWrQptCm0uhrS1DW0uPj3stInuBulCdE5pjptquClZFt6d4UshNyaVnWk9y3XKNDF9GB7a49ewyMEb+q4oJZPFBsm0CY/3zJBsYw3a942LOE3eNyHn2oMDohMT4wOiJC5TxgdFjGHgM8Hrc9e6xRuT/ONz/CnEO8sRew90W+f8w9ryGYcRdI3qemEAbu18k0IIRs58bdmP+q/HEtC/2PB4j8v/yzuPaSqJh+nrgH6ZpPgikmKZ5DXAecE6btSxBwfff74CrJh+UFNoaLjcZ2iLrIh/pNrHc2T/yFZHOr/7HutWhanbUOT3O2+t2UFa3g+pQjRuebFI96WT7cuid0ossfw5Z3hxSvGk7g2AQdgRgB7UxQS/+4+nYwNiS4BkbEiE+MMb1SrZCYGzByNntysPOsBQNcLGhKxLk2LlcP0jGBcaYdZH/G2NDWFwojdkv9r/vuABHvfPEBL24cEu99sZcM+646H71A20kE7R9YJSmtWQ68TE44bkfsAZ42LKsz9qwbYmwS774ouWhK8H9FdpEZG8Ttm26devO5s1b4noY43se44Of3SDouduI9CrauxUYowEu9jxxQXL3A+POHtDY8zjbasLV1ITLqQmXU+1+DcXc+JhipJPmyY55ZOEz2nYWz0gQrB8Y4z+ijukJdPc1Yp8TE8DY9XkiQa+xXsKd2w23RzKmBzQ2SMacJxpuGzmPEdPe+oEx/nXEtEeBUTpYU0PjaZxpEZF2EgrbTm+fHXnuhMC45+4224ZgE/s7x1Bvm/M86B4bCtuE7JhjwhByQ2hWVhYVFRUd/XbEBcb6YSmuRxHiQ2K93juID4yeJs6DbVNnV1EVKqMqVE5lqJyqUBkhOxQ9Z4Yvi2xfDtm+HLL8OWT7svF5/A0CY+M9oM19lF4/0NYLjEZ8mBSRzme3xpk2TfPmprZZlnVD8s0SEWl7thtSw24gDbmBNBIww+62UL3gGrLt+GPCOwNv2A2woWggbiQYxwbeVuq48LkhzOtx6xoNA0/Mc7/XwGMYeN1Q5vHsfB45pmvXXMp22DtDKfFBML5nsn4P6M6PnyGxwNjYedq6dzFsh6kIVERH09hRt4PyQDkhQuADr99DgT+bHH//6FTb2f5svIaGohORlkm0ZrpvveXeOONM/1/rNkdE9jbxoTU+wDYdWmNDbyToRs4V33sbaqRnNy7ohlun7jNSo+n1NAypkdpLn8fA4/PsDLpNBF5n0jej3vP4c3ljQrNzfOv1WOblZbPZX7vrHfcQoXCI8mB5dDSNSHC23eIRr+Elx59DYWZhNDhn+TQUnYi0joTCtGVZZ9VfZ5rm0cDsVm+RiLSa+mUFITu2fKBeSG20dGDntmDY7eGNC7aN9+zWLyvYXdHQGRtmo2HV6XFNiQmc8SG2iQAbd66dz70ep3c1co5ICNZH751DIBxwhqKLGcO5IrCzZMXv8ZPjz2FA9oDoGM4ZvgzV2YpIm0l4OvFG/AtnJkQRqSdSVhDtPa0XNHeG1oalANFe2Xo9tY0G2CbKCkKt1BsLTlmBJzbANlJW4G0QbneG1Nge2gYB1t1mGDi9uo3s3x4lAdI51YXq4so0ygJlcUPRpXpTyfHn0Dutd3QM53Sfpj8QkfaVaM30/vVWZQCn4ozqIdKpxAbM5m7yql9WUL/2tbmbvBq/YczZv7V6Yz3U+/i/kVIAv8eD4YuUAkR6UWOfN1dW0Fgpght4W7msQGRXakI1cWUaZYEyakI10e3p3nRyUnIozCgkJyWHHH8Oad60DmyxiIgj0Z7p5TgjCUX+Z60CFgE/S/RCblnI3TiT5D5iWdYfm9jvZOB5YJxlWZ82to90Tonf5BUfViOlB9He22Zu8krkhrHWsKuyAl8zN3k1VVbQZP1sbICNqZ9Vb6zsjWzbpipUtbNMw/0aCO8cii7Tl0m31G7RMo3slGxSPG07FJ2ISLISrZnerbs0TNP0AvcDU4ES4BPTNOdblvVtvf2ygV8BH+3O9fZFzd3k1VRva1wZQRM3eTV+w1jb3eRlEB8wI4EzthQgJaZOtrHSgfplAj5P7I1cDYNxwzIEhViR1hC2w1QGK+Pqm8sCZQTDQcAZNSTLn0WvtF7RMo1sfzY+z+5UIIqItK/2+o01HlhuWdYKANM0nwWmA9/W2+93wG3Ale3Urlaxq7FjI/Wr0SDazE1e9bc1fsNY29zkFVtW4ITXnc8jodPv88T0pDZRVuBxxoVtrH7W26D2VmUFInuDkB3aORSdO3NgeaCckB0CnBE1sv3Z5KfnR4Nzlj9LQ9GJyB6vyTBtmuYaEph23rKs/RK4TgHx9dUlwIR61zsQ6GtZ1j9M02wyTJumOQeY416bLl27xQXPcNiOTloQDIfje1Njygh21sm6X90QG4zdN9xYT258SE6+rMCI+xrpffV63DIBj4HPcHph40YkcJfjgmqDdbEjExhxxzV+np3hWUT2fj6fj7y8vKSPD4aD7Kjdwfa67WyvdR5ldWVEJgHzeX10yehC/9T+dEntQpfULmT5NRSdiOydmuuZPr29GmGapge4EzhzV/talvUQ8JC7aD/+7tLdunYiY8fGfvzvjzz34o4D6xyb6Nix9c8bWZe4yAS5Ce4Wjl8Vch8isu/Ky8tj8+bNCe1bF66jvK48blSNymBldLvf4yfXn0uPlB7kpjhjOGd4Y4aiq4W62jq2srUtXoqISLtxZ0BsoMkwbVnW2614/VLiJ34pdNdFZAMjgLdM0wRnUpj5pmkev6ubEIf3ytDYsSIiraA2VBt3U2BZXRnVoero9jRvGjn+HPIz8p2bA1NyNaKGiOzzEq6ZNk1zNPAjII+YeckTnE78E6DINM0BOCF6Fs7QepFz7HDPG7nWW8AViYzmUdQjI7EXICIiUdXB6vjgHCijNrRzVsQMXwa5Kbnsl7JfdFSNFK9G1BARqS/RcabnAH/GmajlGOA14CfAy4kcb1lW0DTNXwKv4wyN9zfLsr4xTfNm4FPLsuYn03gRkb1dKBwiZIcIE44+D9khwnY4+jz2UX99Y8tGmcG2sm3Ra2T5s+ie2j061XaOPwe/x9+Br1pEZM9h2AncQGea5nLgLMuy3jVNc5tlWV1N0zwGmGVZVsJjTbcBe+3atR14eRHZF7VFwK3/PPI1WV7Di9fw4jE80eeRR+/uvbGrbHJTcsn2ZeP1aEQNEZFdcWumG9QHJ1rm0dOyrHfd52HTND2WZb1mmuZTrdVAEZHdsScHXJ/hI9Wbigd3vcfdD8/O5zHH1D++seXmtOQGRBERaV6iYbrENM3+lmWtApYC003T3AzUtVnLRGSP1x4BN/I8WZ0l4IqIyJ4p0TD9/4ChwCrgZpzpvlOAi9umWSLSFmzb3tnLqoCrgCsiIrstoZrp+kzTTAFSLMuqaP0mtYhqpmWPt6cH3EioTSbgNnYuBdy2pzIPEZGWa6pmOtEbEO8CnrIs65NWb9nuUZiWNhEbcJsLpwq4sidSmBYRabndvQHRAF42TbMSeBp42rKsJa3XPJFd29MD7u6UKCjgioiIdE4Jl3m4U37/GJgNnAiswOmtvrPtmrdL6pnuYHt6wFUPruyL1DMtItJyu1XmUZ9pmgXAY8CPLcvqyAFKFaYb0VzAbSrkdvaA2yDsKuCKJE1hWkSk5Xa3zAPTNDNxeqRnA5OBt4GOnLBlj7KnB1yf4SPFk9J8b64CroiIiOxjEp1OfB7ONOILgWeAn1mWtcd3ayjgKuCKiIiI7I5Ee6Y/AS63LOuHtmxMMlaUr+iwgNtYIFXAFREREdl3JFUz3YnYj3z8SHShyfrb+usVcEVkH6aaaRGRltvtmunOamr+VAVcEREREekQe3yY9nv8Hd0EEREREdlHqTtXRERERCRJCtMiIiIiIklSmBYRERERSZLCtIiIiIhIkhSmRURERESSpDAtIiIiIpIkhWkRERERkSQpTIuIiIiIJElhWkREREQkSQrTIiIiIiJJUpgWEREREUmSwrSIiIiISJIUpkVEREREkqQwLSIiIiKSJIVpEREREZEkKUyLiIiIiCRJYVpEREREJEkK0yIiIiIiSVKYFhERERFJksK0iIiIiEiSFKZFRERERJKkMC0iIiIikiSFaRERERGRJClMi4iIiIgkSWFaRERERCRJCtMiIiIiIklSmBYRERERSZLCtIiIiIhIkhSmRURERESSpDAtIiIiIpIkhWkRERERkSQpTIuIiIiIJElhWkREREQkSQrTIiIiIiJJUpgWEREREUmSwrSIiIiISJJ87XUh0zSPBu4GvMAjlmX9sd72y4BfAEFgE/Bzy7JWt1f7RERERERaql16pk3T9AL3A8cAw4DZpmkOq7fbIuAgy7IOAJ4H/l97tE1EREREJFnt1TM9HlhuWdYKANM0nwWmA99GdrAsa0HM/h8Cp7dT20REREREktJeYboAWBOzXAJMaGb/s4HXGttgmuYcYA6AZVnk5eW1VhtFRPYJPp9PvztFRFpJu9VMJ8o0zdOBg4DDG9tuWdZDwEPuor158+b2apqIyF4hLy8P/e4UEWmZ/Pz8Rte3V5guBfrGLBe66+KYpjkFuBY43LKs2nZqm4iIiIhIUtorTH8CFJmmOQAnRM8CTo3dwTTNMcBfgaMty9rYTu0SEREREUlau4zmYVlWEPgl8Dqw2FllfWOa5s2maR7v7nY7kAXMM03zc9M057dH20REREREkmXYtt3Rbdgd9tq1azu6DSIiexTVTIuItJxbM23UX68ZEEVEREREkqQwLSIiIiKSJIVpEREREZEkKUyLiIiIiCRJYVpEREREJEkK0yIiIiIiSVKYFhERERFJksK0iIiIiEiSFKZFRERERJKkMC0iIiIikiSFaRERERGRJClMi4iIiIgkSWFaRERERCRJCtMiIiIiIklSmBYRERERSZLCtIiIiIhIkhSmRURERESSpDAtIiIiIpIkhWkRERERkSQpTIuIiIiIJElhWkREREQkSQrTIiIiIiJJUpgWEREREUmSwrSIiIiISJIUpkVEREREkqQwLSIiIiKSJIVpEREREZEkKUyLiIiIiCRJYVpEREREJEkK0yIiIiIiSVKYFhERERFJksK0iIiIiEiSFKZFRERERJKkMC0iIiIikiSFaRERERGRJClMi4iIiIgkSWFaRERERCRJCtMiIiIiIklSmBYRERERSZLCtIiIiIhIkhSmRURERESSpDAtIiIiIpIkhWkRERERkSQpTIuIiIiIJElhWkREREQkSQrTIiIiIiJJUpgWEREREUmSwrSIiIiISJIUpkVEREREkqQwLSIiIiKSJF97Xcg0zaOBuwEv8IhlWX+stz0V+F9gLLAFmGlZ1qr2ap+IiIiISEu1S8+0aZpe4H7gGGAYMNs0zWH1djsb2GZZ1iDgz8Bt7dE2EREREZFktVeZx3hguWVZKyzLqgOeBabX22c68IT7/Hngx6ZpGu3UPhERERGRFmuvMF0ArIlZLnHXNbqPZVlBYAfQvV1aJyIiIiKShHarmW4tpmnOAeYAWJZFfn5+B7dIRGTPo9+dIiKto716pkuBvjHLhe66RvcxTdMH5OLciBjHsqyHLMs6yLKsg0zT/Aww9Gidh2maD3d0G/aWh95LvZ+d+aHfna3+furnU+9lp3zo/Wz19/MzGtFePdOfAEWmaQ7ACc2zgFPr7TMf+BnwATADeNOyLLud2ieOVzq6AXsRvZetS++ndGb6+Ww9ei9bl97PdtAuPdNuDfQvgdeBxc4q6xvTNG82TfN4d7dHge6maS4HLgN+3R5tk50sy9I/ulai97J16f2Uzkw/n61H72Xr0vvZPtqtZtqyrFeBV+utuyHmeQ1wSgtP+1ArNE1EZF+j350iIi3X6O9Ow7ZVSSEiIiIikow9bjQPaTnTNG3gKcuyTneXfcA64CPLso7r0Mbt4UzTrLAsK6uj27G32dX7aprmW8AVlmV92n6tkn2VaZrX4tznEwLCwLmWZX3Usa3ac5mmWYgzkdswnHLTvwNXuvNQNLb/JcBDlmVVtVsj9wDu/+13WpZ1ubt8BZBlWdaNHdqwfVB7jeYhHasSGGGaZrq7PJWGo6mIiEg9pmkeDBwHHGhZ1gHAFOLnTZAWcCdjexF4ybKsIqAYyAJuaeawS4CMtm/dHqcWOMk0zbyObsi+Tj3T+45XgZ/izC45G3gG+BGAaZrjgbuBNKAaOMuyrCWmab4DXGxZ1ufufv8FLrQs64v2b37nZZrmZJxe0uPc5fuATy3Letw0zVU4M3tOA/zAKZZlfddRbd2TNPe+dmS7ZJ/TB9hsWVYtgGVZmwFM0xwL3IkTBDcDZ1qWtc791OQL4HCc/2N/blnWxx3R8E7qSKDGsqzHACzLCpmmeSmw0jTNG4GbgKNxPgGIDOuWDywwTXOzZVlHdEyzO6UgTg3vpcC1sRtM0+wP/A3IAzYBZ+FMhvclMMCyrLBpmpnAd8D+lmUF2rHdex31TO87ngVmmaaZBhwAxH5E+R3wI8uyxgA3ALe66x8FzgQwTbMYSFOQTspmy7IOBB4ArujoxohIi/wL6Gua5lLTNP9imubhpmn6gXuBGZZljcUJLbE9qxmWZY0GLnC3yU7Dgbixei3LKgN+AH4B9AdGu58CPGVZ1j3AWuAIBelG3Q+cZppmbr319wJPRN5H4B7LsnYAn+P8oQfOJy6vK0jvPoXpfYRlWV/i/JKaTb1RVXAmyJlnmubXwJ9xftkBzAOOc//j+DnweLs0du/zovv1M5zvgYjsISzLqgDG4sy8uwl4DjgXGAG8YZrm58B1OJORRTzjHvsOkGOaZpd2bPKebDLwV3c4XSzL2tqxzen83D9E/he4uN6mg4Gn3edPAoe5z58DZrrPZ7nLsptU5rFvmQ/8CecXVveY9b8DFliWdaL70dBbAJZlVZmm+QYwHTBx/kORhoLE/2GaVm97rfs1hP7NtcSu3leRdmFZVgjn9+Jbpml+BVwIfGNZ1sFNHFJ/mCwNm7XTtzgTs0WZppkD7Aes6ogG7QXuAhYCjyWw73zgVtM0u+H8n/5mG7Zrn6Ge6X3L34CbLMv6qt76XHbekHhmvW2PAPcAn1iWta1tm7fHWg0MM00z1e2B+nEHt2dvofdVOpxpmoNN0yyKWTUaZ/KxHu7NiZim6TdNc3jMPjPd9YcBO9yP18XxHyDDNM3/ATBN0wvcgfPJ5+vAue6IU7iBD6AcyG7/pu4Z3B58Czg7ZvX7OD3PAKcB77r7VuDMSn038Hf3D0XZTQrT+xDLskrc+rP6/h/wB9M0F1Gv59SyrM+AMhL7i3ef4v7Cr7Usaw3OL7Kv3a+LOrRhezi9r9LJZAFPmKb5rWmaX+IM53YDTu/qbaZpfoFTh3pIzDE17u/TB4kPOPs8y7Js4ETgFNM0lwFLgRrgNzidNz8AX7rv66nuYQ8B/zRNc0EHNHlPcQfOzYYRFwFnuT+zZwC/itn2HHA6KvFoNZq0RZplmmY+zsebQyzLCndwczoV0zRHAQ9bljW+o9uyN9H7KnsyjYEusu9Rz7Q0yf0Y7iPgWgXpeKZpnodzk9F1Hd2WvYneVxER2dOoZ1pEREREJEkaWWAvZppmX5whc3rh3E3+kGVZd7s3dTyHM0zbKsC0LGubaZpDcGqjD8Tpjf5Tc+dp55cjIiIi0umozGPvFgQutyxrGDARuNA0zWHAr4H/uFO5/sddBtiKM1blnxI8j4iIiMg+TWF6L2ZZ1jrLsha6z8txhnMqwBk3+gl3tyeAE9x9NlqW9QkQSPA8IiIiIvs0hel9hDsZyxicGwp7WZa1zt20Hqd8I5nziIiIiOzTFKb3AaZpZgEvAJe4U49GuWN+JnQXanPnEREREdkXKUzv5UzT9OME4Kcsy3rRXb3BNM0+7vY+wMYkzyMiIiKyT1OY3ouZpmkAjwKLLcu6M2bTfOBn7vOfAS8neR4RERGRfZrGmd6LmaZ5GPAu8BUQmXTlNzj1zhawH7AaZ2i8raZp9gY+BXLc/Stwps49oLHzWJb1aju9FBEREZFOSWFaRERERCRJKvMQEREREUmSwrSIiIiISJIUpkVEREREkqQwLSIiIiKSJIVpEREREZEkKUyLiHQipmk+aJrm9Z2gHTeapjm3Bfv/xjTNR9qyTSIinZGvoxsgIrK3ME1zFfALy7L+new5LMs6r/Va1DZM05wMzLUsqzCyzrKsWzuuRSIiHUc90yIi7cQ0TXVgiIjsZfSLXUSkFZim+STOrKKvmKYZAm7GmWl0JfAL4LfAKmCSaZrzgB8B6cAXwPmWZX3jnudxoMSyrOsiPcDAn4GrgRDO7KOPNdGGXOBO4Fic2UofA35rWVbINM0z3XZ8CJwNbAcusCzrNffYAcDjwIHuPkuauEYm8BqQappmhbu6GJgDDLIs63TTNPu7r/vn7vuQBVwDfAY86r5Pcy3L+mXMeX8OXAn0Bj4G5liWtbrxd1tEpPNQz7SISCuwLOsM4AdgmmVZWZZl/b+YzYcDQ4Gj3OXXgCKgJ7AQeKqZU/cGcoECnBB8v2maXZvY93EgCAwCxgA/wQnQERNwQnIe8P+AR03TNNxtT+OE3Tzgd8DPmnidlcAxwFr3dWZZlrW2ifZMcF/nTOAu4FpgCjAcME3TPBznyXTgN8BJQA/gXeCZJs4pItKpKEyLiLS9Gy3LqrQsqxrAsqy/WZZVbllWLXAjMMrtVW5MALjZsqyAZVmvAhXA4Po7mabZC6dH+hL3WhtxerRnxey22rKshy3LCgFPAH2AXqZp7geMA663LKvWsqx3gFda4XX/zrKsGsuy/gVUAs9YlrXRsqxSnMA8xt3vPOAPlmUttiwrCNwKjDZNs18rtEFEpE2pzENEpO2tiTwxTdML3AKcgtMLG3Y35QE7Gjl2ixswI6pwyibq6wf4gXWmaUbWeWKvDayPPLEsq8rdL8u99ja31zliNdB3Vy9sFzbEPK9uZDnyOvoBd5umeUfMdgOnN16lHiLSqSlMi4i0HjuB9acC03HKHVbhlHBswwmPu2MNUAvk1QvfiVgHdDVNMzMmUO9HYq+nNawBbrEsq7lyFxGRTkllHiIirWcDsP8u9snGCb1bgAyckobdZlnWOuBfwB2maeaYpukxTXNgpC55F8euBj4FbjJNM8U0zcOAac0csgHo3kxpSks9CFxjmuZwcG6kNE3zlFY6t4hIm1KYFhFpPX8ArjNNc7tpmlc0sc//4pQulALf4oyc0Vr+B0hxz7sNeB6nLjoRp+LcMLgVZ+SR/21qR8uyvsO5QXCF+1rzd6fRlmX9H3Ab8KxpmmXA1zg3OYqIdHqGbbf2p3UiIiIiIvsG9UyLiIiIiCRJYVpEREREJEkK0yIiIiIiSVKYFhERERFJksK0iIiIiEiSFKZFRERERJKkMC0iIiIikiSFaRERERGRJClMi4iIiIgk6f8DCXlXHZ0/OywAAAAASUVORK5CYII=\n" + }, + "metadata": { + "needs_background": "light" + } + } + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "Pi0-YE3VYcVH" + }, + "source": [ + "With our first default setting, we don't filter out models because `max_from_best=1.0` and `threshold_value=0.0` are the loosest criteria.\n", + "\n", + "- The first graph shows us the fraction of models worse than the best model by distance with respect to the metric of interest.\n", + "- The second graph shows us the performance of a model group over time. The dashed line is the best case at that time period." + ] + }, + { + "cell_type": "code", + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/" + }, + "id": "qQMTtcJ5ZORw", + "outputId": "86ae4891-d9da-4128-c565-57623729ca93" + }, + "source": [ + "# model groups that conform to the current filters\n", + "aud.thresholded_model_group_ids" + ], + "execution_count": 32, + "outputs": [ + { + "output_type": "execute_result", + "data": { + "text/plain": [ + "{1, 2, 3}" + ] + }, + "metadata": {}, + "execution_count": 32 + } + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "7vfJKljfZWZS" + }, + "source": [ + "We won't delve further into the model selection process here, but see the resources noted above for a deeper look." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "SPm3NKoz4BLo" + }, + "source": [ + "## Bias Audit\n", + "\n", + "Finally, we turn to the results of the bias audit using the toolkit [aequitas](http://www.datasciencepublicpolicy.org/our-work/tools-guides/aequitas/), which can be found in the table `test_results.aequitas`:" + ] + }, + { + "cell_type": "code", + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/", + "height": 300 + }, + "id": "LH5CgdzO4GX3", + "outputId": "371776b2-1992-4232-98bb-3da41a0ed87d" + }, + "source": [ + "pd.read_sql(\"\"\"SELECT * FROM test_results.aequitas LIMIT 5\"\"\", conn)" + ], + "execution_count": 33, + "outputs": [ + { + "output_type": "execute_result", + "data": { + "text/plain": [ + " model_id subset_hash tie_breaker evaluation_start_time evaluation_end_time \\\n", + "0 1 worst 2012-04-01 2012-06-30 \n", + "1 1 worst 2012-04-01 2012-06-30 \n", + "2 1 worst 2012-04-01 2012-06-30 \n", + "3 1 worst 2012-04-01 2012-06-30 \n", + "4 1 worst 2012-04-01 2012-06-30 \n", + "\n", + " matrix_uuid parameter attribute_name attribute_value \\\n", + "0 6a751eabcf4722abda70f77c0d9d712d 25_abs teacher_prefix Mr. \n", + "1 6a751eabcf4722abda70f77c0d9d712d 25_abs teacher_prefix Mrs. \n", + "2 6a751eabcf4722abda70f77c0d9d712d 25_abs teacher_prefix Ms. \n", + "3 6a751eabcf4722abda70f77c0d9d712d 50_abs teacher_prefix Mr. \n", + "4 6a751eabcf4722abda70f77c0d9d712d 50_abs teacher_prefix Mrs. \n", + "\n", + " total_entities ... Impact_Parity FDR_Parity FPR_Parity FOR_Parity \\\n", + "0 388 ... None None None None \n", + "1 388 ... None None None None \n", + "2 388 ... None None None None \n", + "3 388 ... None None None None \n", + "4 388 ... None None None None \n", + "\n", + " FNR_Parity TypeI_Parity TypeII_Parity Equalized_Odds \\\n", + "0 None None None None \n", + "1 None None None None \n", + "2 None None None None \n", + "3 None None None None \n", + "4 None None None None \n", + "\n", + " Unsupervised_Fairness Supervised_Fairness \n", + "0 None None \n", + "1 None None \n", + "2 None None \n", + "3 None None \n", + "4 None None \n", + "\n", + "[5 rows x 62 columns]" + ], + "text/html": [ + "\n", + "
\n", + "
\n", + "
\n", + "\n", + "\n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + "
model_idsubset_hashtie_breakerevaluation_start_timeevaluation_end_timematrix_uuidparameterattribute_nameattribute_valuetotal_entities...Impact_ParityFDR_ParityFPR_ParityFOR_ParityFNR_ParityTypeI_ParityTypeII_ParityEqualized_OddsUnsupervised_FairnessSupervised_Fairness
01worst2012-04-012012-06-306a751eabcf4722abda70f77c0d9d712d25_absteacher_prefixMr.388...NoneNoneNoneNoneNoneNoneNoneNoneNoneNone
11worst2012-04-012012-06-306a751eabcf4722abda70f77c0d9d712d25_absteacher_prefixMrs.388...NoneNoneNoneNoneNoneNoneNoneNoneNoneNone
21worst2012-04-012012-06-306a751eabcf4722abda70f77c0d9d712d25_absteacher_prefixMs.388...NoneNoneNoneNoneNoneNoneNoneNoneNoneNone
31worst2012-04-012012-06-306a751eabcf4722abda70f77c0d9d712d50_absteacher_prefixMr.388...NoneNoneNoneNoneNoneNoneNoneNoneNoneNone
41worst2012-04-012012-06-306a751eabcf4722abda70f77c0d9d712d50_absteacher_prefixMrs.388...NoneNoneNoneNoneNoneNoneNoneNoneNoneNone
\n", + "

5 rows × 62 columns

\n", + "
\n", + " \n", + " \n", + " \n", + "\n", + " \n", + "
\n", + "
\n", + " " + ] + }, + "metadata": {}, + "execution_count": 33 + } + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "YJPLLBF4d-4h" + }, + "source": [ + "For each model, the table contains the full set of confusion matrix values for the thresholds and each attribute value specified in your `bias_audit_config`, as well as disparities relative to reference group identified in that configuration file. Note that these disparities are calculated as ratios with the reference group as the denominator.\n", + "\n", + "Audition provides some tools for visualizing these results. If we assume the appropriate fairness metric for this context is true positive rate (TPR) disparity, let's take a look at how model 12 performed at the 10% threshold:" + ] + }, + { + "cell_type": "code", + "metadata": { + "id": "T-X0aJWBiZIV" + }, + "source": [ + "import triage.component.postmodeling.fairness.aequitas_utils as au\n", + "import aequitas.plot as ap\n", + "\n", + "# Load Data\n", + "bdf = au.get_aequitas_results(conn, parameter = \"0.1_pct\", model_id = 12)\n", + "\n", + "# attribute of interest\n", + "attribute = 'teacher_prefix'\n", + "\n", + "# fairness metric we care about\n", + "metrics = ['tpr']\n", + "\n", + "# tolerance for disparities\n", + "disparity_tolerance = 1.30" + ], + "execution_count": 34, + "outputs": [] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "w6XHYI28i84I" + }, + "source": [ + "The `disparity` plot shows disparities for each group on the selected metric relative to the reference group:" + ] + }, + { + "cell_type": "code", + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/", + "height": 338 + }, + "id": "wSvZaJGIjPpO", + "outputId": "6bea8c3b-b3f2-49a3-8033-2cfe71505563" + }, + "source": [ + "ap.disparity(bdf, metrics, attribute, fairness_threshold = disparity_tolerance)" + ], + "execution_count": 35, + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": [ + "\u001b[32m2022-06-21 18:47:40\u001b[0m - \u001b[1;30m INFO\u001b[0m NumExpr defaulting to 2 threads.\n" + ] + }, + { + "output_type": "execute_result", + "data": { + "text/html": [ + "\n", + "
\n", + "" + ], + "text/plain": [ + "alt.LayerChart(...)" + ] + }, + "metadata": {}, + "execution_count": 35 + } + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "VyUpTI_RjeJK" + }, + "source": [ + "The `absolute` plot will show the value of the true positive rate itself:" + ] + }, + { + "cell_type": "code", + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/", + "height": 245 + }, + "id": "Q13a7KUjjryc", + "outputId": "b93af95b-6570-4857-86fa-e6f0195f213b" + }, + "source": [ + "ap.absolute(bdf, metrics, attribute, fairness_threshold = disparity_tolerance)" + ], + "execution_count": 36, + "outputs": [ + { + "output_type": "execute_result", + "data": { + "text/html": [ + "\n", + "
\n", + "" + ], + "text/plain": [ + "alt.LayerChart(...)" + ] + }, + "metadata": {}, + "execution_count": 36 + } + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "zrl9izw7jxRw" + }, + "source": [ + "We might also want to look at the disparities across models and how they relate to our metric for evaluating model accuracy to consider trade-offs between these goals, which we can do by joining the aequitas table to the evaluations:" + ] + }, + { + "cell_type": "code", + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/", + "height": 175 + }, + "id": "USxfuZrvlW9c", + "outputId": "30edec2a-cc83-43e5-f00e-27c013668162" + }, + "source": [ + "acc_metric = 'precision@'\n", + "disp_metric = 'tpr_disparity'\n", + "acc_thresh = '10_pct'\n", + "disp_thresh = '0.1_pct'\n", + "attribute_name = 'teacher_prefix'\n", + "attribute_value = 'Mrs.'\n", + "\n", + "df = pd.read_sql(f\"\"\"\n", + " WITH last_end_time AS (\n", + " SELECT max(train_end_time) AS train_end_time\n", + " FROM triage_metadata.models\n", + " )\n", + " , last_models AS (\n", + " SELECT model_id\n", + " FROM triage_metadata.models\n", + " JOIN last_end_time USING(train_end_time)\n", + " )\n", + " , eval_metrics AS (\n", + " SELECT e.model_id, e.stochastic_value\n", + " FROM test_results.evaluations e\n", + " JOIN last_models m USING(model_id)\n", + " WHERE e.metric = '{acc_metric}' AND e.parameter = '{acc_thresh}'\n", + " )\n", + " , fair_metrics AS (\n", + " SELECT a.model_id, a.{disp_metric}\n", + " FROM test_results.aequitas a\n", + " JOIN last_models m USING(model_id)\n", + " WHERE a.parameter = '{disp_thresh}'\n", + " AND a.attribute_name = '{attribute_name}'\n", + " AND a.attribute_value = '{attribute_value}'\n", + " AND tie_breaker = 'worst'\n", + " )\n", + " SELECT * \n", + " FROM eval_metrics e\n", + " JOIN fair_metrics f USING(model_id)\n", + " ORDER BY e.stochastic_value DESC\n", + " \"\"\", conn)\n", + "\n", + "df" + ], + "execution_count": 37, + "outputs": [ + { + "output_type": "execute_result", + "data": { + "text/plain": [ + " model_id stochastic_value tpr_disparity\n", + "0 9 0.6032 0.2242\n", + "1 7 0.5016 0.7585\n", + "2 8 0.4704 0.8427\n", + "3 12 0.4444 0.7846" + ], + "text/html": [ + "\n", + "
\n", + "
\n", + "
\n", + "\n", + "\n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + "
model_idstochastic_valuetpr_disparity
090.60320.2242
170.50160.7585
280.47040.8427
3120.44440.7846
\n", + "
\n", + " \n", + " \n", + " \n", + "\n", + " \n", + "
\n", + "
\n", + " " + ] + }, + "metadata": {}, + "execution_count": 37 + } + ] + }, + { + "cell_type": "code", + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/", + "height": 299 + }, + "id": "8JiXPz-2obK1", + "outputId": "17da472b-20b7-4368-cc6d-083ff180c149" + }, + "source": [ + "df.plot('stochastic_value', disp_metric, kind='scatter')" + ], + "execution_count": 38, + "outputs": [ + { + "output_type": "execute_result", + "data": { + "text/plain": [ + "" + ] + }, + "metadata": {}, + "execution_count": 38 + }, + { + "output_type": "display_data", + "data": { + "text/plain": [ + "
" + ], + "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYUAAAEJCAYAAAB7UTvrAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8QVMy6AAAACXBIWXMAAAsTAAALEwEAmpwYAAAb8UlEQVR4nO3df5QcZZ3v8XfIGBAJl3VbxElQggY1AoIK7G4WQQQJuoJ71c9JgCO4SPRghN3gD/wFERT5oUTuIa6E4A9YMH5B5cQlGBBF9oK44bqgBhaMUSAEr0z4FS5CSJj7x1NT6TTdM1OZ7urq8HmdMyddVc9Uf2bS09+up6qeZ9zg4CBmZmYA23Q7gJmZVYeLgpmZ5VwUzMws56JgZmY5FwUzM8v1dTvAGPnSKTOzLTOu2cpeLwqsWbOm6fparcbAwEDJaUbmXMVVNVtVc0F1szlXcZ3I1t/f33Kbu4/MzCznomBmZjkXBTMzy7komJlZzkXBzMxyLgpmZpbr+UtSX6geWree+beuYd0zG5m47XjmTu9nlx0mdDuWmfU4Hyn0qPm3ruGegadZs+5Z7hl4mvm3NL9fw8ysCBeFHrXumY2bLT/RsGxmtiVcFHrUxG3HD7tsZrYlXBR61Nzp/byuth39E1/Ea2vbMXd669vWzcxGyyeae9QuO0zg3MN363YMM9vK+EjBzMxyLgpmZpZzUTAzs5yLgpmZ5VwUzMws56JgZmY5FwUzM8uVdp+CpBnAhcB4YFFEnNOw/ZXAd4CdsjanRcTSsvKZmVlJRwqSxgMLgCOAacAsSdMamn0OiIjYF5gJfL2MbGZmtklZ3Uf7AysjYlVErAcWA0c1tBkEdswe/w/Aw36amZWsrO6jScADdcurgQMa2swDrpf0MeAlwKHNdiRpNjAbICKo1WpNn7Cvr6/ltm5yruKqmq2quaC62ZyruLKzVWnso1nAtyPiq5L+Frhc0p4R8Vx9o4hYCCzMFgcHBgaa7qxWq9FqWzc5V3FVzVbVXFDdbM5VXCey9fe3HkCzrO6jB4Fd65YnZ+vqnQAEQET8AtgOqGbpNjPbSpV1pLAcmCppCqkYzASObmhzP/B24NuSXk8qCg+3O4insTQza62UI4WI2ADMAZYBd6dVsULSmZKOzJqdCpwo6U7gu8DxETHY7iyextLMrLXSzilk9xwsbVh3et3ju4Dpnc7haSzNzFp7wd3R7Gkszcxae8EVBU9jaWbWWpUuSS2Fp7E0M2vtBXekYGZmrbkomJlZ7gXXfWS9yfeXmJXDRwrWE3x/iVk5XBSsJ/j+ErNyuChYT/D9JWblcFGwnuD7S8zK4RPN1hN8f4lZOXykYGZmORcFMzPLuSiYmVnORcHMzHIuCmZmlnNRMDOznIuCmZnlXBTMzCznomBmZjkXBTMzy7komJlZzkXBzMxypQ2IJ2kGcCEwHlgUEec0bJ8PvC1b3B7YOSJ2KiufmZmVVBQkjQcWAIcBq4HlkpZExF1DbSLiX+rafwzYt4xsZma2SVndR/sDKyNiVUSsBxYDRw3Tfhbw3VKSmZlZrqzuo0nAA3XLq4EDmjWU9CpgCvDTFttnA7MBIoJardb0Cfv6+lpu6ybnKq6q2aqaC6qbzbmKKztbFSfZmQlcHRFNJ+GNiIXAwmxxcGBgoOlOarUarbZ1k3MVV9VsVc0F1c3mXMV1Ilt/f+uZC8vqPnoQ2LVueXK2rpmZuOvIzKwryjpSWA5MlTSFVAxmAkc3NpL0OuCvgF+UlMvMzOqUcqQQERuAOcAy4O60KlZIOlPSkXVNZwKLI2KwjFxmZra50s4pRMRSYGnDutMblueVlcfMzJ7PdzSbmVnORcHMzHIuCmZmlnNRMDOznIuCmZnlXBTMzCznomBmZjkXBTMzy7komJlZzkXBzMxyLgpmZpZzUTAzs9yoi4Kk+ZL26WAWMzPrsiKjpI4Hlkl6GLgcuCIiVncmlpmZdcOojxQi4mSgHzgN2Ae4W9JPJH1A0g4dymdmZiUqNJ9CNm/yvwP/LukNwJXAt4GvS1oMnBERrabZNDOziitUFCTtCLwfOBbYG/g+cBJwP3AqcF223szMetCoi4Kkq4HDgZuBbwDXRMQzddvnAo+3PaGZmZWmyJHCbcCciPhTs40R8Zykl7cnlpmZdUOR+xQObFYQJP1g6HFEPNWWVGZm1hVFisLbWqw/uA05zMysAkbsPpJ0ZvZwQt3jIbsD97U9lZmZdcVozinsmv27Td1jgEHgAWBemzOZmVmXjFgUIuKDAJJujYhLtvSJJM0ALiTdGb0oIs5p0kakIjMI3BkRR2/p85mZWXHDFgVJu0XEH7PFGyXt3qxdRKwaYT/jgQXAYcBqYLmkJRFxV12bqcCngekR8aiknUf/Y5iZWTuMdKTwG2Bi9ngl6RP8uIY2g6RP/8PZH1g5VDyyu5+PAu6qa3MisCAiHgWIiD+PmN7MzNpq2KIQERPrHo9lmO1JpPMPQ1YDBzS02QNA0i2kIjMvIn7cuCNJs4HZWSZqtVrTJ+zr62u5rZucq7iqZqtqLqhuNucqruxso7p5Lev+uReYVn8XcweyTCVd4joZuFnSXhHxWH2jiFgILMwWBwcGBprurFar0WpbNzlXcVXNVtVcUN1szlVcJ7L19/e33DaqT//ZQHgbgRdvYYYH2fzKpcnZunqrgSUR8WxE/IFUhKZu4fOZmdkWKDLMxdeA70k6m/QGPji0YaQTzcByYKqkKaRiMBNovLLoGmAW8C1JNVJ30kj7NTOzNipynuAi0tVDPwN+RzrxvDJ7PKyI2ADMAZYBd6dVsULSmZKOzJotA9ZKuit7jk9ExNoC+czMbIzGDQ4OjtyqugbXrFnTdENV+widq7iqZqtqLqhuNucqroPnFBqvJAWKHSmYmdlWrsh8Cn2kCXUOAmrUVZmIeGv7o5mZWdmKHCnMBz5MmmTnzaRZ13YGftqBXGZm1gVFisL/BI6IiAuBDdm/76H1kNpmZtZjihSF7dl0V/JfJG0fEf8N7Nv+WGZm1g1F7lO4G9gP+E/gdmCepCd4/k1oZmbWo4oUhVNIdzUDzAX+lTRY3ux2hzIzs+4YdVGIiOV1j38HHNqRRGZm1jVFjhSQdAhpKIp+YA2wOCJu7EQwMzMr36hPNEs6FVgMPAJcC6wFrszWm5nZVqDIkcJc4JCI+O3QCkmXAzcAX213MDMzK1/RYS5WNiyvom60VDMz621FjhTmAZdKmkcaOntX4PPAGZLy4hIRz7UzoJmZladIUbg4+3cWm8/VfEy2bRyjm6/ZzMwqqkhRmNKxFGZmVglF7lO4r35Z0ouB5zo4Z7OZmZWsyCWpX5G0f/b4XaRLUx+V9O5OhTMzs3IVufroGGDoctTTgWOBI4Gz2x3KzMy6o8g5he0j4ilJfw3sHhHfB5D0qs5EMzOzshUpCvdKOgZ4DemGNSTVgL90IpiZmZWvSFE4CbgQWA+ckK07HLi+3aHMzKw7io6S+ncN664Armh3KDMz645hi4Kkt0bEzdnjQ1q1iwjP02xmthUY6Ujh68Ce2eNLW7QZBHYf6YkkzSB1P40HFkXEOQ3bjwfOZ9NMbhdFxKKR9mtmZu0zbFGIiD3rHm/xHc2SxgMLgMNI4yYtl7QkIu5qaPq9iJizpc9jZmZjU3SU1C21P7AyIlZFxHrSvAxHlfTcZmY2SiOdU3iAUQyNHRGvHKHJJOCBuuXVwAFN2r1X0luBe4F/iYgHGhtImk02L3REUKvVmj5hX19fy23d5FzFVTVbVXNBdbM5V3FlZxvpnMKxdY/3A44D/hdwH/AqYA5wWZuy/Aj4bkQ8I+nDwHeA553cjoiFwMJscXBgYKDpzmq1Gq22dZNzFVfVbFXNBdXN5lzFdSJbf39/y20jnVP4+dBjSQuAwyPiwbp11wE/ZuSZ1x4kzb8wZDKbTigPPdfausVFwHkj7NPMzNqsyDmFfuDJhnVPkrqGRrIcmCppiqQJwExgSX0DSa+oWzwSuLtANjMza4MidzQvAZZI+iKbZl77NA1v7s1ExAZJc4BlpEtSvxkRKySdCdweEUuAkyUdCWwgjcB6fKGfxMzMxqxIUfgIaUrOb5COGtYAVwFfGM03R8RSYGnDutPrHn+aVGTMzKxLigxz8TRwWvbVlKTTGm9KMzOz3tHu+xQ+0+b9mZlZidpdFMa1eX9mZlaidheFEW90MzOz6iprmAszM+sB7j4yM7PcqK4+ykY5PQ64IiKeGabpf7QllZmZdcWojhQiYiNwwQgFgYh4Z1tSmZlZVxTpPvqRpHd3LImZmXVdkTuatwOulvQL0jDY+ZVGEfGBdgczM7PyFSkKv82+zMxsK1VkmItRjXFkZma9q8iRApIOAWaxaUC8xRFxYyeCmZlZ+UZ9olnSqaS5lR8BrgXWAldm683MbCtQ5EhhLnBIROTnFSRdDtzAyDOvmZlZDyh6R/PKhuVVeLwjM7OtRpEjhXnApZLmsWnmtc8DZ0jKi0tEPNfOgGZmVp4iReHi7N9ZpKODoXGOjs62jcvWj29bOjMzK1WRovApIJqsfx9wdXvimJlZNxUpCp+PiPMbV0r6bET4RLOZ2VZgxKKQ3ZsA0CfpbWw+PPbuwLpOBDMzs/KN5kjh0uzfbYFv1q0fBP4EfKzdoczMrDtGLAoRMQVA0mUe+M7MbOtWZOyjMRUESTOAC0lXJy2KiHNatHsv6cT1fhFx+1ie08zMiilljuZs5rYFwBHANGCWpGlN2k0ETgF+WUYuMzPbXClFAdgfWBkRqyJiPWkMpaOatDsLOBd4uqRcZmZWp9AoqWMwiTQxz5DVwAH1DSS9Cdg1Iq6V9IlWO5I0G5gNEBHUarWm7fr6+lpu6ybnKq6q2aqaC6qbzbmKKztbWUVhWNkwGRcAx4/UNiIWAguzxcGBgYGm7Wq1Gq22dZNzFVfVbFXNBdXN5lzFdSJbf39/y21ldR89SBoracjkbN2QicCewE2S/gj8DbBE0ltKymdmZpR3pLAcmCppCqkYzCSNmQRARDwO5MdHkm4CPu6rj8zMylXKkUJEbADmAMuAu9OqWCHpTElHlpHBzMxGVto5hYhYCixtWHd6i7YHl5HJzMw2V9Y5BTMz6wEuCmZmlnNRMDOznIuCmZnlXBTMzCznomBmZjkXBTMzy7komJlZzkXBzMxyLgpmZpZzUTAzs5yLgpmZ5VwUzMws56JgZmY5FwUzM8u5KJiZWc5FwczMci4KZmaWc1EwM7Oci4KZmeVcFMzMLOeiYGZmORcFMzPL9ZX1RJJmABcC44FFEXFOw/aPAB8FNgJPArMj4q6y8pmZWUlHCpLGAwuAI4BpwCxJ0xqaXRkRe0XEPsB5wAVlZDMzs03K6j7aH1gZEasiYj2wGDiqvkFEPFG3+BJgsKRsZmaWKav7aBLwQN3yauCAxkaSPgrMBSYAhzTbkaTZwGyAiKBWqzV9wr6+vpbbusm5iqtqtqrmgupmc67iys5W2jmF0YiIBcACSUcDnwOOa9JmIbAwWxwcGBhouq9arUarbd3kXMVVNVtVc0F1szlXcZ3I1t/f33JbWd1HDwK71i1Pzta1shh4TycDmZnZ85VVFJYDUyVNkTQBmAksqW8gaWrd4ruA35WUzczMMqV0H0XEBklzgGWkS1K/GRErJJ0J3B4RS4A5kg4FngUepUnXkZmZdVZp5xQiYimwtGHd6XWPTykri5mZNec7ms3MLOeiYGZmORcFMzPLuSiYmVnORcHMzHIuCmZmlnNRMDOznIuCmZnlXBTMzCznomBmZjkXBTMzy7komJlZzkXBzMxyLgpmZpZzUTAzs5yLgpmZ5UqbZMfMzMbuoXXrmX/rGtY9s5GJ245n7vR+dtlhQtv27yMFM7MeMv/WNdwz8DRr1j3LPQNPM/+WNW3dv4uCmVkPWffMxs2Wn2hYHisXBTOzHjJx2/HDLo+Vi4KZWQ+ZO72f19W2o3/ii3htbTvmTu9v6/59otnMrIfsssMEzj18t47tv7SiIGkGcCEwHlgUEec0bJ8LfAjYADwM/FNE3FdWPjMzK6n7SNJ4YAFwBDANmCVpWkOz/wLeEhF7A1cD55WRzczMNinrSGF/YGVErAKQtBg4CrhrqEFE/Kyu/W3AsSVlMzOzTFknmicBD9Qtr87WtXICcF1HE5mZ2fNU7kSzpGOBtwAHtdg+G5gNEBHUarWm++nr62u5rZucq7iqZqtqLqhuNucqruxsZRWFB4Fd65YnZ+s2I+lQ4LPAQRHxTLMdRcRCYGG2ODgwMND0CWu1Gq22dZNzFVfVbFXNBdXN5lzFdSJbf3/ry1jHDQ4OtvXJmpHUB9wLvJ1UDJYDR0fEiro2+5JOMM+IiN+NctedD29mtnUa12xlKecUImIDMAdYBtydVsUKSWdKOjJrdj6wA3CVpDskLRnFrse1+pL0f4bb3q0v59p6slU1V5WzOVelsjVV2jmFiFgKLG1Yd3rd40PLymJmZs15mAszM8ttzUVh4chNusK5iqtqtqrmgupmc67iSs1WyolmMzPrDVvzkYKZmRXkomBmZrnK3dE8kpFGW61r917SfQ/7RcTtdetfSRpzaV5EfKUq2STtDVwM7Ag8l217upu5JL0IWAS8ifRauSwivtyOTKPJJel40qXKQzc6XhQRi7JtxwGfy9Z/MSK+065cY8kmaR/gX0n/jxuBL0XE97qdq277jqTX/zURMacKubK/yUWkG1wHgXdGxB8rku084F2kD9A3AKdERFv63EfzdylJwDzS7+XOiDg6W9+x139PHSmMcrRVJE0ETgF+2WQ3F9CBcZXGki27ue/fgI9ExBuAg4Fnu50LeD+wbUTsBbwZ+LCk3crMBXwvIvbJvob+UF8KnAEcQBps8QxJf9WOXGPNBjwFfCD7f5wBfE3SThXINeQs4OZ25GljrsuA8yPi9aT/zz9XIZukvwOmA3sDewL70WL4nU7kkjQV+DQwPXs9/XO2vqOv/54qCtSNthoR64Gh0VYbnQWcC2z2SVvSe4A/ACuafE83s70D+HVE3AkQEWsjol0Tr44l1yDwkqxovRhYDzxRcq5mDgduiIhHIuJR0ie4GW3KNaZsEXHv0B35EbGG9Ab3sm7nApD0ZuDlwPVtyjPmXNkbYV9E3AAQEU9GxFNVyEZ6/W8HTAC2BV4E/N8Sc50ILMhe40TEULHs6Ou/14rCiKOtSnoTsGtEXNuwfgfgU8AXqpYN2AMYlLRM0q8kfbIiua4G/h/wEHA/8JWIeKSsXJn3Svq1pKslDY2fVXTU3TKz5STtT3pD+X23c0naBvgq8PE2ZWlLLtJr/zFJP5D0X5LOzz5Fdz1bRPwC+Bnp9f8QsCwi7i4x1x7AHpJukXRb1t002u/dYr1WFIaVvfAvAE5tsnkeMD8iniw1VGaEbH3A3wPHZP/+o6S3VyDX/qR+8X5gCnCqpN3LyJX5EbBbNvHSDUBbzxuM0bDZJL0CuBz4YEQ8V4FcJwFLI2J1iVlGk6sPOJBUrPYDdgeOr0I2Sa8BXk8awHMScIikA0vM1QdMJXUnzwIuaVdX5EhP2ktGGm11Iqnv76Z0foZdgCXZ+EoHAO/LThztBDwn6emIuKgC2VYDN0fEAICkpaSTuzd2OdfRwI8j4lngz5JuIQ1rvqqEXETE2rrFRWyaje9B0h9K/ffe1IZM7cg2dDL3WuCzEXFbRXL9LXCgpJNIY4xNkPRkRJzW5VyrgTvqJuC6Bvgb4NI25Bprtn8Ebhv6ICnpOtLv8T/KyEX63fwy+/v7g6R7SUWio6//XisKy4GpkqaQfjEzSW9cAETE40A+8Likm4CPZ1f4HFi3fh7wZBsLwpiySfo98ElJ25P67Q8C5lcg19uBQ4DLJb2E9Mf6tTJyZVleEREPZYtHkgZThDSw4tl1J9feQToh1y5bnE3SBOCHpCu1rm5jpjHliohj6tocT5r6th0FYUy5su/dSdLLIuJh0uvtdtpnLNnuB06U9GXSAHIHUeLrH7iGdITwLUk1UnfSKlJ3ZMde/z3VfRSjG22157JlJ4suIL1Q7gB+1aR/v/RcpKsjdpC0Isv2rYj4dYm5Tpa0QtKdwMlk3QrZeY2zskzLgTPbeK5jTNkAAW8Fjlca7fcOpctUu52rY8b4f7mR1HV0o6TfkN58L6lCNtI5td8DvwHuJF0S+qMScy0D1kq6i3Ru4xPZRSgdff17mAszM8v11JGCmZl1louCmZnlXBTMzCznomBmZjkXBTMzy7komJlZzkXBXpAkDWbDGHT6ea5TGua4VJIOltStIS2sh/XaHc32Apbdif6aiDi221maaZYvIo7oXiKz4nykYGZmOR8pWCVJ+hRpyIEdgTXAXOAzwDileTF+HxFvlNQPfIM0uuwjwLkRcUm2j/Gk4dJPAHYG7gXeExFDww4fmg1y9jLgCmBORAxKejVpqIU3ksbUXwZ8NCIea5HtJNJY+83y3QT8W93ELSdmP8tk0vDHx0bEr4b5HewXEe+rW3chMC4iTpb0QeCT2b4ezn72i1vsaxCYGhErs+VvA6sj4nPZ8j8AXwR2I83M9pF2DWlivcVHClY5kl5LGhdmv4iYSJpU5L+Bs0kzZO0QEW/Mmi8mjSbZD7yPNFDYIdm2uaQBxd5JegP/J9LMaEP+gTRc896kMYsOz9aPA76c7fP1pNEs5w2T7Y8R8eMW+ep/rvdn+/lAludIYG1juzqLgXcqzYo3VOQEXJlt/3P2M+wIfBCYrzQ3RiGS9gW+CXwY+GvStLBLJG1bdF/W+3ykYFW0kTTT1TRJD0c2X282tHdOaTKU6cC7Is1nfYekRaQ33Z8CHwI+GRH3ZN9yZ8PznJN9+n9M0s+AfUhDha8EVmZtHpZ0AWn6w5bZRulDwHkRsTxbXjlc44i4T9KvSEM4X0YaQfSpoeG4GwZN/Lmk60mjATc98hjGbODiiBiaivU7kj5DGhX35wX3ZT3ORcEqJyJWSvpn0qfqN0haRvrU36gfeCQi1tWtu4805wOkT/jDzXr2p7rHT5HmGUDSy0kTqh9Imm9iG2BoSsSm2SJNvTmSkfI0cyXpaOcy0tDKQ0cJSDqCVKz2yDJuTxrRs6hXAcdJ+ljdugmk36+9wLj7yCopIq6MiL8nvWENkuaPbhzSdw3w0qHulcwr2TRZyQPAq7fg6c/OnmuviNgROJbUpTRcNprka7Qlea4CDpY0mXTEcCVA1rXzfeArwMsjYidgaX3OBk+RisaQXRpyfSkidqr72j4ivlswq20FfKRglZP1208CbgGeBv4CjCdNmn6YpG0i4rmIeEDSrcCXJX2c9In5BNK0ppBm0TorG49+JbAX8GDDTFvNTAQeBx6XNAn4xCiy0ZivyX4XARdI+t+kLp5XA89GxH2tgkTEw9nJ6m8Bf4hNcwQPTSb/MLAhO2p4B/DbFru6Azg6mxvjMNKEMUOT2VwC/FDST4D/JBWPg0mzAa57/q5sa+YjBauibYFzgAFSF8/OpJmlrsq2r8362iF1rexGOmr4IXBGRPwk23YBEMD1wBOkKR5fPIrn/wJpOtTHSdNq/mAU2WiRLxcRVwFfIn3aX0eaWeulo8hzJXAodV1H2Zv1ydnP9yipa2nJMPs4BXg38BipaF5Tt6/bgROBi7J9raT8eZKtIjzJjpmZ5XykYGZmOZ9TMOsiSa8k3SzWzLSIuL/MPGbuPjIzs5y7j8zMLOeiYGZmORcFMzPLuSiYmVnu/wOBnBGRkM1QEAAAAABJRU5ErkJggg==\n" + }, + "metadata": { + "needs_background": "light" + } + } + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "LFLsJwjSojeM" + }, + "source": [ + "This final plot helps us visualize the fairness-accuracy trade-offs across the grid of models we ran, with the disparity metric on the y-axis (values closer to 1 don't favor either group) and the accuracy metric on the x-axis (higher values being more accurate).\n", + "\n", + "Although you might choose a model from this set based on these trade-offs, you might also want to consider some of the many fairness-enhancing methods that have been developed for machine learning recently. For an introduction, you may want to explore the materials from our [hands-on fairness tutorial](https://dssg.github.io/fairness_tutorial/)." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "TqYZSRXn6ni9" + }, + "source": [ + "## Where Do I Go From Here?\n", + "\n", + "You've reached the end of this online `triage` tutorial, but have a few potential options for where to go next:\n", + "- Feel free to tinker with the parameters, features, and outputs of the DonorsChoose example here for a working sandbox to start from (keeping in mind the resource constraints of colab, of course)\n", + "- For a more comprehensive tutorial that explores `triage` concepts and functionality in much more depth, see our [Dirty Duck Tutorial](https://dssg.github.io/triage/dirtyduck/)\n", + "- If you're ready to get started with using `triage` for your own project, check out our [Quickstart Guide](https://dssg.github.io/triage/quickstart/)" + ] + } + ] +} \ No newline at end of file