Skip to content

Commit

Permalink
version 0.1.2
Browse files Browse the repository at this point in the history
  • Loading branch information
@allisonportis
allisonportis committed Aug 14, 2019
1 parent 0c55fef commit 989d103
Show file tree
Hide file tree
Showing 2 changed files with 290 additions and 1 deletion.
289 changes: 289 additions & 0 deletions autonormalize/demos/Editing Dependnecies Demo.ipynb
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,289 @@
{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Demo with Editing Dependencies"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"This is a quick demo for a situation where you may want to edit the dependencies AutoNormalize discovers. In this example, extra dependencies are detected due to lack of data. "
]
},
{
"cell_type": "code",
"execution_count": 28,
"metadata": {},
"outputs": [],
"source": [
"import pandas as pd\n",
"import autonormalize as an"
]
},
{
"cell_type": "code",
"execution_count": 29,
"metadata": {
"scrolled": true
},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>team</th>\n",
" <th>city</th>\n",
" <th>state</th>\n",
" <th>roster_size</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>tigers</td>\n",
" <td>boston</td>\n",
" <td>MA</td>\n",
" <td>20</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>elephants</td>\n",
" <td>chicago</td>\n",
" <td>IL</td>\n",
" <td>21</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>foxes</td>\n",
" <td>miami</td>\n",
" <td>FL</td>\n",
" <td>20</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>snakes</td>\n",
" <td>austin</td>\n",
" <td>TX</td>\n",
" <td>20</td>\n",
" </tr>\n",
" <tr>\n",
" <th>4</th>\n",
" <td>dolphins</td>\n",
" <td>honolulu</td>\n",
" <td>HI</td>\n",
" <td>19</td>\n",
" </tr>\n",
" <tr>\n",
" <th>5</th>\n",
" <td>eagles</td>\n",
" <td>houston</td>\n",
" <td>TX</td>\n",
" <td>21</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" team city state roster_size\n",
"0 tigers boston MA 20\n",
"1 elephants chicago IL 21\n",
"2 foxes miami FL 20\n",
"3 snakes austin TX 20\n",
"4 dolphins honolulu HI 19\n",
"5 eagles houston TX 21"
]
},
"execution_count": 29,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"rows = [['tigers', 'boston', 'MA', 20],\n",
" ['elephants', 'chicago', 'IL', 21],\n",
" ['foxes', 'miami', 'FL', 20],\n",
" ['snakes', 'austin', 'TX', 20],\n",
" ['dolphins', 'honolulu', 'HI', 19],\n",
" ['eagles', 'houston', 'TX', 21]]\n",
"df = pd.DataFrame(rows, columns=['team', 'city', 'state', 'roster_size'])\n",
"df"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"As you can see, because we have such little data, there is only one team represented per city. Logically, we know that there can be more than one team per city, but the algorithm doesn't. Thus it detects the dependencies {city} --> team, and {city} --> roster_size. "
]
},
{
"cell_type": "code",
"execution_count": 25,
"metadata": {},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"100%|██████████| 2/2 [00:00<00:00, 141.40it/s]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
" {city} --> team\n",
" {team} --> city\n",
" {team} {city} --> state\n",
" {team} {city} --> roster_size\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\n"
]
}
],
"source": [
"deps = an.find_dependencies(df)\n",
"print(deps)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Because we logically know this is incorrect, we just remove these dependencies before proceeding to normalization and creating our `EntitySet`."
]
},
{
"cell_type": "code",
"execution_count": 26,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
" --> team\n",
" {team} --> city\n",
" {team} {city} --> state\n",
" {team} --> roster_size\n"
]
}
],
"source": [
"deps.remove_dep('team', ['city'])\n",
"deps.remove_dep('roster_size', ['city'])\n",
"print(deps)"
]
},
{
"cell_type": "code",
"execution_count": 27,
"metadata": {},
"outputs": [
{
"data": {
"image/svg+xml": [
"<?xml version=\"1.0\" encoding=\"UTF-8\" standalone=\"no\"?>\n",
"<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n",
" \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n",
"<!-- Generated by graphviz version 2.40.1 (20161225.0304)\n",
" -->\n",
"<!-- Title: %3 Pages: 1 -->\n",
"<svg width=\"141pt\" height=\"176pt\"\n",
" viewBox=\"0.00 0.00 141.40 176.00\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n",
"<g id=\"graph0\" class=\"graph\" transform=\"scale(1 1) rotate(0) translate(4 172)\">\n",
"<title>%3</title>\n",
"<polygon fill=\"#ffffff\" stroke=\"transparent\" points=\"-4,4 -4,-172 137.4004,-172 137.4004,4 -4,4\"/>\n",
"<!-- team -->\n",
"<g id=\"node1\" class=\"node\">\n",
"<title>team</title>\n",
"<polygon fill=\"none\" stroke=\"#000000\" points=\"0,-95.5 0,-167.5 133.4004,-167.5 133.4004,-95.5 0,-95.5\"/>\n",
"<text text-anchor=\"middle\" x=\"66.7002\" y=\"-152.3\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">team</text>\n",
"<polyline fill=\"none\" stroke=\"#000000\" points=\"0,-145.5 133.4004,-145.5 \"/>\n",
"<text text-anchor=\"start\" x=\"8\" y=\"-130.3\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">team : index</text>\n",
"<text text-anchor=\"start\" x=\"8\" y=\"-116.3\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">city : id</text>\n",
"<text text-anchor=\"start\" x=\"8\" y=\"-102.3\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">roster_size : numeric</text>\n",
"</g>\n",
"<!-- city -->\n",
"<g id=\"node2\" class=\"node\">\n",
"<title>city</title>\n",
"<polygon fill=\"none\" stroke=\"#000000\" points=\"9.7275,-.5 9.7275,-58.5 123.6729,-58.5 123.6729,-.5 9.7275,-.5\"/>\n",
"<text text-anchor=\"middle\" x=\"66.7002\" y=\"-43.3\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">city</text>\n",
"<polyline fill=\"none\" stroke=\"#000000\" points=\"9.7275,-36.5 123.6729,-36.5 \"/>\n",
"<text text-anchor=\"start\" x=\"17.7275\" y=\"-21.3\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">city : index</text>\n",
"<text text-anchor=\"start\" x=\"17.7275\" y=\"-7.3\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">state : categorical</text>\n",
"</g>\n",
"<!-- team&#45;&gt;city -->\n",
"<g id=\"edge1\" class=\"edge\">\n",
"<title>team&#45;&gt;city</title>\n",
"<path fill=\"none\" stroke=\"#000000\" d=\"M66.7002,-95.2514C66.7002,-95.2514 66.7002,-68.5905 66.7002,-68.5905\"/>\n",
"<polygon fill=\"#000000\" stroke=\"#000000\" points=\"70.2003,-68.5904 66.7002,-58.5905 63.2003,-68.5905 70.2003,-68.5904\"/>\n",
"<text text-anchor=\"middle\" x=\"56.2036\" y=\"-84.7209\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">city</text>\n",
"</g>\n",
"</g>\n",
"</svg>\n"
],
"text/plain": [
"<graphviz.dot.Digraph at 0x1060b11d0>"
]
},
"execution_count": 27,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"es = an.make_entityset(df, deps)\n",
"es.plot()"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.7.4"
}
},
"nbformat": 4,
"nbformat_minor": 2
}
2 changes: 1 addition & 1 deletion setup.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@

setup(
name='autonormalize',
version='0.0.0',
version='0.1.2',
description='a library for automated table normalization',
url='http://featuretools.com',
license='BSD 3-clause',
Expand Down

0 comments on commit 989d103

Please sign in to comment.