Skip to content
This repository has been archived by the owner on Aug 29, 2024. It is now read-only.

Commit

Permalink
Upgrade ipywidgets and widgetsnbextension for auto-cube notebook (#362)
Browse files Browse the repository at this point in the history 
* fixed broken ipywidget

* Upgrade ipywidgets and widgetsnbextension

* formatted auto-cube

* Update auto-cube
  • Loading branch information
@HuifangYeo
HuifangYeo authored Sep 6, 2022
1 parent 296da5f commit f991938
Show file tree
Hide file tree
Showing 5 changed files with 2,776 additions and 469 deletions.
Binary file added notebooks/auto-cube/content/content.mv.db
View file
Binary file not shown.
156 changes: 114 additions & 42 deletions notebooks/auto-cube/main-advanced.ipynb
View file
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@
"\n",
"[atoti](https://www.atoti.io/) is a free Python BI analytics platform for Quants, Data Analysts, Data Scientists & Business Users to collaborate better, analyze faster and translate their data into business KPIs. \n",
"\n",
"This notebook is an extension of [main.ipynb](main.ipynb), demonstrating how users could customize the data type of each column. This is useful particularly for columns storing an array list. We will also the atoti session and its attributes in this notebook after the BI application is created (with reference to the [VaR dataset](https://s3.eu-west-3.amazonaws.com/data.atoti.io/notebooks/auto-cube/var_dataset.csv)). \n",
"This notebook is an extension of [main.ipynb](main.ipynb), demonstrating how users could customize the data type of each column. This is useful particularly for columns storing an array list. We will also the atoti session and its attributes in this notebook after the BI application is created (with reference to the [VaR dataset](https://s3.eu-west-3.amazonaws.com/data.atoti.io/notebooks/auto-cube/var_dataset.csv)). \n",
"\n",
"<img src=\"https://data.atoti.io/notebooks/auto-cube/spin-up-cube.gif\" width=\"70%\" />\n",
"\n",
Expand All @@ -34,7 +34,24 @@
"execution_count": 1,
"id": "c9a014b6-e1ff-4d7f-a5d7-0cc4429de899",
"metadata": {},
"outputs": [],
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Welcome to atoti 0.7.0!\n",
"\n",
"By using this community edition, you agree with the license available at https://docs.atoti.io/latest/eula.html.\n",
"Browse the official documentation at https://docs.atoti.io.\n",
"Join the community at https://www.atoti.io/register.\n",
"\n",
"atoti collects telemetry data, which is used to help understand how to improve the product.\n",
"If you don't wish to send usage data, set the ATOTI_DISABLE_TELEMETRY environment variable to True.\n",
"\n",
"You can hide this message by setting the ATOTI_HIDE_EULA_MESSAGE environment variable to True.\n"
]
}
],
"source": [
"import functools\n",
"import io\n",
Expand Down Expand Up @@ -110,8 +127,14 @@
"tags": []
},
"source": [
"## Steps to creating BI analytics platform with atoti\n",
"\n",
"## Steps to creating BI analytics platform with atoti"
]
},
{
"cell_type": "markdown",
"id": "40b67f80-20dc-449d-83d2-cde2bc3c3493",
"metadata": {},
"source": [
"In the following function, the key steps to create an atoti web application are defined:\n",
"- Instantiate atoti session (web application is created upon instantiation)\n",
"- Create atoti table by loading the Pandas DataFrame (atoti also accepts other datasources such as CSV, Parquet, SQL, Spark DataFrame etc.)\n",
Expand Down Expand Up @@ -182,10 +205,18 @@
{
"cell_type": "markdown",
"id": "1cd0ed02-093d-4cd4-864e-4bbe28ebd0b5",
"metadata": {
"tags": []
},
"source": [
"## Data processing prior to BI platform creation"
]
},
{
"cell_type": "markdown",
"id": "4a7e09aa-b14d-4135-a514-aa3392bcb0d6",
"metadata": {},
"source": [
"## Data processing prior to BI platform creation\n",
"\n",
"Using iPyWidget, users are able to:\n",
"- interactively select CSV for upload\n",
"- choose keys for table column and set specific data type for columns where necessary\n",
Expand Down Expand Up @@ -276,10 +307,18 @@
{
"cell_type": "markdown",
"id": "fd5c8c62-5947-4aa2-b22f-8ec3d69b07b3",
"metadata": {
"tags": []
},
"source": [
"## Set the stage with ipywidget"
]
},
{
"cell_type": "markdown",
"id": "164996dc-e90d-4b29-829a-4f71ef1a393d",
"metadata": {},
"source": [
"## Set the stage with ipywidget\n",
"\n",
"Using ipywidget, we can interact with the uploaded data to:\n",
"1. choose keys for the atoti table that we are creating\n",
"2. choose datatype for column (to override the default type inferred by Pandas)"
Expand All @@ -298,14 +337,13 @@
"def on_upload_change(change):\n",
" out.clear_output()\n",
" display(fp)\n",
" print(\"Starting cube creation for \", list(change[\"new\"].keys()))\n",
" print(\"Starting cube creation for \", change[\"new\"][0].name)\n",
"\n",
" fp.value = 0\n",
" print(\"- Reading file\")\n",
" input_file = list(change[\"new\"].values())[0]\n",
" input_file = list(change[\"new\"])[0]\n",
" content = input_file[\"content\"]\n",
" content = io.StringIO(content.decode(\"utf-8\"))\n",
" df = pd.read_csv(content)\n",
" df = pd.read_csv(io.BytesIO(content))\n",
"\n",
" fp.value = 1\n",
" columns = df.columns.tolist()\n",
Expand Down Expand Up @@ -380,16 +418,17 @@
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "13ba1bf7518d4fb89eda34e466d3ce9f",
"model_id": "8331ccb8308c4e0f87c444e3c1ad3cc7",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"Output()"
]
},
"execution_count": 8,
"metadata": {},
"output_type": "display_data"
"output_type": "execute_result"
}
],
"source": [
Expand All @@ -404,8 +443,14 @@
"tags": []
},
"source": [
"## Technology behind atoti \n",
"\n",
"## Technology behind atoti "
]
},
{
"cell_type": "markdown",
"id": "98831de3-822b-44f7-982d-b3ad6882661a",
"metadata": {},
"source": [
"<img src=\"https://data.atoti.io/notebooks/auto-cube/img/atoti-tech-stack.png\" width=\"50%\"/> \n",
"\n",
"### In-memory multidimensional data cube\n",
Expand All @@ -417,9 +462,17 @@
"- drill-through for investigation"
]
},
{
"cell_type": "markdown",
"id": "77c7ed78-afb7-4a7a-8092-c5b25a7e4f5f",
"metadata": {},
"source": [
"__NOTE__: Change the `raw` cells below to `code` cells after you have created your cube. The below code snippets are based on the data cube created using the [VaR dataset](https://s3.eu-west-3.amazonaws.com/data.atoti.io/notebooks/auto-cube/var_dataset.csv)."
]
},
{
"cell_type": "raw",
"id": "469a6c62-f577-424e-87af-7e97aa92b288",
"id": "f90b3889-fb36-4758-8b75-6a54d7ea6a6f",
"metadata": {},
"source": [
"cube"
Expand All @@ -440,13 +493,12 @@
},
{
"cell_type": "raw",
"id": "cd824d22-af32-4b76-b1e7-6e6ef7e1dfe4",
"id": "5b9660ff-840f-487f-9f71-b73b38a670bd",
"metadata": {
"atoti": {
"widget": {
"columnWidths": {
"[Measures].[pnl_vector.VALUE]": 228.80621337890625,
"[table].[instrument_code].[instrument_code]": 178.390625
"[Measures].[pnl_vector.VALUE]": 309.92498779296875
},
"mapping": {
"columns": [
Expand All @@ -456,11 +508,12 @@
"[Measures].[pnl_vector.VALUE]"
],
"rows": [
"[table].[book].[book]",
"[table].[instrument_code].[instrument_code]"
]
},
"query": {
"mdx": "SELECT NON EMPTY Hierarchize(Descendants({[table].[instrument_code].[AllMember]}, 1, SELF_AND_BEFORE)) ON ROWS, NON EMPTY {[Measures].[pnl_vector.VALUE]} ON COLUMNS FROM [table] CELL PROPERTIES VALUE, FORMATTED_VALUE, BACK_COLOR, FORE_COLOR, FONT_FLAGS",
"mdx": "SELECT NON EMPTY {[Measures].[pnl_vector.VALUE]} ON COLUMNS, NON EMPTY Crossjoin(Hierarchize(Descendants({[table].[book].[AllMember]}, 1, SELF_AND_BEFORE)), Hierarchize(Descendants({[table].[instrument_code].[AllMember]}, 1, SELF_AND_BEFORE))) ON ROWS FROM [table] CELL PROPERTIES VALUE, FORMATTED_VALUE, BACK_COLOR, FORE_COLOR, FONT_FLAGS",
"updateMode": "once"
},
"serverKey": "default",
Expand All @@ -483,7 +536,7 @@
},
{
"cell_type": "raw",
"id": "318c0a5c-ed1c-4e86-be4c-25d60a27cef4",
"id": "cea555e2-12ef-4bc0-81c6-128afb3fc89b",
"metadata": {},
"source": [
"h, l, m = cube.hierarchies, cube.levels, cube.measures"
Expand All @@ -499,29 +552,33 @@
},
{
"cell_type": "raw",
"id": "f2673684-a4bd-4ebf-9341-1a173d026b75",
"id": "53563532-ba17-4599-9f70-98e0b1e27e01",
"metadata": {},
"source": [
"m[\"scaled_pnl_vector\"] = m[\"quantity.SUM\"] * m[\"pnl_vector.VALUE\"]"
]
},
{
"cell_type": "raw",
"id": "0300e606-3ad3-44e4-a63d-c4f59373023c",
"id": "5d7277b0-13aa-45b8-bcea-efca8003d921",
"metadata": {},
"source": [
"m[\"Position vector\"] = tt.agg.sum(m[\"scaled_pnl_vector\"], scope=tt.OriginScope(l[\"instrument_code\"], l[\"book_id\"]))"
"m[\"Position vector\"] = tt.agg.sum(\n",
" m[\"scaled_pnl_vector\"], scope=tt.OriginScope(l[\"instrument_code\"], l[\"book_id\"])\n",
")"
]
},
{
"cell_type": "raw",
"id": "23f45d39-a876-4695-819f-c1898c084e0e",
"id": "bf3c6a76-a8ee-4fb6-9a76-f28573a6759e",
"metadata": {
"atoti": {
"height": 487,
"widget": {
"columnWidths": {
"[Measures].[Position vector]": 317,
"[Measures].[pnl_vector.VALUE]": 189.21875,
"[Measures].[quantity.SUM]": 96,
"[Measures].[scaled_pnl_vector]": 292.22503662109375,
"[confidence_simulation].[confidence_simulation].[90%],[Measures].[Position vector]": 208,
"[confidence_simulation].[confidence_simulation].[90%],[Measures].[scaled_pnl_vector]": 176.77496337890625,
Expand All @@ -534,6 +591,8 @@
"ALL_MEASURES"
],
"measures": [
"[Measures].[quantity.SUM]",
"[Measures].[pnl_vector.VALUE]",
"[Measures].[scaled_pnl_vector]",
"[Measures].[Position vector]"
],
Expand All @@ -542,7 +601,7 @@
]
},
"query": {
"mdx": "SELECT NON EMPTY Hierarchize(Descendants({[table].[instrument_code].[AllMember]}, 1, SELF_AND_BEFORE)) ON ROWS, NON EMPTY {[Measures].[scaled_pnl_vector], [Measures].[Position vector]} ON COLUMNS FROM [table] CELL PROPERTIES VALUE, FORMATTED_VALUE, BACK_COLOR, FORE_COLOR, FONT_FLAGS",
"mdx": "SELECT NON EMPTY Hierarchize(Descendants({[table].[instrument_code].[AllMember]}, 1, SELF_AND_BEFORE)) ON ROWS, NON EMPTY {[Measures].[quantity.SUM], [Measures].[pnl_vector.VALUE], [Measures].[scaled_pnl_vector], [Measures].[Position vector]} ON COLUMNS FROM [table] CELL PROPERTIES VALUE, FORMATTED_VALUE, BACK_COLOR, FORE_COLOR, FONT_FLAGS",
"updateMode": "once"
},
"serverKey": "default",
Expand All @@ -557,43 +616,51 @@
},
{
"cell_type": "markdown",
"id": "8210c392-701f-4bae-8745-2428a7baddc2",
"id": "09aba44e-5c70-4530-b4d3-a927b25fde65",
"metadata": {},
"source": [
"### Running simulations"
]
},
{
"cell_type": "raw",
"id": "3cefdeca-f9f1-4234-9035-221399fed514",
"id": "80d4cc38-f705-4a1d-ba2a-4281524fa834",
"metadata": {},
"source": [
"# m[\"VaR\"] = tt.array.quantile(m[\"Position vector\"], 0.95)"
]
},
{
"cell_type": "raw",
"id": "75fa1171-7ae3-4279-a8b4-5eea3cdee213",
"metadata": {},
"source": [
"confidence_simulation = cube.create_parameter_simulation(\n",
" \"confidence_simulation\",\n",
" measures={\"Confidence level\": 0.95},\n",
" base_scenario_name=\"95%\"\n",
" base_scenario_name=\"95%\",\n",
")"
]
},
{
"cell_type": "raw",
"id": "8c6620fe-1019-4455-8ae3-8dd3dbc7140e",
"id": "54a6d767-08f8-48c1-945f-a679ef79bc68",
"metadata": {},
"source": [
"cube.query(m[\"Confidence level\"])"
]
},
{
"cell_type": "raw",
"id": "8e533b42-b31c-4bdb-abda-a83b0432370b",
"id": "1992eede-83ae-44e5-9233-92d646bb9ca8",
"metadata": {},
"source": [
"m[\"VaR\"] = tt.array.quantile(m[\"Position vector\"], m[\"Confidence level\"])"
]
},
{
"cell_type": "raw",
"id": "cfc51b8f-0d2a-4b79-8673-ec2f1e897f96",
"id": "0d71de2f-6ec8-4a95-a53d-70e89da5b243",
"metadata": {
"atoti": {
"height": 350,
Expand Down Expand Up @@ -636,7 +703,7 @@
},
{
"cell_type": "raw",
"id": "ebc531f4-c764-4f78-b2ff-d1d74736df97",
"id": "7b5a8910-3454-4c65-85ed-030d2dc79e8e",
"metadata": {},
"source": [
"confidence_simulation += (\"90%\", 0.90)\n",
Expand All @@ -645,7 +712,7 @@
},
{
"cell_type": "raw",
"id": "59fda9ab-9ce8-45f4-a056-5a789bc6d95e",
"id": "bc744520-00b8-4417-927a-856a2692bb4e",
"metadata": {
"atoti": {
"height": 474,
Expand Down Expand Up @@ -683,24 +750,29 @@
{
"cell_type": "markdown",
"id": "147c33c9-3cb2-4dde-a30a-62fd7969853e",
"metadata": {},
"metadata": {
"tags": []
},
"source": [
"## Find out more about atoti\n",
"\n",
"<table style=\"width: 100%\">\n",
"<thead>\n",
" <tr>\n",
" <td rowspan=\"4\" style=\"width: 420px\"><img src=\"https://data.atoti.io/notebooks/auto-cube/img/qr-code.png\" width=\"400px\"/></td>\n",
" <td><div style=\"display: inline-block;\"><img src=\"https://data.atoti.io/notebooks/covid-analytics/img/linkedin.png\" width = \"50px\"/></div><div style=\"font-size:30px;display: inline-block;padding-left: 15px; vertical-align: top;\">LinkedIn https://www.linkedin.com/company/atoti</div></td>\n",
" <td rowspan=\"5\" style=\"width: 420px\"><img src=\"https://data.atoti.io/notebooks/auto-cube/img/qr-code.png\" width=\"400px\"/></td>\n",
" <td><div style=\"display: inline-block; vertical-align: bottom;\"><img src=\"https://data.atoti.io/notebooks/banners/logo.png\" width = \"50px\"/></div><div style=\"font-size:30px;display: inline-block;padding-left: 15px; vertical-align: top;\">https://www.atoti.io/</div></td>\n",
" </tr>\n",
" <tr>\n",
" <td><div style=\"display: inline-block;\"><img src=\"https://data.atoti.io/notebooks/covid-analytics/img/linkedin.png\" width = \"50px\"/></div><div style=\"font-size:30px;display: inline-block;padding-left: 15px; vertical-align: top;\">https://www.linkedin.com/company/atoti</div></td>\n",
" </tr>\n",
" <tr>\n",
" <td><div style=\"display: inline-block;\"><img src=\"https://data.atoti.io/notebooks/covid-analytics/img/twitter.png\" width = \"50px\"/></div><div style=\"font-size:30px;display: inline-block;padding-left: 15px; vertical-align: top;\">Twitter https://twitter.com/atoti_io</div></td>\n",
" <td><div style=\"display: inline-block;\"><img src=\"https://data.atoti.io/notebooks/covid-analytics/img/twitter.png\" width = \"50px\"/></div><div style=\"font-size:30px;display: inline-block;padding-left: 15px; vertical-align: top;\">https://twitter.com/atoti_io</div></td>\n",
" </tr>\n",
" <tr>\n",
" <td><div style=\"display: inline-block;\"><img src=\"https://data.atoti.io/notebooks/covid-analytics/img/youtube.png\" width = \"50px\"/></div><div style=\"font-size:30px;display: inline-block;padding-left: 15px; vertical-align: top;\">YouTube https://www.youtube.com/c/atoti</div></td>\n",
" <td><div style=\"display: inline-block;\"><img src=\"https://data.atoti.io/notebooks/covid-analytics/img/youtube.png\" width = \"50px\"/></div><div style=\"font-size:30px;display: inline-block;padding-left: 15px; vertical-align: top;\">https://www.youtube.com/c/atoti</div></td>\n",
" </tr>\n",
" <tr>\n",
" <td><div style=\"display: inline-block;\"><img src=\"https://data.atoti.io/notebooks/covid-analytics/img/medium.png\" width = \"50px\"/></div><div style=\"font-size:30px;display: inline-block;padding-left: 15px; vertical-align: top;\">Medium https://medium.com/atoti</div></td>\n",
" <td><div style=\"display: inline-block;\"><img src=\"https://data.atoti.io/notebooks/covid-analytics/img/medium.png\" width = \"50px\"/></div><div style=\"font-size:30px;display: inline-block;padding-left: 15px; vertical-align: top;\">https://medium.com/atoti</div></td>\n",
" </tr>\n",
"</thead>\n",
"</table>\n",
Expand Down Expand Up @@ -739,7 +811,7 @@
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.9.9"
"version": "3.10.6"
}
},
"nbformat": 4,
Expand Down
Loading

0 comments on commit f991938

Please sign in to comment.