Python_for_data_science_best_practices.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ce745f84",
   "metadata": {},
   "outputs": [],
   "source": [
    "# pip install guppy3"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3498fb00",
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd \n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d4395887",
   "metadata": {},
   "source": [
    "### 1. Configuring setting and importing libraries\n",
    "\n",
    "* When starting a Jupyter notebook, it's considered good practice to configure settings such as plot sizes and display formats at the beginning. \n",
    "  * This only makes sense if settings are used throughout the notebook, not to modify a single plot\n",
    "\n",
    "* This practice helps maintain consistency throughout the notebook and ensures that all visualizations and outputs are standardizes across notebooks\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c48dd193",
   "metadata": {},
   "outputs": [],
   "source": [
    "example_df = pd.DataFrame({\"A\": np.arange(100), \"B\": np.linspace(0, 55, 100)})\n",
    "\n",
    "example_df.head()\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c576b728",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Modify the column B so it only contains two decimal points\n",
    "example_df = pd.DataFrame({\"A\": np.arange(100), \"B\": np.linspace(0, 55, 100)})\n",
    "example_df['B'] = example_df['B'].round(2)\n",
    "example_df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "57c2cae3",
   "metadata": {},
   "outputs": [],
   "source": [
    "pd.options.display.float_format = '{:,.2f}'.format"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a4bcf334",
   "metadata": {},
   "outputs": [],
   "source": [
    "example_df = pd.DataFrame({\"A\": np.arange(100), \"B\": np.linspace(0, 55, 100)})\n",
    "example_df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3d4d8ef7",
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.plot(example_df['B'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4d13cd20",
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.figure(figsize=(14,4))\n",
    "plt.plot(example_df['B'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d5825f8c",
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.rcParams"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ec0ad092",
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.rcParams[\"figure.figsize\"] = (14,4)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0a7b71e9",
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.plot(example_df['B'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9b3c5317",
   "metadata": {},
   "outputs": [],
   "source": [
    "pd.options.display.float_format = None"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ae799d26",
   "metadata": {},
   "outputs": [],
   "source": [
    "df = pd.DataFrame({\n",
    "    'Large Numbers': [100000000000000000000000000.0]\n",
    "})\n",
    "df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d91d7e0c",
   "metadata": {},
   "outputs": [],
   "source": [
    "pd.options.display.max_rows = 5"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2a02ce17",
   "metadata": {},
   "outputs": [],
   "source": [
    "example_df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2303c23d",
   "metadata": {},
   "outputs": [],
   "source": [
    "# You can also set the display format to suppress scientific notation for floating-point numbers\n",
    "pd.options.display.float_format = '{:.2f}'.format"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "00de0606",
   "metadata": {},
   "outputs": [],
   "source": [
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "721bd35f",
   "metadata": {},
   "source": [
    "\n",
    "### 2. Avoiding Unintended References\n",
    "\n",
    "* A view in pandas is a subset of the original object ( DataFrame or Series ) linked to the original source. \n",
    " * When you create a view, you are not creating a new copy of the data, but rather a reference to the original data. \n",
    " * Changes you make to the view will be reflected in the original data, and vice versa.\n",
    "\n",
    "* When you work directly with views on the data, instead of simple column assignment, you may end up with a reference to the original data rather than a copy. \n",
    "* Also, many of you have experienced the \"SettingWithCopyWarning.\" \n",
    "  *  triggered when you try to modify a DataFrame that might be a view on another DataFrame\n",
    "   * Pandas cannot definitively tell which. \n",
    "\n",
    "* If you want to ensure that your modifications do not affect the original DataFrame when working with slices or subsets, you should explicitly create a copy using the .copy() method."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bab27641",
   "metadata": {},
   "outputs": [],
   "source": [
    "example_df['C'] = example_df['A'] * 2\n",
    "example_df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e9f2af2d",
   "metadata": {},
   "outputs": [],
   "source": [
    "example_df.loc[0, 'A'] =  10\n",
    "example_df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ed4360db",
   "metadata": {},
   "outputs": [],
   "source": [
    "example_df['D'] = example_df['A']\n",
    "example_df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e96dcfeb",
   "metadata": {},
   "outputs": [],
   "source": [
    "example_df.loc[0, 'A'] =  20\n",
    "example_df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "428f46a2",
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "slice_df = example_df.loc[0:3]  \n",
    "slice_df\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f3db26db",
   "metadata": {},
   "outputs": [],
   "source": [
    "slice_df.iloc[0] = (0, 0, 0, 0)\n",
    "slice_df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "80042820",
   "metadata": {},
   "outputs": [],
   "source": [
    "example_df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "27157384",
   "metadata": {},
   "outputs": [],
   "source": [
    "slice_df = example_df.loc[0:3].copy()  # This creates a view on the original DataFrame\n",
    "slice_df\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7560a8ac",
   "metadata": {},
   "outputs": [],
   "source": [
    "slice_df.iloc[0] = (1, 1, 1, 1)\n",
    "slice_df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e98e35b5",
   "metadata": {},
   "outputs": [],
   "source": [
    "example_df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1fbc7c15",
   "metadata": {},
   "outputs": [],
   "source": [
    "df = pd.DataFrame({'A': [1, 2, 3], 'B': [4, 5, 6]})\n",
    "\n",
    "# Chained indexing, potentially leading to SettingWithCopyWarning\n",
    "example_2_df = df[df['A'] > 1]\n",
    "example_2_df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "eb85e579",
   "metadata": {},
   "outputs": [],
   "source": [
    "example_2_df['B'] = [50, 60] "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1668201e",
   "metadata": {},
   "outputs": [],
   "source": [
    "example_2_df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c253b3c3",
   "metadata": {},
   "outputs": [],
   "source": [
    "df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "598dd968",
   "metadata": {},
   "outputs": [],
   "source": [
    "df = pd.DataFrame({'A': [1, 2, 3], 'B': [4, 5, 6]})\n",
    "\n",
    "# Chained indexing, potentially leading to SettingWithCopyWarning\n",
    "example_2_df = df[df['A'] > 1]\n",
    "example_2_df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "aad87574",
   "metadata": {},
   "outputs": [],
   "source": [
    "df.loc[df['A'] > 1, 'B'] = [50, 60]\n",
    "df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "458c592f",
   "metadata": {},
   "outputs": [],
   "source": [
    "df = pd.DataFrame({'A': [1, 2, 3], 'B': [4, 5, 6]})\n",
    "\n",
    "subset_df = df.loc[df['A'] > 1]\n",
    "subset_df['C'] = subset_df['B'] + 10  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ce2c6085",
   "metadata": {},
   "outputs": [],
   "source": [
    "subset_df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "405a745e",
   "metadata": {},
   "outputs": [],
   "source": [
    "df = pd.DataFrame({'A': [1, 2, 3], 'B': [4, 5, 6]})\n",
    "\n",
    "df.loc[df['A'] > 1, 'C'] = df.loc[df['A'] > 1, 'B'] + 10"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "550343d7",
   "metadata": {},
   "outputs": [],
   "source": [
    "df = pd.DataFrame({'A': [1, 2, 3], 'B': [4, 5, 6]})\n",
    "df[df['A']> 1]['B']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "037fda6c",
   "metadata": {},
   "outputs": [],
   "source": [
    "x = df[df['A']> 1]['B']\n",
    "x._is_view"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5a601a3a",
   "metadata": {},
   "outputs": [],
   "source": [
    "x = example_df.loc[example_df['A']> 1, 'B']\n",
    "x._is_view"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ab114411",
   "metadata": {},
   "source": [
    "##### example\n",
    "```python\n",
    "import pandas, numpy\n",
    "num_states = 50\n",
    "def PROCESSDATA(file_name, int):\n",
    "    global df\n",
    "    df = pd.read_csv(file_name)\n",
    "    df = df.fillna(df.median())\n",
    "    if int > 0: df = df.sample(int)\n",
    "```\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cc4061fe",
   "metadata": {},
   "source": [
    "### Improving Readability and Maintainability by Adhering to PEP8\n",
    "* The PEP8 is Python's official style guide,\n",
    "  * Python Enhancement Proposal: a design document providing information to the Python community, or describing a new feature for Python or its processes or environment. \n",
    "  * PEPs are intended as the primary mechanisms for proposing major new features, collecting community input on an issue, and documenting the design decisions that have gone into Python. T\n",
    " * PEP8:\n",
    "   * Naming vairables and importing packages\n",
    "   * Avoiding compound statements\n",
    "   * Avoiding builtin functions or keywords as variable names\n",
    "   * Avoiding global variables"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "36e5d8ad",
   "metadata": {},
   "source": [
    "### example \n",
    "* Avoid compound aliases and non-standard aliases\n",
    "\n",
    "```python\n",
    "import padas as ada, numpy as ny, matplotlib.pyplot as mtb\n",
    "```\n",
    "* One import per line, use standard aliases\n",
    "\n",
    "```python\n",
    "import padas as pd, numpy as np, matplotlib.pyplot as plt\n",
    "import padas as ada, numpy as ny, matplotlib.pyplot as mtb\n",
    "```\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e0540c07",
   "metadata": {},
   "source": [
    "### Important PEP8  Recommendations\n",
    "\n",
    "* Avoid compound statement\n",
    "```python\n",
    "if x > 0: df1 = df.sample(10); df2 = df.sample(20); x +=1;\n",
    "```    \n",
    "\n",
    "```python\n",
    "if income > 0:\n",
    "    df = df.sample(nb)\n",
    "```    \n",
    "* Write variable names appropriately\n",
    "  * Use snake_case for variables and functions, UPPER_CASE for constants, and lowercase for general variables.\n",
    "* Functions should be in lowercase and use snake_case to improve readability.\n",
    "* Classes should be CamelCase and start with an uppercase letter to distinguish them from functions and variables.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "50785957",
   "metadata": {},
   "outputs": [],
   "source": [
    "for i in range(10):\n",
    "    sum += i\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0c491daf",
   "metadata": {},
   "outputs": [],
   "source": [
    "a = input()\n",
    "print(a)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6057dbeb",
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a60f10ad",
   "metadata": {},
   "outputs": [],
   "source": [
    "a = pd.DataFrame({'col_1': [1,2,3]})\n",
    "\n",
    "def f(some_df):\n",
    "    some_df['col_1'] = [4,5,6]\n",
    "    return some_df\n",
    "f(a)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cc8016b0",
   "metadata": {},
   "outputs": [],
   "source": [
    "print(a)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "31779a00",
   "metadata": {},
   "source": [
    "### Important PEP8  Recommendations - Cont'd\n",
    "\n",
    "* Never use a builtin function or keyword as a variable name\n",
    "  * I see it often with `str`, `list`, `id`, `sum`, `min`, `max`, `input` etc. \n",
    "  * Make sure you know what variables or function your package is importing so you don't overwrite them\n",
    "\n",
    "* Avoid global variables\n",
    " * considered bad practice from a styling perspective \n",
    "   * Changes to global variables can be made from anywhere in the program, making it difficult to understand how and when their values are modified.\n",
    "   *  Functions that rely on global variables are closely tied to those variables\n",
    "     * Reducing modularity and making the code harder to maintain or reuse.\n",
    "  * Testing functions that depend on global variables can be challenging \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2721be96",
   "metadata": {},
   "outputs": [],
   "source": [
    "a=1+2\n",
    "a = 1 + 2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b797c7dd",
   "metadata": {},
   "outputs": [],
   "source": [
    "class X:\n",
    "    pass\n",
    "\n",
    "\n",
    "class Y:\n",
    "    pass\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "puc\n",
    "\n",
    "def z:\n",
    "    pass\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1214a48d",
   "metadata": {},
   "source": [
    "### Important PEP8  Recommendations - Cont'd\n",
    "\n",
    "* Use Spaces Appropriately:\n",
    "  * Around arithmetic operators (+, -, *, /)\n",
    "  * Comparison operators (==, !=, <, >, <=, >=)\n",
    "  * Assignments (=) for better readability. \n",
    "   ```x = 1 + 2``\n",
    "   \n",
    "* After commas\n",
    "  `my_function(arg1, arg2, arg3)` not `my_function(arg1,arg2,arg3)`\n",
    "\n",
    "* Separate functions and classes with two blank lines\n",
    "\n",
    "* Single blank line to separate logical sections inside functions\n",
    "\n",
    "* Avoid excessive blank lines"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fbd72d6b",
   "metadata": {},
   "source": [
    "#### Use Function and Modularize Your Code\n",
    "* Define functions to encapsulate reusable code.\n",
    "  \n",
    "* Functions in modules help break down complex code into smaller, manageable parts making the notebook more organized and easier to read.\n",
    "\n",
    "\n",
    "* Functions can be individually tested to ensure they perform as expected. \n",
    "  * This is much harder with code that is not broken down into modules or functions.\n",
    "  \n",
    "* Use defensive programming when your code will be used by others "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ffc742a5",
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "* In a separate python file (e.g., `modulename.py`)\n",
    "```python\n",
    "\n",
    "import random\n",
    "def magic_addition(a, b)\n",
    "    retun a + b + random.randint(a, b)\n",
    "```\n",
    "\n",
    "* In your notebook \n",
    "```\n",
    "import modulename as mn\n",
    "mn.magic_addition(10, 20)\n",
    "```\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f27be979",
   "metadata": {},
   "outputs": [],
   "source": [
    "import random\n",
    "ran"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7262ed72",
   "metadata": {},
   "source": [
    "### Document and Test your Functions Appropriately\n",
    "\n",
    "* Type hints can be simple classes like float or str or be more complex. \n",
    "  * The typing module provides a vocabulary of more advanced type hints.\n",
    "  * Using pydantic to crete more advanced type hints and handle validation\n",
    "* Increasingly critical to document complex functions\n",
    "  * Makes it easy to know what the function expects as input, returns and what it does.\n",
    "  * Makes it easier for generative code to know to interpret and match natural language to your code\n",
    "  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "70f2b2c5",
   "metadata": {},
   "outputs": [],
   "source": [
    "### Specifying Parameter Data Types in a Function\n",
    "import random\n",
    "def magic_addition(a:int, b:int) -> int:\n",
    "    return a + b + random.randint(a, b)\n",
    "\n",
    "magic_addition(\"A\", \"B\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c825d97d",
   "metadata": {},
   "outputs": [],
   "source": [
    "#magic_addition()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c4b06598",
   "metadata": {},
   "outputs": [],
   "source": [
    "magic_addition(1, 10.0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e0fca319",
   "metadata": {},
   "outputs": [],
   "source": [
    "magic_addition(\"one\", \"test\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9fd30d4a",
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "os.path.isfile(\"/Users/mahdi/Documents/GitHub/phaage-g2p/data/ENA.40.clstr/\")\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2752b071",
   "metadata": {},
   "outputs": [],
   "source": [
    "import random\n",
    "def magic_addition(a:int, b:int) -> int:\n",
    "    if type(a) is not int or type(b) is not int:\n",
    "        raise TypeError(f\"the inputs a and b must be ints> you provided a: {a} and b: {b} \")\n",
    "    return a + b + random.randint(a, b)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ba0a1499",
   "metadata": {},
   "outputs": [],
   "source": [
    "magic_addition(1, \"Test\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "063f3f59",
   "metadata": {},
   "source": [
    "* Using more complex type hints\n",
    "\n",
    "```python\n",
    "type Vector = list[float]\n",
    "\n",
    "def scale(scalar: float, vector: Vector) -> Vector:\n",
    "    return [scalar * num for num in vector]\n",
    "```\n",
    "\n",
    "```python\n",
    "from collections.abc import Sequence\n",
    "\n",
    "type ConnectionOptions = dict[str, str]\n",
    "type Address = tuple[str, int]\n",
    "type Server = tuple[Address, ConnectionOptions]\n",
    "\n",
    "def broadcast_message(message: str, servers: Sequence[Server]) -> None:\n",
    "    pass\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "62c626c3",
   "metadata": {},
   "outputs": [],
   "source": [
    "from pydantic import BaseModel\n",
    "\n",
    "\n",
    "class Foo(BaseModel):\n",
    "    a: str\n",
    "    b: int\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "963db3be",
   "metadata": {},
   "outputs": [],
   "source": [
    "f = Foo(a = \"A\", b=2)\n",
    "f.model_dump_json()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3b63d89f",
   "metadata": {},
   "outputs": [],
   "source": [
    "from pydantic import BaseModel\n",
    "\n",
    "\n",
    "class Foo(BaseModel):\n",
    "    a: str\n",
    "    b: int\n",
    "\n",
    "class Bar(BaseModel):\n",
    "    c: str\n",
    "    d: int\n",
    "\n",
    "def some_function(x: Foo) -> Bar:\n",
    "    print(f\"passed {x}\")\n",
    "    \n",
    "\n",
    "f =Foo(a=\"A\", b=1)\n",
    "a = 1\n",
    "some_function(a)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "83c571d6",
   "metadata": {},
   "outputs": [],
   "source": [
    "f = Foo(a=1, b=1)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9042dda1",
   "metadata": {},
   "outputs": [],
   "source": [
    "f.model_dump()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "928e3f2b",
   "metadata": {},
   "outputs": [],
   "source": [
    "f.model_dump_json()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cb7f0bb9",
   "metadata": {},
   "outputs": [],
   "source": [
    "import random\n",
    "help(random.randint)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9ea7f902",
   "metadata": {},
   "source": [
    "### Documenting your Modules and Functions\n",
    "\n",
    "```python\n",
    "\"\"\"\n",
    "This module provides utility functions for mathematical operations.\n",
    "\n",
    "The functions include simple arithmetic operations like addition and multiplication.\n",
    "\n",
    "Dependencies:\n",
    "math (standard library)\n",
    "\n",
    "Usage:\n",
    "from my_module import add\n",
    "result = add(5, 3)\n",
    "\"\"\"\n",
    "\n",
    "def add(a, b):\n",
    "    \"\"\"\n",
    "    Add two numbers and return the result.\n",
    "\n",
    "    Parameters:\n",
    "    a (int or float): The first number.\n",
    "    b (int or float): The second number.\n",
    "\n",
    "    Returns:\n",
    "    int or float: The sum of the two numbers.\n",
    "    \"\"\"\n",
    "    return a + b\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2bc35b39",
   "metadata": {},
   "source": [
    "### Working with Classes\n",
    "\n",
    "* Benefits of object oriented programming are similar to those listed for functions and modules above.\n",
    "  * create modular resuable code . \n",
    "  * Reduce redundancy and increases efficiency.\n",
    "  * Improve maintainability\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8aed1407",
   "metadata": {},
   "outputs": [],
   "source": [
    "from typing import List\n",
    "import numpy as np\n",
    "class Data:    \n",
    "    def __init__(self, data: List):\n",
    "        if type(data) is not np.array:\n",
    "            data = np.array(data)\n",
    "        self.data = data    \n",
    "        \n",
    "    def start_at_zero(self):\n",
    "        \"\"\"doc goes here\"\"\"\n",
    "        return self.data - self.data[0]\n",
    "        \n",
    "        "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cd0b7d34",
   "metadata": {},
   "outputs": [],
   "source": [
    "d = Data([3, 7, 2, 3])\n",
    "d.data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a1e9665f",
   "metadata": {},
   "outputs": [],
   "source": [
    "d.start_at_zero()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e250988b",
   "metadata": {},
   "outputs": [],
   "source": [
    "class DataCleaner:\n",
    "    def __init__(self, df):\n",
    "        self.df = df\n",
    "\n",
    "    def _remove_missing_values(self):\n",
    "        \"\"\"Remove rows from the DataFrame that contain missing values.\"\"\"\n",
    "        self.df.dropna(inplace=True)\n",
    "        return self.df\n",
    "\n",
    "    def _replace_values(self, column, old_value, new_value):\n",
    "        \"\"\"Replace specific values in a DataFrame column.\"\"\"\n",
    "        self.df[column] = self.df[column].replace(old_value, new_value)\n",
    "        return self.df\n",
    "    \n",
    "    def to_custom_json(self):\n",
    "        json_repr = {x: data[x].tolist() for x in data.columns}\n",
    "        json_repr[\"index\"]  = self.df.index.tolist()\n",
    "        \n",
    "        return json_repr\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d714e582",
   "metadata": {},
   "outputs": [],
   "source": [
    "data = pd.DataFrame({\n",
    "    'Age': [25, 30, 35, 40, None],\n",
    "    'Gender': ['Male', 'Female', 'Female', 'Male', 'Male'],\n",
    "    'Income': [50000, 60000, 75000, 65000, 62000]\n",
    "})\n",
    "\n",
    "#{x: data[x].tolist() for x in data.columns}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e3e713e5",
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "# Instantiate the DataCleaner with the DataFrame\n",
    "df_cleaner = DataCleaner(data)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "66a30854",
   "metadata": {},
   "outputs": [],
   "source": [
    "df_cleaner.to_custom_json()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5d7ff7d8",
   "metadata": {},
   "outputs": [],
   "source": [
    "data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8e60a801",
   "metadata": {},
   "outputs": [],
   "source": [
    "df_cleaner._remove_missing_values()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7858a4da",
   "metadata": {},
   "outputs": [],
   "source": [
    "df_cleaner._replace_values('Gender', 'Male', 'M')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8cb59c8d",
   "metadata": {},
   "outputs": [],
   "source": [
    "class IncomeGenderDataCleaner(DataCleaner):\n",
    "\n",
    "    \n",
    "    \n",
    "    def clean_data(self):\n",
    "        self.df = self._replace_values('Gender', 'Male', 'M')\n",
    "        self.df = self._replace_values('Gender', 'Female', 'F')\n",
    "        self.df = self._remove_missing_values()\n",
    "        \n",
    "        \n",
    "        \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "71e8d0f7",
   "metadata": {},
   "outputs": [],
   "source": [
    "income_clearner = IncomeGenderDataCleaner(data)\n",
    "income_clearner.df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "43ff3967",
   "metadata": {},
   "outputs": [],
   "source": [
    "json.dumps({1:[1,2,4]})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "53b60eb6",
   "metadata": {},
   "outputs": [],
   "source": [
    "import json \n",
    "\n",
    "class IncomeGenderDataCleanerDefault(IncomeGenderDataCleaner):\n",
    "\n",
    "    def __init__(self, df):\n",
    "        super().__init__(df)\n",
    "        self._clean_data()\n",
    "\n",
    "        \n",
    "    \n",
    "    def _clean_data(self):\n",
    "        self.df = self._replace_values('Gender', 'Male', 'M')\n",
    "        self.df = self._replace_values('Gender', 'Female', 'F')\n",
    "        self.df = self._remove_missing_values()\n",
    "        \n",
    "    def __repr__(self):\n",
    "        return json.dumps(self.to_custom_json())\n",
    "        \n",
    "        \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b62f0a9e",
   "metadata": {},
   "outputs": [],
   "source": [
    "custom_income_clearner = IncomeGenderDataCleanerDefault(data)\n",
    "custom_income_clearner "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4d745a48",
   "metadata": {},
   "source": [
    "# Memory Profiling and Managment\n",
    "\n",
    "* Priofiling is importatn for identifying memory consumption by different parts of your code.\n",
    " * Memory profiling helps optimize resource usage and improve performance.\n",
    " \n",
    " * Various packages available for detailed memory usage analysis.\n",
    "   * Perform tasks such as line-by-line profiling or aggregate data type memory usage.\n",
    "   * `guppy` is a popular package for profiling\n",
    "\n",
    "* Profile is useful for \n",
    "  * Quickly identification of potential memory overuse.\n",
    "  * Helps in optimizing data handling and storage efficiency.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "c7304d7a",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Partition of a set of 363704 objects. Total size = 42837782 bytes.\n",
       " Index  Count   %     Size   % Cumulative  % Kind (class / dict of class)\n",
       "     0 105960  29 11183078  26  11183078  26 str\n",
       "     1  74038  20  5324928  12  16508006  39 tuple\n",
       "     2  21066   6  3747087   9  20255093  47 types.CodeType\n",
       "     3  40824  11  3132524   7  23387617  55 bytes\n",
       "     4   8594   2  2984616   7  26372233  62 dict (no owner)\n",
       "     5   2795   1  2794200   7  29166433  68 type\n",
       "     6  19048   5  2590528   6  31756961  74 function\n",
       "     7   2795   1  1358392   3  33115353  77 dict of type\n",
       "     8    979   0  1350984   3  34466337  80 dict of module\n",
       "     9   9851   3   948144   2  35414481  83 list\n",
       "<1074 more rows. Type e.g. '_.more' to view.>"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from guppy import hpy \n",
    "data_type_summary = hpy()\n",
    "data_type_summary.heap()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "4647710e",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Partition of a set of 511284 objects. Total size = 64924173 bytes.\n",
       " Index  Count   %     Size   % Cumulative  % Kind (class / dict of class)\n",
       "     0 144931  28 19106687  29  19106687  29 str\n",
       "     1 109374  21  7978944  12  27085631  42 tuple\n",
       "     2  31373   6  5579572   9  32665203  50 types.CodeType\n",
       "     3  15967   3  5048608   8  37713811  58 dict (no owner)\n",
       "     4  59621  12  4695669   7  42409480  65 bytes\n",
       "     5  28617   6  3891912   6  46301392  71 function\n",
       "     6   3856   1  3627576   6  49928968  77 type\n",
       "     7   1422   0  2203664   3  52132632  80 dict of module\n",
       "     8   3411   1  1736040   3  53868672  83 dict of type\n",
       "     9  10631   2  1063720   2  54932392  85 list\n",
       "<1340 more rows. Type e.g. '_.more' to view.>"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import pandas as pd\n",
    "data_type_summary.heap()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "216cf20a",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "52.659894265209715"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "64924173/1204/1024"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "bf88b880",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['A', 'B', 'C', 'D', 'E', 'F', 'G', 'H', 'I', 'J', 'K', 'L', 'M']"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "col_names = chars = [chr(i) for i in range(ord('A'), ord('M') + 1)]\n",
    "col_names"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a4fdb00a",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "37d0858b",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>A</th>\n",
       "      <th>B</th>\n",
       "      <th>C</th>\n",
       "      <th>D</th>\n",
       "      <th>E</th>\n",
       "      <th>F</th>\n",
       "      <th>G</th>\n",
       "      <th>H</th>\n",
       "      <th>I</th>\n",
       "      <th>J</th>\n",
       "      <th>K</th>\n",
       "      <th>L</th>\n",
       "      <th>M</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>3</td>\n",
       "      <td>3</td>\n",
       "      <td>3</td>\n",
       "      <td>3</td>\n",
       "      <td>3</td>\n",
       "      <td>3</td>\n",
       "      <td>3</td>\n",
       "      <td>3</td>\n",
       "      <td>3</td>\n",
       "      <td>3</td>\n",
       "      <td>3</td>\n",
       "      <td>3</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>4</td>\n",
       "      <td>4</td>\n",
       "      <td>4</td>\n",
       "      <td>4</td>\n",
       "      <td>4</td>\n",
       "      <td>4</td>\n",
       "      <td>4</td>\n",
       "      <td>4</td>\n",
       "      <td>4</td>\n",
       "      <td>4</td>\n",
       "      <td>4</td>\n",
       "      <td>4</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>999995</th>\n",
       "      <td>999995</td>\n",
       "      <td>999995</td>\n",
       "      <td>999995</td>\n",
       "      <td>999995</td>\n",
       "      <td>999995</td>\n",
       "      <td>999995</td>\n",
       "      <td>999995</td>\n",
       "      <td>999995</td>\n",
       "      <td>999995</td>\n",
       "      <td>999995</td>\n",
       "      <td>999995</td>\n",
       "      <td>999995</td>\n",
       "      <td>999995</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>999996</th>\n",
       "      <td>999996</td>\n",
       "      <td>999996</td>\n",
       "      <td>999996</td>\n",
       "      <td>999996</td>\n",
       "      <td>999996</td>\n",
       "      <td>999996</td>\n",
       "      <td>999996</td>\n",
       "      <td>999996</td>\n",
       "      <td>999996</td>\n",
       "      <td>999996</td>\n",
       "      <td>999996</td>\n",
       "      <td>999996</td>\n",
       "      <td>999996</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>999997</th>\n",
       "      <td>999997</td>\n",
       "      <td>999997</td>\n",
       "      <td>999997</td>\n",
       "      <td>999997</td>\n",
       "      <td>999997</td>\n",
       "      <td>999997</td>\n",
       "      <td>999997</td>\n",
       "      <td>999997</td>\n",
       "      <td>999997</td>\n",
       "      <td>999997</td>\n",
       "      <td>999997</td>\n",
       "      <td>999997</td>\n",
       "      <td>999997</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>999998</th>\n",
       "      <td>999998</td>\n",
       "      <td>999998</td>\n",
       "      <td>999998</td>\n",
       "      <td>999998</td>\n",
       "      <td>999998</td>\n",
       "      <td>999998</td>\n",
       "      <td>999998</td>\n",
       "      <td>999998</td>\n",
       "      <td>999998</td>\n",
       "      <td>999998</td>\n",
       "      <td>999998</td>\n",
       "      <td>999998</td>\n",
       "      <td>999998</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>999999</th>\n",
       "      <td>999999</td>\n",
       "      <td>999999</td>\n",
       "      <td>999999</td>\n",
       "      <td>999999</td>\n",
       "      <td>999999</td>\n",
       "      <td>999999</td>\n",
       "      <td>999999</td>\n",
       "      <td>999999</td>\n",
       "      <td>999999</td>\n",
       "      <td>999999</td>\n",
       "      <td>999999</td>\n",
       "      <td>999999</td>\n",
       "      <td>999999</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>1000000 rows × 13 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "             A       B       C       D       E       F       G       H   \n",
       "0            0       0       0       0       0       0       0       0  \\\n",
       "1            1       1       1       1       1       1       1       1   \n",
       "2            2       2       2       2       2       2       2       2   \n",
       "3            3       3       3       3       3       3       3       3   \n",
       "4            4       4       4       4       4       4       4       4   \n",
       "...        ...     ...     ...     ...     ...     ...     ...     ...   \n",
       "999995  999995  999995  999995  999995  999995  999995  999995  999995   \n",
       "999996  999996  999996  999996  999996  999996  999996  999996  999996   \n",
       "999997  999997  999997  999997  999997  999997  999997  999997  999997   \n",
       "999998  999998  999998  999998  999998  999998  999998  999998  999998   \n",
       "999999  999999  999999  999999  999999  999999  999999  999999  999999   \n",
       "\n",
       "             I       J       K       L       M  \n",
       "0            0       0       0       0       0  \n",
       "1            1       1       1       1       1  \n",
       "2            2       2       2       2       2  \n",
       "3            3       3       3       3       3  \n",
       "4            4       4       4       4       4  \n",
       "...        ...     ...     ...     ...     ...  \n",
       "999995  999995  999995  999995  999995  999995  \n",
       "999996  999996  999996  999996  999996  999996  \n",
       "999997  999997  999997  999997  999997  999997  \n",
       "999998  999998  999998  999998  999998  999998  \n",
       "999999  999999  999999  999999  999999  999999  \n",
       "\n",
       "[1000000 rows x 13 columns]"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data = pd.DataFrame({x:range(1000000) for x in col_names})\n",
    "data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "b9a36476",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Partition of a set of 511554 objects. Total size = 272976926 bytes.\n",
       " Index  Count   %     Size   % Cumulative  % Kind (class / dict of class)\n",
       "     0     80   0 104009772  38 104009772  38 numpy.ndarray\n",
       "     1      1   0 104000144  38 208009916  76 pandas.core.frame.DataFrame\n",
       "     2 145150  28 19126019   7 227135935  83 str\n",
       "     3 109392  21  7981624   3 235117559  86 tuple\n",
       "     4  31425   6  5589384   2 240706943  88 types.CodeType\n",
       "     5  16016   3  5061128   2 245768071  90 dict (no owner)\n",
       "     6  59729  12  4706940   2 250475011  92 bytes\n",
       "     7  28515   6  3878040   1 254353051  93 function\n",
       "     8   3859   1  3633192   1 257986243  95 type\n",
       "     9   1424   0  2205480   1 260191723  95 dict of module\n",
       "<1355 more rows. Type e.g. '_.more' to view.>"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data_type_summary.heap()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "5f810fdc",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "99.18226623535156"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "104000144/1024/1024"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "c254b18f",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "A    int64\n",
       "B    int64\n",
       "C    int64\n",
       "D    int64\n",
       "E    int64\n",
       "F    int64\n",
       "G    int64\n",
       "H    int64\n",
       "I    int64\n",
       "J    int64\n",
       "K    int64\n",
       "L    int64\n",
       "M    int64\n",
       "dtype: object"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data.dtypes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ff8dda25",
   "metadata": {},
   "outputs": [],
   "source": [
    "len(col_names)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "8b113cbb",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(1000000, 13)"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "4bf12f7f",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "104000000"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data.shape[0] * data.shape[1] * 8 "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "d5668a91",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "99.18212890625"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "104000000 / 1024 / 1024"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a4dc438e",
   "metadata": {},
   "outputs": [],
   "source": [
    "data.index.dtype"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "7afbe3d7",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Index        128\n",
       "A        8000000\n",
       "B        8000000\n",
       "C        8000000\n",
       "D        8000000\n",
       "E        8000000\n",
       "F        8000000\n",
       "G        8000000\n",
       "H        8000000\n",
       "I        8000000\n",
       "J        8000000\n",
       "K        8000000\n",
       "L        8000000\n",
       "M        8000000\n",
       "dtype: int64"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "detailed_memory_usage = data.memory_usage(deep=True, )\n",
    "detailed_memory_usage"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "455c087b",
   "metadata": {},
   "outputs": [],
   "source": [
    "8000000/1024/1024 * len(col_names)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "5d8bc370",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 1000000 entries, 0 to 999999\n",
      "Data columns (total 13 columns):\n",
      " #   Column  Non-Null Count    Dtype\n",
      "---  ------  --------------    -----\n",
      " 0   A       1000000 non-null  int64\n",
      " 1   B       1000000 non-null  int64\n",
      " 2   C       1000000 non-null  int64\n",
      " 3   D       1000000 non-null  int64\n",
      " 4   E       1000000 non-null  int64\n",
      " 5   F       1000000 non-null  int64\n",
      " 6   G       1000000 non-null  int64\n",
      " 7   H       1000000 non-null  int64\n",
      " 8   I       1000000 non-null  int64\n",
      " 9   J       1000000 non-null  int64\n",
      " 10  K       1000000 non-null  int64\n",
      " 11  L       1000000 non-null  int64\n",
      " 12  M       1000000 non-null  int64\n",
      "dtypes: int64(13)\n",
      "memory usage: 99.2 MB\n"
     ]
    }
   ],
   "source": [
    "data.info()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d10d227b",
   "metadata": {},
   "source": [
    "### Using  The Correct DataType\n",
    "```\n",
    "    int8: -128 to 127\n",
    "    int16: -32,768 to 32,767\n",
    "    int32: -2,147,483,648 to 2,147,483,647\n",
    "    int64: -9,223,372,036,854,775,808 to 9,223,372,036,854,775,807\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "35cd6af4",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 1000000 entries, 0 to 999999\n",
      "Data columns (total 13 columns):\n",
      " #   Column  Non-Null Count    Dtype\n",
      "---  ------  --------------    -----\n",
      " 0   A       1000000 non-null  int32\n",
      " 1   B       1000000 non-null  int32\n",
      " 2   C       1000000 non-null  int32\n",
      " 3   D       1000000 non-null  int32\n",
      " 4   E       1000000 non-null  int32\n",
      " 5   F       1000000 non-null  int32\n",
      " 6   G       1000000 non-null  int32\n",
      " 7   H       1000000 non-null  int32\n",
      " 8   I       1000000 non-null  int32\n",
      " 9   J       1000000 non-null  int32\n",
      " 10  K       1000000 non-null  int32\n",
      " 11  L       1000000 non-null  int32\n",
      " 12  M       1000000 non-null  int32\n",
      "dtypes: int32(13)\n",
      "memory usage: 49.6 MB\n"
     ]
    }
   ],
   "source": [
    "data_int_32 = pd.DataFrame({x:range(1000000) for x in col_names}, dtype='int32')\n",
    "data_int_32.info()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "8c05c172",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 1000000 entries, 0 to 999999\n",
      "Data columns (total 13 columns):\n",
      " #   Column  Non-Null Count    Dtype\n",
      "---  ------  --------------    -----\n",
      " 0   A       1000000 non-null  int8 \n",
      " 1   B       1000000 non-null  int8 \n",
      " 2   C       1000000 non-null  int8 \n",
      " 3   D       1000000 non-null  int8 \n",
      " 4   E       1000000 non-null  int8 \n",
      " 5   F       1000000 non-null  int8 \n",
      " 6   G       1000000 non-null  int8 \n",
      " 7   H       1000000 non-null  int8 \n",
      " 8   I       1000000 non-null  int8 \n",
      " 9   J       1000000 non-null  int8 \n",
      " 10  K       1000000 non-null  int8 \n",
      " 11  L       1000000 non-null  int8 \n",
      " 12  M       1000000 non-null  int8 \n",
      "dtypes: int8(13)\n",
      "memory usage: 12.4 MB\n"
     ]
    }
   ],
   "source": [
    "import random\n",
    "data_int_8 = pd.DataFrame({x:[random.randint(0,10) for i in range(1000000)] for x in col_names}, dtype='int8')\n",
    "data_int_8.info()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c65c7151",
   "metadata": {},
   "source": [
    "### Using Progress Bars\n",
    "\n",
    "* Feedback on long operations is considered a good practive.\n",
    "  * When running scripts that take time, having a progress bar improves the user experience \n",
    "  * Reassurance that process is ongoing and providing a sense of how much longer it will take.\n",
    "  \n",
    "* Very easy to add progress bars across scenarios"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "aa3006f6",
   "metadata": {},
   "outputs": [],
   "source": [
    "from tqdm import \n",
    "\n",
    "import time\n",
    "for i in tqdm(range(100), desc=\"Doing my magic\"):\n",
    "    time.sleep(1)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f4bd85e1",
   "metadata": {},
   "outputs": [],
   "source": [
    "for i in tqdm(range(100), desc=\"Doing my magic\"):\n",
    "    time.sleep(0.1)\n",
    "\n",
    "for i in tqdm(range(100), desc=\"Doing something else\"):\n",
    "    time.sleep(0.1)    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "520e245e",
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "tqdm.pandas()\n",
    "\n",
    "df['processed_column'] = df['raw_column'].progress_apply(some_processing_function)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "239a932c",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.16"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}