ipums-microdata-extract-api-ipumspy-demo.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "f83ec79e-66f9-4920-a26d-6bca2352ef4c",
   "metadata": {},
   "source": [
    "## Creating an IPUMS CPS extract using the IPUMS Microdata Extract API and ipumspy\n",
    "\n",
    "This notebook contains sample code for creating, submitting, downloading, and reading an IPUMS CPS extract via the IPUMS Microdata Extract API using the `ipumspy` Python library.\n",
    "\n",
    "To be able to run the code in this notebook, you will need 1) an [IPUMS CPS user account](https://cps.ipums.org/cps-action/menu), 2) an [IPUMS API key](https://account.ipums.org/api_keys), and 3) to have version 0.2.1 of `ipumspy` installed.\n",
    "\n",
    "First, we will import all the necessary libraries."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "2d9417e4-6bc7-4abb-984d-734f989cea1d",
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "\n",
    "import pandas as pd\n",
    "\n",
    "from ipumspy import IpumsApiClient, CpsExtract, readers"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5f93e7eb-aff6-4b13-903d-d9e59ed5ea33",
   "metadata": {},
   "source": [
    "Next we will pass our API key to the `IpumsApiClient`. In this example, I have stored my API key as a conda environment variable. This is considered best practice, but you may also replace the `my_api_key` variable in the second line of the next code block with your API key string in quotes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "d5e9da89-c3eb-455e-8d34-4bd6449fdd26",
   "metadata": {},
   "outputs": [],
   "source": [
    "my_api_key = os.getenv(\"IPUMS_MICRODATA_API_KEY\")\n",
    "ipums = IpumsApiClient(my_api_key)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "180cfa67-cc67-405d-b390-12a88640c66b",
   "metadata": {},
   "source": [
    "Next we will define an extract. For the purposes of this demonstration, we will only choose one sample and a small number of variables. An extract is defined by an IPUMS collection id, a list of sample ids, and a list of variable names. You may also specify an extract description. This is not required, but highly recommended! In the code chunk below using the `CpsExtract` class indicates that we wish to make an IPUMS CPS extract. Note that IPUMS does not currently offer a metadata API, so if you do not know the sample IDs and variable names that you want to include, you can find these by browsing the list of IPUMS CPS sample IDs and the list of available IPUMS CPS variables."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "a68748e0-a613-4e3d-83f5-8adaecd808ef",
   "metadata": {},
   "outputs": [],
   "source": [
    "extract = CpsExtract([\"cps2022_05b\"],\n",
    "                     [\"AGE\", \"SEX\", \"RACE\"],\n",
    "                     description=\"My first API extract!\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ad43b65f-cb99-41f7-9c2d-02c177eb6d28",
   "metadata": {},
   "source": [
    "The next step is to submit our `extract` object to the IPUMS Microdata Extract API using the `ipums` API client instance."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "fca52aac-7877-4508-95d1-a317874c7f4a",
   "metadata": {},
   "outputs": [],
   "source": [
    "submitted_extract = ipums.submit_extract(extract)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "51add6df-81be-4f74-8639-0d395ad2c255",
   "metadata": {},
   "source": [
    "Now that the extract has been submitted to the IPUMS extract system, we chan check its status and see that it has been received and is in line to be processed."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "00c09ced-e0e8-4055-a1b8-e9487532fe3a",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'queued'"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ipums.extract_status(submitted_extract)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "16e55a75-d554-4223-a853-b61e37de5eac",
   "metadata": {},
   "source": [
    "To save ourselves a few lines of code, we can use the `wait_for_extract()` method to let us know when the extract has been completed and is ready for download. Once the extract is completed, we can use the `download_extract()` method to download the data file and its accompanying DDI codebook in to our current working directory. The codeblock below also demonstrates use of `collection` and `extract_id` attributes of our IPUMS CPS extract. Each extract a user submitts is assigned its own unique ID number by the IPUMS extract system. This number can be used to easily read in your downloaded files, or to re-download this extract at a later time if needed."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "7eef3b00-7732-4f3b-9038-dd5fdab344c8",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "cps number 4 is complete!\n"
     ]
    }
   ],
   "source": [
    "ipums.wait_for_extract(submitted_extract)\n",
    "print(f\"{submitted_extract.collection} number {submitted_extract.extract_id} is complete!\")\n",
    "ipums.download_extract(submitted_extract)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0b6365a9-c858-46a5-8234-67b130f7dde9",
   "metadata": {},
   "source": [
    "Now that the necessary files are downloaded, we can use the `readers` module to parse the DDI codebook and read the extract data file into a Pandas DataFrame. Note that the first line uses the `submitted_extract` object attributes to grab the correct file without requiring any intermediate steps. Note that the naming convention for IPUMS extract files is `[lowercase collection name]_[extract ID number, left-padded to 5 digits]`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "29b04511-ff9c-48e3-9bf6-a58588d5da32",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/pkg/ipums/programming/conda/current/envs/ipumspy-demo/lib/python3.9/site-packages/ipumspy/readers.py:46: CitationWarning: Use of data from IPUMS is subject to conditions including that users should cite the data appropriately.\n",
      "See the `ipums_conditions` attribute of this codebook for terms of use.\n",
      "See the `ipums_citation` attribute of this codebook for the appropriate citation.\n",
      "  warnings.warn(\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>YEAR</th>\n",
       "      <th>SERIAL</th>\n",
       "      <th>MONTH</th>\n",
       "      <th>HWTFINL</th>\n",
       "      <th>CPSID</th>\n",
       "      <th>PERNUM</th>\n",
       "      <th>WTFINL</th>\n",
       "      <th>CPSIDP</th>\n",
       "      <th>AGE</th>\n",
       "      <th>SEX</th>\n",
       "      <th>RACE</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2022</td>\n",
       "      <td>1</td>\n",
       "      <td>5</td>\n",
       "      <td>1892.0459</td>\n",
       "      <td>20220500000100</td>\n",
       "      <td>1</td>\n",
       "      <td>1892.0459</td>\n",
       "      <td>20220500000101</td>\n",
       "      <td>49</td>\n",
       "      <td>2</td>\n",
       "      <td>100</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2022</td>\n",
       "      <td>3</td>\n",
       "      <td>5</td>\n",
       "      <td>1962.7180</td>\n",
       "      <td>20220500000300</td>\n",
       "      <td>1</td>\n",
       "      <td>1962.7180</td>\n",
       "      <td>20220500000301</td>\n",
       "      <td>29</td>\n",
       "      <td>1</td>\n",
       "      <td>100</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2022</td>\n",
       "      <td>3</td>\n",
       "      <td>5</td>\n",
       "      <td>1962.7180</td>\n",
       "      <td>20220500000300</td>\n",
       "      <td>2</td>\n",
       "      <td>2450.4601</td>\n",
       "      <td>20220500000302</td>\n",
       "      <td>25</td>\n",
       "      <td>2</td>\n",
       "      <td>100</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>2022</td>\n",
       "      <td>3</td>\n",
       "      <td>5</td>\n",
       "      <td>1962.7180</td>\n",
       "      <td>20220500000300</td>\n",
       "      <td>3</td>\n",
       "      <td>1885.9137</td>\n",
       "      <td>20220500000303</td>\n",
       "      <td>30</td>\n",
       "      <td>1</td>\n",
       "      <td>100</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>2022</td>\n",
       "      <td>6</td>\n",
       "      <td>5</td>\n",
       "      <td>1785.2980</td>\n",
       "      <td>20220300001000</td>\n",
       "      <td>1</td>\n",
       "      <td>1630.7870</td>\n",
       "      <td>20220300001001</td>\n",
       "      <td>80</td>\n",
       "      <td>1</td>\n",
       "      <td>200</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   YEAR  SERIAL  MONTH    HWTFINL           CPSID  PERNUM     WTFINL  \\\n",
       "0  2022       1      5  1892.0459  20220500000100       1  1892.0459   \n",
       "1  2022       3      5  1962.7180  20220500000300       1  1962.7180   \n",
       "2  2022       3      5  1962.7180  20220500000300       2  2450.4601   \n",
       "3  2022       3      5  1962.7180  20220500000300       3  1885.9137   \n",
       "4  2022       6      5  1785.2980  20220300001000       1  1630.7870   \n",
       "\n",
       "           CPSIDP  AGE  SEX  RACE  \n",
       "0  20220500000101   49    2   100  \n",
       "1  20220500000301   29    1   100  \n",
       "2  20220500000302   25    2   100  \n",
       "3  20220500000303   30    1   100  \n",
       "4  20220300001001   80    1   200  "
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "extract_file_name = f\"{submitted_extract.collection}_{str(submitted_extract.extract_id).zfill(5)}\"\n",
    "\n",
    "ddi = readers.read_ipums_ddi(f\"{extract_file_name}.xml\")\n",
    "\n",
    "df = readers.read_microdata(ddi, f\"{extract_file_name}.dat.gz\")\n",
    "\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a195e97a-75e3-4a38-8287-8f2c6fe47c72",
   "metadata": {},
   "source": [
    "Ta Da! Now we have an IPUMS CPS extract all ready for analysis in Python! Forgot a variable? Just add it to the cell where the extract is defined and re-run the notebook! Want to do the same analysis next month when the most recent data is available? Just add the newest sample id to that same cell and re-run! As you can imagine, getting IPUMS extracts via API opens up lots of interesting possibilities for efficient and reproduceable workflows! "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4518d6c3-17ba-4e2d-81c1-a7cc53ab5590",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python3 (ipumspy-demo)",
   "language": "python",
   "name": "ipumspy-demo"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}