build-laterality-dataframe.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "2273cf47-b5bf-43cc-b7ac-1794779c2668",
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c300e3b3-6cbd-48ea-81b4-d8bd06143b44",
   "metadata": {},
   "outputs": [],
   "source": [
    "# load the dataframe\n",
    "df = pd.read_csv('./tractmeasures.csv')\n",
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8c22afc6-aa67-458f-bdba-fd4ec46c6ecf",
   "metadata": {},
   "source": [
    "# Building a laterality dataframe\n",
    "Now that we have a better understanding of the data and what can be done with pandas, we now need to get to the fun part. We need to compute the laterality index for each tract and build our laterality dataframe!\n",
    "\n",
    "As a refesher, laterality is the concept that the properties of the two hemispheres differ. This follows along with functional work that have found hemispheric dominance in certain cognitive abilities, such as language processing, comprehension, and production.\n",
    "\n",
    "In white matter, this can be examined by looking at the difference between structural properties of white matter tracts between the two hemispheres.\n",
    "\n",
    "This index is just that...an index. Thus, we define laterality index as the ratio between the difference between the hemispheres and the sum of the hemispheres. We can quantify this using the following formula:\n",
    "    \n",
    "```math\n",
    "laterality index = (left - right) / (left + right)\n",
    "```\n",
    "\n",
    "where left and right correspond to the microstructural property for a the left hemisphere tract and the right hemisphere tract.\n",
    "\n",
    "So, now that we know how to define laterality index, we need to talk about some caveats to computing laterality index with this dataset:"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "caed0053-69fc-4802-80e4-435e1a45e2ab",
   "metadata": {},
   "source": [
    "### caveats\n",
    "1. Nan's exist in this data. This is because outlier removal was already performed on this dataset. This detection was done  on a measure and structure basis. Because of this, hemisphere data may be uneven. We will have to do some specific things for this.\n",
    "2. There exists some tracts that cross hemispheres (corpus callosal tracts). These include: anterioFrontalCC, forcepsMajor, forcepsMinor, middleFrontalCC, parietalCC. Fortunately, all the other tracts have a 'left' or 'right' suffix, which we can use to our advantage.\n",
    "\n",
    "So, with this knowledge, here's the roadmap ahead of us.\n",
    "\n",
    "First, we need to define functions that will do the following:\n",
    "\n",
    "## What do we need to do\n",
    "1. Seperate the dataframes \n",
    "2. Identify when there are uneven numbers between the left and right hemispheres for a given measure (i.e. **caveat 1**), and reduce it down to only the common subjects. \n",
    "3. compute laterality index between the left and right hemispheric measures and return the laterality index value and return it as a dataframe\n",
    "\n",
    "First, let's define all the functions that we will need to build our laterality index dataframe! It is good practice to build code that you need to repeat into functions, and to define those at the very beginning."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7487a737-0a9b-444f-94b0-5ddd52208ab3",
   "metadata": {},
   "source": [
    "## Define laterality index function\n",
    "So, let's take this one step at a time. And we can make the first one a bit easier to start.\n",
    "\n",
    "Let's first define a function that will compute laterality index for us. This function will take in as input the following values: 'left' which is the value of the microstructural property from the left hemisphere tract, and 'right' which is the same but for the right hemisphere. These are a series of numerical values of length N where N is the number of rows (data points). This function should then return the laterality index as a series of the same length."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ad3836e4-9b10-43d9-bced-6902011f0e45",
   "metadata": {},
   "outputs": [],
   "source": [
    "# define laterality functions\n",
    "def laterality_index(left,right):\n",
    "    \n",
    "    # HINT: look back at the forula from above\n",
    "    li = \n",
    "    \n",
    "    return li"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f0f6eaf9-c9e8-4b61-8a38-904b8065a36b",
   "metadata": {},
   "source": [
    "Great! We now have a function to define laterality index!\n",
    "\n",
    "Now, let's move onto probably the most complicated function to write.\n",
    "\n",
    "## Define function to reduce dataframes to same size\n",
    "Because of **caveat 1**, it is likely that there will be uneven numbers of subjects between the left hemisphere and right hemisphere data for a given tract. In these situations, we need to identify the common subjects between the two hemispheres and set both the left and right hemispheres to have the same subjects and same size. \n",
    "\n",
    "For this, we can write a function! This function takes in as inputs dataframes corresponding to the left and right hemisphere data. This function should then output dataframes for left hemisphere and right hemisphere tracts that are even in lenght and contain the same subjects and tracts. We will need to identify when the dataframes are of uneven length, and then subselect only the tracts and subjects that are common between the two hemispheres. We can then concatenate across all subjects to return a left and right dataframe of equal length."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2b4b693d-0f26-4dec-a092-bad636346587",
   "metadata": {},
   "outputs": [],
   "source": [
    "# define function to grab some participants and structures from reference datasets (need to be done because outliers were detected on a measure and structure basis. hemisphere data may be uneven\n",
    "def reduce_dataframes_to_same(left, right):\n",
    "    \n",
    "    # setup blank dataframes. one for left hemisphere and one for right (see pd.DataFrame)\n",
    "    final_df_left = \n",
    "    final_df_right = \n",
    "    \n",
    "    # set a \"stem name\" for each tract (i.e. doesn't have the left or right prefix). (see list comprehension with pandas)\n",
    "    left['stem'] = [ f.split('left')[1] for f in left['structureID'] ]\n",
    "    right['stem'] = \n",
    "\n",
    "    # write a logical statement that compares the lengths of left and right dataframes. if left is greater than right, set a new variable corresponding to the final subjects\n",
    "    # 'sub' to the unique subjectIDs found in the right hemisphere. Else, set it to the subjectIDs from the left hemisphere (see pandas .unique())\n",
    "    if len(left) > len(right):\n",
    "        subs = \n",
    "    else:\n",
    "        subs = \n",
    "        \n",
    "    # loop through all the subjects to extract the data and build the final left and right dataframes. this may be slow but that's okay!\n",
    "    for i in subs:\n",
    "        # set a temporary dataframe for left and right for only the i subjects data (see pd.loc)\n",
    "        tmp_left =  # subset data on subject basis: left hemisphere\n",
    "        tmp_right = right.loc[right['subjectID'] == i] # subset data on subject basis: right hemisphere\n",
    "\n",
    "        # generate lists of the tract stem names for both left and right hemispheres (see pandas tolist())\n",
    "        tmp_left_stems = \n",
    "        tmp_right_stems = \n",
    "        \n",
    "        # use list comprehension to identify the common stem names between left and right (see list comprehension)\n",
    "        common_structures = \n",
    "\n",
    "        # subselect all the data corresponding to only the tracts that are found in the common structures list for both left and right hemispheres (see pandas .loc and pandas .isin)\n",
    "        tmp_left_sub = tmp_left.loc[tmp_left['stem'].isin(common_structures)]\n",
    "        tmp_right_sub = \n",
    "\n",
    "        # concatenate data to final dataframe for each hemisphere (see pd.concat)\n",
    "        final_df_left =     \n",
    "        final_df_right = \n",
    "    \n",
    "    return final_df_left, final_df_right"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dd03e27f-7b50-477c-aac7-6f77deaee35c",
   "metadata": {},
   "source": [
    "## Define function to build a laterality dataframe\n",
    "The final function we need to write is one that actually creates the laterality dataframe. This function will first generate two dataframes, one for left hemisphere tracts and one for right. It will also remove any missing data values. It will then need to check the lengths of the two dataframes to  see if they are unequal in length. If so, run the function we just defined above! If not, do a step that's in the function above which is adding the stem name column. Then, it will grab the values for the inputted measure, compute the laterality index, and return the laterality index dataframe."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "2a9f329f-f8e4-4995-811d-e8fa424c5153",
   "metadata": {},
   "outputs": [],
   "source": [
    "# define function to build laterality dataframe\n",
    "def laterality_dataframe(data, measure):\n",
    "\n",
    "    # seperate hemispheric data (see multiple conditional .loc statments with pandas)\n",
    "    left = data.loc[(data[\"hemisphere\"] == ) & (~data[measure].isna())]\n",
    "    right = \n",
    "\n",
    "    # check to make sure left and right comparisons are same. if not, call the function from above. if so, add a stem name column to each\n",
    "    if len(left) != len(right):\n",
    "        left, right = # call function we created above for reducing the dataframes to the same subjects and tracts\n",
    "    else:\n",
    "        left['stem'] = # see how this was done in previous function\n",
    "        right['stem'] = \n",
    "\n",
    "    # grab data for given measure (see pandas .values)\n",
    "    left_data = left[measure].\n",
    "    right_data = \n",
    "    \n",
    "    # compute laterality index and make dataframe. HINT: you can just subselect the 'subjectID', 'classID', 'structureID' columns from the left dataframe.\n",
    "    # (see multiple column indexing in pandas)\n",
    "    li = (left_data, right_data)\n",
    "    laterality = left[] # subselect the appropriate columns\n",
    "    \n",
    "    # update with 'structureID' name with the 'stem' name and add the laterality index values\n",
    "    laterality[\"structureID\"] = \n",
    "    laterality[\"laterality_index\"] = \n",
    "    \n",
    "    return laterality"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "037957ca-c6f8-489a-89d7-0aebdca908fb",
   "metadata": {},
   "source": [
    "Excellent! We've now defined all the functions we need!\n",
    "\n",
    "We can now start building our dataframes!\n",
    "\n",
    "But first, let's do a couple more things to the starting dataframe. We will first set a 'hemisphere' column where the values will either be 'left' if the string 'left' is present in the structureID, 'right' if the string 'right' is present, otherwise NA. Remember we used this column to split the tracts in our previous function. Following this, we will compute some age bins (for replication purposes) using bins of 0-20, 20-40, 40-60, 60-80, 80-100.\n",
    "\n",
    "## Add a hemisphere column and compute age bins"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8e6cdc0e-e11b-40c7-a108-953abf89cc81",
   "metadata": {},
   "outputs": [],
   "source": [
    "# set hemispheres. if 'left' in structureID, set value as 'left'. if 'right', set it as 'right'. if neither, set as 'NA' (see list comprehension and pandas columns)\n",
    "df['hemisphere'] = [  if _ in f else _ if _ in f else 'NA' for f in df.structureID ] # make sure to replace the _ with appropriate things\n",
    "\n",
    "# compute age bin. first set list of bins [0,20,40,60,80,100]. then see pd.cut to create a new column called 'age_bin'\n",
    "bins = \n",
    "df['age_bin'] = pd.cut()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "88bd0926-6361-44d2-9a03-7a16273671a0",
   "metadata": {},
   "source": [
    "We are finally ready to generate our laterality dataframe!\n",
    "\n",
    "## Generate laterality dataframe"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "175e849a-0461-40f8-b2ef-434205393576",
   "metadata": {},
   "outputs": [],
   "source": [
    "# let's set a variable named measure to 'fa' to compute the laterality dataframe for the fa measure. we will then call our laterality_dataframe function\n",
    "measure = 'fa'\n",
    "laterality = laterality_dataframe()\n",
    "\n",
    "# finally, let's merge subject-related demogrpahic data back to laterality dataframe and then save it to a csv (see pd.merge). before saving, we will remove duplicates\n",
    "# see pd.drop_duplicates\n",
    "laterality = pd.merge(,combined[['subjectID','classID','age','age_bin']],on=[])\n",
    "laterality = \n",
    "\n",
    "# save as csv (see pandas .to_csv). make sure to not save the index. \n",
    "laterality."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d6b910e6-13b5-4f71-b171-d2240dd37828",
   "metadata": {},
   "source": [
    "Now let's take a look at our final dataframe!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "43c6d0fc-b564-49f3-a6c7-5a6eef241021",
   "metadata": {},
   "outputs": [],
   "source": [
    "laterality"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}