00_Intro.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Exploratory Multivariate Analysis of Geochemical Datasets\n",
    "Compiled by [Morgan Williams](mailto:morgan.williams@csiro.au) for C3DIS 2018 "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This collection of Jupyter notebooks illustrates some common simple problems encountered with geochemical data, and some solutions. They cover the majority of the workflow outlined below, but represent what is generally a work in progress. Associated data is sourced solely from the [EarthChem data portal](http://ecp.iedadata.org/), and is here stored in a S3 bucket for simplicity."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### The Workflow"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The data analysis workflow denoted below lists some common necessary tasks to derive useful insight from geochemical data. Much of this is common to any data science workflow, but due to the nature of the geochemical data itself, a few of these processes are still current research problems. Our research aims not to introduce radical change in methodology, but instead to simply streamline and standardise the process, such that we can use geochemistry in a robust way to address geological problems."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![Workflow Image](images/Workflow.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### The Problem"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Much has happened since our planet was a primitive ball of molten rock,  including the origin of plate tectonics, the modern atmosphere and life. This extended geological history has been encoded into chemical signatures of rocks and minerals, which may then used to (partially) reconstruct the past.\n",
    "\n",
    "Inverting geochemistry to infer the geological past is commonly an underdetermined problem (especially prior to the advent of modern geochemical analysis instrumentation), and is hindered by complex geological histories.\n",
    "\n",
    "Modern analytical methods have higher throughput and greater sensitivity and precision. As a result, established publicly-accessible geochemical databases are growing steadily. However, the potential value of aggregating the increasing volume of high-quality data has not yet been fully realised."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### The Other Problems.."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Before we can tackle the geological problems, we must first have a dataset which is consistently formatted and which contains relevant data of sufficient accuracy (lest we achieve simply *\"garbage in, garbage out\"*). These notebooks illustrate some of these processing steps, and demonstrate some approaches for the initial stages of data exploration."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### The Data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you wish to download a subset of the EarthChem data to this binder server (approx 300 MB as a sparse dataframe) such that it can be acessed in later notebooks, do so below. If you do not, it will instead be downloaded *on-run* as necessary. Please note this can take more than a minute even on a good day."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "%load_ext autoreload\n",
    "%load_ext memory_profiler\n",
    "%autoreload 2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Wall time: 1min 48s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "import sys\n",
    "sys.path.insert(0, './src')\n",
    "from datasource import download_data, load_df\n",
    "download_data('EarthChemData.pkl', 'EarthChemData.pkl')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "Int64Index: 1073034 entries, 0 to 2092330\n",
      "Data columns (total 71 columns):\n",
      "SampleID       1073028 non-null object\n",
      "Source         1073034 non-null object\n",
      "Reference      1073034 non-null object\n",
      "CruiseID       180711 non-null object\n",
      "Latitude       1073034 non-null float64\n",
      "Longitude      1073034 non-null float64\n",
      "LocPrec        1073034 non-null float64\n",
      "MinAge         614923 non-null float64\n",
      "Age            607557 non-null float64\n",
      "MaxAge         625776 non-null float64\n",
      "Method         1073034 non-null object\n",
      "Material       1073034 non-null object\n",
      "Type           1073019 non-null object\n",
      "Composition    1073034 non-null object\n",
      "RockName       1073034 non-null object\n",
      "Na2O           378229 non-null float64\n",
      "MgO            375983 non-null float64\n",
      "Al2O3          375158 non-null float64\n",
      "SiO2           381264 non-null float64\n",
      "P2O5           343454 non-null float64\n",
      "K2O            391758 non-null float64\n",
      "CaO            375813 non-null float64\n",
      "TiO2           375673 non-null float64\n",
      "MnO            349445 non-null float64\n",
      "FeOT           485685 non-null float64\n",
      "Li             44788 non-null float64\n",
      "Be             74981 non-null float64\n",
      "B              42947 non-null float64\n",
      "Mg             92342 non-null float64\n",
      "Cl             37633 non-null float64\n",
      "K              58797 non-null float64\n",
      "Ca             103305 non-null float64\n",
      "Sc             227228 non-null float64\n",
      "Ti             109537 non-null float64\n",
      "V              249099 non-null float64\n",
      "Cr             278450 non-null float64\n",
      "Mn             106043 non-null float64\n",
      "Fe             120046 non-null float64\n",
      "Co             209541 non-null float64\n",
      "Ni             281268 non-null float64\n",
      "Cu             228744 non-null float64\n",
      "Zn             220164 non-null float64\n",
      "Ga             126015 non-null float64\n",
      "Rb             275938 non-null float64\n",
      "Sr             367161 non-null float64\n",
      "Y              308961 non-null float64\n",
      "Zr             337013 non-null float64\n",
      "Nb             240845 non-null float64\n",
      "Mo             37700 non-null float64\n",
      "Cs             95928 non-null float64\n",
      "Ba             341793 non-null float64\n",
      "La             264928 non-null float64\n",
      "Ce             232241 non-null float64\n",
      "Pr             89315 non-null float64\n",
      "Nd             199149 non-null float64\n",
      "Sm             175005 non-null float64\n",
      "Eu             162006 non-null float64\n",
      "Gd             117043 non-null float64\n",
      "Tb             138647 non-null float64\n",
      "Dy             104030 non-null float64\n",
      "Ho             90438 non-null float64\n",
      "Er             99464 non-null float64\n",
      "Tm             86574 non-null float64\n",
      "Yb             186035 non-null float64\n",
      "Lu             143638 non-null float64\n",
      "Hf             133165 non-null float64\n",
      "Ta             121178 non-null float64\n",
      "Pb             201956 non-null float64\n",
      "Th             190403 non-null float64\n",
      "U              147985 non-null float64\n",
      "TotalAlkali    362866 non-null float64\n",
      "dtypes: float64(62), object(9)\n",
      "memory usage: 589.4+ MB\n",
      "peak memory: 1527.42 MiB, increment: 1423.32 MiB\n"
     ]
    }
   ],
   "source": [
    "%%memit\n",
    "df = load_df('EarthChemData.pkl')\n",
    "df.info()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}