3-split-data-into-training-and-test-sets.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "eab9ec67-891b-4ecc-92ab-34f07aa505cd",
   "metadata": {},
   "source": [
    "![Split Data into Training and Test Sets banner](./images/3_split_data_into_training_and_test_sets.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "13886a88-569a-4ccb-a254-e5aa29eac62e",
   "metadata": {},
   "source": [
    "# 3. Split Data into Training and Test Sets\n",
    "\n",
    "In supervised learning, we have a dataset with labeled examples, where each instance consists of input features (independent variables) and a corresponding target variable (dependent variable). The goal is to learn a mapping function from the input features to the target variable, which can then be used to make predictions on new, unseen data.\n",
    "\n",
    "However, if we train and evaluate our model on the same data, we risk overfitting, where the model memorizes the training data instead of learning the underlying patterns. This leads to poor generalization performance on new data.\n",
    "\n",
    "To avoid this, we split the dataset into two mutually exclusive sets:\n",
    "\n",
    "- **Training Set**: Used to train the model, allowing it to learn the patterns and relationships between the input features and the target variable.\n",
    "\n",
    "- **Test Set**: Used to evaluate the trained model's performance on unseen data, providing an unbiased estimate of its generalization ability.\n",
    "\n",
    "Failing to separate the data into training and test sets can result in overly optimistic performance estimates, as the model has already seen and memorized the data it's being evaluated on."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d945e876-f26b-4932-ae21-da12033bab03",
   "metadata": {},
   "source": [
    "## 3.1. Holdout Method\n",
    "\n",
    "The holdout method is a simple and widely used technique for splitting the dataset into training and test sets. Here's how it works:\n",
    "\n",
    "1. Shuffle the dataset randomly to ensure that the examples are not ordered in any systematic way (depending on the type of data).\n",
    "\n",
    "2. Determine the desired split ratio, such as 80/20 or 70/30, where the first number represents the percentage of data for the training set, and the second number represents the percentage for the test set.\n",
    "\n",
    "3. Split the dataset according to the chosen ratio, ensuring that the training and test sets are mutually exclusive (no overlapping instances).\n",
    "\n",
    "![Split Data into Training and Test Sets holdout](./images/3_split_data_into_training_and_test_sets_holdout.png)\n",
    "\n",
    "For classification problems, it's essential to maintain the class proportions in both the training and test sets. This process is called **stratification**, and it ensures that the target variable's distribution is similar in both sets, preventing any class imbalance issues.\n",
    "\n",
    "![Split Data into Training and Test Sets stratification](./images/3_split_data_into_training_and_test_sets_stratification.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5c18f5a2-f479-4771-8742-ecdef256f63b",
   "metadata": {},
   "source": [
    "-----\n",
    "\n",
    "## 3.2. Cross-Validation\n",
    "\n",
    "While the holdout method is simple and widely used, it can lead to unstable performance estimates, especially for small datasets. Cross-validation is an alternative technique that can provide more reliable performance estimates and help mitigate overfitting.\n",
    "\n",
    "The idea behind cross-validation is to split the dataset into multiple folds (subsets), train the model on a combination of folds, and evaluate it on the remaining fold(s). This process is repeated for different fold combinations, and the performance metrics are averaged across all iterations.\n",
    "\n",
    "One popular cross-validation technique is **k-fold cross-validation**, where the dataset is divided into k equal-sized folds. The model is trained on **k-1** folds and evaluated on the remaining fold. This process is repeated k times, with each fold serving as the test set once.\n",
    "\n",
    "![Split Data into Training and Test Sets cross validation](./images/3_split_data_into_training_and_test_sets_cross_validation.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "99cf99c2-bdab-419f-a952-a5f27b534c4b",
   "metadata": {},
   "source": [
    "-----\n",
    "\n",
    "## 3.3. Data Leakage\n",
    "\n",
    "Data leakage occurs when information from the test set is inadvertently used during the training process, leading to overly optimistic performance estimates and poor generalization on new, unseen data.\n",
    "\n",
    "There are two main types of data leakage:\n",
    "\n",
    "- **Target Leakage**: This occurs when the target variable (or a close approximation) is used as an input feature for training the model. For example, in a credit risk prediction problem, using the customer's current credit score as a feature would be considered target leakage, as the model would essentially be learning to predict the target variable from itself.\n",
    "\n",
    "![Split Data into Training and Test Sets target leakage](./images/3_split_data_into_training_and_test_sets_target_leakage.png)\n",
    "\n",
    "- **Feature Leakage**: This happens when information from the test set is used to create or transform features during the training process. For example, if you calculate the mean or standard deviation of a feature using the entire dataset (including the test set) and then use these statistics to normalize the feature values, you have introduced feature leakage.\n",
    "\n",
    "![Split Data into Training and Test Sets feature leakage](./images/3_split_data_into_training_and_test_sets_feature_leakage.png)\n",
    "\n",
    "To avoid data leakage, it's crucial to ensure that the test set remains completely unseen during the model training and selection process. This includes:\n",
    "- Splitting the data into training and test sets before any feature engineering or preprocessing steps.\n",
    "- Performing all feature transformations (e.g., scaling, encoding) using only the training set, and then applying the same transformations to the test set.\n",
    "- Avoiding peeking at the test set during model selection, hyperparameter tuning, or any other step that could potentially introduce bias.\n",
    "\n",
    "Data leakage can lead to overly optimistic performance estimates and poor generalization on new data, as the model has effectively seen and learned from information it should not have had access to during training."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ba708095-1a4a-4096-9203-b2b613aadf56",
   "metadata": {},
   "source": [
    "-----\n",
    "\n",
    "## 3.4. Best Practices\n",
    "\n",
    "Here are some best practices to follow when splitting data into training and test sets:\n",
    "\n",
    "- **Shuffle the data**: Before splitting the dataset, ensure that the instances are shuffled randomly to avoid any systematic ordering that could introduce bias - unless the order of the samples is wanted, e.g. time-series forecasting.\n",
    "  \n",
    "- **Use a consistent random_state**: When using functions like train_test_split() or cross_val_score(), set a consistent random_state value to ensure reproducibility of the splits across different runs.\n",
    "\n",
    "- **Avoid peeking at the test set**: Never use the test set for any step that could potentially introduce bias, such as feature engineering, model selection, or hyperparameter tuning.\n",
    "\n",
    "- **Consider cross-validation for small datasets**: If you have a small dataset or if the performance estimates from the holdout method are unstable, consider using cross-validation techniques like k-fold cross-validation to obtain more reliable performance estimates.\n",
    "\n",
    "- **Maintain class proportions (stratification)**: For classification problems, ensure that the class proportions are maintained in both the training and test sets by using stratification (e.g., stratify=y in train_test_split()).\n",
    "\n",
    "- **Evaluate multiple train/test split**: To account for potential variability in the splits, consider evaluating your model's performance on multiple train/test splits and reporting the average or distribution of performance metrics.\n",
    "\n",
    "- **Document and version control**: Maintain clear documentation and version control for your data splitting and preprocessing steps to ensure reproducibility and transparency in your modeling process.\n",
    "\n",
    "By following these best practices, you can ensure that your data splitting process is robust, unbiased, and provides reliable performance estimates for your supervised learning models."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a3c10810-3318-41c2-99d7-cd891375663d",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}