fraud_detection.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "outdoor-arizona",
   "metadata": {},
   "source": [
    "# Business Case - Fraud detection model"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "psychological-leather",
   "metadata": {},
   "source": [
    "Author: Emilio Macias"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "demonstrated-migration",
   "metadata": {},
   "source": [
    "Note: the dataset can be downloaded from www.kaggle.com/ealtman2019/credit-card-transactions/download\n",
    "\n",
    "This notebook expects the dataset to be located at: data/credit_card_transactions-ibm_v2.csv"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "auburn-presence",
   "metadata": {},
   "source": [
    "## Table of Contents\n",
    "\n",
    "* [Data exploration](#exploration)\n",
    "* [Feature engineering](#feature_eng)\n",
    "    * [Handling of missing values](#missing_val)\n",
    "    * [Transforming categorical features](#cat_features)\n",
    "* [Data balancing](#data_balancing)\n",
    "* [Modelling](#modelling)\n",
    "    * [Model preparation](#model_prep)\n",
    "    * [Training and evaluation](#model_train)\n",
    "* [Region analysis](#region_analysis)\n",
    "* [Merchant analysis](#merchant_analysis)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "blond-tobacco",
   "metadata": {},
   "source": [
    "## Data exploration <a class=\"anchor\" id=\"exploration\"></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "collected-burst",
   "metadata": {},
   "source": [
    "We will start with an exploratory analysis of the data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "actual-niagara",
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "\n",
    "import category_encoders as ce\n",
    "\n",
    "from imblearn.over_sampling import SMOTE\n",
    "\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.tree import DecisionTreeClassifier\n",
    "from sklearn.ensemble import RandomForestClassifier\n",
    "from sklearn import linear_model\n",
    "from sklearn.preprocessing import StandardScaler\n",
    "from sklearn.metrics import (confusion_matrix, precision_recall_curve, auc,\n",
    "                             roc_curve, recall_score, classification_report, f1_score,\n",
    "                             precision_recall_fscore_support, accuracy_score)\n",
    "\n",
    "import matplotlib.pyplot as plt\n",
    "from matplotlib.colors import ListedColormap\n",
    "%matplotlib inline\n",
    "from pylab import rcParams\n",
    "\n",
    "import seaborn as sns\n",
    "sns.set(style='whitegrid', palette='muted', font_scale=1.5)\n",
    "\n",
    "rcParams['figure.figsize'] = 14, 8"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "occupational-cruise",
   "metadata": {},
   "outputs": [],
   "source": [
    "df = pd.read_csv(\"data/credit_card_transactions-ibm_v2.csv\")\n",
    "#df = pd.read_csv(\"data/medium_dataset.csv\")\n",
    "#df = df.sample(n=100000, random_state=111)\n",
    "#df.to_csv('data/medium_dataset.csv')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "black-zoning",
   "metadata": {},
   "source": [
    "The following line displays the number of rows (transactions) and columns (variables) from the dataset:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "concerned-distance",
   "metadata": {},
   "outputs": [],
   "source": [
    "df.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "detailed-seafood",
   "metadata": {},
   "source": [
    "Below we can see what some of the transactions look like:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "parental-leonard",
   "metadata": {},
   "outputs": [],
   "source": [
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "right-crossing",
   "metadata": {},
   "source": [
    "The following table points out the descriptive statistics of the different fields:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "consolidated-orchestra",
   "metadata": {},
   "outputs": [],
   "source": [
    "df.describe(include='all')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "demographic-valve",
   "metadata": {},
   "source": [
    "As we can see above, some of the features contain null variables (represented as NaN) which will be handled in the following section.\n",
    "From the previous table, it is important to note some facts from certain features:\n",
    "- There are 2,000 different users and each of them can use up to 9 cards.\n",
    "- The transactions were recorded from the year 1991 to 2020.\n",
    "- There are 3 types of transactions: Swipe Transaction, Online Transaction and Chip Transaction.\n",
    "- The transactions were paid in 223 different states including US states and world countries.\n",
    "- There are 23 different errors that occurred during the transactions."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "diverse-brook",
   "metadata": {},
   "source": [
    "Below are the cities with the highest number of transactions. As we can see all of them are cities in the US except the most frequent value which are online transactions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "applied-halloween",
   "metadata": {},
   "outputs": [],
   "source": [
    "df['Merchant City'].value_counts().head(10).to_frame('Number of transactions')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "recovered-attention",
   "metadata": {},
   "source": [
    "Below we list the categorical features that will have to be transformed into numerical values since the machine learning algorithms are math-based:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "piano-default",
   "metadata": {},
   "outputs": [],
   "source": [
    "df_types = df.dtypes.to_frame('Type')\n",
    "print([feature for feature in df_types[df_types['Type'] == 'object'].index])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "flush-finder",
   "metadata": {},
   "source": [
    "How different are the amount of money used in different transaction classes?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "invisible-pleasure",
   "metadata": {},
   "outputs": [],
   "source": [
    "# remove $ character to make Amount a numeric field\n",
    "df['Amount'] = df['Amount'].apply(lambda x: float(x.strip('$')))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "thorough-integral",
   "metadata": {},
   "outputs": [],
   "source": [
    "df.Amount.describe()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "express-bishop",
   "metadata": {},
   "outputs": [],
   "source": [
    "# drop the transactions with negative amount\n",
    "index_negatives = df[df['Amount'] < 0 ].index\n",
    "df.drop(index_negatives, inplace = True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "coral-treatment",
   "metadata": {},
   "outputs": [],
   "source": [
    "df.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "intimate-heritage",
   "metadata": {},
   "outputs": [],
   "source": [
    "df.Amount.max()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "clinical-complaint",
   "metadata": {},
   "outputs": [],
   "source": [
    "frauds = df[df['Is Fraud?'] == 'Yes']\n",
    "legitimates = df[df['Is Fraud?'] == 'No']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "polyphonic-kitchen",
   "metadata": {},
   "outputs": [],
   "source": [
    "f, (ax1, ax2) = plt.subplots(2, 1, sharex=True)\n",
    "f.suptitle('Amount per transaction by class')\n",
    "\n",
    "bins = 50\n",
    "\n",
    "ax1.hist(frauds.Amount, bins = bins)\n",
    "ax1.set_title('Fraud')\n",
    "\n",
    "ax2.hist(legitimates.Amount, bins = bins)\n",
    "ax2.set_title('Legitimate')\n",
    "\n",
    "plt.xlabel('Amount ($)')\n",
    "plt.ylabel('Number of Transactions')\n",
    "plt.xlim([0, 10000])\n",
    "plt.yscale('log')\n",
    "plt.show();"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "beneficial-pencil",
   "metadata": {},
   "source": [
    "## Feature Engineering <a class=\"anchor\" id=\"feature_eng\"></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "acute-convertible",
   "metadata": {},
   "source": [
    "The following sections cover the different steps needed to convert our data into a format useable by machine learning algorithms."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "crazy-potato",
   "metadata": {},
   "source": [
    "### Handling of missing values <a class=\"anchor\" id=\"missing_val\"></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "maritime-latex",
   "metadata": {},
   "source": [
    "As we discovered in our descriptive analysis earlier, our dataset contains null values that have to be treated. These are the features that contain null values:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "peripheral-sapphire",
   "metadata": {},
   "outputs": [],
   "source": [
    "[column for column in list(df) if df[column].isnull().values.any()]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "verbal-construction",
   "metadata": {},
   "source": [
    "It turns out that when a transaction is online, the merchant state and zip are filled with null values. In order for the machine learning algorithms to work correctly we should fill these values, and one option is to use the same \"ONLINE\" string as for the Merchant City."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "organizational-compiler",
   "metadata": {},
   "outputs": [],
   "source": [
    "df.loc[df['Merchant City'] == 'ONLINE', 'Merchant State'] = 'ONLINE'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ordered-network",
   "metadata": {},
   "outputs": [],
   "source": [
    "df.loc[df['Merchant City'] == 'ONLINE', 'Zip'] = 0"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "distinguished-stations",
   "metadata": {},
   "source": [
    "We found there are over 157K in-person transactions without a Zip code. We are going to fill them following this approach:\n",
    "1. Get the most frequent Zip from the transactions for the same merchant name and city.\n",
    "2. Get the most frequent Zip from the transactions for the same merchant city only.\n",
    "3. Get the most frequent Zip from the transactions for the same merchant state.\n",
    "4. Get the most frequent Zip from all the transactions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bibliographic-constant",
   "metadata": {},
   "outputs": [],
   "source": [
    "len(df[df['Zip'].isnull()])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "stupid-waterproof",
   "metadata": {},
   "outputs": [],
   "source": [
    "df['Zip'].isnull().values.any()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "growing-employment",
   "metadata": {},
   "outputs": [],
   "source": [
    "value_counts = df.groupby(['Merchant Name', 'Merchant City'])['Zip'].value_counts().to_frame('count')\n",
    "value_counts = value_counts.reset_index()\n",
    "value_counts.columns\n",
    "#value_counts"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "armed-mills",
   "metadata": {},
   "outputs": [],
   "source": [
    "value_counts.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "nearby-poland",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Fill missing Zip with most frequent for given Merchant name and city\n",
    "if df['Zip'].isnull().values.any():\n",
    "    df['Zip'] = df.groupby(['Merchant Name', 'Merchant City'])['Zip'].transform(lambda x: x.fillna(x.value_counts().idxmax() if x.value_counts().max() > 0 else np.nan))\n",
    "\n",
    "# If there are still missing Zips, fill them with most frequent for given Merchant city only\n",
    "if df['Zip'].isnull().values.any():\n",
    "    df['Zip'] = df.groupby('Merchant City')['Zip'].transform(lambda x: x.fillna(x.value_counts().idxmax() if x.value_counts().max() > 0 else np.nan))\n",
    "\n",
    "# If there are still missing Zips, fill them with most frequent for given Merchant state\n",
    "if df['Zip'].isnull().values.any():\n",
    "    df['Zip'] = df.groupby('Merchant State')['Zip'].transform(lambda x: x.fillna(x.value_counts().idxmax() if x.value_counts().max() > 0 else np.nan))\n",
    "\n",
    "# If there are still missing Zips, fill them with most frequent for any merchant\n",
    "if df['Zip'].isnull().values.any():\n",
    "    freq_zip = df['Zip'].value_counts().idxmax()\n",
    "    df['Zip'].fillna(freq_zip, inplace=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "racial-clothing",
   "metadata": {},
   "source": [
    "Finally, when a transaction is legitimate, the field \"Errors?\" is null. We'll fill these values with an \"OK\" string."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "current-toronto",
   "metadata": {},
   "outputs": [],
   "source": [
    "df['Errors?'].fillna('OK', inplace=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "processed-finger",
   "metadata": {},
   "outputs": [],
   "source": [
    "df.isnull().values.any()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "grave-throw",
   "metadata": {},
   "source": [
    "### Transforming categorical features <a class=\"anchor\" id=\"cat_features\"></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "finite-girlfriend",
   "metadata": {},
   "source": [
    "As mentioned earlier there is a set of features that are categorical instead of numeric:\n",
    "\n",
    "- Nominal: Use Chip, Merchant City, Merchant State, Errors?, Is Fraud?\n",
    "- Ordinal: Time"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "extra-anger",
   "metadata": {},
   "source": [
    "The time can be converted into an hour-only integer feature since minutes will have low or no impact on the fraud detection."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "neutral-inquiry",
   "metadata": {},
   "outputs": [],
   "source": [
    "df['Time'] = df['Time'].apply(lambda x: int(x.split(':')[0]))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "executive-copyright",
   "metadata": {},
   "source": [
    "The \"Use Chip\" variable can be easily converted with one-hot encoding since there are only 3 types of transaction: swipe, online and chip."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "weekly-lodging",
   "metadata": {},
   "outputs": [],
   "source": [
    "use_chip_dummies = pd.get_dummies(df['Use Chip'])\n",
    "df = pd.concat([df, use_chip_dummies], axis='columns')\n",
    "df.drop('Use Chip', axis='columns', inplace=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "helpful-instrument",
   "metadata": {},
   "source": [
    "We are going to apply the same technique on the \"Errors?\" feature since there is a limited amount of error types. However, as shown below, some transactions have multiple errors combined into one single error string. This would lead to a loss of potentially valuable information if we apply one-hot encoding directly without splitting the errors into single-error boolean features. Therefore we will have to apply it manually."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "alike-rider",
   "metadata": {},
   "outputs": [],
   "source": [
    "df['Errors?'].value_counts().to_frame('Number transactions')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "considerable-orientation",
   "metadata": {},
   "source": [
    "It turns out that there are only 7 different errors that can occur during a transaction. We'll create a different dummy column for each of them:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "owned-complex",
   "metadata": {},
   "outputs": [],
   "source": [
    "error_types = list(set([single_error for error in df['Errors?'].unique().tolist() for single_error in error.split(',') if single_error != 'OK']))\n",
    "error_types"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "unauthorized-finnish",
   "metadata": {},
   "outputs": [],
   "source": [
    "# create a zero-filled column for each type of error\n",
    "for error in error_types:\n",
    "    df[error] = 0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "solar-honor",
   "metadata": {},
   "outputs": [],
   "source": [
    "# function to set the error columns for the given transaction row\n",
    "def set_error(row):\n",
    "    if row['Errors?'] != 'OK':\n",
    "        for error_type in error_types:\n",
    "            if error_type in row['Errors?']:\n",
    "                row[error_type] = 1\n",
    "    return row\n",
    "\n",
    "df = df.apply(lambda row: set_error(row), axis=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "hired-farmer",
   "metadata": {},
   "source": [
    "Below we can see some transactions with the new error columns:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "conditional-taxation",
   "metadata": {},
   "outputs": [],
   "source": [
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "appropriate-refund",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Remove Errors categorical column\n",
    "df.drop('Errors?', axis='columns', inplace=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "configured-barcelona",
   "metadata": {},
   "source": [
    "The next two categorical variables are going to be treated in the same way. Both Merchant City and Merchant State have a high cardinality and applying one-hot encoding on them would create far too many columns which might lead to overfitting of the future tree-based model. Therefore we are going to convert them into numeric features via binary encoding, by which each category is converted into binary digits and each digit creates one feature column.\n",
    "\n",
    "We can make use of the library category_encoders for the binary encoding: https://contrib.scikit-learn.org/category_encoders/"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "armed-buying",
   "metadata": {},
   "outputs": [],
   "source": [
    "print ('Merchant cities: ' + str(len(df['Merchant City'].unique())))\n",
    "print ('Merchant states: ' + str(len(df['Merchant State'].unique())))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "atmospheric-sculpture",
   "metadata": {},
   "outputs": [],
   "source": [
    "ce_be = ce.BinaryEncoder(cols=['Merchant City', 'Merchant State']);\n",
    "df = ce_be.fit_transform(df);"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "worst-quebec",
   "metadata": {},
   "source": [
    "The new columns resulting from the binary encoding can be seen below:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "floppy-learning",
   "metadata": {},
   "outputs": [],
   "source": [
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "accessory-cornell",
   "metadata": {},
   "source": [
    "Finally, for the target variable \"Is Fraud?\", we can just map it to an integer where 0 represents a legitimate transaction and 1, a fraudulent transaction."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fleet-russell",
   "metadata": {},
   "outputs": [],
   "source": [
    "df['Is Fraud?'] = df['Is Fraud?'].apply(lambda x: 0 if x == 'No' else 1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "coral-growing",
   "metadata": {},
   "outputs": [],
   "source": [
    "df.head(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "selective-grill",
   "metadata": {},
   "source": [
    "With the above transformations, all our features are now numerical and can be used to train a machine learning model. However, we should still filter out some of the features that might not be relevant enough to predict the target \"Is Fraud\" value."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "flexible-tactics",
   "metadata": {},
   "source": [
    "### Data balancing <a class=\"anchor\" id=\"data_balancing\"></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "czech-alabama",
   "metadata": {},
   "source": [
    "- Dataset is highly imbalanced - need to do something:\n",
    "    * check ROC AUC (ability to distinguish between classes) should be close to 1 (instead of 0.5 which means it can only predict half the classes I.e. legitimate transactions)\n",
    "    * Could use  imbalance-learn library - ADASYN and SMOTE oversampling techniques in the minority class\n",
    "    * Confusion matrix and FP/FN"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "renewable-medicare",
   "metadata": {},
   "outputs": [],
   "source": [
    "def plot_balance(data):\n",
    "    LABELS = [\"Ok\", \"Fraud\"]\n",
    "    count_classes = pd.value_counts(data['Is Fraud?'])\n",
    "    count_classes.plot(kind = 'bar', rot=0)\n",
    "    plt.title(\"Fraud distribution\")\n",
    "    plt.xticks(range(2), LABELS)\n",
    "    plt.xlabel(\"Is Fraud\")\n",
    "    plt.ylabel(\"Frequency\");\n",
    "    \n",
    "plot_balance(df)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "normal-charger",
   "metadata": {},
   "outputs": [],
   "source": [
    "frauds = df[df['Is Fraud?'] == 1]\n",
    "normal = df[df['Is Fraud?'] == 0]\n",
    "print(f'Num fraudulent transactions:{len(frauds)}')\n",
    "print(f'Num legitimate transactions:{len(normal)}')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "stupid-corner",
   "metadata": {},
   "outputs": [],
   "source": [
    "print(f'Only {round(len(frauds)/len(df) * 100, 2)}% of transactions are fraudulent')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "initial-electric",
   "metadata": {},
   "source": [
    "It seems that we have a highly imbalanced dataset on our hands since legitimate transactions overwhelm the fraudulent ones by a large margin. This imbalance of the target class might decrease the performance of the classification algorithm (the model might fail to identify fraud) so we'll need to create more samples of the minority class i.e. the fraudulent transactions.\n",
    "\n",
    "In particular, we are going to apply an oversampling method called Synthetic Minority Oversampling Technique (SMOTE) which takes into account characteristics of the fraudulent class to create synthetic duplicates. The objective is to increase the ratio of the fraudulent class from 0.6% to 10% of the total transactions.\n",
    "\n",
    "We can use the library imbalanced-learn for this task: https://pypi.org/project/imbalanced-learn/"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "outstanding-eligibility",
   "metadata": {},
   "outputs": [],
   "source": [
    "X = df.loc[:, df.columns != 'Is Fraud?']\n",
    "y = df['Is Fraud?']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "palestinian-criminal",
   "metadata": {},
   "outputs": [],
   "source": [
    "method = SMOTE(sampling_strategy=0.1)\n",
    "X_resampled, y_resampled = method.fit_sample(X, y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "banned-paintball",
   "metadata": {},
   "outputs": [],
   "source": [
    "X_resampled = pd.DataFrame(X_resampled, columns=X.columns)\n",
    "\n",
    "y_resampled = pd.DataFrame(y_resampled, columns=['Is Fraud?'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "exciting-lotus",
   "metadata": {},
   "outputs": [],
   "source": [
    "df_resampled = pd.concat([X_resampled, y_resampled], axis='columns')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "rental-poison",
   "metadata": {},
   "source": [
    "Below we can see that the distribution of fraudulent and legimitate transactions has now changed to have a ratio of 90-10:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "gorgeous-commercial",
   "metadata": {},
   "outputs": [],
   "source": [
    "plot_balance(df_resampled)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "postal-southeast",
   "metadata": {},
   "outputs": [],
   "source": [
    "frauds = df_resampled[df_resampled['Is Fraud?'] == 1]\n",
    "normal = df_resampled[df_resampled['Is Fraud?'] == 0]\n",
    "print(f'Num fraudulent transactions:{len(frauds)}')\n",
    "print(f'Num genuine transactions:{len(normal)}')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "premium-raise",
   "metadata": {},
   "source": [
    "## Modelling <a class=\"anchor\" id=\"modelling\"></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "compliant-bracelet",
   "metadata": {},
   "source": [
    "### Model preparation <a class=\"anchor\" id=\"model_prep\"></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "positive-dinner",
   "metadata": {},
   "source": [
    "We are going to try 3 different classification algorithms (logistic regression, decision tree and random forest) but fefore that, we need to split our data into train (80%) and test (20%), and normalize it with the standard scaler."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "altered-optimization",
   "metadata": {},
   "outputs": [],
   "source": [
    "y = df_resampled['Is Fraud?']\n",
    "X = df_resampled.drop(['Is Fraud?'], axis=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "minus-steal",
   "metadata": {},
   "outputs": [],
   "source": [
    "RANDOM_SEED = 15\n",
    "\n",
    "X_train, X_test, y_train, y_test = \\\n",
    "        train_test_split(X, y, test_size=0.2, random_state=RANDOM_SEED)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "greek-baseball",
   "metadata": {},
   "outputs": [],
   "source": [
    "1 in y_test"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "prompt-deployment",
   "metadata": {},
   "outputs": [],
   "source": [
    "scaler = StandardScaler()\n",
    "T = scaler.fit(X_train)\n",
    "\n",
    "X_train = scaler.transform(X_train.values)\n",
    "X_test = scaler.transform(X_test.values)\n",
    "\n",
    "y_test = y_test.values\n",
    "y_train = y_train.values"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "protected-cholesterol",
   "metadata": {},
   "source": [
    "Below are the sizes of the 2 subsets"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "spectacular-midwest",
   "metadata": {},
   "outputs": [],
   "source": [
    "print(f'Train size: {str(len(X_train))}')\n",
    "print(f'Test size: {str(len(X_test))}')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "capital-convertible",
   "metadata": {},
   "source": [
    "### Model training and evaluation <a class=\"anchor\" id=\"model_train\"></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "tutorial-cosmetic",
   "metadata": {},
   "source": [
    "Below we can see the parameters we are using for our 3 classification models:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "basic-tournament",
   "metadata": {},
   "outputs": [],
   "source": [
    "names = ['Logistic Regression', 'Decision Tree', 'Random Forest']\n",
    "\n",
    "classifiers = [\n",
    "    linear_model.LogisticRegression(),\n",
    "    DecisionTreeClassifier(max_depth=5),\n",
    "    RandomForestClassifier(max_depth=5, n_estimators=50)]\n",
    "\n",
    "classifiers"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "explicit-knowing",
   "metadata": {},
   "source": [
    "Now let's run the different algorithms and generate their confusion matrices. Confusion matrix is an essential evaluation method for classification problems such as our fraud detection system. With this method we can easily visualise the false/true negatives and positives, which will help us understand the precision and recall of our model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "built-hormone",
   "metadata": {},
   "outputs": [],
   "source": [
    "LABELS = ['Legitimate', 'Fraud']\n",
    "\n",
    "i = 1\n",
    "figure = plt.figure(figsize=(27, 7))\n",
    "\n",
    "half_point = int(len(classifiers)/2)\n",
    "\n",
    "if len(classifiers) % 2 == 1:\n",
    "    half_point += 1\n",
    "\n",
    "probs =[]\n",
    "for name, clf in zip(names, classifiers):\n",
    "        ax = plt.subplot(1, len(classifiers) + 1, i)\n",
    "        clf.fit(X_train, y_train)\n",
    "        score = clf.score(X_test, y_test)\n",
    "        y_pred = clf.predict(X_test) \n",
    "                \n",
    "        Z = clf.predict_proba(X_test)[:, 1]\n",
    "        \n",
    "        probs.append(Z)\n",
    "        \n",
    "        conf_matrix = confusion_matrix(y_test, y_pred)\n",
    "\n",
    "        show_bar = False\n",
    "        if i == len(classifiers):\n",
    "            show_bar = True\n",
    "            \n",
    "        sns.heatmap(conf_matrix, xticklabels=LABELS, yticklabels=LABELS, annot=True, \n",
    "                    cbar = show_bar, fmt=\"d\");\n",
    "\n",
    "        ax.set_title(name + \": \" + str(\"{0:.4f}\".format(score)))\n",
    "        \n",
    "        if i == 1:\n",
    "            plt.ylabel('True class')\n",
    "            \n",
    "        if i == half_point:\n",
    "            plt.xlabel('Predicted class')\n",
    "            \n",
    "        i += 1\n",
    "\n",
    "\n",
    "plt.tight_layout()\n",
    "plt.show()        "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "everyday-shirt",
   "metadata": {},
   "source": [
    "Even though the plots above can give us a good summary of our models performance, we should dive deeper specially taking into account how imbalanced our dataset was originally.\n",
    "\n",
    "One way to investigate further is by using ROC curves to understand the performance of our binary classifier."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "accredited-feeding",
   "metadata": {},
   "outputs": [],
   "source": [
    "figure = plt.figure(figsize=(27, 5))\n",
    "i = 1\n",
    "\n",
    "for name, clf in zip(names, classifiers):\n",
    "    fpr, tpr, thresholds = roc_curve(y_test, probs[i-1])\n",
    "    roc_auc = auc(fpr, tpr)\n",
    "    ax = plt.subplot(1, len(classifiers) + 1, i)\n",
    "    \n",
    "# \n",
    "    ax.plot(fpr, tpr, label= 'AUC= %0.4f'% roc_auc)\n",
    "    ax.legend(loc='lower right')\n",
    "    ax.plot([0,1],[0,1],'r--')\n",
    "    ax.set_xlim([-0.01, 1])\n",
    "    ax.set_ylim([0, 1.05])\n",
    "    ax.set_title(name)\n",
    "    if i == 1:\n",
    "        ax.set_ylabel('True Positive Rate')\n",
    "    if i == half_point:\n",
    "       \n",
    "        ax.set_xlabel('False Positive Rate')\n",
    "    i += 1\n",
    "\n",
    "plt.show();"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fantastic-female",
   "metadata": {},
   "source": [
    "# Region analysis <a class=\"anchor\" id=\"region_analysis\"></a>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ordered-saudi",
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "import re\n",
    "import folium"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "chinese-digest",
   "metadata": {},
   "outputs": [],
   "source": [
    "df = pd.read_csv(\"data/credit_card_transactions-ibm_v2.csv\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "animal-thanks",
   "metadata": {},
   "outputs": [],
   "source": [
    "# group by merchant country and fraud class, and rename columns\n",
    "df_fraud_per_state = df.groupby(['Merchant State', 'Is Fraud?']).size().unstack(fill_value=0).reset_index()\n",
    "df_fraud_per_state.rename(columns = {'No':'Legitimate', 'Yes':'Fraudulent'}, inplace = True)\n",
    "df_fraud_per_state.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "hired-catalyst",
   "metadata": {},
   "outputs": [],
   "source": [
    "df_fraud_per_state.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "opposite-amateur",
   "metadata": {},
   "outputs": [],
   "source": [
    "# calculate fraud ratio per country\n",
    "df_fraud_per_state['Fraud Ratio'] = df_fraud_per_state.apply(lambda x: x['Fraudulent'] / (x['Fraudulent'] + x['Legitimate']), axis='columns')\n",
    "df_fraud_per_state.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ordinary-sterling",
   "metadata": {},
   "source": [
    "Since we will be performing an analysis at country level, we are going to treat all the transactions from an US state as part of the United States. Therefore we will convert all 2-letter US state names into \"United States of America\"."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "critical-accused",
   "metadata": {},
   "outputs": [],
   "source": [
    "# update country name for all the US states\n",
    "df_fraud_per_state['Merchant State'] = df_fraud_per_state['Merchant State'].apply(lambda x: 'United States of America' if re.search(r'\\b[A-Z]{2}\\b', x) else x)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "frank-research",
   "metadata": {},
   "outputs": [],
   "source": [
    "# combine the amounts for all the US states into one\n",
    "us_transactions = df_fraud_per_state[df_fraud_per_state['Merchant State'] == 'United States of America'].sum(numeric_only=True).to_frame().transpose()\n",
    "us_transactions['Merchant State'] = 'United States of America'\n",
    "us_transactions['Fraud Ratio'] = us_transactions.apply(lambda x: x['Fraudulent'] / (x['Fraudulent'] + x['Legitimate']), axis='columns')\n",
    "us_transactions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "portuguese-nevada",
   "metadata": {},
   "outputs": [],
   "source": [
    "# drop US-states rows from dataframe (they'll be replaced by 1 US-row)\n",
    "index_us_states = df_fraud_per_state[df_fraud_per_state['Merchant State'] == 'United States of America'].index\n",
    "df_fraud_per_state.drop(index_us_states, inplace = True)\n",
    "\n",
    "# append US-row\n",
    "df_fraud_per_state = df_fraud_per_state.append(us_transactions)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "novel-economy",
   "metadata": {},
   "outputs": [],
   "source": [
    "df_fraud_per_state.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ordered-wrestling",
   "metadata": {},
   "source": [
    "Below is the list of countries with the highest fraud rate:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "falling-recording",
   "metadata": {},
   "outputs": [],
   "source": [
    "df_fraud_per_state[df_fraud_per_state['Merchant State'] == 'Italy']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "afraid-briefs",
   "metadata": {},
   "outputs": [],
   "source": [
    "# show countries with the highest ratio of fraud\n",
    "df_fraud_per_state[['Merchant State', 'Fraud Ratio']].sort_values('Fraud Ratio', ascending=False).head(10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "dated-contact",
   "metadata": {},
   "outputs": [],
   "source": [
    "# load file with world country coords\n",
    "world_geo = r'data/world-countries.json' # geojson file\n",
    "\n",
    "# create a plain world map\n",
    "world_map = folium.Map(location=[0, 0], zoom_start=2, tiles='Mapbox Bright')\n",
    "\n",
    "# create a numpy array of length 4 and has linear spacing from the minimum fraud ratio to the maximum value\n",
    "threshold_scale = np.linspace(start=0.0, stop=1.0, num=5, endpoint=True, dtype=float)\n",
    "threshold_scale = threshold_scale.tolist() # change the numpy array to a list\n",
    "threshold_scale[-1] = threshold_scale[-1] + 0.001 # ensure last value of list is greater than max fraud ratio\n",
    "\n",
    "# let Folium determine the scale.\n",
    "world_map = folium.Map(location=[0, 0], zoom_start=2, tiles='Mapbox Bright')\n",
    "world_map.choropleth(\n",
    "    geo_data=world_geo,\n",
    "    data=df_fraud_per_state,\n",
    "    columns=['Merchant State', 'Fraud Ratio'],\n",
    "    key_on='feature.properties.name',\n",
    "    threshold_scale=threshold_scale,\n",
    "    fill_color='YlOrRd', \n",
    "    fill_opacity=0.7, \n",
    "    line_opacity=0.2,\n",
    "    legend_name='Fraud ratio',\n",
    "    reset=True\n",
    ")\n",
    "world_map"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "loose-space",
   "metadata": {},
   "source": [
    "Apart from discovering the ratio of fraud in each country, it might be interesting to find out what are the amounts (in USD) for which fraud has been commited. Perhaps only a few fraudulent transactions have been commited in a country but if those transactions included large amounts of money then it will be relevant for our analysis."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "beautiful-scope",
   "metadata": {},
   "outputs": [],
   "source": [
    "# drop legitimate and online transactions since we're only interested in the total fraudulent amount per merchant country\n",
    "drop_transactions = df[(df['Is Fraud?'] == 'No') | (df['Merchant City'] == 'ONLINE')].index\n",
    "fraudulent_df = df.drop(drop_transactions)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "similar-seattle",
   "metadata": {},
   "outputs": [],
   "source": [
    "# keep only columns Merchant State and Amount\n",
    "fraudulent_df.drop(fraudulent_df.columns.difference(['Merchant State','Amount']), axis='columns', inplace=True)\n",
    "fraudulent_df['Amount'] = fraudulent_df['Amount'].apply(lambda x: float(x.strip('$')))\n",
    "fraudulent_df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "specific-interim",
   "metadata": {},
   "outputs": [],
   "source": [
    "# group by merchant country and calculate the sum of amounts\n",
    "df_fraud_amount_per_state = fraudulent_df.groupby(['Merchant State']).sum().reset_index()\n",
    "df_fraud_amount_per_state.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cellular-crown",
   "metadata": {},
   "source": [
    "Since we will be performing an analysis at country level, we are going to treat all the fraudulent amount from an US state as part of the United States. Therefore we will convert all 2-letter US state names into \"United States of America\"."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "annoying-moses",
   "metadata": {},
   "outputs": [],
   "source": [
    "# merge all US states into 1 United States row\n",
    "df_fraud_amount_per_state[df_fraud_amount_per_state['Merchant State'].apply(lambda x: True if re.search(r'\\b[A-Z]{2}\\b', x) else False)]['Amount'].sum()\n",
    "\n",
    "df_fraud_amount_per_state['Merchant State'] = df_fraud_amount_per_state['Merchant State'].apply(lambda x: 'United States of America' if re.search(r'\\b[A-Z]{2}\\b', x) else x)\n",
    "\n",
    "us_fraud_amount = df_fraud_amount_per_state[df_fraud_amount_per_state['Merchant State'] == 'United States of America'].sum(numeric_only=True).to_frame().transpose()\n",
    "us_fraud_amount['Merchant State'] = 'United States of America'\n",
    "us_fraud_amount"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "hired-coaching",
   "metadata": {},
   "outputs": [],
   "source": [
    "# drop US-states rows from dataframe (they'll be replaced by 1 US-row)\n",
    "index_us_states = df_fraud_amount_per_state[df_fraud_amount_per_state['Merchant State'] == 'United States of America'].index\n",
    "df_fraud_amount_per_state.drop(index_us_states, inplace = True)\n",
    "\n",
    "# append US-row\n",
    "df_fraud_amount_per_state = df_fraud_amount_per_state.append(us_fraud_amount)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ancient-cookie",
   "metadata": {},
   "source": [
    "Below is the list of countries with the highest fraud amount:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "incorporated-ancient",
   "metadata": {},
   "outputs": [],
   "source": [
    "df_fraud_amount_per_state.sort_values('Amount', ascending=False).head(10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "based-trade",
   "metadata": {},
   "outputs": [],
   "source": [
    "df_fraud_amount_per_state['Amount'].max()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "promotional-meter",
   "metadata": {},
   "outputs": [],
   "source": [
    "# load world countries with coords\n",
    "world_geo = r'data/world-countries.json' # geojson file\n",
    "\n",
    "# create a plain world map\n",
    "world_map = folium.Map(location=[0, 0], zoom_start=2, tiles='Mapbox Bright')\n",
    "\n",
    "# create a numpy array of length 6 and has linear spacing from the minimum fraud ratio to the maximum value\n",
    "threshold_scale = np.linspace(start=0, stop=500000, num=6, endpoint=True, dtype=float)\n",
    "threshold_scale = threshold_scale.tolist() # change the numpy array to a list\n",
    "threshold_scale[-1] = threshold_scale[-1] + 0.001 # ensure last value of list is greater than max fraud ratio\n",
    "\n",
    "# let Folium determine the scale.\n",
    "world_map = folium.Map(location=[0, 0], zoom_start=2, tiles='Mapbox Bright')\n",
    "world_map.choropleth(\n",
    "    geo_data=world_geo,\n",
    "    data=df_fraud_amount_per_state,\n",
    "    columns=['Merchant State', 'Amount'],\n",
    "    key_on='feature.properties.name',\n",
    "    threshold_scale=threshold_scale,\n",
    "    fill_color='YlOrRd', \n",
    "    fill_opacity=0.7, \n",
    "    line_opacity=0.2,\n",
    "    legend_name='Fraudulent amount (in USD)',\n",
    "    reset=True\n",
    ")\n",
    "world_map"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bibliographic-talent",
   "metadata": {},
   "source": [
    "# Merchant analysis <a class=\"anchor\" id=\"merchant_analysis\"></a>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "absent-storm",
   "metadata": {},
   "outputs": [],
   "source": [
    "# for HTTP requests\n",
    "import requests  \n",
    "\n",
    "# for HTML scrapping \n",
    "from bs4 import BeautifulSoup "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "asian-handbook",
   "metadata": {},
   "outputs": [],
   "source": [
    "df = pd.read_csv(\"data/credit_card_transactions-ibm_v2.csv\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "environmental-adams",
   "metadata": {},
   "outputs": [],
   "source": [
    "# URL of website from which to scrap tabular data.\n",
    "mcc_url = \"https://docs.checkout.com/resources/codes/merchant-category-codes\"\n",
    "\n",
    "# if the request was successful, reponse should be 200.\n",
    "response = requests.get(mcc_url)\n",
    "assert response.status_code == 200\n",
    "\n",
    "# parse response content to HTML\n",
    "soup = BeautifulSoup(response.content, 'html.parser')\n",
    "\n",
    "# title of website\n",
    "title = soup.title.string\n",
    "print(f'Page title: {title}') \n",
    "\n",
    "# find the right table to scrap\n",
    "mcc_table=soup.find('table')\n",
    "\n",
    "# get the 1st row of the table i.e. the header\n",
    "row0 = mcc_table.findAll(\"tr\")[0]\n",
    "\n",
    "# show the column names\n",
    "header = [th.text.rstrip() for th in row0.find_all('th')]\n",
    "print(f'Column names: {header}') "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "female-motorcycle",
   "metadata": {},
   "outputs": [],
   "source": [
    "# construct dictionary of MCCs\n",
    "merchant_category_codes = {}\n",
    "\n",
    "# iterate through the rows of the table\n",
    "for row in mcc_table.findAll(\"tr\"):    \n",
    "    cells = row.findAll('td')\n",
    "    if len(cells)==2:\n",
    "        code = int(cells[0].find(text=True))\n",
    "        desc =  cells[1].find(text=True)\n",
    "        merchant_category_codes[code] = desc\n",
    "    \n",
    "print(f'Number of merchant codes: {len(merchant_category_codes)}')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "framed-dover",
   "metadata": {},
   "outputs": [],
   "source": [
    "# group by MCC and fraud class, and rename columns\n",
    "df_fraud_per_mcc = df.groupby(['MCC', 'Is Fraud?']).size().unstack(fill_value=0).reset_index()\n",
    "df_fraud_per_mcc.rename(columns = {'No':'Legitimate', 'Yes':'Fraudulent'}, inplace = True)\n",
    "df_fraud_per_mcc.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "surface-hayes",
   "metadata": {},
   "outputs": [],
   "source": [
    "df_fraud_per_mcc.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "attached-trick",
   "metadata": {},
   "source": [
    "Below is the total number of fraudulent transactions:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "alleged-emperor",
   "metadata": {},
   "outputs": [],
   "source": [
    "# calculate total number of fraudulent transactions\n",
    "total_fraud_transactions = df_fraud_per_mcc['Fraudulent'].sum()\n",
    "total_fraud_transactions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "irish-thread",
   "metadata": {},
   "outputs": [],
   "source": [
    "# get the categories of merchant with the highest number of fraudulent transactions\n",
    "top_merchant_fraud = df_fraud_per_mcc.sort_values('Fraudulent', ascending=False).head(9)\n",
    "top_merchant_fraud"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "split-moscow",
   "metadata": {},
   "outputs": [],
   "source": [
    "rest_fraud_transactions = total_fraud_transactions - top_merchant_fraud['Fraudulent'].sum()\n",
    "y = np.array(top_merchant_fraud['Fraudulent'].append(pd.Series(rest_fraud_transactions)))\n",
    "top_mcc_labels = [merchant_category_codes[mcc] for mcc in top_merchant_fraud['MCC']]\n",
    "top_mcc_labels.append('Other')\n",
    "\n",
    "title = plt.title('Fraud by type of merchant')\n",
    "pie = plt.pie(y)\n",
    "plt.axis('equal')\n",
    "plt.legend(pie[0],top_mcc_labels, bbox_to_anchor=(1.5,0.5), loc=\"center right\",\n",
    "            fontsize=14, bbox_transform=plt.gcf().transFigure)\n",
    "plt.subplots_adjust(left=0.0, bottom=0.1, right=0.5)"
   ]
  }
 ],
 "metadata": {
  "language_info": {
   "name": "python",
   "pygments_lexer": "ipython3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}