4-Classification.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "f0981766",
   "metadata": {},
   "source": [
    "## Classification with wine quality dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a5f33e9b",
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd \n",
    "\n",
    "url = \"https://archive.ics.uci.edu/ml/machine-learning-databases/wine-quality/winequality-red.csv\"\n",
    "\n",
    "df = pd.read_csv(url, sep=';')\n",
    "\n",
    "df.head(10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "aa3b8b00",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Defining a binary quality Q:\n",
    "# Q=1 when quality > 5 and is considered as high quality\n",
    "# Q=0 when quality <=5 and is considered as low quality\n",
    "df['Q'] = 0\n",
    "df.loc[df['quality'] > 5, 'Q'] = 1\n",
    "df.drop(columns='quality', inplace=True)\n",
    "\n",
    "df.head(20)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "058bd08b",
   "metadata": {},
   "outputs": [],
   "source": [
    "df.corr()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c3fe1d17",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Let's choose the two most correlating features for Q: alcohol and volatile acidity"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7be4ab50",
   "metadata": {},
   "source": [
    "## Logistic Regression\n",
    "Despite its name, is not a regression algorithm. It is a classification algorithm based on a logistic function\n",
    "\n",
    "L(S) = 1 / (1 + e ^ (-S))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d0eb769f",
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "\n",
    "def logistic(s): \n",
    "    return 1. / (1 + np.exp(-s))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e153ccb9",
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "xs = np.arange(-10, 10, 0.1)\n",
    "plt.plot(xs, logistic(xs));"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f6aee9f7",
   "metadata": {},
   "source": [
    "Therefore whatever number you pass to logistic function, it will clamp it between 0 and 1\n",
    "In the case of a binary classification, you can interpret this behavior as follows:\n",
    "\n",
    "- output of 0: negative\n",
    "- output of 1: positive\n",
    "- in between: the probability of positive\n",
    "\n",
    "In Logistic Regression, the s is typically a linear combination of features:\n",
    "\n",
    "s = b + w0 X0 + w1 X1 + ...\n",
    "\n",
    "loss function: https://developers.google.com/machine-learning/crash-course/logistic-regression/model-training\n",
    "\n",
    "Take the data set below:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "99c542d4",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Generating a random datasets with two features and a binary class target using scikit-learn datasets module\n",
    "from sklearn.datasets import make_classification\n",
    "\n",
    "X, y = make_classification(\n",
    "    n_samples = 500,\n",
    "    n_features = 2,\n",
    "    n_informative=2,\n",
    "    n_redundant=0,\n",
    "    n_repeated = 0,\n",
    "    n_classes = 2,\n",
    "    random_state = 6,\n",
    "    class_sep = 1.1,\n",
    "    n_clusters_per_class=1,\n",
    "    shift=[10, 10],\n",
    "    scale=[1, 2]\n",
    ")\n",
    "\n",
    "# print(X[:5])\n",
    "# print(y[:5])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cc7016ec",
   "metadata": {},
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "\n",
    "\n",
    "p = X[y == 1]\n",
    "n = X[y == 0]\n",
    "\n",
    "plt.scatter(p[:, 0], p[:, 1], color='blue', label='Positive')\n",
    "plt.scatter(n[:, 0], n[:, 1], color='orange', label='Negative');\n",
    "\n",
    "plt.xlabel('X0');plt.ylabel('X1');plt.legend();"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cccd9cd5",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.preprocessing import StandardScaler\n",
    "\n",
    "# Train test splitting\n",
    "X_train, X_test, y_train, y_test = train_test_split(\n",
    "    X, y, test_size=0.2, random_state=46)\n",
    "\n",
    "# Using a scaler object to bring all features in the same range\n",
    "scaler = StandardScaler().fit(X_train)\n",
    "\n",
    "# Try plotting the data points above with this scaled version of data\n",
    "X_train_scaled = scaler.transform(X_train)\n",
    "X_test_scaled = scaler.transform(X_test)\n",
    "\n",
    "# Creating and training the classifier\n",
    "clf = linear_model.LogisticRegression().fit(X_train_scaled, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bdac28b9",
   "metadata": {},
   "outputs": [],
   "source": [
    "## Classifier training score relates to the percentage of correctly classified datapoints\n",
    "clf.score(X_train_scaled, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c1e89215",
   "metadata": {},
   "outputs": [],
   "source": [
    "## Classifier test score\n",
    "clf.score(X_test_scaled, y_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f22f794e",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Coefficients\n",
    "W = clf.coef_[0]\n",
    "b = clf.intercept_[0]\n",
    "print(f'W is {W}')\n",
    "print(f'b is {b}')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bcb6a234",
   "metadata": {},
   "source": [
    "### Decision boundary\n",
    "is the line separating these two regions where the logistic is 0.5 which means s = 0\n",
    "\n",
    "0 = b + W[0] * X[0] + W[1] * X[1]\n",
    "\n",
    "in terms of the plot:\n",
    "X[1] = -(b + W[0] * X[0]) / W[1]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cc3ecd02",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Decision Boundary line equation using lambda function\n",
    "\n",
    "DBLine = lambda x0: -(b + W[0] * x0) / W[1]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f1ca9bd6",
   "metadata": {},
   "outputs": [],
   "source": [
    "p = X_train_scaled[y_train == 1]\n",
    "n = X_train_scaled[y_train == 0]\n",
    "plt.scatter(p[:, 0], p[:, 1], color='blue', label='Positive')\n",
    "plt.scatter(n[:, 0], n[:, 1], color='orange', label='Negative')\n",
    "\n",
    "x0Range = np.arange(-4,4,1)\n",
    "plt.plot(x0Range, DBLine(x0Range), color='red')\n",
    "\n",
    "plt.xlabel('X0');plt.ylabel('X1');plt.legend();"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1bba3e27",
   "metadata": {},
   "source": [
    "## Nonlinear relationships\n",
    "Similar to linear regression, with logistic regression we can produce curved decision boundaries by adding more pseudo-features. To be concrete we can add higher order terms like second order of features like x0^2, x1^2 and x0*x1\n",
    "\n",
    "Take the circular dataset below. It is not possible to efficively define separation with a line\n",
    "- Try rerunning the cells above with X and y below to see how the scores and the line\n",
    "- Because Logistic regression is too biased, it performs awful, we need to add variance by adding more features"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f019bb1d",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.datasets import make_circles\n",
    "\n",
    "X, y = make_circles(n_samples=500, shuffle=True, noise=0.2, random_state=0, factor=.3)\n",
    "\n",
    "p = X[y == 1]\n",
    "n = X[y == 0]\n",
    "\n",
    "plt.scatter(p[:, 0], p[:, 1], color='blue', label='Positive')\n",
    "plt.scatter(n[:, 0], n[:, 1], color='orange', label='Negative')\n",
    "\n",
    "plt.xlabel('X0');plt.ylabel('X1');plt.legend();"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0b53c4a4",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Let's add higher order psuedo-features to the data set using numpy.stack function\n",
    "\n",
    "X = np.stack((X[:,0], X[:,1], X[:,0]**2, X[:,1]**2, X[:,0] * X[:,1]), axis = 1)\n",
    "\n",
    "X[0:3]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "318cced6",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Train test splitting\n",
    "X_train, X_test, y_train, y_test = train_test_split(\n",
    "    X, y, test_size=0.2, random_state=46)\n",
    "\n",
    "# Creating and training the classifier\n",
    "clf = linear_model.LogisticRegression().fit(X_train, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "04800876",
   "metadata": {},
   "outputs": [],
   "source": [
    "clf.score(X_test, y_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9a34d040",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Coefficients\n",
    "W = clf.coef_[0]\n",
    "b = clf.intercept_[0]\n",
    "print(f'W is {W}')\n",
    "print(f'b is {b}')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8cad612c",
   "metadata": {},
   "outputs": [],
   "source": [
    "p = X_test[y_test == 1]\n",
    "n = X_test[y_test == 0]\n",
    "\n",
    "plt.scatter(p[:, 0], p[:, 1], color='blue', label='Positive')\n",
    "plt.scatter(n[:, 0], n[:, 1], color='orange', label='Negative')\n",
    "\n",
    "# Plotting the polynomail equation using contours\n",
    "delta = 0.025\n",
    "xrange = np.arange(-2, 2, delta)\n",
    "yrange = np.arange(-2, 2, delta)\n",
    "x, y = np.meshgrid(xrange, yrange)\n",
    "equation = b + W[0]*x + W[1]*y + W[2]*x**2 + W[3]*y**2 + W[4]*x*y\n",
    "plt.contour(x, y, equation, levels=[0])\n",
    "\n",
    "plt.xlabel('X0');plt.ylabel('X1');plt.legend();"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bce6272e",
   "metadata": {},
   "source": [
    "## Confusion matrix\n",
    "Used for analyzing classifier prediction results"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2421d0fb",
   "metadata": {},
   "outputs": [],
   "source": [
    "y_true = y_test\n",
    "y_true"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0421efde",
   "metadata": {},
   "outputs": [],
   "source": [
    "y_pred = clf.predict(X_test)\n",
    "y_pred"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5cbf5762",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.metrics import confusion_matrix\n",
    "\n",
    "cm = confusion_matrix(y_true, y_pred)\n",
    "cm"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e21ddad2",
   "metadata": {},
   "outputs": [],
   "source": [
    "import seaborn as sn\n",
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "df_cm = pd.DataFrame(cm, [0, 1], [0, 1])\n",
    "plt.figure(figsize=(5,4))\n",
    "sn.set(font_scale=1.4) # for label size\n",
    "sn.heatmap(df_cm, annot=True, annot_kws={\"size\": 16}, cmap=\"Blues\") # font size\n",
    "plt.xlabel('Predicted');plt.ylabel('True value');"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "49313791",
   "metadata": {},
   "source": [
    "The link below explains the confusion matrix and alternative measures for evaluation\n",
    "\n",
    "https://medium.com/analytics-vidhya/what-is-a-confusion-matrix-d1c0f8feda5"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "86711683",
   "metadata": {},
   "outputs": [],
   "source": [
    "[[TN, FP], [FN, TP]] = cm\n",
    "\n",
    "print(\n",
    "f'''\n",
    "True Negative: {TN}\n",
    "True Positive: {TP}\n",
    "\n",
    "False Positive: {FP} (TYPE I error)\n",
    "True Negative: {FN} (TYPE II error)\n",
    "'''\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f6c9878a",
   "metadata": {},
   "source": [
    "### Common evaluation metrics\n",
    "Accuracy is simplay the percentage of correct predictions. This may not be a good value to evaluate a model specially if the data is not balanced"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "13730dd7",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Accuracy: (TP + TN) / All\n",
    "acc = (TP + TN) / (TP + TN + FP + FN)\n",
    "acc"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fd67def6",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Precision: TP / All predicted positive -> application: Email spam or fraud transaction\n",
    "precision = TP / (TP + FP)\n",
    "precision"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "010b5118",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Recall: TP / All true positives -> important medical test\n",
    "recall = TP / (TP + FN)\n",
    "recall"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "686c4e09",
   "metadata": {},
   "outputs": [],
   "source": [
    "# F1 score: 2 * precision * recall / (precision + recall)\n",
    "F1 = 2 * precision * recall / (precision + recall)\n",
    "F1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "198b9a89",
   "metadata": {},
   "outputs": [],
   "source": [
    "# There's always a trade-off between precision and recall and you can adjust your hyperparameters to move towards one side"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "40d89e34",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}