Customer_Segmentation_Analysis.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 5,
  "metadata": {
    "noteable-chatgpt": {
      "create_notebook": {
        "openai_conversation_id": "3121cb33-28e6-587a-b866-09149f632b47",
        "openai_ephemeral_user_id": "29511849-3e0b-581a-889a-69c08a397a30",
        "openai_subdivision1_iso_code": "US-NJ"
      }
    },
    "noteable": {
      "last_transaction_id": "d54f49d5-8cfd-4919-922f-0703afb289a5"
    },
    "selected_hardware_size": "small"
  },
  "cells": [
    {
      "id": "6b03f6fc-4267-4912-b0e5-3d6281f83f4d",
      "cell_type": "code",
      "metadata": {
        "noteable": {
          "cell_type": "code",
          "output_collection_id": "0da70c8b-40d3-4aa2-ba55-6e2ba933fceb"
        },
        "ExecuteTime": {
          "end_time": "2023-06-30T17:43:09.964844+00:00",
          "start_time": "2023-06-30T17:42:55.254616+00:00"
        }
      },
      "execution_count": null,
      "source": "!pip install pandas\n!pip install matplotlib\n!pip install seaborn\nimport pandas as pd\nimport matplotlib.pyplot as plt\nimport seaborn as sns\nimport numpy as np\nfrom sklearn.model_selection import train_test_split\nfrom sklearn.linear_model import LogisticRegression\nfrom sklearn.metrics import accuracy_score, precision_score, recall_score, f1_score, roc_auc_score\nfrom sklearn.preprocessing import StandardScaler, LabelEncoder\nfrom scipy.stats import chi2_contingency\nfrom scipy.stats import ttest_ind\nimport warnings\nwarnings.filterwarnings('ignore')",
      "outputs": []
    },
    {
      "id": "7f770a75-bffd-47b8-a31b-b05a9493d397",
      "cell_type": "code",
      "metadata": {
        "noteable": {
          "cell_type": "code",
          "output_collection_id": "578d326c-30e6-4cce-92a9-9a0640194fe5"
        },
        "ExecuteTime": {
          "end_time": "2023-06-30T17:43:28.682119+00:00",
          "start_time": "2023-06-30T17:43:28.339474+00:00"
        },
        "datalink": {
          "6a16c64d-dfff-4231-bf4e-8a89d8effd37": {
            "applied_filters": [],
            "dataframe_info": {
              "default_index_used": true,
              "orig_num_cols": 8,
              "orig_num_rows": 5,
              "orig_size_bytes": 360,
              "truncated_num_cols": 8,
              "truncated_num_rows": 5,
              "truncated_size_bytes": 360,
              "truncated_string_columns": []
            },
            "display_id": "6a16c64d-dfff-4231-bf4e-8a89d8effd37",
            "dx_settings": {
              "ALLOW_NOTEABLE_ATTRS": true,
              "COLUMN_SAMPLING_METHOD": "outer",
              "DB_LOCATION": ":memory:",
              "DEV_MODE": false,
              "DISPLAY_MAX_COLUMNS": 100,
              "DISPLAY_MAX_ROWS": 50000,
              "DISPLAY_MODE": "simple",
              "ENABLE_ASSIGNMENT": true,
              "ENABLE_DATALINK": true,
              "FLATTEN_COLUMN_VALUES": true,
              "FLATTEN_INDEX_VALUES": false,
              "GENERATE_DEX_METADATA": false,
              "HTML_TABLE_SCHEMA": false,
              "LOG_LEVEL": 30,
              "MAX_RENDER_SIZE_BYTES": 104857600,
              "MAX_STRING_LENGTH": 250,
              "NUM_PAST_SAMPLES_TRACKED": 3,
              "RANDOM_STATE": 12648430,
              "RESET_INDEX_VALUES": false,
              "ROW_SAMPLING_METHOD": "random",
              "SAMPLING_FACTOR": 0.1,
              "SAMPLING_METHOD": "random",
              "STRINGIFY_COLUMN_VALUES": true,
              "STRINGIFY_INDEX_VALUES": false
            },
            "sample_history": [],
            "sampling_time": "2023-06-30T17:43:28.524847",
            "user_variable_name": null,
            "variable_name": "unk_dataframe_72b1d9bb8a7b45fc94e02ffe053add3d"
          }
        }
      },
      "execution_count": null,
      "source": "# Importing the dataset\nurl = 'https://raw.githubusercontent.com/Krishna2709/Customer-Segmentation-And-Retention-System/master/synthetic_customer_data.csv'\ndf = pd.read_csv(url)\ndf.head()",
      "outputs": []
    },
    {
      "id": "1544a230-83d2-4e4a-bd34-019edcfac040",
      "cell_type": "code",
      "metadata": {
        "noteable": {
          "cell_type": "code",
          "output_collection_id": "1c806ffc-3954-478a-949a-5b234cec7714"
        },
        "ExecuteTime": {
          "end_time": "2023-06-30T17:44:10.950385+00:00",
          "start_time": "2023-06-30T17:44:10.413867+00:00"
        }
      },
      "execution_count": null,
      "source": "# Target Variable Analysis\nplt.figure(figsize=(6, 4))\nsns.countplot(x='churned', data=df)\nplt.title('Distribution of Churned')\nplt.show()",
      "outputs": []
    },
    {
      "id": "466bf675-4612-4d5e-a8e5-ebf78a057776",
      "cell_type": "code",
      "metadata": {
        "noteable": {
          "cell_type": "code",
          "output_collection_id": "220d4254-d14e-492d-b730-fe4b6271959c"
        },
        "ExecuteTime": {
          "end_time": "2023-06-30T17:44:35.480876+00:00",
          "start_time": "2023-06-30T17:44:35.292952+00:00"
        },
        "datalink": {
          "0a02303a-b422-481b-828b-8ba758a163cb": {
            "applied_filters": [],
            "dataframe_info": {
              "default_index_used": false,
              "orig_num_cols": 1,
              "orig_num_rows": 8,
              "orig_size_bytes": 128,
              "truncated_num_cols": 1,
              "truncated_num_rows": 8,
              "truncated_size_bytes": 128,
              "truncated_string_columns": []
            },
            "display_id": "0a02303a-b422-481b-828b-8ba758a163cb",
            "dx_settings": {
              "ALLOW_NOTEABLE_ATTRS": true,
              "COLUMN_SAMPLING_METHOD": "outer",
              "DB_LOCATION": ":memory:",
              "DEV_MODE": false,
              "DISPLAY_MAX_COLUMNS": 100,
              "DISPLAY_MAX_ROWS": 50000,
              "DISPLAY_MODE": "simple",
              "ENABLE_ASSIGNMENT": true,
              "ENABLE_DATALINK": true,
              "FLATTEN_COLUMN_VALUES": true,
              "FLATTEN_INDEX_VALUES": false,
              "GENERATE_DEX_METADATA": false,
              "HTML_TABLE_SCHEMA": false,
              "LOG_LEVEL": 30,
              "MAX_RENDER_SIZE_BYTES": 104857600,
              "MAX_STRING_LENGTH": 250,
              "NUM_PAST_SAMPLES_TRACKED": 3,
              "RANDOM_STATE": 12648430,
              "RESET_INDEX_VALUES": false,
              "ROW_SAMPLING_METHOD": "random",
              "SAMPLING_FACTOR": 0.1,
              "SAMPLING_METHOD": "random",
              "STRINGIFY_COLUMN_VALUES": true,
              "STRINGIFY_INDEX_VALUES": false
            },
            "sample_history": [],
            "sampling_time": "2023-06-30T17:44:35.324934",
            "user_variable_name": null,
            "variable_name": "unk_dataframe_543a38f643c5490ea7d045f85e720c2c"
          }
        }
      },
      "execution_count": null,
      "source": "# Checking for missing values\ndf.isnull().sum()",
      "outputs": []
    },
    {
      "id": "16fc1d12-bb81-4bcf-93dc-55ce4e7e1f96",
      "cell_type": "code",
      "metadata": {
        "noteable": {
          "cell_type": "code",
          "output_collection_id": "7ce4315d-b9b7-4cd2-8ec0-f0a4291b51d7"
        },
        "ExecuteTime": {
          "end_time": "2023-06-30T17:45:08.344021+00:00",
          "start_time": "2023-06-30T17:45:08.150806+00:00"
        },
        "datalink": {
          "3415ef18-c5c2-48a3-aea4-a27b3aa40609": {
            "applied_filters": [],
            "dataframe_info": {
              "default_index_used": false,
              "orig_num_cols": 1,
              "orig_num_rows": 8,
              "orig_size_bytes": 128,
              "truncated_num_cols": 1,
              "truncated_num_rows": 8,
              "truncated_size_bytes": 128,
              "truncated_string_columns": []
            },
            "display_id": "3415ef18-c5c2-48a3-aea4-a27b3aa40609",
            "dx_settings": {
              "ALLOW_NOTEABLE_ATTRS": true,
              "COLUMN_SAMPLING_METHOD": "outer",
              "DB_LOCATION": ":memory:",
              "DEV_MODE": false,
              "DISPLAY_MAX_COLUMNS": 100,
              "DISPLAY_MAX_ROWS": 50000,
              "DISPLAY_MODE": "simple",
              "ENABLE_ASSIGNMENT": true,
              "ENABLE_DATALINK": true,
              "FLATTEN_COLUMN_VALUES": true,
              "FLATTEN_INDEX_VALUES": false,
              "GENERATE_DEX_METADATA": false,
              "HTML_TABLE_SCHEMA": false,
              "LOG_LEVEL": 30,
              "MAX_RENDER_SIZE_BYTES": 104857600,
              "MAX_STRING_LENGTH": 250,
              "NUM_PAST_SAMPLES_TRACKED": 3,
              "RANDOM_STATE": 12648430,
              "RESET_INDEX_VALUES": false,
              "ROW_SAMPLING_METHOD": "random",
              "SAMPLING_FACTOR": 0.1,
              "SAMPLING_METHOD": "random",
              "STRINGIFY_COLUMN_VALUES": true,
              "STRINGIFY_INDEX_VALUES": false
            },
            "sample_history": [],
            "sampling_time": "2023-06-30T17:45:08.187853",
            "user_variable_name": null,
            "variable_name": "unk_dataframe_05df2b8d1515466f855e3f7ff735fdd6"
          }
        }
      },
      "execution_count": null,
      "source": "# Imputing missing values with the most frequent category\ndf['product_owned'].fillna(df['product_owned'].mode()[0], inplace=True)\ndf['competitor_product_owned'].fillna(df['competitor_product_owned'].mode()[0], inplace=True)\n\n# Checking if there are any more missing values\ndf.isnull().sum()",
      "outputs": []
    },
    {
      "id": "279f9b11-ee8a-4ccf-9114-7de3ddb768b7",
      "cell_type": "code",
      "metadata": {
        "noteable": {
          "cell_type": "code",
          "output_collection_id": "71204361-245c-4f29-8135-76b64fb73988"
        },
        "ExecuteTime": {
          "end_time": "2023-06-30T17:45:44.242556+00:00",
          "start_time": "2023-06-30T17:45:42.845387+00:00"
        }
      },
      "execution_count": null,
      "source": "# Numerical Variable Analysis\nnumerical_features = ['age', 'income', 'last_purchase', 'product_knowledge']\n\nfor feature in numerical_features:\n    df[feature].hist(bins=25)\n    plt.title(feature)\n    plt.show()",
      "outputs": []
    },
    {
      "id": "6b63d245-5735-46b3-8bb4-a5067867042b",
      "cell_type": "code",
      "metadata": {
        "noteable": {
          "cell_type": "code",
          "output_collection_id": "ba44e7b0-4b24-482b-9825-267c6eda64b9"
        },
        "ExecuteTime": {
          "end_time": "2023-06-30T17:53:12.474978+00:00",
          "start_time": "2023-06-30T17:53:11.387697+00:00"
        }
      },
      "execution_count": null,
      "source": "# Boxplot for numerical variables to check for outliers\nfor feature in numerical_features:\n    sns.boxplot(df[feature])\n    plt.title(feature)\n    plt.show()",
      "outputs": []
    },
    {
      "id": "4ce8f39c-dfb6-4b4f-b6e0-6a9e8ccfd6f6",
      "cell_type": "code",
      "metadata": {
        "noteable": {
          "cell_type": "code",
          "output_collection_id": "23de3113-6811-47b3-9612-0c7eaf02d37e"
        },
        "ExecuteTime": {
          "end_time": "2023-06-30T17:53:47.239385+00:00",
          "start_time": "2023-06-30T17:53:47.078868+00:00"
        }
      },
      "execution_count": null,
      "source": "# Categorical Variable Analysis\ncategorical_features = ['product_owned', 'competitor_product_owned']\n\nfor feature in categorical_features:\n    print(f'\\nCardinality of {feature}: {df[feature].nunique()}')\n    print(df[feature].value_counts())",
      "outputs": []
    },
    {
      "id": "cb039c8a-681b-4e29-a30d-54676f7a553e",
      "cell_type": "code",
      "metadata": {
        "noteable": {
          "cell_type": "code",
          "output_collection_id": "bd85286b-1a9c-4068-ba09-568c29ef22ff"
        },
        "ExecuteTime": {
          "end_time": "2023-06-30T17:54:41.735417+00:00",
          "start_time": "2023-06-30T17:54:39.593112+00:00"
        }
      },
      "execution_count": null,
      "source": "# Preprocessing\n\n# One-hot encoding for categorical variables\ndf_encoded = pd.get_dummies(df, columns=categorical_features)\n\n# Standard scaling for numerical variables\nscaler = StandardScaler()\ndf_encoded[numerical_features] = scaler.fit_transform(df_encoded[numerical_features])\n\n# Splitting the dataset into training and testing sets\nX = df_encoded.drop(['churned', 'customer_id', 'index'], axis=1)\ny = df_encoded['churned']\nX_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)\n\n# Model Building\nmodel = LogisticRegression()\nmodel.fit(X_train, y_train)\n\n# Model Evaluation\ny_pred = model.predict(X_test)\naccuracy = accuracy_score(y_test, y_pred)\nprecision = precision_score(y_test, y_pred)\nrecall = recall_score(y_test, y_pred)\nf1 = f1_score(y_test, y_pred)\nroc_auc = roc_auc_score(y_test, y_pred)\n\nprint(f'Accuracy: {accuracy}')\nprint(f'Precision: {precision}')\nprint(f'Recall: {recall}')\nprint(f'F1 Score: {f1}')\nprint(f'ROC AUC Score: {roc_auc}')",
      "outputs": []
    },
    {
      "id": "2f9db7e7-eea2-47ea-a5d4-d3abb3221256",
      "cell_type": "code",
      "metadata": {
        "noteable": {
          "cell_type": "code",
          "output_collection_id": "8d47b0cc-e7d8-4143-a9ac-ec7cdae599b8"
        },
        "ExecuteTime": {
          "end_time": "2023-06-30T17:55:19.833896+00:00",
          "start_time": "2023-06-30T17:55:19.643494+00:00"
        }
      },
      "execution_count": null,
      "source": "# Preprocessing\n\n# One-hot encoding for categorical variables\ndf_encoded = pd.get_dummies(df, columns=categorical_features)\n\n# Standard scaling for numerical variables\nscaler = StandardScaler()\ndf_encoded[numerical_features] = scaler.fit_transform(df_encoded[numerical_features])\n\n# Splitting the dataset into training and testing sets\nX = df_encoded.drop(['churned', 'customer_id'], axis=1)\ny = df_encoded['churned']\nX_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)\n\n# Model Building\nmodel = LogisticRegression()\nmodel.fit(X_train, y_train)\n\n# Model Evaluation\ny_pred = model.predict(X_test)\naccuracy = accuracy_score(y_test, y_pred)\nprecision = precision_score(y_test, y_pred)\nrecall = recall_score(y_test, y_pred)\nf1 = f1_score(y_test, y_pred)\nroc_auc = roc_auc_score(y_test, y_pred)\n\nprint(f'Accuracy: {accuracy}')\nprint(f'Precision: {precision}')\nprint(f'Recall: {recall}')\nprint(f'F1 Score: {f1}')\nprint(f'ROC AUC Score: {roc_auc}')",
      "outputs": []
    },
    {
      "id": "e2ddf666-e26c-4b80-ac63-244cdd855a72",
      "cell_type": "markdown",
      "source": "## A/B Testing and Statistical Analysis\n\nA/B testing is a method of comparing two versions of a webpage or other user experience to determine which one performs better. It is a way to test changes to your webpage against the current design and determine which one produces better results. It's a concept in statistics that can inform you whether changes to a certain variable will improve the performance.\n\nIn our case, we can design an A/B test for a new customer retention strategy. For example, we can introduce a new feature or service for a subset of customers and compare their churn rate with the rest of the customers. The null hypothesis would be that the new feature or service does not affect the churn rate, while the alternative hypothesis would be that the new feature or service reduces the churn rate.\n\nWe can use a statistical test such as the chi-square test for independence to analyze the results of the A/B test. The chi-square test for independence compares two variables in a contingency table to see if they are related. In a more general sense, it tests to see whether distributions of categorical variables differ from each another.\n\nThe reason for choosing the chi-square test is that it is appropriate for comparing categorical variables, and it can tell us whether the observed difference between the groups in the A/B test is statistically significant or not. If the p-value from the chi-square test is less than a chosen significance level (e.g., 0.05), we would reject the null hypothesis and conclude that the new feature or service has a significant effect on the churn rate.",
      "metadata": {
        "noteable": {
          "cell_type": "markdown"
        }
      }
    },
    {
      "id": "d38cee7e-75e5-4649-8ae4-d80efefae70e",
      "cell_type": "markdown",
      "source": "## Trends and Anomalies\n\nFrom the data analysis, we can observe the following trends and anomalies:\n\n1. **Balanced Target Variable:** The target variable 'churned' is balanced. This is good as it means our model will have a fair representation of both classes to learn from.\n\n2. **Missing Values:** There were missing values in the 'product_owned' and 'competitor_product_owned' columns. We handled these by imputing the most frequent category in each column.\n\n3. **Numerical Variables:** The numerical variables 'age', 'income', 'last_purchase', and 'product_knowledge' do not have significant outliers. However, they are not normally distributed. In a more advanced analysis, we could consider applying transformations to these variables to make them more normally distributed.\n\n4. **Categorical Variables:** The categorical variables 'product_owned' and 'competitor_product_owned' do not have high cardinality or rare labels. This simplifies the preprocessing steps as we don't need to handle high cardinality or rare labels.\n\n5. **Baseline Model Performance:** The baseline model has moderate performance. The recall is relatively high, which means the model is able to identify a good proportion of the positive class (churned customers) correctly. However, the precision is low, which means among the customers that the model predicted as churned, less than half are actually churned. The accuracy, F1 score, and ROC AUC score are also not very high, indicating there is room for improvement in the model.",
      "metadata": {
        "noteable": {
          "cell_type": "markdown"
        }
      }
    }
  ]
}