notebooks/membership_inference_black_box_rule_based_attack.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Rf_QO8flWx6_"
   },
   "source": [
    "# Membership Inference Attack Rule-Based Examples"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "WKAYn2M4Wx7F"
   },
   "source": [
    "<table align=\"left\">\n",
    "  <td>\n",
    "    <a target=\"_blank\" href=\"https://colab.research.google.com/github/privML/privacy-evaluator/blob/main/notebooks/membership_inference_black_box_rule_based_attack.ipynb\"><img src=\"https://raw.githubusercontent.com/privML/privacy-evaluator/main/notebooks/images/colab_logo_32px.png\" />Run in Google Colab</a>\n",
    "  </td>\n",
    "  <td>\n",
    "    <a target=\"_blank\" href=\"https://github.com/privML/privacy-evaluator/blob/main/notebooks/membership_inference_black_box_rule_based_attack.ipynb\"><img src=\"https://raw.githubusercontent.com/privML/privacy-evaluator/main/notebooks/images/GitHub-Mark-32px.png\" />View source on GitHub</a>\n",
    "  </td>\n",
    "</table>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Ia0VmlzeWx7H"
   },
   "source": [
    "\n",
    "## Overview\n",
    "\n",
    "In this notebook, we want to show you how to use the `privacy-evaluator` tool to perform the Membership Inference Attacks Black Box Rule-Based Attack on both, a provided PyTorch and a provided Tensorflow model.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Enable Logging"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import logging\n",
    "logging.basicConfig(level = logging.DEBUG)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "DH6rB9KSWx7J"
   },
   "source": [
    "## Setup\n",
    "\n",
    "First, you should set the notebook's runtime to use a GPU (e.g. if Colab is used go to ***Runtime > Change runtime type > Hardware accelerator***). Now we can install the `privacy-evaluator` package and import all needed modules."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "lNU7L3GCWx7M"
   },
   "outputs": [],
   "source": [
    "!pip3 install git+https://github.com/privML/privacy-evaluator"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "9d7ab468"
   },
   "outputs": [],
   "source": [
    "import tensorflow as tf\n",
    "import torch\n",
    "import numpy as np\n",
    "\n",
    "import tensorflow.python.ops.numpy_ops.np_config as np_config\n",
    "np_config.enable_numpy_behavior()\n",
    "\n",
    "import privacy_evaluator.models.torch.dcti.dcti as torch_dcti\n",
    "import privacy_evaluator.models.tf.dcti.dcti as tf_dcti \n",
    "\n",
    "from privacy_evaluator.datasets.tf.cifar10 import TFCIFAR10\n",
    "from privacy_evaluator.datasets.torch.cifar10 import TorchCIFAR10\n",
    "\n",
    "from privacy_evaluator.classifiers.classifier import Classifier\n",
    "\n",
    "from privacy_evaluator.attacks.membership_inference.black_box_rule_based import MembershipInferenceBlackBoxRuleBasedAttack\n",
    "from privacy_evaluator.attacks.membership_inference import MembershipInferenceAttackAnalysis\n",
    "\n",
    "from privacy_evaluator.attacks.membership_inference import MembershipInferenceAttackAnalysis\n",
    "\n",
    "from privacy_evaluator.attacks.membership_inference.data_structures.attack_input_data import AttackInputData\n",
    "from privacy_evaluator.attacks.membership_inference.data_structures.slicing import Slicing"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "mUH4YQ4tWx7Q"
   },
   "source": [
    "## Conduct Membership Inference Rule-Based Attacks\n",
    "\n",
    "Now we can start with conducting the Membership Inference Rule-Based Attacks. Therefore, we prepared two instances of the attack: one attacking a PyTorch model and attacking a TensorFlow model. For both attacks, we implemented a simple neural network trained on the CIFAR-10 dataset. For details about the provided network have a look at the following paper: https://www.scitepress.org/Papers/2018/67520/67520.pdf)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "b2da0876"
   },
   "source": [
    "### PyTorch\n",
    "\n",
    "We start the evaluation of the PyTorch version of the model."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "7aQIN6UMWx7W"
   },
   "source": [
    "#### Prepare target model\n",
    "\n",
    "Now, we need to initialize our pre-trained Lightweight Deep Convolutional Neural Network (short DCTI) as a generic `Classifier`. Therefore we need to specify the loss function used to train the model (in our case the `torch.nn.CrossEntropyLoss`), the number of classes and the input shape of our CIFAR-10 dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "fz4UuSxMWx7Y"
   },
   "outputs": [],
   "source": [
    "# Initalize PyTorch model as a Classifier\n",
    "target_model = Classifier(\n",
    "    torch_dcti.load_dcti(), # PyTorch DCTI \n",
    "    loss=torch.nn.CrossEntropyLoss(reduction=\"none\"), # Loss function of the PyTorch model\n",
    "    nb_classes=TorchCIFAR10.N_CLASSES, # Number of classes of the CIFAR10 dataset\n",
    "    input_shape=TorchCIFAR10.INPUT_SHAPE # Input shape of the CIFAR10 dataset\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "sv_fGg8jWx7S"
   },
   "source": [
    "#### Load CIFAR10 Dataset\n",
    "\n",
    "Before we can start to conduct the membership inference attacks, we need to load the dataset. The CIFAR10 dataset needs to be preprocessed in a specific manner to work for the PyTorch model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "dcae11d8"
   },
   "outputs": [],
   "source": [
    "# Load CIFAR10 dataset as numpy array\n",
    "x_train, y_train, x_test, y_test = TorchCIFAR10.numpy()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "76a432f6"
   },
   "source": [
    "#### Perform Membership Inference Black Box Rule Based Attack\n",
    "\n",
    "Next, we want to perform the `MembershipInferenceBlackBoxRuleBasedAttack`. First, we initialize the attack with our respective `target_model`. In this case, we do not need to fit the attack because this approach is fully rule-based and depends only on the attacked data points, and the target model. That means, every time the target model classifies a data point correctly, the attack model identifies the datapoint as a member and vice versa. Following these steps, we can now run the attack via the `attack` method. The `attack` method takes a datasets as parameter. This dataset contains the data (`x_train`) and its corresponding labels (`y_train`) which we want to attack. As a result, we get an array holding the inferred membership status for each data point, whereas 1 indicates a member and 0 indicates non-member."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "35c9fc67"
   },
   "outputs": [],
   "source": [
    "attack = MembershipInferenceBlackBoxRuleBasedAttack(target_model)\n",
    "attack.attack(x_train[:100], y_train[:100])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "f4ecb6b3"
   },
   "source": [
    "#### Get pythonic attack statistics\n",
    "\n",
    "Again, we want to get a better overview of the attack results. This time we want to receive the result in a more pythonic manner and we are only interested in the attack model's accuracy. Therefore, we filter the output and convert it into a dictionary."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "274272d3"
   },
   "outputs": [],
   "source": [
    "output = attack.attack_output(\n",
    "    x_train[:100], \n",
    "    y_train[:100],\n",
    "    x_train, \n",
    "    y_train, \n",
    "    x_test, \n",
    "    y_test,\n",
    "    np.ones((len(y_train[:100]),))\n",
    ")\n",
    "\n",
    "output.to_dict()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "IxquXY0x8CF_"
   },
   "source": [
    "#### Explanation of the outcome:\n",
    "\n",
    "##### Attack Model Accuracy:\n",
    "The attack model accuracy specifies how well the membership attack model performs in predicting if a given data point was used for training the target model. Since we have a two-class classification problem that the attack model solves (member or non-member), the baseline accuracy is 50% (random guessing for each sample). The best accuracy is at 100% if the model predicts every data point it sees right as member or non-member."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "9Ka1UONoDFlY"
   },
   "source": [
    "#### Perpare attack analysis\n",
    "\n",
    "Next, we prepare our attack analysis. To initialize our attack analysis we define the Membership Inference Attack method we want to perform (in this case we use the `MembershipInferenceBlackBoxRuleBasedAttack`) and the Attack Input Data. The Attack Input Data consists of two different sets. The first contains the data (`x_train`) and its corresponding labels (`y_train`) which were used to train the target model. The second contains the data (`x_test`) and its corresponding labels (`y_test`) which were not part of the training process of the target model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "aE3cakPzDmBd"
   },
   "outputs": [],
   "source": [
    "attack_analysis = MembershipInferenceAttackAnalysis(\n",
    "    MembershipInferenceBlackBoxRuleBasedAttack, \n",
    "    AttackInputData(\n",
    "        x_train[:100], \n",
    "        y_train[:100], \n",
    "        x_test[:100], \n",
    "        y_test[:100]\n",
    "    )\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "AK8vw9v-Dm7G"
   },
   "source": [
    "#### Define the slicing\n",
    "\n",
    "Now we can define the slicing for our analysis. The slicing defines how the data will be sliced. Each slice will then be analysed by the attack separately, so see if the inference works better on certain parts of data than on others.\n",
    "\n",
    "The types of slices can be chosen:\n",
    "##### entire_dataset:\n",
    "This options allows to pick the whole dataset as a single slice.\n",
    "##### by_class:\n",
    "If this option is chosen, the analysis will be done for each class individually, resulting in as many slices as there are classes.\n",
    "##### by_classification_correctness:\n",
    "If this option is chosen, two slices will be analyzed. The first slice is the set of all correctly classified samples, and the second slice is the set of all incorrectly classified samples.\n",
    "\n",
    "If all three choices are selected, a total of 3 + n slices will be analyzed, where n is the amount of classes of the dataset.\n",
    "\n",
    "For further information, please see the explanation of the outcome below the analysis."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "59Z9WVRaDoIb"
   },
   "outputs": [],
   "source": [
    "slicing = Slicing(\n",
    "    entire_dataset=True, \n",
    "    by_class=True, \n",
    "    by_classification_correctness=True\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "XdavMRYlDp35"
   },
   "source": [
    "#### Perform Membership Inference Attack Analysis\n",
    "\n",
    "Finally, we can perform our Membership Inference Attack Analysis. Therefore, we input the target model, the data that should be analysed, the membership labels (i.e. the labels which correctly describe if a data point is a member of the training dataset or not) and the splicing specification into the `analyse()` method. As a result, we get for each slice the indices of the corresponding data points, a human-readable description of the slice and the advantage score of the Membership Inference Attack (for more details about the advantage score, please read the following paper: https://arxiv.org/abs/1709.01604)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# For the advantage score, we need to use data points from both train and test sets.\n",
    "# Otherwise the advantage score cannot be calculated correctly if all data samples are\n",
    "# from the same class (member vs non-member).\n",
    "x = np.concatenate((x_train[:100], x_test[:100]))\n",
    "y = np.concatenate((y_train[:100], y_test[:100]))\n",
    "membership = np.concatenate((np.ones(len(x_train[:100])), np.zeros(len(x_test[:100]))))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "pdsW4uNcDrkU",
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "result = attack_analysis.analyse(\n",
    "    target_model, \n",
    "    x, \n",
    "    y, \n",
    "    membership, \n",
    "    slicing\n",
    ")\n",
    "\n",
    "print(\"\\n\".join((str(r) for r in result)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "A6NNaOOfDvCk"
   },
   "source": [
    "#### Explanation of the outcome:\n",
    "##### Attacker Advantage:\n",
    "The attacker advantageis a score that relies on comparing the model output on member and non-member data points. The model outputs are probability values over all classes, and they are often different on member and non-member data points. Usually, the model is more confident on member data points, because it has seen them during training. When trying to find a threshold value to tell apart member and non-member samples by their different model outputs, the attacker has interest in finding the best ratio between false positives “fpr” (non-members that are classified as members) and true positives “tpr” (members that are correctly identifies as members). \n",
    "\n",
    "This best ratio is calculated as the max(tpr-fpr) over all threshold values and represents the attacker advantage. \n",
    "\n",
    "##### Slicing: Incorrectly classified:\n",
    "It is normal that the attacker is more successful to deduce membership on incorrectly classified samples than on correctly classified ones. This results from the fact, that model predictions are often better on training than on test data points, whereby your attack model might learn to predict incorrectly classified samples as non-members. If your model overfits the training data, this assumption might hold true often enough to make the attack seem more successful on this slice. If you wish to reduce that, pay attention to reducing your model’s overfitting.\n",
    "\n",
    "##### Slicing: Specific classes more vulnerable: \n",
    "It seems that the membership inference attack is more successful on your class X than on the other classes. Research has shown that the class distribution (and also the distribution of data points within one class) are factors that influence the vulnerability of a class for membership inference attacks [1].\n",
    "\n",
    "Also, small classes (belonging to minority groups) can be more prone to membership inference attacks [2]. One reason for this could be, that there is less data for that class, and therefore, the model overfits within this class. It might make sense to look into the vulnerable classes of your model again, and maybe add more data to them, use private synthetic data, or introduce privacy methods like Differential Privacy [2]. Attention, the use of Differential Privacy could have a negative influence on the performance of your model for the minority classes.\n",
    "\n",
    "\n",
    "(For References, please see last box)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "xG4nYKGzWx7i"
   },
   "source": [
    "### TensorFlow\n",
    "\n",
    "Now we do the same with the TensorFlow model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Note: to force run on CPU uncomment the line below\n",
    "# os.environ[\"CUDA_VISIBLE_DEVICES\"] = \"\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "_7ty54AKWx7k"
   },
   "source": [
    "#### Prepare target model\n",
    "\n",
    "Now, we need to initialize our pre-trained Lightweight Deep Convolutional Neural Network (short DCTI) as a generic `Classifier`. Therefore we need to specify the loss function used to train the model (in our case the `tf.keras.losses.CategoricalCrossentropy`), the number of classes and the input shape of our CIFAR-10 dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "Uhrn5znOWx7k"
   },
   "outputs": [],
   "source": [
    "# Initalize TensorFlow target model\n",
    "target_model = Classifier(\n",
    "    tf_dcti.load_dcti(), # TensorFlow DCTI\n",
    "    loss=tf.keras.losses.CategoricalCrossentropy(), # Loss function of the TensorFlow target model\n",
    "    nb_classes=TFCIFAR10.N_CLASSES, # Number of classes of the CIFAR10 dataset\n",
    "    input_shape=TFCIFAR10.INPUT_SHAPE # Input shape of the CIFAR10 dataset\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "UBfeudg_Wx7i"
   },
   "source": [
    "#### Load CIFAR10 Dataset\n",
    "\n",
    "Again, before we can start to conduct the membership inference attacks, we need to load the dataset. The CIFAR10 dataset needs to be preprocessed in a specific manner to work for the TensorFlow model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "kAJgfSgoWx7j"
   },
   "outputs": [],
   "source": [
    "# Load CIFAR10 dataset as numpy array\n",
    "x_train, y_train, x_test, y_test = TFCIFAR10.numpy()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "VMOD4MsfWx7o"
   },
   "source": [
    "#### Perform Membership Inference Black Box Rule Based Attack\n",
    "\n",
    "Next, we want to perform the `MembershipInferenceBlackBoxRuleBasedAttack`. First, we initialize the attack with our respective `target_model`. In this case, we do not need to fit the attack because this approach is fully rule-based and depends only on the attacked data points, and the target model. That means, every time the target model classifies a data point correctly, the attack model identifies the datapoint as a member and vice versa. Following these steps, we can now run the attack via the `attack` method. The `attack` method takes a datasets as parameter. This dataset contains the data (`x_train`) and its corresponding labels (`y_train`) which we want to attack. As a result, we get an array holding the inferred membership status for each data point, whereas 1 indicates a member and 0 indicates non-member."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "M6q1WZSAWx7p"
   },
   "outputs": [],
   "source": [
    "attack = MembershipInferenceBlackBoxRuleBasedAttack(target_model)\n",
    "attack.attack(x_train, y_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "8_K4-ErBWx7q"
   },
   "source": [
    "#### Get pythonic attack statistics\n",
    "\n",
    "Again, we want to get a better overview of the attack results. This time we want to receive the result in a more pythonic manner and we are only interested in the attack model's accuracy. Therefore, we filter the output and convert it into a dictionary."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "qY4g4xHZWx7q"
   },
   "outputs": [],
   "source": [
    "output = attack.attack_output(\n",
    "    x_train[:100], \n",
    "    y_train[:100], \n",
    "    x_train, \n",
    "    y_train, \n",
    "    x_test, \n",
    "    y_test,\n",
    "    np.ones((len(y_train[:100]),))\n",
    ")\n",
    "\n",
    "output.to_dict()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "pa-PyGbB8A35"
   },
   "source": [
    "#### Explanation of the outcome:\n",
    "\n",
    "##### Attack Model Accuracy:\n",
    "The attack model accuracy specifies how well the membership attack model performs in predicting if a given data point was used for training the target model. Since we have a two-class classification problem that the attack model solves (member or non-member), the baseline accuracy is 50% (random guessing for each sample). The best accuracy is at 100% if the model predicts every data point it sees right as member or non-member."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "GTUIFR5gDJ4Z"
   },
   "source": [
    "#### Perpare attack analysis\n",
    "\n",
    "Next, we prepare our attack analysis. To initialize our attack analysis we define the Membership Inference Attack method we want to perform (in this case we use the `MembershipInferenceBlackBoxRuleBasedAttack`) and the Attack Input Data. The Attack Input Data consists of two different sets. The first contains the data (`x_train`) and its corresponding labels (`y_train`) which were used to train the target model. The second contains the data (`x_test`) and its corresponding labels (`y_test`) which were not part of the training process of the target model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "Cldis2weDMI-"
   },
   "outputs": [],
   "source": [
    "analysis = MembershipInferenceAttackAnalysis(\n",
    "    MembershipInferenceBlackBoxRuleBasedAttack, \n",
    "    AttackInputData(\n",
    "        x_train[:100], \n",
    "        y_train[:100], \n",
    "        x_test[:100], \n",
    "        y_test[:100]\n",
    "    )\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "7QL_9rSDDNkG"
   },
   "source": [
    "#### Define the slicing\n",
    "\n",
    "Now we can define the slicing for our analysis. The slicing defines how the data will be sliced. Each slice will then be analysed by the attack separately, so see if the inference works better on certain parts of data than on others.\n",
    "\n",
    "The types of slices can be chosen:\n",
    "##### entire_dataset:\n",
    "This options allows to pick the whole dataset as a single slice.\n",
    "##### by_class:\n",
    "If this option is chosen, the analysis will be done for each class individually, resulting in as many slices as there are classes.\n",
    "##### by_classification_correctness:\n",
    "If this option is chosen, two slices will be analyzed. The first slice is the set of all correctly classified samples, and the second slice is the set of all incorrectly classified samples.\n",
    "\n",
    "If all three choices are selected, a total of 3 + n slices will be analyzed, where n is the amount of classes of the dataset.\n",
    "\n",
    "For further information, please see the explanation of the outcome below the analysis."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "Dp_TlL9XDQK1"
   },
   "outputs": [],
   "source": [
    "slicing = Slicing(\n",
    "    entire_dataset=True, \n",
    "    by_class=True, \n",
    "    by_classification_correctness=True\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "u7_B_FWkDR5D"
   },
   "source": [
    "#### Perform Membership Inference Attack Analysis\n",
    "\n",
    "Finally, we can perform our Membership Inference Attack Analysis. Therefore, we input the target model, the data that should be analysed, the membership labels (i.e. the labels which correctly describe if a data point is a member of the training dataset or not) and the splicing specification into the `analyse()` method. As a result, we get for each slice the indices of the corresponding data points, a human-readable description of the slice and the advantage score of the Membership Inference Attack (for more details about the advantage score, please read the following paper: https://arxiv.org/abs/1709.01604)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "Cc_DLnPODVOP"
   },
   "outputs": [],
   "source": [
    "result = analysis.analyse(\n",
    "    target_model, \n",
    "    np.concatenate((x_train[:100], x_test[:100])), \n",
    "    np.concatenate((y_train[:100], y_test[:100])), \n",
    "    np.concatenate((np.ones(len(x_train[:100])), np.zeros(len(x_test[:100])))), \n",
    "    slicing\n",
    ")\n",
    "\n",
    "print(\"\\n\".join((str(r) for r in result)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "nLlQkyZYDaFo"
   },
   "source": [
    "#### Explanation of the outcome:\n",
    "##### Attacker Advantage:\n",
    "The attacker advantageis a score that relies on comparing the model output on member and non-member data points. The model outputs are probability values over all classes, and they are often different on member and non-member data points. Usually, the model is more confident on member data points, because it has seen them during training. When trying to find a threshold value to tell apart member and non-member samples by their different model outputs, the attacker has interest in finding the best ratio between false positives “fpr” (non-members that are classified as members) and true positives “tpr” (members that are correctly identifies as members). \n",
    "\n",
    "This best ratio is calculated as the max(tpr-fpr) over all threshold values and represents the attacker advantage. \n",
    "\n",
    "##### Slicing: Incorrectly classified:\n",
    "It is normal that the attacker is more successful to deduce membership on incorrectly classified samples than on correctly classified ones. This results from the fact, that model predictions are often better on training than on test data points, whereby your attack model might learn to predict incorrectly classified samples as non-members. If your model overfits the training data, this assumption might hold true often enough to make the attack seem more successful on this slice. If you wish to reduce that, pay attention to reducing your model’s overfitting.\n",
    "\n",
    "##### Slicing: Specific classes more vulnerable: \n",
    "It seems that the membership inference attack is more successful on your class X than on the other classes. Research has shown that the class distribution (and also the distribution of data points within one class) are factors that influence the vulnerability of a class for membership inference attacks [1].\n",
    "\n",
    "Also, small classes (belonging to minority groups) can be more prone to membership inference attacks [2]. One reason for this could be, that there is less data for that class, and therefore, the model overfits within this class. It might make sense to look into the vulnerable classes of your model again, and maybe add more data to them, use private synthetic data, or introduce privacy methods like Differential Privacy [2]. Attention, the use of Differential Privacy could have a negative influence on the performance of your model for the minority classes.\n",
    "\n",
    "\n",
    "(For References, please see last box)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "LjKjKcOtFple"
   },
   "source": [
    "[1] Stacey Truex, Ling Liu, Mehmet Emre Gursoy, Lei Yu, and Wenqi Wei. 2019.Demystifying Membership Inference Attacks in Machine Learning as a Service.IEEE Transactions on Services Computing(2019)\n",
    "\n",
    "[2] Suriyakumar, Vinith M., Nicolas Papernot, Anna Goldenberg, and Marzyeh Ghassemi. \"Chasing Your Long Tails: Differentially Private Prediction in Health Care Settings.\" In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, pp. 723-734. 2021."
   ]
  }
 ],
 "metadata": {
  "accelerator": "GPU",
  "colab": {
   "collapsed_sections": [],
   "name": "membership_inference_black_box_rule_based_attack.ipynb",
   "provenance": []
  },
  "interpreter": {
   "hash": "bdaae9e836e6796a4363c97979c836477756ef9f67b8698af83fef8b5f4ebf2e"
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}