23411016_Ishita Agrawal

23411016_Ishita Agrawal/assignment 2/assign2.ipynb

@@ -0,0 +1,292 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "SOLUTION 1 <br><br>\n",
+    "PART 1 <br>\n",
+    "\n",
+    "x ~ $N(\\theta,4)$<br>\n",
+    "$\\theta$ ~ $N(5,9)$<br>\n",
+    "\n",
+    "bayesian update scheme :- <br>\n",
+    "$p(\\theta | x) \\propto L(x| \\theta) * p(\\theta)$\n",
+    "\n",
+    " L(x|$\\theta$) = $ \\frac{1}{\\sqrt{2\\pi 4 }} \\ e^{-\\frac{(x-\\theta)^2}{2 * 4}} = \\frac{1}{\\sqrt{8\\pi}}e^{- \\frac{(x - \\theta)^2}{8}} $   ....(Likelihood function)\n",
+    "\n",
+    "$P(\\theta) = \\frac{1}{\\sqrt{2\\pi 9 }} e^{- \\frac{(\\theta - 5)^2}{2*4}} = \\frac{1}{3\\sqrt{2\\pi}}e^{- \\frac{(\\theta - 5)^2}{18}}$\n",
+    "\n",
+    "$P(\\theta | x ) \\propto e^{- \\frac{(x - \\theta)^2}{8} - \\frac{(\\theta - 5 )^2}{18}}$  (ignoring the constant term)<br>\n",
+    "\n",
+    "$P(\\theta | x ) \\propto e^{- \\frac{(6 - \\theta)^2}{8} - \\frac{(\\theta - 5 )^2}{18}}$\n",
+    "\n",
+    "$P(\\theta | x ) \\propto e^{-\\frac{1}{8}(36 - 12\\theta + \\theta^2) + \\frac{1}{18}(\\theta^2 - 10\\theta + 25)}$\n",
+    "\n",
+    "$P(\\theta | x ) \\propto e^{-\\frac{9}{72}(36 - 12\\theta + \\theta^2) + \\frac{4}{72}(\\theta^2 - 10\\theta + 25)}$\n",
+    "\n",
+    "$P(\\theta | x ) \\propto e^{-\\frac{324}{72} + \\frac{148}{72}\\theta - \\frac{13}{72} \\theta^2 - \\frac{100}{72}}$\n",
+    "\n",
+    "$P(\\theta | x ) \\propto e^{-\\frac{13}{72}\\theta^2 + \\frac{148}{72}\\theta - \\frac{424}{72}}$\n",
+    "\n",
+    "$P(\\theta | x ) \\propto e^{-\\frac{13}{72}(\\theta^2) - \\frac{148}{13}\\theta + (\\frac{148}{13*2})^2 - (\\frac{148}{13*2})^2 + \\frac{424}{13}}$\n",
+    "\n",
+    "$P(\\theta | x ) \\propto e^{-\\frac{13}{72}[(\\theta - \\frac{74}{13})^2]}$\n",
+    "\n",
+    "$P(\\theta | x ) \\propto e^{-\\frac{1}{2*\\frac{36}{13}}(\\theta - \\frac{74}{13})^2}$\n",
+    "\n",
+    "from this we get \n",
+    "$\\theta$ ~ $N(\\frac{74}{13} , \\frac{36}{13})$\n",
+    "\n",
+    "\n",
+    "\n",
+    "\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "SOLUTION 2 \n",
+    "\n",
+    "Probability density function for gaussian distribution is  p($\\mu , \\sigma | x) = \\frac{1}{\\sigma\\sqrt{2\\pi}} e^-\\frac{(x-\\mu)^2}{2 \\sigma^2} $                   ...... (1.1)\n",
+    "<br>\n",
+    "Likelihood function for datapoints $x_1 , x_2 , ... , x_n $ is : <br>\n",
+    "\n",
+    "$ L(\\mu , \\sigma | x_1 , x_2 , ... , x_n) = \\prod_{i=1} ^{n} p (\\mu,\\sigma | x_i)$                 ..... (1.2)\n",
+    "\n",
+    "Finding maximum likelihood function(MLE):<br><br>\n",
+    "Taking log on both sides in equation (1.2) to ease the process <br>\n",
+    "$ Log[L(\\mu , \\sigma | x_1 , x_2 , ... , x_n)] = \\sum_{i=1}^{n} log[p(\\mu, \\sigma|x_i)] $ <br>\n",
+    "\n",
+    "substituting value of p($\\mu , \\sigma |x$) from eqn (1.1)\n",
+    "\n",
+    "Log[L($\\mu , \\sigma | x_1 ,...,x_n )] = - \\frac{n}{2}Log(2\\pi) - nLog(\\sigma) - \\frac{(x_1 - \\mu)^2}{2\\sigma^2} - ... - \\frac{(x_n - \\mu)^2}{2\\sigma^2} $ <br><br>\n",
+    "\n",
+    "differenting Log[L($\\mu , \\sigma | x_1 ,...,x_n $)] with respect to $\\mu $, keeping $\\sigma$ constant : \n",
+    "\n",
+    "$ \\frac{\\partial Log[L(\\mu , \\sigma | x_1 ,...,x_n )]}{\\partial \\mu} $ = $\\frac{1}{\\sigma^2}[(x_1,...,x_n) - n\\mu]$\n",
+    "\n",
+    "$ \\frac{\\partial Log[L(\\mu , \\sigma | x_1 ,...,x_n )]}{\\partial \\sigma} $ = $ -\\frac{n}{\\sigma} + \\frac{(x_1 - \\mu^2)}{\\sigma^3} + ... + \\frac{(x_n - \\mu)^2}{\\sigma^3}$\n",
+    "\n",
+    "Inorder to maximize liklihood function: <br> \n",
+    "$ \\frac{\\partial Log[L(\\mu , \\sigma | x_1 ,...,x_n )]}{\\partial \\mu} $ = 0 and  $ \\frac{\\partial Log[L(\\mu , \\sigma | x_1 ,...,x_n )]}{\\partial \\sigma} $ = 0\n",
+    "<br><br>\n",
+    "thus we get :<br><br>\n",
+    "$$ \\mu = \\frac{x_1 , x_2,...,x_n}{n}$$ \n",
+    "$$ \\sigma = \\sqrt{\\frac{(x_1 - \\mu)^2 + ... + (x_n - \\mu)^2}{n}}$$"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 1,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "true mean of the data was -  0 and estimated mean is 0.02654906487108962\n",
+      "true variance of the data was -  1 and estimated variance is 1.014919037516215\n"
+     ]
+    }
+   ],
+   "source": [
+    "import numpy as np\n",
+    "obsn = 1000\n",
+    "true_mean = 0\n",
+    "true_variance = 1\n",
+    "data = np.random.normal(true_mean , true_variance, obsn)\n",
+    "\n",
+    "mean_data = np.mean(data)\n",
+    "var_data = np.var(data)\n",
+    "\n",
+    "print(\"true mean of the data was - \", true_mean , \"and estimated mean is\" , mean_data)\n",
+    "\n",
+    "print(\"true variance of the data was - \", true_variance , \"and estimated variance is\" , var_data)\n",
+    "\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Through the above code , we get to know the effectiveness of MLE method to get the unknown variables by seeing how close the values of estimated parameters are to the true parameters (mean and variance) "
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "SOLUTION 3 <br><br>\n",
+    "\n",
+    "Let <br>\n",
+    "y = vector of binary labels(0 or 1)<br>\n",
+    "x = design matrix with features for each data point<br> \n",
+    "w = weight vector of logistic regression model<br>\n",
+    "$\\sigma^2$ = variance of gaussian prior <br>\n",
+    "<br>\n",
+    "$sigmoid(w^T x_i) = \\frac{1}{1+e^{-w^T x_i}}$<br>\n",
+    "probability density function :-<br>\n",
+    "$P(y_i | x_i , w) = sigmoid(w^T x_i)$  ...if $y_i = 0$  <br>\n",
+    "$P(y_i | x_i , w) = 1 - sigmoid(w^T x_i)$  ...if $y_i = 1$\n",
+    "<br><br>\n",
+    "(A)Likelihood function :-<br>\n",
+    "Likelihood function is the product of individual probabilities across all datapoints . <br>\n",
+    "Thus $L(y_i | x_i , w) = \\prod_{i = 1}^{n}  (\\frac{1}{1+e^{-w^T x_i}})^{y_i} \\  (1-\\frac{1}{1+e^{-w^T x_i}})^{1-y_i}$<br><br>\n",
+    "\n",
+    "(B)Prior distribution :- <br>\n",
+    "Given that there is gaussian prior on model parameters , thus <br>\n",
+    "p(w) = $\\frac{1}{\\sigma\\sqrt{2\\pi}} e^-\\frac{(w-\\mu)^2}{2 \\sigma^2} $  \n",
+    "<br><br>\n",
+    "\n",
+    "(C)Posterior distribution :-<br>\n",
+    "Probability of w given data <br>\n",
+    "Posterior distribution is proportional to product of likelihood and prior<br>\n",
+    "P(w|y,x) = L(y|x,w) x P(w)     ...(1.1)<br><br>\n",
+    "\n",
+    "Log posterior(taking log on both sides of equation (1.1)) : <br>\n",
+    "\n",
+    "Log(P(w|y,x)) = $\\sum_{i =1}^{n} [y_i \\ log \\ (sigmoid(w^T x_i)) + (1-y_i) \\ log \\ (1-sigmoid(w^T x_i))]$ - $\\frac{w^T w}{2\\sigma^2}$\n",
+    "<br><br>\n",
+    "\n",
+    "To find MAP estimate:<br>\n",
+    "we maximize the log posterior with respect to w <br>\n",
+    "\n",
+    "\n",
+    "\n",
+    "\n",
+    "\n",
+    "\n",
+    "\n",
+    "\n",
+    "\n",
+    "\n",
+    "\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "SOLUTION 5 <br><br>\n",
+    "PART 1 <br>\n",
+    "KL divergence = $\\int p(x) \\ log \\frac{p(x)}{q(x)} \\ dx$ \n",
+    "\n",
+    "p(x)= $N(\\mu_1, \\sigma_1 ^2)$ = $\\frac{1}{\\sigma_1\\sqrt{2\\pi}} e^{-\\frac{(x-\\mu_1)^2}{2 \\sigma_1^2}} $ <br>\n",
+    "q(x)= $N(\\mu_2, \\sigma_2 ^2)$ = $\\frac{1}{\\sigma_2\\sqrt{2\\pi}} e^{-\\frac{(x-\\mu_2)^2}{2 \\sigma_2^2}} $ <br>\n",
+    "\n",
+    "KL divergence = $\\int p(x) \\ log [p(x)] \\ dx$  + $\\int p(x) \\ log [q(x)] \\ dx$ .....(1.1)\n",
+    "<br>\n",
+    "\n",
+    "$\\int p(x) \\ log [p(x)] \\ dx$ = $- \\frac{(1 + log \\ 2\\pi\\sigma_1 ^2)}{2}$ ....(1.2)\n",
+    "<br><br>\n",
+    "$- \\int p(x) \\ log [q(x)] \\ dx$ = $ - \\int p(x)$ log $[\\frac{1}{\\sigma_2\\sqrt{2\\pi}} e^{-\\frac{(x-\\mu_2)^2}{2 \\sigma_2^2}}]$ dx  \n",
+    "<br>\n",
+    "$\\ \\ \\ \\ $= $\\frac{1}{2}log(2\\pi\\sigma_2 ^2) - \\int p(x) \\ log \\ e^{-\\frac{(x-\\mu)^2}{2\\sigma_2 ^2}} dx$<br>\n",
+    "\n",
+    "$\\ \\ \\ \\ $= $\\frac{1}{2}log(2\\pi\\sigma_2 ^2) +  \\int p(x) \\ \\frac{(x-\\mu)^2}{2\\sigma^2} dx$<br>\n",
+    "$\\ \\ \\ \\ $= $\\frac{1}{2}log(2\\pi\\sigma_2 ^2) + \\frac{1}{2\\sigma_2 ^2}[\\int p(x)x^2 dx + \\int p(x) \\mu_2 ^2 dx - 2\\int p(x)x\\mu_2 dx]$<br>\n",
+    "\n",
+    "$\\ \\ \\ \\ $= $\\frac{1}{2}log(2\\pi\\sigma_2 ^2) + \\frac{1}{2\\sigma_2 ^2}[<x^2> + \\mu_2 ^2 - 2\\mu_2 <x> dx]$\n",
+    "<br>\n",
+    "since ,<br> $\\int p(x) dx = 1$<br>\n",
+    "$\\int p(x) x dx = <x>$ <br>\n",
+    "$\\int p(x) x^2 dx = <x^2> $ <br> \n",
+    "where <.> denotes the expectation operator <br>\n",
+    "\n",
+    "also<br> \n",
+    "$<x>=\\mu_1$  and<br>\n",
+    "$<x^2> = \\sigma_1 ^2 + \\mu_1 ^2 $\n",
+    "<br><br>\n",
+    "\n",
+    "$\\ \\ \\ \\ $= $\\frac{1}{2}log(2\\pi\\sigma_2 ^2) + \\frac{1}{2\\sigma_2 ^2}[\\mu_1 ^2 + \\sigma_1 ^2 + \\mu_2 ^2- 2\\mu_2\\mu_1]$<br>\n",
+    "$- \\int p(x) \\ log [q(x)] \\ dx$ = $\\frac{1}{2}log(2\\pi\\sigma_2 ^2) + \\frac{1}{2\\sigma_2 ^2}[(\\mu_1 - \\mu_2)^2 + \\sigma_1 ^2] $ .....(1.3)<br>\n",
+    "\n",
+    "using eqn (1.2) and eqn (1.3) in eqn (1.1)<br>\n",
+    "\n",
+    "KL divergence = $- \\frac{(1 + log \\ 2\\pi\\sigma_1 ^2)}{2}$ + $\\frac{1}{2}log(2\\pi\\sigma_2 ^2) + \\frac{1}{2\\sigma_2 ^2}[(\\mu_1 - \\mu_2)^2 + \\sigma_1 ^2] $<br>\n",
+    "\n",
+    "\n",
+    "$\\ \\ \\ \\ = log \\frac{\\sigma_2}{sigma_1} + \\frac{\\sigma_1 ^2 + (\\mu_1 - \\mu_2)^2}{2\\sigma_2 ^2} - \\frac{1}{2}$ <br>\n",
+    "\n",
+    "\n",
+    "\n",
+    "\n",
+    "\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "PART 2 <br><br>\n",
+    "\n",
+    "KL divergence = $\\int p(x) \\ log \\frac{p(x)}{q(x)} \\ dx$ "
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 1,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "KL divergence for the data provided =  0.3181551004141449\n"
+     ]
+    }
+   ],
+   "source": [
+    "import pandas as pd\n",
+    "import numpy as np\n",
+    "import math\n",
+    "\n",
+    "df = pd.read_csv(\"data_KL.csv\")\n",
+    "length = len(df)\n",
+    "\n",
+    "p_x = np.array(df.P)\n",
+    "q_x = np.array(df.Q)\n",
+    "\n",
+    "KL_div = 0 \n",
+    "for i in range(length): #i goes from 0 to 99\n",
+    "    a = p_x[i]/q_x[i]\n",
+    "\n",
+    "    KL_div += p_x[i] * math.log(a)\n",
+    "\n",
+    "print(\"KL divergence for the data provided = \" , KL_div)\n",
+    "   \n",
+    "\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": []
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.12.1"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}

23411016_Ishita Agrawal/assignment 2/data_KL.csv

@@ -0,0 +1,101 @@
+,P,Q
+0,7.88860905221016e-31,1.486720385427983e-07
+1,7.88860905221004e-29,2.438962174264809e-07
+2,3.904861480844006e-27,3.9613014109573e-07
+3,1.275588083742379e-25,6.369828909339788e-07
+4,3.093301103075289e-24,1.0140858004452227e-06
+5,5.939138117904556e-23,1.5983750467744769e-06
+6,9.403635353348723e-22,2.4942485897551474e-06
+7,1.2627738903068471e-20,3.853521931013182e-06
+8,1.4679746474817013e-19,5.894310227640482e-06
+9,1.5005963063146228e-18,8.92617094530064e-06
+10,1.365542638746319e-17,1.3383030414223315e-05
+11,1.117262158974263e-16,1.986555877348687e-05
+12,8.286361012392394e-16,2.919470967694707e-05
+13,5.6092289930040604e-15,4.2478051932228865e-05
+14,3.4857351599382385e-14,6.119022884726562e-05
+15,1.998488158364584e-13,8.726832061302681e-05
+16,1.061696834131184e-12,0.0001232219890119193
+17,5.2460314157070536e-12,0.0001722569947872011
+18,2.4190033750204664e-11,0.0002384089597700325
+19,1.0439909302719975e-10,0.00032668209694045915
+20,4.2281632676015655e-10,0.00044318510074393025
+21,1.6107288638482235e-09,0.0005952535905862057
+22,5.783980920182251e-09,0.0007915456218645029
+23,1.9615239642357194e-08,0.001042094091741784
+24,6.293222718589578e-08,0.0013582977188518668
+25,1.91313970645124e-07,0.0017528310758975296
+26,5.518672230147828e-07,0.0022394543410145357
+27,1.5125249815960546e-06,0.0028327054331262276
+28,3.943368702018315e-06,0.0035474613621840857
+29,9.790432639493707e-06,0.004398362173931372
+30,2.317069058013514e-05,0.0053990998132866916
+31,5.232091421320849e-05,0.006561585320241969
+32,0.00011281697127223027,0.007895020453786327
+33,0.000232471334742778,0.009404913245650147
+34,0.00045810527728723993,0.011092089963997388
+35,0.0008638556657416476,0.012951767151755816
+36,0.00155973939647798,0.01497275533232727
+37,0.002697927604718676,0.017136869240940846
+38,0.004472879976244128,0.019418616870780607
+39,0.007110732269926539,0.02178523046171507
+40,0.01084386671163798,0.024197086622871247
+41,0.01586907323654335,0.026608540573093736
+42,0.02229226954657284,0.02896917224186394
+43,0.030068642644214553,0.031225411623752156
+44,0.03895255978909616,0.03332247980439818
+45,0.04847429662643073,0.0352065532950506
+46,0.05795839814029768,0.036827035597983955
+47,0.06659049999098023,0.03813880388193751
+48,0.07352701040670744,0.03910429229886631
+49,0.07802866410507721,0.039695277995131034
+50,0.07958923738717877,0.039894251404101355
+51,0.07802866410507721,0.039695277995131034
+52,0.07352701040670744,0.03910429229886631
+53,0.06659049999098024,0.03813880388193751
+54,0.05795839814029768,0.036827035597983955
+55,0.048474296626430734,0.0352065532950506
+56,0.03895255978909616,0.03332247980439818
+57,0.030068642644214504,0.031225411623752156
+58,0.022292269546572777,0.02896917224186394
+59,0.01586907323654335,0.026608540573093736
+60,0.010843866711637961,0.024197086622871247
+61,0.007110732269926539,0.02178523046171507
+62,0.004472879976244128,0.019418616870780607
+63,0.002697927604718676,0.017136869240940846
+64,0.00155973939647798,0.01497275533232727
+65,0.0008638556657416476,0.012951767151755816
+66,0.00045810527728723993,0.011092089963997388
+67,0.00023247133474277795,0.009404913245650147
+68,0.00011281697127223027,0.007895020453786327
+69,5.232091421320849e-05,0.006561585320241969
+70,2.3170690580135137e-05,0.0053990998132866916
+71,9.790432639493707e-06,0.004398362173931372
+72,3.943368702018315e-06,0.0035474613621840857
+73,1.5125249815960546e-06,0.0028327054331262276
+74,5.518672230147828e-07,0.0022394543410145357
+75,1.91313970645124e-07,0.0017528310758975296
+76,6.293222718589578e-08,0.0013582977188518668
+77,1.961523964235719e-08,0.001042094091741784
+78,5.78398092018225e-09,0.0007915456218645029
+79,1.6107288638482239e-09,0.0005952535905862057
+80,4.2281632676015655e-10,0.00044318510074393025
+81,1.0439909302719976e-10,0.00032668209694045915
+82,2.4190033750204664e-11,0.0002384089597700325
+83,5.2460314157070536e-12,0.0001722569947872011
+84,1.061696834131184e-12,0.0001232219890119193
+85,1.9984881583645844e-13,8.726832061302681e-05
+86,3.4857351599382385e-14,6.119022884726562e-05
+87,5.609228993004061e-15,4.2478051932228865e-05
+88,8.2863610123923925e-16,2.919470967694707e-05
+89,1.117262158974263e-16,1.986555877348687e-05
+90,1.3655426387463192e-17,1.3383030414223315e-05
+91,1.5005963063146224e-18,8.92617094530064e-06
+92,1.4679746474817013e-19,5.894310227640482e-06
+93,1.2627738903068471e-20,3.853521931013182e-06
+94,9.403635353348723e-22,2.4942485897551474e-06
+95,5.939138117904558e-23,1.5983750467744769e-06
+96,3.093301103075289e-24,1.0140858004452227e-06
+97,1.2755880837423788e-25,6.369828909339788e-07
+98,3.904861480844006e-27,3.9613014109573e-07
+99,7.888609052210037e-29,2.438962174264809e-07