38-CNN-mnist-tensorflow.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"http://akhavanpour.ir/notebook/images/srttu.gif\" alt=\"SRTTU\" style=\"width: 150px;\"/>\n",
    "\n",
    "[![Azure Notebooks](https://notebooks.azure.com/launch.png)](https://notebooks.azure.com/import/gh/Alireza-Akhavan/class.vision)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div style=\"text-align: right;direction:rtl;font-family:tahoma\">\n",
    "\n",
    "# استفاده از راهکار شبکه عصبی ژرف"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import tensorflow as tf\n",
    "\n",
    "#Start interactive session\n",
    "sess = tf.InteractiveSession()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Extracting MNIST_data\\train-images-idx3-ubyte.gz\n",
      "Extracting MNIST_data\\train-labels-idx1-ubyte.gz\n",
      "Extracting MNIST_data\\t10k-images-idx3-ubyte.gz\n",
      "Extracting MNIST_data\\t10k-labels-idx1-ubyte.gz\n"
     ]
    }
   ],
   "source": [
    "from tensorflow.examples.tutorials.mnist import input_data\n",
    "mnist = input_data.read_data_sets('MNIST_data', one_hot=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div style=\"text-align: right;direction:rtl;font-family:tahoma\">\n",
    "\n",
    "### پارامترهای اولیه"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div style=\"text-align: right;direction:rtl;font-family:tahoma\">\n",
    "ایجاد پارامترهای عمومی برای مدل:\n",
    "</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "width = 28 # width of the image in pixels \n",
    "height = 28 # height of the image in pixels\n",
    "flat = width * height # number of pixels in one image \n",
    "class_output = 10 # number of possible classifications for the problem"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div style=\"text-align: right;direction:rtl;font-family:tahoma\">\n",
    "### ورودی  و خروجی"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div style=\"text-align: right;direction:rtl;font-family:tahoma\">\n",
    "ایجاد place holder برای ورودی ها و خروجی ها\n",
    "</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "x  = tf.placeholder(tf.float32, shape=[None, flat])\n",
    "y_ = tf.placeholder(tf.float32, shape=[None, class_output])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div style=\"text-align: right;direction:rtl;font-family:tahoma\">\n",
    "### وزن ها"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div style=\"text-align: right;direction:rtl;font-family:tahoma\">\n",
    "تعریف تابع برای تولید وزنها:\n",
    "</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "def weight_variable(shape):\n",
    "    initial = tf.truncated_normal(shape, stddev=0.1)\n",
    "    return tf.Variable(initial)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div style=\"text-align: right;direction:rtl;font-family:tahoma\">\n",
    "### بایاس"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div style=\"text-align: right;direction:rtl;font-family:tahoma\">\n",
    "تعریف تابع برای تولید بایاس:\n",
    "<hr>\n",
    "باتوجه به ویژگی ReLU، نورون‌ها در طول آموزش بسیار حساس هستند.\n",
    "برای جلوگیری از نورون‌های مرده، مقدار دهی اولیه با مقادیر مثبت کوچک انجام شده است.\n",
    "</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "def bias_variable(shape):\n",
    "    initial = tf.constant(0.1, shape=shape)\n",
    "    return tf.Variable(initial)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div style=\"text-align: right;direction:rtl;font-family:tahoma\">\n",
    "### لایه های کانولوشنالی"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div style=\"text-align: right;direction:rtl;font-family:tahoma\">\n",
    "تعریف یک تابع برای ایجاد لایه های کانولوشنالی\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "<img src=\"lecture_images/stride.png\" alt=\"HTML5 Icon\" style=\"width:600px;\"> \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "def conv2d(x, W):\n",
    "    return tf.nn.conv2d(x, W, strides=[1, 1, 1, 1], padding='SAME')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div style=\"text-align: right;direction:rtl;font-family:tahoma\">\n",
    "### ادغام یا Max Pooling"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div style=\"text-align: right;direction:rtl;font-family:tahoma\">\n",
    "تعریف تابع برای انجام عمل max pooling یا ادغام بیشینه.\n",
    "<ul>\n",
    "<li>\n",
    "__ اندازه Kernel: __ \n",
    "<br>\n",
    "2x2  (اگر تصویر یک ماتریس 2x2 باشد، آنگاه در خروجی یک پیکسل نتیجه میدهد)\n",
    "</li>\n",
    "<li>\n",
    "__Strides: __ \n",
    "<br>\n",
    "گام های حرکت. در این مورد کرنل 2 پیکسل در هر حرکت جا به جا میشود، با توجه به اندازه ی کرنل، ادغام بدون همپوشانی است.\n",
    "</li>\n",
    "</ul>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "def max_pool_2x2(x):\n",
    "    return tf.nn.max_pool(x, ksize=[1, 2, 2, 1],\n",
    "                        strides=[1, 2, 2, 1], padding='SAME')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div style=\"text-align: right;direction:rtl;font-family:tahoma\">\n",
    "### اولین لایه ی کانولوشنالی (convolutional) و ادغام بیشینه ( max-pooling )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div style=\"text-align: right;direction:rtl;font-family:tahoma\">\n",
    "#### وزنها و بایاس"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div style=\"text-align: right;direction:rtl;font-family:tahoma\">\n",
    "    <ul>\n",
    "        <li>\n",
    "        اندازه فیلتر یا کرنل: 5x5 \n",
    "        </li>\n",
    "        <li>\n",
    "        تعداد کانال ورودی: 1 (سیاه و سفید)\n",
    "        </li>\n",
    "        <li>\n",
    "        تعداد feature mapها: 32 \n",
    "        </li>\n",
    "    </ul>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "W_conv1 = weight_variable([5, 5, 1, 32])\n",
    "b_conv1 = bias_variable([32]) # need 32 biases for 32 outputs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div style=\"text-align: right;direction:rtl;font-family:tahoma\">\n",
    "#### تبدیل تصاویر  مجموعه داده  به تنسور"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div style=\"text-align: right;direction:rtl;font-family:tahoma\">\n",
    "28 پیکسل در 28 پیکسل و  1 کانال (سیاه سفید)\n",
    "<br>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "x_image = tf.reshape(x, [-1,28,28,1])  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div style=\"text-align: right;direction:rtl;font-family:tahoma\">\n",
    "####  کانوالو کردن تصویر با ماتریس وزن ها و افزودن بایاس"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "convolve1= conv2d(x_image, W_conv1) + b_conv1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div style=\"text-align: right;direction:rtl;font-family:tahoma\">\n",
    "#### اعمال تابع فعالیت ReLU"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "<img src=\"lecture_images/relu.png\" alt=\"HTML5 Icon\" style=\"width:600px;\"> \n",
    "<div style=\"text-align:center\">\n",
    "<div style=\"direction:rtl;font-family:tahoma\">\n",
    "واحد یکسو ساز خطی (ReLU)</div>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div style=\"text-align: right;direction:rtl;font-family:tahoma\">\n",
    "یک تابع غیر خطی که معمولا برای مسائل طبقه بندی استفاده میشود (rectified linear unit)\n",
    "</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "h_conv1 = tf.nn.relu(convolve1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor 'Relu:0' shape=(?, 28, 28, 32) dtype=float32>"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "h_conv1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div style=\"text-align: right;direction:rtl;font-family:tahoma\">\n",
    "#### اعمال max pooling"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div style=\"text-align: right;direction:rtl;font-family:tahoma\">\n",
    "این تابع را قبلا تعریف کرده ایم.\n",
    "</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "h_pool1 = max_pool_2x2(h_conv1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div style=\"text-align: right;direction:rtl;font-family:tahoma\">\n",
    "#### لایه ی نخست به اتمام رسید\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "layer1= h_pool1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "<div style=\"text-align: right;direction:rtl;font-family:tahoma\">\n",
    "### دومین لایه ی کانولوشنالی (convolutional) و ادغام بیشینه ( max-pooling )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div style=\"text-align: right;direction:rtl;font-family:tahoma\">\n",
    "#### وزنها و بایاس"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div style=\"text-align: right;direction:rtl;font-family:tahoma\">\n",
    "    <ul>\n",
    "        <li>\n",
    "        اندازه فیلتر یا کرنل: 5x5 \n",
    "        (25 پیکسل)\n",
    "        </li>\n",
    "        <li>\n",
    "         تعداد کانال ورودی: 32 (خروجی لایه ی نخست)        </li>\n",
    "        <li>\n",
    "        تعداد feature mapها: 64 \n",
    "        </li>\n",
    "    </ul>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "W_conv2 = weight_variable([5, 5, 32, 64])\n",
    "b_conv2 = bias_variable([64]) #need 64 biases for 64 outputs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div style=\"text-align: right;direction:rtl;font-family:tahoma\">\n",
    "#### تبدیل تصاویر  مجموعه داده  به تنسور"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div style=\"text-align: right;direction:rtl;font-family:tahoma\">\n",
    "قبلا در لایه ی نخست تبدیل شده...<br>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div style=\"text-align: right;direction:rtl;font-family:tahoma\">\n",
    "####  کانوالو کردن تصویر با ماتریس وزن ها و افزودن بایاس"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [],
   "source": [
    "convolve2= conv2d(layer1, W_conv2) + b_conv2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div style=\"text-align: right;direction:rtl;font-family:tahoma\">\n",
    "#### اعمال تابع فعالیت ReLU"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [],
   "source": [
    "h_conv2 = tf.nn.relu(convolve2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div style=\"text-align: right;direction:rtl;font-family:tahoma\">\n",
    "#### اعمال max pooling"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [],
   "source": [
    "h_pool2 = max_pool_2x2(h_conv2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div style=\"text-align: right;direction:rtl;font-family:tahoma\">\n",
    "#### لایه ی دوم به اتمام رسید\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [],
   "source": [
    "layer2= h_pool2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div style=\"text-align: right;direction:rtl;font-family:tahoma\">\n",
    "### لایه ی سوم"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div style=\"text-align: right;direction:rtl;font-family:tahoma\">\n",
    "این لایه تمام-متصل یا fully connected است.\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div style=\"text-align: right;direction:rtl;font-family:tahoma\">\n",
    "#### وزنها و بایاس"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div style=\"text-align: right;direction:rtl;font-family:tahoma\">\n",
    "از لایه ی قبل 64 عدد feature map با سایز 7*7 داشتیم.\n",
    "</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [],
   "source": [
    "W_fc1 = weight_variable([7 * 7 * 64, 1024])\n",
    "b_fc1 = bias_variable([1024])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div style=\"text-align: right;direction:rtl;font-family:tahoma\">\n",
    "#### مسطح کردن خروجی لایه دوم"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [],
   "source": [
    "layer2_matrix = tf.reshape(layer2, [-1, 7*7*64])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div style=\"text-align: right;direction:rtl;font-family:tahoma\">\n",
    "#### ضرب ماتریسی (اعمال وزنها و بایاس)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [],
   "source": [
    "matmul_fc1=tf.matmul(layer2_matrix, W_fc1) + b_fc1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div style=\"text-align: right;direction:rtl;font-family:tahoma\">\n",
    "#### اعمال تابع فعالیت ReLU"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [],
   "source": [
    "h_fc1 = tf.nn.relu(matmul_fc1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div style=\"text-align: right;direction:rtl;font-family:tahoma\">\n",
    "#### لایه ی سوم به اتمام رسید\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [],
   "source": [
    "layer3= h_fc1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div style=\"text-align: right;direction:rtl;font-family:tahoma\">\n",
    "### فاز اختیاری برای کاهش Over-fitting خواهد - Dropout"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div style=\"text-align: right;direction:rtl;font-family:tahoma\">\n",
    "    برای شبکه های عصبی بسیار بزرگ بسیار مفید است. در این مرحله برخی از ویژگی ها فراموش میشوند.\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "<img src=\"lecture_images/dropout.png\" alt=\"HTML5 Icon\" style=\"width:250px;\"> \n",
    "<div style=\"text-align:center\">\n",
    "<div style=\"direction:rtl;font-family:tahoma\">\n",
    "Dropout</div>\n",
    "</div>\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [],
   "source": [
    "keep_prob = tf.placeholder(tf.float32)\n",
    "layer3_drop = tf.nn.dropout(layer3, keep_prob)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div style=\"text-align: right;direction:rtl;font-family:tahoma\">\n",
    "### لایه ی نهایی"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div style=\"text-align: right;direction:rtl;font-family:tahoma\">\n",
    "یک لایه ی تمام متصل Softmax\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div style=\"text-align: right;direction:rtl;font-family:tahoma\">\n",
    "#### وزنها و بایاس"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div style=\"text-align: right;direction:rtl;font-family:tahoma\">\n",
    "کانال ورودی: 1024 (نورونها از لایه 3)؛ تعداد خروجی:10\n",
    "</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [],
   "source": [
    "W_fc2 = weight_variable([1024, 10]) #1024 neurons\n",
    "b_fc2 = bias_variable([10]) # 10 possibilities for digits [0,1,2,3,4,5,6,7,8,9]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div style=\"text-align: right;direction:rtl;font-family:tahoma\">\n",
    "#### ضرب ماتریسی (اعمال وزنها و بایاس)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [],
   "source": [
    "matmul_fc2=tf.matmul(layer3_drop, W_fc2) + b_fc2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div style=\"text-align: right;direction:rtl;font-family:tahoma\">\n",
    "#### اعمال تابع فعال Softmax"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [],
   "source": [
    "y_conv= tf.nn.softmax(matmul_fc2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div style=\"text-align: right;direction:rtl;font-family:tahoma\">\n",
    "#### لایه ی آخر به اتمام رسید\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [],
   "source": [
    "layer4= y_conv"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div style=\"text-align: right;direction:rtl;font-family:tahoma\">\n",
    "# خلاصه ای از شبکه عصبی کانولوشن عمیق طراحی شده"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 0) Input - MNIST dataset\n",
    "#### 1) Convolutional and Max-Pooling\n",
    "#### 2) Convolutional and Max-Pooling\n",
    "#### 3) Fully Connected Layer\n",
    "#### 4) Processing - Dropout\n",
    "#### 5) Readout layer - Fully Connected\n",
    "#### 6) Outputs - Classified digits"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div style=\"text-align: right;direction:rtl;font-family:tahoma\">\n",
    "# تعریف توابع و آموزش مدل"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div style=\"text-align: right;direction:rtl;font-family:tahoma\">\n",
    "#### تعریف معیار خطا یا loss function"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [],
   "source": [
    "cross_entropy = -tf.reduce_sum(y_ * tf.log(layer4), reduction_indices=[1])\n",
    "cost = tf.reduce_mean(cross_entropy)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div style=\"text-align: right;direction:rtl;font-family:tahoma\">\n",
    "#### تعریف بهینه ساز"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_step = tf.train.AdamOptimizer(1e-4).minimize(cost)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div style=\"text-align: right;direction:rtl;font-family:tahoma\">\n",
    "#### پیش بینی شبکه"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [],
   "source": [
    "correct_prediction = tf.equal(tf.argmax(layer4,1), tf.argmax(y_,1))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div style=\"text-align: right;direction:rtl;font-family:tahoma\">\n",
    "#### دقت شبکه"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [],
   "source": [
    "accuracy = tf.reduce_mean(tf.cast(correct_prediction, tf.float32))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div style=\"text-align: right;direction:rtl;font-family:tahoma\">\n",
    "#### مقداردهی اولیه ی متغیرها"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [],
   "source": [
    "sess.run(tf.global_variables_initializer())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div style=\"text-align: right;direction:rtl;font-family:tahoma\">\n",
    "#### آموزش"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "step 0, training accuracy 0.06\n",
      "step 100, training accuracy 0.82\n",
      "step 200, training accuracy 0.94\n",
      "step 300, training accuracy 0.94\n",
      "step 400, training accuracy 0.9\n",
      "step 500, training accuracy 0.88\n",
      "step 600, training accuracy 0.88\n",
      "step 700, training accuracy 0.92\n",
      "step 800, training accuracy 1\n",
      "step 900, training accuracy 0.96\n",
      "step 1000, training accuracy 0.94\n",
      "step 1100, training accuracy 0.98\n"
     ]
    }
   ],
   "source": [
    "for i in range(1101):\n",
    "    batch = mnist.train.next_batch(50)\n",
    "    if i%100 == 0:\n",
    "        train_accuracy = accuracy.eval(feed_dict={x:batch[0], y_: batch[1], keep_prob: 1.0})\n",
    "        print(\"step %d, training accuracy %g\"%(i, float(train_accuracy)))\n",
    "    train_step.run(feed_dict={x: batch[0], y_: batch[1], keep_prob: 0.5})\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "<div style=\"text-align: right;direction:rtl;font-family:tahoma\">\n",
    "# ارزیابی مدل"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "test accuracy 0.9654\n"
     ]
    }
   ],
   "source": [
    "print(\"test accuracy %g\"%accuracy.eval(feed_dict={x: mnist.test.images, y_: mnist.test.labels, keep_prob: 1.0}))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [],
   "source": [
    "sess.close() #finish the session"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-success alertsuccess\" style=\"margin-top: 20px;text-align: right;direction:rtl;font-family:tahoma\">\n",
    "<font size = 3><strong>\n",
    "با تشکر از توجه شما\n",
    "</strong></font>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-block alert-info\">\n",
    "<div style=\"direction:rtl;text-align:right;font-family:B Lotus, B Nazanin, Tahoma\"> دانشگاه تربیت دبیر شهید رجایی<br>مباحث ویژه 2 - یادگیری عمیق پیشرفته<br>علیرضا اخوان پور<br>97-98<br>\n",
    "</div>\n",
    "<a href=\"https://www.srttu.edu/\">SRTTU.edu</a> - <a href=\"http://class.vision\">Class.Vision</a> - <a href=\"http://AkhavanPour.ir\">AkhavanPour.ir</a>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr>\n",
    "<div style=\"text-align: right;direction:rtl;font-family:tahoma\">\n",
    "منابع:\n",
    "</div>\n",
    "https://bigdatauniversity.com/courses/deep-learning-tensorflow/\n",
    "<br>\n",
    "http://tensorflow.org/\n",
    "<br>\n",
    "https://gotocon.com/dl/goto-london-2016/slides/MartinGorner_TensorflowAndDeepLearningWithoutAPhD.pdf\n",
    "<br>\n",
    "http://sebastianruder.com/optimizing-gradient-descent/index.html#batchgradientdescent\n",
    "<br>\n",
    "http://yann.lecun.com/exdb/mnist/  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}