4.2. 模型参数的访问、初始化和共享.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "4.2. 模型参数的访问、初始化和共享.ipynb",
      "provenance": [],
      "collapsed_sections": [],
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/chongzicbo/Dive-into-Deep-Learning-tf.keras/blob/master/4.2.%20%E6%A8%A1%E5%9E%8B%E5%8F%82%E6%95%B0%E7%9A%84%E8%AE%BF%E9%97%AE%E3%80%81%E5%88%9D%E5%A7%8B%E5%8C%96%E5%92%8C%E5%85%B1%E4%BA%AB.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "MQD3aovaG6eM",
        "colab_type": "text"
      },
      "source": [
        "##4.2. 模型参数的访问、初始化和共享"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "z8J5B9qwri__",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 62
        },
        "outputId": "7b057ebe-f2e1-4a72-8454-f629443af3a3"
      },
      "source": [
        "import tensorflow as tf\n",
        "from tensorflow import keras\n",
        "from tensorflow.keras import Sequential\n",
        "from tensorflow.keras import layers"
      ],
      "execution_count": 1,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<p style=\"color: red;\">\n",
              "The default version of TensorFlow in Colab will soon switch to TensorFlow 2.x.<br>\n",
              "We recommend you <a href=\"https://www.tensorflow.org/guide/migrate\" target=\"_blank\">upgrade</a> now \n",
              "or ensure your notebook will continue to use TensorFlow 1.x via the <code>%tensorflow_version 1.x</code> magic:\n",
              "<a href=\"https://colab.research.google.com/notebooks/tensorflow_version.ipynb\" target=\"_blank\">more info</a>.</p>\n"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "wxw8tcBir96h",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "tf.enable_eager_execution() #启用动态图计算"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "mjRBq0Lpvn-0",
        "colab_type": "text"
      },
      "source": [
        "&emsp;&emsp;我们先定义一个与上一节中相同的含单隐藏层的多层感知机。我们依然使用默认方式初始化它的参数，并做一次前向计算。"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "LRfrOEAEsBez",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "net=Sequential([layers.Dense(256,activation='relu',input_shape=(20,)),layers.Dense(10)])\n",
        "X=tf.random.uniform(shape=(2,20))\n",
        "Y=net(X) #前向计算"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Ag8kImnasWT2",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "82f38410-640f-48ff-8995-a5e4ee5f1a13"
      },
      "source": [
        "Y.shape"
      ],
      "execution_count": 4,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "TensorShape([Dimension(2), Dimension(10)])"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 4
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6ptFDlJbvuHx",
        "colab_type": "text"
      },
      "source": [
        "###4.2.1. 访问模型参数\n",
        "&emsp;&emsp;keras中访问模型参数有多种方法，首先我们可以通过net.trainable_weights属性获取模型的可训练参数。每个Dense层包括权重W和偏置bias,因此在trainable_weights中总共包括四个weight"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "LeDf_WnvsX47",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 84
        },
        "outputId": "b6b5274a-0258-43e0-ffbb-bbb2c75add3a"
      },
      "source": [
        "for weight in net.trainable_weights:\n",
        "  print(weight.shape)"
      ],
      "execution_count": 5,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "(20, 256)\n",
            "(256,)\n",
            "(256, 10)\n",
            "(10,)\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "SFJrcJjYyJUx",
        "colab_type": "text"
      },
      "source": [
        "&emsp;&emsp;我们也可以通过net.get_weights()方法来获取模型参数"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "WM-mlCWTwbX8",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 84
        },
        "outputId": "a445967d-d554-4c3f-e457-727f1ad631c6"
      },
      "source": [
        "for weight in net.get_weights():\n",
        "  print(weight.shape)"
      ],
      "execution_count": 6,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "(20, 256)\n",
            "(256,)\n",
            "(256, 10)\n",
            "(10,)\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ibyoN9CyyT5z",
        "colab_type": "text"
      },
      "source": [
        "&emsp;&emsp;此外，还可以先获取模型中的所有层，然后再获取各个层的参数。"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "YVU_eC1Gse_c",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 151
        },
        "outputId": "1149688b-c77d-4e80-e527-137423d3b174"
      },
      "source": [
        "for layer in net.layers:\n",
        "  layer_name=layer.name\n",
        "  print(layer_name)\n",
        "  layer=net.get_layer(name=layer_name)\n",
        "  print(len(layer.weights))\n",
        "  print(layer.weights[0].shape)\n",
        "  print(layer.weights[1].shape)"
      ],
      "execution_count": 7,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "dense\n",
            "2\n",
            "(20, 256)\n",
            "(256,)\n",
            "dense_1\n",
            "2\n",
            "(256, 10)\n",
            "(10,)\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "P8TkGYkOyvJH",
        "colab_type": "text"
      },
      "source": [
        "&emsp;&emsp;Dense中默认使用bias，且默认初始化为0。因为模型现在还没进行训练，所以bias全都为0。"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "fFbw5HrRt553",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 286
        },
        "outputId": "44482472-bfcc-4f6a-c3fe-fc3dc5e4afa0"
      },
      "source": [
        "net.get_layer('dense').get_weights()[1]"
      ],
      "execution_count": 8,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "array([0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,\n",
              "       0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,\n",
              "       0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,\n",
              "       0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,\n",
              "       0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,\n",
              "       0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,\n",
              "       0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,\n",
              "       0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,\n",
              "       0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,\n",
              "       0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,\n",
              "       0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,\n",
              "       0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,\n",
              "       0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,\n",
              "       0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,\n",
              "       0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,\n",
              "       0.], dtype=float32)"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 8
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2o2eu2fh3MiK",
        "colab_type": "text"
      },
      "source": [
        "###4.2.2 自定义模型参数初始化方法\n",
        "&emsp;&emsp;模型参数初始化方法在3.15节已经讲过，这里主要介绍下自定义初始化方法。\\\n",
        "&emsp;&emsp;如果传递一个自定义的可调用函数，那么它必须使用参数 shape（需要初始化的变量的尺寸）和 dtype（数据类型）："
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "HfG7UOx93uTN",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def my_init(shape,dtype=None):\n",
        "  return tf.random.normal(shape=shape,dtype=dtype)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "8piMf0ZZzuWO",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "net=Sequential()\n",
        "net.add(layers.Dense(64,kernel_initializer=my_init,input_shape=(10,)))"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "DBHoiEnS0Amx",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 185
        },
        "outputId": "34adfc8b-e731-4b1f-d12c-507188de3362"
      },
      "source": [
        "net.summary()"
      ],
      "execution_count": 11,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Model: \"sequential_1\"\n",
            "_________________________________________________________________\n",
            "Layer (type)                 Output Shape              Param #   \n",
            "=================================================================\n",
            "dense_2 (Dense)              (None, 64)                704       \n",
            "=================================================================\n",
            "Total params: 704\n",
            "Trainable params: 704\n",
            "Non-trainable params: 0\n",
            "_________________________________________________________________\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "A1Vr5oBu0FBC",
        "colab_type": "text"
      },
      "source": [
        "###4.2.3. 共享模型参数\n",
        "&emsp;&emsp;在有些情况下，我们希望在多个层之间共享模型参数。但是在前向传播函数call()中无法通过多次调用同一个层来计算，多次调用会报错。因此只能通过在call()函数中，将一个层的参数修改为同另一个层一样，以使两个层具有相同的模型参数。该功能通过set_weights()方法实现。"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "17EW2ZbR14Es",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "class MLPModel(keras.Model):\n",
        "  def __init__(self):\n",
        "    super(MLPModel,self).__init__()\n",
        "    self.hidden=keras.layers.Dense(2,activation='relu') #隐藏层\n",
        "    self.hidden1=keras.layers.Dense(2,activation='relu')   \n",
        "    self.out=keras.layers.Dense(2) #输出层\n",
        "\n",
        "  def call(self,inputs):\n",
        "    x=self.hidden(inputs)\n",
        "    x=self.hidden1(x)\n",
        "    print(self.hidden.get_weights()) #将hidden1的参数修改为同hidden一样\n",
        "    self.hidden1.set_weights(self.hidden.get_weights())\n",
        "    x=self.out(x)\n",
        "    return x #前向传播"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "-6NRF-Pz2Wg-",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "x_in=tf.random.normal(shape=(2,2))\n",
        "net=MLPModel()"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "HEespabb2lxb",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 101
        },
        "outputId": "da88b393-dae0-43c1-d1f0-e79ea96ceb74"
      },
      "source": [
        "net(x_in)"
      ],
      "execution_count": 14,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "[array([[ 0.34257436, -0.21097231],\n",
            "       [ 1.221043  ,  0.51139   ]], dtype=float32), array([0., 0.], dtype=float32)]\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<tf.Tensor: id=216, shape=(2, 2), dtype=float32, numpy=\n",
              "array([[0., 0.],\n",
              "       [0., 0.]], dtype=float32)>"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 14
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "NWv72YGEGL8c",
        "colab_type": "text"
      },
      "source": [
        "输出hidden和hidden1的参数，可以发现两个层的模型参数一样。"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "oK1DGsGI_DHj",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 168
        },
        "outputId": "ae04afeb-d3b8-4ce0-aa71-98868b7c171f"
      },
      "source": [
        "net.get_weights()"
      ],
      "execution_count": 15,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[array([[ 0.34257436, -0.21097231],\n",
              "        [ 1.221043  ,  0.51139   ]], dtype=float32),\n",
              " array([0., 0.], dtype=float32),\n",
              " array([[ 0.34257436, -0.21097231],\n",
              "        [ 1.221043  ,  0.51139   ]], dtype=float32),\n",
              " array([0., 0.], dtype=float32),\n",
              " array([[ 0.33180118, -0.6456049 ],\n",
              "        [-0.7495526 ,  0.8344692 ]], dtype=float32),\n",
              " array([0., 0.], dtype=float32)]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 15
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Tk16PUHsGokl",
        "colab_type": "text"
      },
      "source": [
        "4.2.4. 小结\n",
        "* 有多种方法来访问模型参数。\n",
        "* 可以自定义模型参数初始化方法。\n",
        "* 模型参数可以共享。"
      ]
    }
  ]
}