pytorch复现.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<center> <font size=6>《Auto-Encoding Variational Bayes》论文复现"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 论文及背景介绍"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "&emsp;&emsp;本篇论文是2013年由Diederik P. Kingma, Max Welling初次发出的著名论文，2014年修订了最终版，最终版在当年被ICLR（International Conference on Learning Representations）收录，本文复现的是最终版。本篇论文提出了变分自编码器（Variational Autoencoder）（VAE），由此拉开了生成模型的热潮，2014年的生成式对抗网络（Generative Adversarial Networks）（GAN），也是在这种背景下诞生的。VAE和GAN都是著名的生成模型，GAN的实际效果比VAE好，但是VAE的数学理论非常优美，并且可解释性强，而GAN的直观性强，现在的趋势是这两种生成模型的融合。\\\n",
    "&emsp;&emsp;在本篇论文中，作者使用了两种算法来训练VAE，一种是Neal在1995年提出的Wake-Sleep算法（醒眠算法），另一种是基于随机梯度的AEVB算法，选择不同的隐变量维度，在两个数据集上对这两种算法进行了比较，一个数据集是著名的手写数字体数据集，即MNIST数据集，另一个是人脸数据集，即Frey Face数据集。\\\n",
    "&emsp;&emsp;由于本人能力有限，只复现了本篇论文的AEVB算法，并没有复现醒眠算法。醒眠算法也是一个非常有趣的启发式算法，最初用于训练信念网络，也叫做贝叶斯网络，是一个有向概率图模型，2006年Hinton将醒眠算法应用到深度信念网络（DBN）上。他的两阶段即Wake阶段训练的目标是正向KL散度，但是Sleep阶段训练的目标是逆向KL散度，因为KL散度并不是一个对称性的距离度量，因此两阶段训练的目标函数并不一致，所以被称为醒眠算法，“醒”的时候目标是我们要的，“眠”的时候并不是我们要的目标。\\\n",
    "&emsp;&emsp;由于运行时间过长，本文只对MNIST数据集做实验，只选择隐变量维度为3、5、10的情况，并且训练样本数达到10^7时就停止。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 变分自编码器理论简介"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 生成模型简介"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "&emsp;&emsp;生成模型相对于判别模型，判别模型是指对$p(y|x)$的条件概率进行建模，而生成模型是对$p(x,y)$的联合概率密度进行建模，根据贝叶斯公式，$p(x,y)=p(x|y)p(y)$因此生成模型可以根据标签生成样本，这是显著不同于判别模型的特点。<br>\n",
    "&emsp;&emsp;生成模型主要应用于无监督学习，也可以应用于监督学习，即$p(x,y)=p(y|x)p(x)$，对于无监督学习，生成模型对数据分布$p(x)$进行建模，对含有隐变量的数据，如假定数据由隐变量$z$生成，相当于是对$p(x,z)$的联合概率密度进行建模。<br>\n",
    "&emsp;&emsp;典型的生成模型有朴素贝叶斯、隐马尔可夫模型（HMM）、高斯混合模型（GMM），典型的判别模型有神经网络、Logistic回归、支持向量机（SVM）。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 概率图模型简介"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "&emsp;&emsp;概率图模型（PGM），是一种用图结构来描述多元随机变量之间条件独立关系的概率模型，对高维空间中的概率模型带来了很大的便捷性，因为条件独立性大大减少参数的数量。\\\n",
    "&emsp;&emsp;图模型由节点，边组成，节点包括观测变量和隐变量，边表示条件概率。概率图具体分为有向图模型和无向图模型，有向图模型是指节点与节点之间是单向的联系，连接边由单向的箭头连接，表示单向的条件概率，无向图模型是指节点与节点之间是双向的交互关系，具有双向的条件概率的表达式。\\\n",
    "&emsp;&emsp;概率图模型的基本问题：\\\n",
    "&emsp;&emsp;（1）表示问题，即如何用图结构的形式表示整个模型。\\\n",
    "&emsp;&emsp;（2）学习问题，有了一个图结构，如何学习模型的参数，即参数估计问题，常见的估计方法为最大似然估计。\\\n",
    "&emsp;&emsp;（3）推断问题，在已知部分变量，计算其他变量的条件概率。对于含有隐变量的模型，推断问题主要是指在观测变量给定的情况下，导致这一现象出现的隐变量概率是多少，即求解$p(z|x)$，常见的求解方法有两大类，精确推断和近似推断，精确推断比如变量消除法，近似推断比如变分推断，MCMC。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## VAE模型"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "&emsp;&emsp;变分自编码器（Variational Autoencoder）是概率图模型和神经网络的结合，是一种基于似然的生成模型。它由两部分组成，编码器和解码器，编码器用于接收数据作为输入，并将其转换为潜在表示，并输入给解码器解码，用以重建数据。它采用变分推断近似模型的后验。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 模型表示"
   ]
  },
  {
   "attachments": {
    "VAE%E7%90%86%E6%83%B3%E6%A8%A1%E5%9E%8B%E5%9B%BE.jpg": {
     "image/jpeg": "iVBORw0KGgoAAAANSUhEUgAAAZUAAAFrCAYAAAAZwxRSAAAAAXNSR0IArs4c6QAAIABJREFUeF7tnQd8FcX2x09C7y0QIBTpTZoaUJqo/BWpT3rvAiKIqCDgU576LBAVeVQRCIgRpCmidGmCtEcvEekQSiAQekuE/+eM9963NyS5u/fu7p3d/c3nk0+UzJxz5ns295eZ2ZkJuXfv3gNCAQEQAAEQAAEdCIRAVHSgCBMgAAIgAAKCAEQFDwIIgAAIgIBuBCAquqGEIRAAARAAAYgKngEQAAEQAAHdCEBUdEMJQyAAAiAAAhAVPAMgAAIgAAK6EYCo6IYShkAABEAABCAqeAZAAARAAAR0IwBR0Q0lDIEACIAACEBU8AyAAAiAAAjoRgCiohtKGAIBEAABEICo4BkAARAAARDQjQBERTeUMAQCIAACIABRwTMAAiAAAiCgGwGIim4oYQgEQAAEQACigmcABEAABEBANwIQFd1QwhAIgAAIgABEBc8ACIAACICAbgQgKrqhhCEQAAEQAAGICp4BEAABEAAB3QhAVHRDCUMgAAIgAAIQFTwDIAACIAACuhGAqOiGEoZAAARAAAQgKngGQAAEQAAEdCMAUdENJQyBAAiAAAhAVPAMgAAIgAAI6EYAoqIbShgCARAAARCAqOAZAAEQAAEQ0I0AREU3lDAEAiAAAiAAUcEzAAIgAAIgoBsBiIpuKGEIBEAABEAAooJnAARAAARAQDcCEBXdUMIQCIAACIAARAXPAAiAAAiAgG4EICq6oYQhEAABEAABqUUlPj6e5syZQ8ePH6e4uDg6c+YMMgYCUhOIiIigYsWKUalSpahjx44UHh4udbwIDgT0JiCtqMTExNCUKVMoKSlJ7z7DHgiYQiBTpkzUv39/6ty5syn+4AQEZCAgpaiwoIwfP14GPogBBAImMGjQIAhLwBRhwCoEpBMVnvJq06aNZ4RSqVIl6tu3LxUvXlx8ZcyY0SpsEafDCCQnJ9Pp06fF19SpUyk2NlYQ4BHLggULMBXmsOfBqd2VTlS+/PJLmjt3rsgHC8q3337r1Nyg3xYn0KVLF4+wdOjQgV5//XWL9wjhg4BvAtKJyuDBg2nr1q0i8rFjx1KDBg189wI1QEBCAhs2bKAhQ4aIyGrXrk3jxo2TMEqEBAL6EpBOVFq3bu15y4unDPgtGhQQsCIBfmuRp3K58FthCxcutGI3EDMIaCIgnag8+eSTng7wiAVrKJryicoSEeA1Fh6huMuWLVskig6hgIAxBKQWlR07dhjTa1gFAZMIPP744xAVk1jDjRwEICpy5AFR2JQARMWmiUW30iQAUcHDAQIGEoCoGAgXpqUkAFGRMi0Iyi4EICp2yST6oZYAREUtKdQDAT8IQFT8gIYmliYAUbF0+hC87AQgKrJnCPHpTQCiojdR2AMBBQGICh4HpxGAqDgt4+ivqQQgKqbihjMJCEBUJEgCQrAvAYiKfXOLnqVOAKKCJwMEDCQAUTEQLkxLSQCiImVaEJRdCEBU7JJJ9EMtAYiKWlKoBwJ+EICo+AENTSxNAKJi6fQheNkJQFRkzxDi05sAREVvorAHAgoCEBU8Dk4jAFFxWsbRX1MJQFRMxQ1nEhCAqEiQBIRgXwIQFfvmFj1LnQBEBU8GCBhIAKJiIFyYlpIAREXKtCAouxCAqNglk+iHWgIQFbWkUA8E/CAAUfEDGppYmgBExdLpQ/CyE4CoyJ4hxKc3AYiK3kRhDwQUBCAqeBycRgCi4rSMo7+mEoComIobziQgAFGRIAkIwb4EICr2zS16ljoBiAqeDBAwkABExUC4MC0lAYiKlGlBUHYhAFGxSybRD7UEICpqSaEeCPhBAKLiBzQ0sTQBiIql04fgZScAUZE9Q4hPbwIQFb2Jwh4IKAhAVPA4OI0ARMVpGUd/TSUAUTEVN5xJQACiIkESEIJ9CUBU7Jtb9Cx1AhAVPBkgYCABiIqBcGFaSgIQFSnTgqDsQgCiYpdMoh9qCUBU1JJCPRDwgwBExQ9oaGJpAhAVS6cPwctOAKIie4YQn94EICp6EzXZ3r3z5+nKpk10Y98+SkpI+Pvr0iXx/V5Cgogmc1gYZeKvAgX+/h4WRjmrVqW8detS5sKFTY7YWe4gKs7KN3pLBFGx2FOQfPUqXdm4ka5u2iTE5OaBAwH1IEeVKkJc8tStS3nr1aOMefIEZA+NvQlAVPBEOI0ARMUiGb916BCdnTGDzk6fbmjERXv3pqK9elH2ChUM9eMU4xAVp2Qa/XQTgKhI/ixc27pVCMmFRYtSjzQkxDPSyBIRIaa4lNNd3Eg5HcbTYnfPnvWMdOjBg1TtFmrdWohL7tq1JSckd3gQFbnzg+j0JwBR0Z+pLhZ5WuvY++9T4po1D9nL/fjjlKdePY+YhGbJ4pfP+3fvesSFp9Ou/fe/D9nJ9+yzVHrUKOJpMhTtBCAq2pmhhbUJQFQkzN+5mTPp8NChlHIUUaBxY+LpqXzPPGNI1Ilr14pR0aXly73th4RQuagoKtKjhyF+7WwUomLn7KJvqRGAqEj0XPx18yYdGTaM4r//3iuq8A4dhJjkqlnTlGiv79olxCV+7lzvONq3p7JjxlCGHDlMicMOTiAqdsgi+qCFAERFCy0D6yauW0dHhg6l28ePe7zkrF5djBByPfaYgZ7TNn19504xYrqxZ4+nUrZSpahsVBTla9gwKDFZzSlExWoZQ7yBEoCoBEpQh/YJS5bQwZ49vSwV7dOHyn76qQ7WAzdxZPhwOjttmpehytHRFNa8eeDGbW4BomLzBKN7DxGAqAT5obi4eDHF9u7tiYL3ifAUE799JVO5sHChmJrjfTLuUmn6dCrYsqVMYUoXC0RFupQgIIMJQFQMBpyeed68uFfxoZy9bFmq8t13lK106SBGlbbr28eO0YHOnenW4cOeStUXLxYbJ1FSJwBRwZPhNAIQlSBl/O65c7S7cWO6e+aMiIA3Gz4aE0NZH3kkSBGpc3vnxAnaz8Jy6JBowHtjaixbRlmKFlVnwGG1ICoOSzi6i2NagvUM7G3Viq5s2CDc85QXfzBnL18+WOFo8nvrzz9p94sveqbC8jZoQNXS2pypybL9KkNU7JdT9Ch9AhipBOEJiZs8mY69+67Hc5WYGCrwwgtBiMR/l5dWrBBTYe5S+sMPqdgrr/hv0KYtISo2TSy6lSYBiIrJD8ftI0doe506RPfvC88l336bSvJGRwuWk1FRdHL06L8jDw2lyN9/p2xly1qwJ8aFDFExji0sy0kAomJyXg4NGEDx8+YJr7z/pObKlSZHoK+7Xc8/T7yfhUt4u3ZUYdIkfR1Y3BpExeIJRPiaCUBUNCPzvwHfebJTccRK9SVLKM9TT/lvUIKWVzdvpj2K/SqPrV0r7mpB+ZsARAVPgtMIQFRMzPgf/fvThQULhMewZs2o8syZJno3ztXBHj0o4eefhYNCbdpQxSlTjHNmMcsQFYslDOEGTACiEjBCdQbuxcfTFsVJvzWWLqXctWqpayx5rWvbttHuJk08UT554ABlDg+XPGpzwoOomMMZXuQhAFExKRdxkybRsffeE97y1q9P1X74wSTP5rjZ+9JLdOW334Sz0h98QMUGDDDHseReICqSJwjh6U4AoqI70tQN7mrUiK7v3i1+WH7sWCrctatJns1xc372bPpzyBDhLFeNGlRz9WpzHEvuBaIieYIQnu4EICq6I33Y4J1Tp2ib66ThkEyZqM7Ro5Qhe3YTPJvn4q9bt+j3MmXoQVKScFpr507KWqKEeQFI6gmiImliEJZhBCAqhqH9n2G+l+TQwIHiH/gmxaquV4pNcG2qi33t21Pir78KnxXGj6fwjh1N9S+jM4iKjFlBTEYSgKgYSddl+9CgQRQ/Z474v1LvvkvFBw82wav5Lk6PG0fHP/xQOGZBYWEJtCQmJtK6desoPDyc6vCmUYsViIrFEoZwAyYAUQkYoW8D/33qKc/JvnzGV+7ISN+NLFjj2vbt4kwwLtnLlaMnNm/2qxdJSUm0fv16ISa//vor3bt3j5o0aUIfugTLL6NBagRRCRJ4uA0aAYiKCeh/K1SIHriOZal76pRh6ylhYWGaepOQkKCpvq/KvK6yybWOEhIaSvUvXPDVxOvnW7ZsEUKyYsUKunbtmtfPICqaUKIyCASNAETFYPR8B8l2134UPia+tuJqXr1daxEVvQXF3Zet1at7jvOP3LbN590w+/fvF0KybNkyOn/+fJpIICp6Py2wBwLGEICoGMPVY/XyqlW037VgHewj4pWiY5SoKPerPDpnDuX/v/97iPCJEyc8QnLkyBFVGYCoqMKESiAQdAIQFYNTcGH+fPrDdSR8MI8wMUNQGKXyKJqKkydTobZtBWEWMR6RLF++nHbt2qWZOkRFMzI0AIGgEICoGIz93MyZdPitt4SXIj16ULnPPjPY48PmU06LGTVKYc+H33yTzs2aJYIo8emnFBsRQatWrRIL74EUiEog9NAWBMwjAFExmHXcxIl0bNQoOpUlC4U2bUqdpk412GP6omKkoLBnPopmRXQ07c6Rg3bnykXJ9+9TaGgo3Xe9qGB65+HQcQT4hQ+U4BGAqBjEnl+D5b/Q502YQPsvXKBnr12jgS+/TCWHDTPIY+pmzZr22rFjh+jvyp9+oqt371IoEf19DRkKCJhLwGxRefLJJ1PtoNlxmEs5bW8QFR0zcfjwYfHBunjxYrGGwCWEiIrdu0f94+OpfP/+4rBFM4tZouLuE4/Kls+YQXswUjEzzfClIGDmh3lagqJMiJnxyPAgQFQCyEJycrJYK+B9FbxJz11CQkLowYMHnv/ve+ECVbp9m4p0707lPv88AI/ampotKBwdrx/xOhKXIv/+Nx0uUYJWr16NNRUictqHi7an1Xq1lYKSMrcpxcZJuYeoaHyWjx075nmL6ejRo+m2zpIlC7WNjKTHY2JEPTPf/jJzcV4JIa23v+Li4mjz5s3i7a/drtOataDHQr0WWqhrNIH0BMXtW00do+MMhn2Iig/qvMDMoxH+4g16PDrh4mvxOUOGDFSzZk364B//oEOuY+7NvEclGKMU5rK3VSu6smGDYJTWPpWDBw8KgeERni9hdqcHohKMjwf4TI2AFrHQUtcutCEqaWSSh6tfffUV7d271+9cT5s2jSrmyWPajnp3oMESFPavdUc9iwuz5rWo+Pj4NFlDVPx+DNFQZwJahEJLXZ3DDJo5iEoK9LzD+6233qLTp08Tjzb++usvv5IzZMgQ6tKli2hr1tlf7CtY017sO5Czv/isL7fArF27lq5fv+7FHaLi12OIRgYQ0CIUWuoaEGpQTEJUUmAfMWIErVy50u9kZMyYURzRPnbsWI8NM08pDuYoRa9Tit3rLzyC2bhxo5hyhKj4/UiioY4E/BEJf9roGLLppiAqqSBv2bIl8QebvyUmJoYqVqzoaW7WfSrBFBTurBH3qfibA1na4eh7WTKhTxz+CIQ/bfSJNjhWICqpcOf9Jjx15V6U15KakSNHUuvWrb2a8AVdLCxc8j33HFX9/nstJlXX1XJKMRvVe3f9vnbtKHHNGhFvhQkTKLxDB9Wx27UiRMVemfVHIPxpY2VqEJU0shcdHU0TJkxQnVtef2nQoAF9lsrZXmbcUa9VUPQWFdxRn/qjAlFR/StkiYr+CIQ/bSwBI40gISrpZK9v377Ex4+oLT/88AOVcF1SlbLNrkaN6Lprf0b5sWOpsOs1Y7W2Za93fvZs+nPIEBFmrho1qObq1bKHbEp8EBVTMJvmRM0O+vSCccImSIhKOk8Av/nFC8Rqpon4qluum1aJmzRJHLbIxcz9Kmb9tinvUeGjaIoNGGCWa6n9QFSkTo/m4CAqvpFBVHwwmjFjBk2cODHdWo0aNaLRo0enW+defDxtqVLFU6fG0qWU23UjpO80yV3j2rZttFshqE8eOECZw8PlDtqk6CAqJoE2yY0/U1n+tDGpO4a4gaikgzU2Ntaz1yQ9+nz0SMGCBX0mSHmESVizZlTZdUaWz4aSVzjYowcl/PyziNLMo2gkxyLCg6hYIUvqY/RHIPxpoz4i+WpCVNLIyZkzZ6hPnz504cIFUSNTpkyUlJT0UG0eofBIRU25sW8f7XzmGU/V6kuWUJ6nnlLTVNo6Vzdvpj3Nm3vie2ztWspZtaq08ZodGETFbOLG+vNHIPxpY2wvjLUOUUmFL+/ufvPNN2nnzp0eQeEjW1577TW6ceOGp4Waaa+U5g8NGEDx8+aJf8712GNUM4CNlsY+Guqs73r+ebru4hTerh1VmDRJXUOH1IKo2CvR/giEP22sTA2ikkr2WFD4PnV3mTx5MtWqVYt+/PFH4gV5d+FDJnPmzKkp/7ePHKHtdeoQuW5CLPn221Ry6FBNNmSpfDIqik6615JCQyny998pW9mysoQnRRwQFSnSoGsQWkRCS11dgwyiMYhKCvjTp0+nSYq/tkeNGkUtWrTw1Bo6dCitWbNGLMyrnfZKmd+4yZPp2Lvvev65SkwMFXjhhSA+BtpdX1qxgg507uxpWPrDD6nYK69oN2TzFhAV+yVYi1BoqWsXUhAVRSYPHDgg1lH4KmAuvE+lX79+D+WaRy6vBPgBqjwiPmOePFRj2TLKXr68JZ6rW3/+SbtffJGSr14V8eZt0ICqLVpkidjNDhKiYjZx4/1pEQotdY2P3BwPEBUF5zfeeMNzQ2HVqlWJj67nAyKNKHfPnhUfzHfPnBHms1eoQI/GxFDWRx4xwp1uNu+cOEH7O3emW4cOCZtZIiKoxvLllKVIEd182MkQRMVO2fxfX9SIhZo6dqQDUXFldd68eV57TfiUYT52xchyddMm2tOypcdF9nLliKfCspUubaRbv23fPnaMDnTqRLeOHPHYqLZ4MeWtW9dvm3ZvCFGxb4bTEw2nCgpnG6JCRKdOnRLTXpcuXRK/AW3btqXhw4eb8ttwcfFiiu3d2+OLp8LKjhlDhVIcSmlKMOk4ubBwIR0ZNswz5cVVK02fTgUVohjsGGX0D1GRMSv6xeRrh70TjmVJSROiQkS8GP+za/NeRESEmPYqVKiQfk+eD0sJS5bQwZ49vWoV7dOHyn76qWkxpOfoyPDhdHbaNK8qlaOjKUyxP0WKQCUMAqIiYVJ0DiktYXGioGCkQiTunf/nP//peczeffdd+sc//qHzY+fbXOK6dXRk6FC6ffy4p3LO6tWpXFSU2M8SjML7Tw4PHUo39uzxuM9WqhSVjYqifA0bBiMky/mEqFguZQg4QAKOHqkkJiaKaa8TJ04IjM899xyNGTMmQKT+N//r5k0xxRSf4r4VvpekaO/elKtmTf+Na2h5fdcuOjt9OsXPnevVKrx9ezE1lyFHDg3WnF0VouLs/Dux944WlaioKJrr+uDMli2bmPZS3tgYrAfi3MyZYoRADx54hVCgcWMhLvkUR73oGWPi2rVCTC4tX+5tNiREjJiK9OihpztH2IKoOCLN6KSCgGNFhRfn27dv79mT8uqrr1KvXr2keThuHjhAx95/33OTojKw3E88QXnq1hVvXfH30CxZ/Ir7/t27xG+gXdm0ia5u3EjXUrk7Jt+zz1LpUaMoh+KEZb+cObQRRMWhiXdwtx0rKl9//TVNmTJFpL5cuXKeEYtsz8K1rVvp7IwZxG9fpVpCQjzikqVoUcoUFkaZw8LE90wFCogmSZcuUVJCAt1LSBD/zXtj3GKScjTk9lGoVSsxKspdu7ZsSCwVD0TFUulCsDoQcKSo8N3zPEpxr6W8/vrr1FXymxh5syGLC09PGVlYSIr26iU2Y6IETgCiEjhDtRZWrVolZh6aNm2qtgnqGUDAkaLCrw/za8Rc+G73OXPmUP78+Q3Aq79JPhrlysaNnpEGT5MFUnhayz2NlrdePeJ9Mij6EYCo6McyLUuXL1+m8ePHizc58+bNK2Yd+DtKcAg4UlQGDhxImzdvFsS7d+8ujrS3arl3/rxYE+G7WniKS3wppru4X8rpMDEtFhYm7jxhMclcuLBVu26JuCEqxqZpyZIlQlD4ugp34ZEKbw1ACQ4Bx4nKjh07xEGRXEJDQ8VfNWXKlAkOfQO98v0vU6dOFR7SOhjTQPcw7SIAUTHmUTh79qwQk7Vr13o5eOmll2jQoEGUPXt2YxzDqk8CjhOVjz76iBa5TtRt3rw5/etf//IJyYoVICpyZA2ion8e5s+fLwTFfZo4eyhRooQQk/r16+vvEBY1EXCUqJw7d04s0N+8eVNA4re/IiMjNQGzSmWIihyZgqjol4djx44JMXFPXbstd+zYUQgKzzygBJ+Ao0QlOjqaJkyYIKjXq1ePxo0bF/wMGBQBRMUgsBrNQlQ0Akuj+uzZs2nixIlePy1fvrwQE7v+YagPOfOtOEpUeJRyxHVs+8cff0wvWOy2RS2PB0RFCy3j6kJUAmN78OBBMTrZtWuXl6GePXumeoFeYN7QWg8CjhEVPjGUd81zqVy5MvFfPnYuEBU5sgtR8T8PfGwSfylLtWrVxOiEL9FDkZOAY0Tlyy+/9AhJjx49xINp5wJRkSO7EBXtedi5c6cYncTGxno15iu8eQsAitwEHCMqHTp0oMOHD4ts/Oc//6G6Nr+tEKIixy8eREVbHnjN89tvv/VqVKtWLfFHIB+nhCI/AUeICv/F06VLF5GNfPnyER/nEBISIn92AogQohIAPB2bQlTUwfz999/F6OS44j6hjBkzEm9U5j8IUaxDwBGiovyAbdSokddd9NZJlbZIISraeBlVG6KSPtm7d+8KMVmwYIFXxQYNGojRSfHixY1KDewaRMARosJH2u9x3V44YsQIatOmjUE45TELUZEjFxCVtPOwZs0aISi8f8xdcuTIIcQkGLevyvHEWD8K24tKQkKC16vDv/zyCxV2wHlXEBU5fjkhKg/n4erVq0JM+GBXZXn++eeFoBQsWFCO5CEKvwjYXlT4SBY+moUL3+oYExPjFyirNYKoyJExiIp3HpYuXSoEha/ydpcCBQoIMWncuLEcSUMUARGwvagMGTKENmzYICDxNJh7r0pA1CzQGKIiR5IgKn/nIT4+XojJ6tWrvRLTokULISi5cuWSI2GIImACthcV5S81H9PCm6ecUCAqcmQZokL0ww8/iNf4b9++7UlKRESEEJOGDRvKkShEoRsBW4sKH8nCR7NwyZYtG23cuFE3cLIbgqjIkSEni8rJkyfF6CTl7127du3Eq8KZM2eWI0mIQlcCthaV5cuX0zvvvCOA8QiFRypOKRAVOTLtVFHh21R5dPLgwQNPIvjeIh6dPPnkk3IkB1EYQsDWosK7c91C0rp1axo5cqQhEGU0ClGRIytOE5VDhw6J0cl///tfrwR07drVMeuZcjx5wYvC1qKiXKQfPnw4tW3bNnikTfYMUTEZeBrunCQq/AccP3fKUqVKFTE6qVGjhhwJQRSGE7C1qPDNjnztKBc+7bRmzZqGA5XFAURFjkw4QVR4YzGPTvbv3+8F/eWXX6bevXvLkQhEYRoB24rKrVu3vK4WXb9+PeXMmdM0sMF2BFEJdgb+9m93UZk8eTLNmjXLCzb3mUcnvC8MxXkEbCsqe/fuJb7Ih0uRIkUe2r1r91RDVOTIsF1FZevWrWJ04r70zk2bxaRz585ywEcUQSFgW1FR7qTnw+nGjh0bFMDBcgpRCRZ5b792E5Xk5GQhJt9//71XR/kqCRaURx55RA7wiCJoBGwrKmPGjPE8+DyvO2DAgKBBDoZjiEowqD/s006iwlPILChxcXGejmbNmlWICb9diQICTMC2otK3b1/asWOHyPInn3xCfFidkwpERY5s20FUbty4IcRk8eLFXlCfe+45sYmRp5dRQMBNwLaiorzpkRcSH330UUdlHaIiR7qtLiorV64Umxj5tG93yZs3rxidNG3aVA7IiEIqArYVlRdffJEuXLggYPNfWMWKFZMKvNHBQFSMJqzOvlVFhUWERycrVqzw6miTJk2EoPANqiggkBoB24pKnTp1iG+V47Ju3TrHnYIKUZHjF96KovLTTz+J0QlPe7kL30HEYsJTXiggkB4BW4oKiwmLCpcMGTLQtm3bHPcUQFTkSLmVRIUX4Hl0wgvyytKqVSshKHwoKwoI+CJgS1HhaS+e/uKSP39+WrVqlS8Otvs5REWOlFpFVObNmycEJSkpyQOOXw9mMeHXhVFAQC0BW4rK4cOHiRfquZQqVYoWLFiglodt6kFU5Eil7KLCmxdZTHgzo7J06tRJCEpISIgcIBGFZQjYUlT4VWJ+pZgLn/fF5345rUBU5Mi4zKLyzTff0KRJk7xA8dEqLCbKuOUgiSisQsCWorJmzRoaOnSoyAHfLPf5559bJR+6xQlR0Q1lQIZkFJUDBw6IhXg+CFJZ+Lpt9x9jAXUajR1NwJaiwteX/vvf/xaJbdmyJb333nuOSzJERY6UyyYqU6dOpRkzZnjBqV69uhidOG0vlxxPiP2isKWo8GZH/kuMS/fu3em1116zX+Z89AiiIkfKZREVnhLm3wm+REtZ+Piibt26yQELUdiCgC1Fhf8SmzhxokgQD+lfffVVWyRLSycgKlpoGVc32KLC1/nyQvx3333n1cnatWuL0UnZsmWN6zwsO5KALUVl7ty5FBUVJRLKb4G511eclGGIihzZDqaobNq0SYxOTp486YGRKVMmISbt2rWTAxCisB0BW4oKH8vywQcfiGS1aNGCRo0aZbvE+eoQRMUXIXN+HgxRuX37thid8PUPyvL0008LQXHakUXmZBpe3ARsKSq82ZHvpOfSqFEjGj16tOMyDlGRI+Vmi8qvv/4qBOX8+fMeALly5RJiwn9goYCA0QRsKSobN26kwYMHC3a8G9i9aG80TJnsQ1TkyIZZonLlyhXxnC9dutSr4y+88IIQlLCwMDmAIArbE7ClqOzcuZNefvllkTxsfiSx96Bfv362f5hl7KAZovLLL7+I0QkLi7sULFhQiInT7hGS8RlwWky2FJXY2Fjq0qWLyGWFChUeevNNvL9dAAAgAElEQVTFCUnGSEWOLBspKjzFxaMT3uyrLLw3iwUlZ86cckBAFI4iYEtR4bdd+GRVLsWLF6cff/zRUUnlzkJU5Ei5UaKycOFCMTq5c+eOp6O8AM9iwgvyKCAQLAK2FJWLFy9S48aNBVOeS0550VCwYJvpF6JiJu20fektKidOnBCjk99//93Lafv27YWgZMyYUY6OIwrHErClqPDlQu6/1rJnz06//fab4xIMUZEj5XqKSkxMjBidKAtvXmQx4c2MKCAgAwFbisr9+/cpMjLSw5ePqHBagajIkXE9ROWPP/4QoxN+AUVZ+AiiV155RY6OIgoQcBGwpahw3/gu7fj4eNFN3FGPt7+C9RsfqKhMnz6dvv76a6/w+eBHHp3wQZAoICAbAduKSv/+/Wn79u2CN98Z4bTpAYxU5PhV81dUdu/eLUYnBw8e9OoIvxres2dPOTqHKEAgFQK2FZVPPvnEc+PjyJEjqXXr1o56ACAqcqTbH1Hhw1Bnz57t1YEnnnhCjE74FXkUEJCZgG1FhRc1v/jiC8HeicffQ1Tk+LXTIipbtmwRC/FHjx71BB8aGkoDBw4kvt4XBQSsQMC2osJvfL3++usiB8899xyNGTPGCvnQLUaIim4oAzKkRlSSkpLEVNf8+fO9fNWrV0+MTkqWLBlQDGgMAmYSsK2onDp1il566SXB0q676i9cuPDQK6buh+fw4cPEX1zKlSsnvlKWMmXKUI8ePcx83hzny5eorFu3TuTwzJkzHjbZsmUTYuLewOs4aOiwpQnYVlQ4K+5faD6uYv369ZZOVFrB88mzyg8kLZ3s2rWrZzSnpR3qqieQlqhcv35djE6WLFniZYxP1WZBCQ8PV+8ENUFAIgK2FhUeqfCIhQsfCZ43b16J0OsTyoQJEyg6OtovY19++SXVr1/fr7ZopI5AaqKyfPlyMTq5dOmSx0i+fPmEmPCr8CggYGUCthYVXlNx76b/5ptvqEqVKlbOVaqx79u3z+8pLJ7DL126tO2YyNQhpajwqIRHJ3zfj7I0a9ZMCEqePHlkCh2xgIBfBGwtKvz2F78FxuXjjz8mvlvCjqVjx470559/auraI488Qt9//z3OitJETXtlpajkyJGDbt686TFSpEgRISbPPvusdsNoAQKSErC1qCxYsIB4vwqXAQMGUO/evSVNQ2BhzZgxg3hvg5bSuXNneuONN7Q0QV0/CChFRdm8TZs2QlCyZMnih1U0AQF5CdhaVHhHPe+s58J3TLz33nvyZiKAyI4fP078IaWl8DQM34qJYhyB7777jj7//HMvB6VKlRJiUqdOHeMcwzIIBJGArUWFz/5yL3zyX4xTp04NImpjXfPtjmoPzuS1JV5jQjGGAE9F8ksQW7du9XLAo0MWFBQQsDMBW4sKJ47nq69evSpyyHdQ2HW6Ye7cuRQVFaXqWe3Tpw9Ot1VFSnul9KYiecc8CgjYnYDtReWf//wnLVu2TOSRpyIaNmxoy5wmJCSIi8kePHjgs392fRPOZ8cNrLB3714xOtmzZ0+aXiAqBiYApqUhYHtRYUFhYeHC6w4jRoyQBr7egbz55ptik2dawhISEkK1atUSpzaj6EeAefIR9cry2GOPiY2l3bp18/wzREU/5rAkLwHbiwpvMHv++edFBiIiIuinn36SNxsBRvbLL7/4fBlh8ODBXh90Abp0dPNt27aJ0cmhQ4e8OLz22mviEFMuvo5pcTRAdN6WBGwvKpw1/gXfv3+/SCCvPaR2DpYdsnv37l1xeObt27dT7Q6vJ/Fr1kWLFrVDd4PWB75ZlMXEvQfKHQi/0cWjEz5TzV0gKkFLExwHiYAjREV5Yu9bb71FvFnQroVfm166dOlDU2A89fX0008/9IqrXTkY1a8NGzYIQTl58qTHRebMmYWYtG/f/iG3EBWjMgG7shJwhKjwq7b8yi0X3pvBezTsWvhDb8iQIal271//+hc1b97crl03tF+3bt2isWPH0qJFi7z88NuFLCg8tZpagagYmhYYl5CAI0SFpyv49Fd+tZgvPVq9erWtz1niviYmJno9bnyuFK+58LHqKNoIrFy5UggKXzXgLrlz5xZiwptq0ysQFW2sUdv6BBwhKpymd955h/h0WC58YRevPdi1cP/4XC9l4REKj1RQ1BO4fPmyEBOeTlQW3lDLglKgQAGfxiAqPhGhgs0IOEZUFi5cKA6V5NKuXTt6++23bZbK/3Vn165dxBsclWXcuHHENwmiqCOwePFisXZy7do1T4NChQoJMdFyMClERR1v1LIPAceIyrFjx6ht27Yic3zce8qrW+2T0r97wtMycXFx4r8LFy4spr5QfBPgC89YTNasWeNVmW9hZEHhk4a1FIiKFlqoawcCjhEVTlaXLl0oNjZW5G3mzJlUtWpVO+RQ9OGvmzcp6dIlSkpIEF/TFy+mua5jQVpHRlL/Vq0oU1jY318FClAGjR+OtgGVTkd4ypAF5d69e55afD887+3hN+f8KRAVf6ihjZUJOEpU+JgWPjmWi9WnwFg4rmzaRFc3bRLfb6XYgHcuc2YaU6SI6OuQc+eohOKDkv8te4UKlLduXcpTt674zmLj1HL06FEhJnw2nLJ06tRJjE4yZMjgNxqIit/o0NCiBBwlKspbErNmzUpz5syhEiVKWCZ1d06epHMzZ1Lihg10I50zptwd+qJIEeKTwN48d85nH3NWr075GjSgIj16UNaSJX3Wt0uFWbNmPfSKeYUKFcTopHbt2gF3E6ISMEIYsBgBR4kK52b48OGe61x79uxJAwcOlD5lLCBnZ8yg865bLFML+EHGjJSUPz8l5clDyXnyiO+b79yh+7w3J2tWynT1KmW8elV8z3T5MoUkJ6fZ78KdO1PRXr2Ihcau5cCBA2J0snPnTq8u8kVufKGbXgWiohdJ2LEKAceJysaNG8VfoVz4lVAerah5NTQYCb2yfj2dnT6dElK80uqO5XqVKnS9WjW6XrWq+P4gNNQrzFuJifTg/n3KkeLV15D79ynX3r2Ua9++v78fOJBq98KaNhXiktfP9YRgMFPjU3nCgrt+9erVxVRXtWrV1JhQXQeiohoVKtqEgONEhfP2yiuvEB8GyIVHKjxika0cGTZMjE5Slms1a1J88+Z0rUYNup81qy5hh965Q7l376bwJUso965dD9lkYSk7ZowuvoJphE9W4NHJwYMHvcJ49dVXqVevXoaEBlExBCuMSkzAkaLCm9neffddkRZeU+HRCq+xyFCubd9OLCg39u3zCudygwZ0oVkzuv7oo4aGmWv/fir088+Uf8MGLz85q1YVwpI7MtJQ/0YZ5306KW+75DUTHp2UL1/eKLc4pdgwsjAsKwFHigong692/eOPP0ReeCMkvw0W7BI3cSIdGzXKK4yE//s/Ot+qFd02efE828mTVHjRIgpbtcorntLvv0/FXn012KhU+9+0aZMYnfA+JXfht7lYTPjtLqMLRipGE4Z92Qg4VlR4dPLZZ5+JfFSuXJlmz54d1NzE9upFFxV3vfDC+8kBA+hi48ZBjavg8uVUctIkr4X9gi1aUKVUpuaCGmgK53wNAIvJvHnzvH7C+01YUMx66w+iItNTgVjMIOBYUbl+/bo4Av+c63XbDz74gJo2bWoG84d8HOzenRIUO9554Z0FxezRSVqd51ELCwsv7LsLL+JXnjUrKLx8Of3111+FoJw9e9ZTlXfCs5jwzngzC0TFTNrwJQMBx4oKw1e+BRQZGUlTpkwxPScHunalS8uWefyea9uW4iR8cYADLBYdTUXmz/fEWuDFF6lKkEd4yoTxKdQsJilv9+SzulhQ+OwuswtExWzi8BdsAo4WFT7niUcrN2/eFHngHfcNGzY0LSdHR46kM1Onevyd6dqVzkp+gVjROXMoQiEkEX37UhnXQZ2mgUvFEZ9txoLCJwu7C78qzmLCpwoHq0BUgkUefoNFwNGiwtCVx8Tz20CTJk0yJRdnp02jI8OHe3zFdetG5zp0MMV3oE6KzJ1Lxb75xmOm7CefUNGXXw7UrF/t4+PjhZjwnSfKwgdqsqDwvSfBLBCVYNKH72AQcLyoHDp0SBwTzzf7cRk2bFiq18LqmZybsbG0o359j0l+w+t4Grc16ulXT1ulxo71ejPs8d9+oxyVKvntgqeu+CIxLYWvM+D7Tm7fvu1pxjcwspjwjYwyFIiKDFlADGYScLyoMOwZM2bQxIkTBXc+Jn769Oniu1FlX7t2lOg6Wv1G5coU63oLzSh/Rtmt9NZblNO1kTDfs89S1RRvWqn1+9dff4nNh19//TXxfe++Ct8Pz2Ly22+/eVXlO+JZUNTY8OVDr59DVPQiCTtWIQBRcWWKRyt8uRUX/nDiEYsR5Vx0NB0eOtRjOvaLL+hGxYpGuDLcZs4//qBKb7zh8VMuKoqK+PGSwYgRI8T9JU888YRH3NMK/ttvvxXTXQ8e8FGZf5cyZcoIMalTp47hfdbqAKKilRjqW50ARMWVQd4k99prr3nyyWsrepxSm/IB2R4ZSbePHxf/fLZDBzrTrZuln6GIb76honPnij5kK1WKIrdv19Qf5X4hbvjRRx9R41T25vBGVRaT7Snsd+/e3StvmpybUBmiYgJkuJCKAERFkY6oqCia6/qANGLRng+HPOK6xphPFN7Ni90pDoGU6ulQE8z9+1SjWzdx8jGXsqNHU9HevdW0JOVVBO4GYWFhtGLFCq/206ZNo8mTJ3v9W5UqVcTo5LHHHlPlK1iVICrBIg+/wSIAUVGQT0hIEIv2p0+fFv+q96K9cpRyuk8fcfyKHQof51J82jRNoxVeXG/Tpg0lJiYS735XFl5kZ4HfvXu3GJ2w+ChLv379qG/fvpZAB1GxRJoQpI4EICopYPLGuffff1/8Ky/W8zQYXykbaElct472tWkjzPB9J7vmzAnUpFTta3bsKO5r4VJ1wQLK52O/zzvvvCPWUZRX9yo71KhRI1q9erVXH/kDmkcnfKyOVQpExSqZQpx6EYCopEKSD5h0f6DVqFFDvA0WaDk0aBDFu4QkvmVLOtWvX6AmpWpf4quvKHzxYhFTeMeOVGH8+DTj4/O4Ro8erSl+vgOnmwXXnyAqmtKMyjYgAFFJJYmHDx+mQYMG0cWLF8VPn3nmGc/hk/7mfGNEBN13TfNY+Y2vtPqvfBMsNEsWqhcXRxQS8lB1vtNEy9RVvXr1xOikVKlS/qIPajuISlDxw3kQCEBU0oDOO7T5VVd3ad26NY0cOdKvFF3ZtIn2tmwp2t4pXpz2ffWVX3Zkb1S1Xz/K6lqPqrZ4MeWtW9crZN7gyFcO8NpVUlKSqu7wKJFHi1YtEBWrZg5x+0sAopIOuZTXzr7xxhviQ1FrOTlmDPEXl4tNmtCJgQO1mrBE/ZITJlAh19XHJYcNI/5Slvfee4/4BOE7d+6o7k94eDjxpWpWLRAVq2YOcftLAKLigxyPVpTnSkVHR2u+x3zvP/5BVzZuFJ6Ovv02XbbZne9uhPnXr6cyrrUSHqXwaMVdeNMi74L3p/AFarzOZcUCUbFi1hBzIAQgKirodejQgXidxV14XUBL2VyxIiUlJIgme2bNonsFC2ppbpm6mS9epOrdu4t4M4WF0VOumzW3bNlCfA98IOX777+nsmXLBmIiKG0hKkHBDqdBJABRUQmfN0MmJyd7av/8889UpEgRn62TExPp93LlRL37WbLQjh9+8NlGa4VPFEe7j1AxVaS1vpZ4Hn/pJQp1vZBQ5/BhSkxOFud6XbhwwYufFptct1ixYrRYMfLR2j5Y9SEqwSIPv8EiAFFRSZ4PMUx5ayAfgOhrR/e1HTto9wsvCC+3SpemAxMmqPSorZpaoVBbT5v3/9WuMnAgZXfdB19jxQr6dOFCWr9+vaZ1FKXvkJAQcc4X71v59NNPif/fSgWiYqVsIVY9CEBUNFDkKTCeClMW3ijZrFmzNK1cXLyYYl3HllyuX5+OKt4o0+BaVVW3YKQ1WjFaUDjIMp98Qvldpwfv6dmTZq1d63X4Y3odcQuIu06lSpWIXynmu1HUjApVQTK5EkTFZOBwF3QCEBWNKeCbBVlYLl265GnZv39/ejmNS6rOf/cd/ek6qNLoe1N8iYYv0dGIItXq7ntW9mfLRtN9XN+rFJHQ0FBxgGeFChWofPny4suqe1OUYCAqejxVsGElAhAVP7LFR4sMGTKEeAHaXfivad4wmS9fPi+LfF0wXxvMJb5FCzrVv78fHtU3SUtYfAmOeg/p1ywxZQplXLqUJoaH05WMGel/B9R7t8uaNSs99dRT4k06t4jkz59frzCksQNRkSYVCMQkAhCVAEDzMe2LFi3yWODF5E6dOnndHHlq7Fg68dFHos65du0orkePADyqa5pyRGKWoHB0xWbOpOVr19LBbNkoWbH+wbc68n0nkZGRHhHJkCGDug5ZuBZExcLJQ+h+EYCo+IXtf43GjRtH3yjua+ef1KxZU5x2/OSTT1IwRIVjUAqLGdNebiInP/qINsXFUd7kZIqsWJGe69OH+Jj6okWLBkjams0hKtbMG6L2nwBExX92npbz58+n7777jk6dOuVlrWnTptS+QAG69OGH4t/NmP5yB6Acnbj/Tc3rxoHgOL17N4WvXEmPb9hAWe/fpzIff0wRFjmiPpB+p9cWomIUWdiVlQBERafMXL9+nWJiYoS43Lx508vq2JMnxf8bvVCfsitmTnu5fbsX6vn/y//nP1S4UyedCFvTDETFmnlD1P4TgKj4zy7VljxamT17tmetpcvTT9Pjrukxo18pVgYUjJEK+1e+Ulxp+nQq6DpIU2fMljEHUbFMqhCoTgQgKjqBTGmGby3k8666Pf00XenZU/zYyM2P6Y1S3D8zevqL/aTc/Jj78ccNImwNsxAVa+QJUepHAKKiH8tULZlxTIuaaS+zFutTHtOSMcUr1gbjls48REW6lCAggwlAVAwGzObNPFAymPtU0jpQ0gTE0rqAqEibGgRmEAGIikFglWb5gi6+qIuL0UffpzciMXq04nX0fb16VO3HH02gK7cLiIrc+UF0+hOAqOjP9CGLyku6LjRpQicNuqTL19tevn4eKIpHJkyggulc0hWofSu2h6hYMWuIORACEJVA6Klsa8Z1wmoFQ209lV3zqubrOmF/bFq9DUTF6hlE/FoJQFS0EvOn/oMHtLFYMbrvumck9osv6EbFiv5YSrWNVqHQWl9NoDn/+IMqvfGGqBqaJQvVO3NGTTPb14Go2D7F6GAKAhAVkx6JQ4MGUfycOcJbfMuWdKpfP5M8m+OmxFdfUbjrEq3wjh2pwvjx5jiW3AtERfIEITzdCUBUdEeausHEdetoX5s24ofJefLQLpfAmOTecDc1O3akjFevCj9VFyygfA0bGu7TCg4gKlbIEmLUkwBERU+aPmxtj4yk28ePi1qn+/Sh861amejdOFeFFy2i4tOmCQfZSpWiyO3bjXNmMcsQFYslDOEGTACiEjBC9QbOTp9OR95+WzRIyp+fdvPxLaGh6g3IWPP+farRrRtlunxZRFd29Ggq6rrpUsZwzY4JomI2cfgLNgGIiskZUI5WznboQGe6dTM5An3dRXzzDRWdOxejlDSwQlT0fd5gTX4CEBWTc3QuOpoODx3q8ar3m2Bmdkf5xhf7LRcVRUVc55yZGYfMviAqMmcHsRlBAKJiBFUfNve1a0eJa9aIWjcqV6bYzz4LQhSBu6z01luU8+BBYSjfs89S1XnzAjdqMwsQFZslFN3xSQCi4hOR/hVuxsbSjvr1PYbNvmdFjx4p701he4//9hvlqFRJD9O2sgFRsVU60RkVBCAqKiAZUeXs11/TkREjPKbjunWjcx06GOFKd5tF5s6lYoorlMt++ikV7dNHdz92MAhRsUMW0QctBCAqWmjpXPfoyJF0ZupUj9UzXbvS2Y4ddfair7mic+ZQxOzZHqN8XTBfG4ySOgGICp4MpxGAqAQ54we6dqVLy5Z5ojjXti3FSbrYXSw6morMn++JtcCLL1IVhcAEGaWU7iEqUqYFQRlIAKJiIFy1pg92704Jv/ziqX69alU6OWAA3S5ZUq0JQ+tlO3mSSk6aRLn27fP4CWvalCrPmmWoXzsYh6jYIYvogxYCEBUttAysG9urF1386SePhwcZMwphudi4sYFefZsuuHy5EJSQ5GRP5YItWlClGTN8N0YNgqjgIXAaAYiKRBmPmziRjo0a5RURvxnGx7mYPWrh0QkfvxK2apVXPKXff5+KvfqqRNTkDgWiInd+EJ3+BCAq+jMNyOK1bdvoyLBhdGP/fi87lxs0oAvNmtH1Rx8NyL6vxrn276dCP/9M+Tds8Kqa89FHqeyYMZS7Vi1fJvBzBQGICh4HpxGAqEia8SNDh9LZ6OiHortWsybFN29O12rUoPtZs+oSfeidO5R7924KX7KEcu/a9ZDNoj17UtmoKF18Oc0IRMVpGUd/ISoSPwNX1q+nszNmeC3iK8O9XqUKXa9WjXhhn78/UHk4Zcj9+5Rr716x8C6+HziQKoWwJk3E4ZB5n35aYkpyhwZRkTs/iE5/AhAV/ZnqbvHGnj1CXM7HxKRpmxf2+eTjpDx5xH0t7u/cgO85yXT1quc7nyisXHhPabRw585UtFcvylm9uu59cZpBiIrTMo7+QlQs9AzcOXmSzs2cSYkbNhALjZ6FBSRfgwZUpEcPyirJq8x69i9YtiAqwSIPv8EiAFEJFvkA/SYlJNCVTZvo6qZN4vutQ4c0WcxeoQLlrVuX8tStK75nCgvT1B6V1RGAqKjjhFr2IQBRsUku/7p5k5IuXSIWG/66x98vXRK9y1SgAGUOCxPCIb4KFKAMOXLYpOdydwOiInd+EJ3+BCAq+jOFRRDwEICo4GFwGgGIitMyjv6aSgCiYipuOJOAAERFgiQgBPsSgKjYN7foWeoEICp4MkDAQAIQFQPhwrSUBCAqUqYFQdmFAETFLplEP9QSgKioJYV6IOAHAYiKH9DQxNIEICqWTh+Cl50AREX2DCE+vQlAVPQmCnsgoCAAUcHj4DQCEBWnZRz9NZUARMVU3HAmAQGIigRJQAj2JQBRsW9u0bPUCUBU8GSAgIEEICoGwoVpKQlAVKRMC4LSi4DyQ33Hjh1pmnXXS6+OPzFBVPyhhjZWJgBRsXL2ELtPAsoPda6clmhAVHyiRAUQUEUAoqIKEypZlQBExaqZQ9xWJQBRsWrmELcqAsoRSHqjEYxUVOFEJRDwSQCi4hMRKliZQGqikto0GETFyllG7DIRgKjIlA3EojuBlGKRlnhAVHRHD4MOJQBRcWjindLt1MRC7b/pwQhvf+lBETasRACiYqVsIVbNBNQKCEYqmtGiAQikSgCiggfD1gTUTndBVGz9GKBzJhKAqJgIG67MJ+BLVNyL9hAV83MDj/YkAFGxZ17RKxcBta8RQ1TwyICAPgQgKvpwhBVJCfgSC7WbI/3tHhbq/SWHdlYlAFGxauYQtyoCEBVVmFAJBHQjAFHRDSUMyUjAl6hwzGoPnfSnfxip+EMNbaxMAKJi5ewhdp8EICo+EaECCOhKAKKiK04YAwFvAhip4IlwGgGIitMyjv6aSgCiYipuOJOAAERFgiQgBPsSgKjYN7foWeoEICp4MkDAQAIQFQPhwrSUBCAqUqYFQdmFAETFLplEP9QSgKioJYV6IOAHAYiKH9DQxNIEICqWTh+Cl50AREX2DCE+vQlAVPQmCnsgoCAAUcHj4DQCEBWnZRz9NZUARMVU3HAmAQGIigRJQAj2JQBRsW9u0bPUCUBU8GSAgIEEICoGwoVpKQlAVKRMC4KyCwGIil0yiX6oJQBRUUsK9UDADwIQFT+goYmlCUBULJ0+BC87AYiK7BlCfHoTgKjoTRT2QEBBAKKCx8FpBCAqTss4+msqAYiKqbjhTAICEBUJkoAQ7EsAomLf3KJnqROAqODJAAEDCUBUDIQL01ISkFpUtm7dShkzZpQSHIICAV8EkpOTqXbt2p5qW7Zs8dUEPwcByxOQTlRat25NZ86cEWAXLFhApUqVsjxkdMCZBI4fP05t2rQRnY+IiKCFCxc6EwR67SgC0onK4MGDiUcoXMaOHUsNGjRwVELQWfsQ2LBhAw0ZMkR0iEcs48aNs0/n0BMQSIOAdKLy5Zdf0ty5c0W4lSpVom+//RbJAwFLEujSpQvFxsaK2Dt06ECvv/66JfuBoEFACwHpRCU+Pl5MGSQlJXmEpW/fvlS8eHHxhTUWLelFXTMJ8BrK6dOnxdfUqVM9gpIpUyYxlRseHm5mOPAFAkEhIJ2oMIWYmBgaP358UIDAKQjoTWDQoEHUuXNnvc3CHghISUBKUXELy5QpUzwjFinpISgQSIcAj1D69+8PQcFT4igC0ooKZ4GnwubMmUP8Fk1cXJznrTBHZQidtRQBfsurWLFi4q3Fjh07YsrLUtlDsHoQkFpU9OggbIAACIAACJhHAKJiHmt4AgEQAAHbE4Co2D7F6CAIgAAImEcAomIea3gCARAAAdsTgKjYPsXoIAiAAAiYRwCiYh5reAIBEAAB2xOAqNg+xeggCIAACJhHAKJiHmt4AgEQAAHbE4Co2D7F6CAIgAAImEcAomIea3gCARAAAdsTgKjYPsXoIAiAAAiYRwCiYh5reAIBEAAB2xOAqNg+xeggCIAACJhHAKJiHmt4AgEQAAHbE4Co2D7F6CAIgAAImEcAomIea3gCARAAAdsTgKjYPsXoIAiAAAiYRwCiYh5reAIBEAAB2xOAqNg+xeggCIAACJhHAKJiHmt4AgEQAAHbE4Co2D7F6CAIgAAImEcAomIea3gCARAAAdsTgKjYPsXoIAiAAAiYRwCiYh5reAIBEAAB2xOAqNg+xeggCIAACJhHAKJiHmt4AgEQAAHbE4Co2D7F6CAIgAAImEcAomIea3gCARAAAdsTgKjYPsXoIAiAAAiYRwCiYhp0XdMAAABeSURBVB5reAIBEAAB2xOAqNg+xeggCIAACJhHAKJiHmt4AgEQAAHbE4Co2D7F6CAIgAAImEcAomIea3gCARAAAdsTgKjYPsXoIAiAAAiYRwCiYh5reAIBEAAB2xP4fxA65j3+B9zBAAAAAElFTkSuQmCC"
    }
   },
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![VAE%E7%90%86%E6%83%B3%E6%A8%A1%E5%9E%8B%E5%9B%BE.jpg](attachment:VAE%E7%90%86%E6%83%B3%E6%A8%A1%E5%9E%8B%E5%9B%BE.jpg)\n",
    "<center> 理想状态"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "&emsp;&emsp;上图是理想状态下的表示方法，$\\theta$为模型参数，$ x, z $ 分别为观测变量和隐变量，我们要建立的模型为$p_\\theta(x,z)$，请注意，这三个变量均为向量。为什么说是理想状态呢，因为从$z$到$x$可以是我们事先确定的机制，比如高斯混合模型，$z$表示不同的高斯分布，$x$是从生成于高斯分布的，他们的关系可以很容易的确定，问题是在$x$到$z$的有向边，我们简单的推导一下,为方便起见，我们省略参数。\n",
    "$$p(z|x) = \\frac{p(z,x)}{p(x)} = \\frac{p(x|z)p(z)}{p(x)} \\tag{1}$$\n",
    "&emsp;&emsp;$p(x)$正是我们要建模的对象，我们假定存在隐变量，因此:\n",
    "$$p(x)=\\int_{z}p(x,z) dx = \\int_{z_1} \\int_{z_2} ... \\int_{z_J}p(x,z_1,z_2,...,z_J)\\;dz_1 dz_2 ... dz_J \\tag{2}$$\n",
    "&emsp;&emsp;$z$的下标指的是$z$的维度，我们假设$z$有$J$维，因此如果$z$只有一两维还是可以直接可解的，但实际上有很多维，求解的复杂度是指数级的，所以当维度很大时，这几乎是个不可解问题，因为$p(x)$不可解，$p(z|x)$也无法直接得到，因此我们更改我们的图表示方法。"
   ]
  },
  {
   "attachments": {
    "VAE%E6%A8%A1%E5%9E%8B%E8%A1%A8%E7%A4%BA-2.jpg": {
     "image/jpeg": "iVBORw0KGgoAAAANSUhEUgAAAjQAAAFrCAYAAADGuxO8AAAAAXNSR0IArs4c6QAAIABJREFUeF7tnQd4FUX3h09CQu8EQgkgNSBVQFADghQRRVSa9KqgoiCo2Mun+KnwR5RPUEGKgBRBitIRpEsxSu9BSugBEjokJP/njMn13pDk7r13y8zub54nD5K7O3POe0byZnZ3NujWrVsphAYCIAACIAACIAACChMIgtAoXD2EDgIgAAIgAAIgIAhAaDARQAAEQAAEQAAElCcAoVG+hEgABEAABEAABEAAQoM5AAIgAAIgAAIgoDwBCI3yJUQCIAACIAACIAACEBrMARAAARAAARAAAeUJQGiULyESAAEQAAEQAAEQgNBgDoAACIAACIAACChPAEKjfAmRAAiAAAiAAAiAAIQGcwAEQAAEQAAEQEB5AhAa5UuIBEAABEAABEAABCA0mAMgAAIgAAIgAALKE4DQKF9CJAACIAACIAACIAChwRwAARAAARAAARBQngCERvkSIgEQAAEQAAEQAAEIDeYACIAACIAACICA8gQgNMqXEAmAAAiAAAiAAAhAaDAHQAAEQAAEQAAElCcAoVG+hEgABEAABEAABEAAQoM5AAIgAAIgAAIgoDwBCI3yJUQCIAACIAACIAACEBrMARAAARAAARAAAeUJQGiULyESAAEQAAEQAAEQgNBgDoAACIAACIAACChPAEKjfAmRAAiAAAiAAAiAAIQGcwAEQAAEQAAEQEB5AhAa5UuIBEAABEAABEAABCA0mAMgAAIgAAIgAALKE4DQKF9CJAACIAACIAACIAChwRwAARAAARAAARBQngCERvkSIgEQAAEQAAEQAAEIDeYACIAACIAACICA8gQgNMqXEAmAAAiAAAiAAAhAaDAHQAAEQAAEQAAElCcAoVG+hEgABEAABEAABEAAQoM5AAIZEDh//jzNmDGDDh8+TLGxsXTy5ElwAgGpCZQsWZIiIiKofPny1LlzZypSpIjU8SI4ENCbAIRGb6LoT3kCM2fOpG+//ZauX7+ufC5IwJkEcuXKRf3796dOnTo5EwCydiQBCI0jy46kMyMwffp0Gj16NACBgC0IDBw4kLp06WKLXJAECHgjAKHxRgifO4bAuXPnqH379nTz5k2Rc+XKlenZZ5+l0qVLi6+QkBDHsECiahFISkqi48ePi6/x48fTgQMHRAI5cuSgOXPmUNGiRdVKCNGCgB8EIDR+QMMp9iTAKzO8QsMtMjLS9d/2zBZZ2ZkAr8rs379fpMj30wwaNMjO6SI3EBAEIDSYCCCQSoD/0d+8ebP42+eff06NGzcGGxBQksCaNWtoyJAhIvYGDRrQl19+qWQeCBoEfCEAofGFFo61NYF27drRiRMnRI6zZ88WT4uggYCKBPjpvA4dOojQS5UqRT/99JOKaSBmEPCJAITGJ1w42M4E7rvvPld6mzZtotDQUDuni9xsTCAxMZHSz2cbp4vUQEAQgNBgIoBAKgH3HwDR0dHgAgJKE6hbt66HoCudDIIHAQ0EIDQaIOEQZxCA0Dijzk7JEkLjlEojzzQCEBrMBRDACg3mgA0JQGhsWFSklCUBCA0mCAhAaDAHbEgAQmPDoiIlCA3mAAhoIYBLTloo4RhVCEBoVKkU4tSLAFZo9CKJfpQnAKFRvoRIwI0AhAbTwWkEIDROqzjyzZQAhAaTw04EIDR2qiZy0UIAQqOFEo5xBAEIjSPK7JgkITSOKTUSTSUAocFUAIFUAhAaTAU7EYDQ2KmayEULAQiNFko4xhEEIDSOKLNjkoTQOKbUSBQrNJgDIOBJAEKDGWEnAhAaO1UTuWghgBUaLZRwjCMIQGgcUWbHJAmhcUypkShWaDAHQAArNJgD9iUAobFvbZFZxgSwQoOZAQKpBLBCg6lgJwIQGjtVE7loIQCh0UIJxziCAITGEWV2TJIQGseUGonikhPmAAjgkhPmgH0JQGjsW1tkhktOmAMgkCUBrNBggtiJAITGTtVELloI4JKTFko4xhEEIDSOKLNjkoTQOKbUSBSXnDAHQACXnDAH7EsAQmPf2iIzXHLCHAABXHLCHHAMAQiNY0qNRLFCgzkAAlihwRywLwEIjX1ri8ywQoM5AAJYocEccAwBCI1jSo1EsUKDOQACWKHBHLAvAQiNfWuLzLBCgzkAAlihwRxwDAEIjWNKjUSxQoM5AAJYocEcsC8BCI19a4vMsEKDOQACWKHBHHAMAQiNY0qNRLFCgzkAAlihwRywLwEIjX1ri8ywQoM5AAJYocEccAwBCI1jSo1EsUKDOQACWKHBHLAvAQiNfWuLzLBCgzkAAlihwRxwDAEIjWNKjUSxQoM5AAJYocEcsC8BCI19a4vMsEKDOQACWKHBHHAMAQiNY0qNRLFCgzkAAlihwRywLwEIjX1ri8ywQoM5AAJYocEccAwBCI1jSo1EsUKDOQACWKEJZA4kxsVR/IYNdHXnTrp1/jzx311f58+LrkOLFKHQsLB/v4oUobw1alDBqCjxPTTjCEBojGOLnuUkEHTr1q0UOUNDVCBgLoH77rvPNWB0dLS5gysw2u2rVylhwwYhMfHr19OV7dsDijpvrVpUsGFDITcFoqIoW548AfWHkz0JQGgwI5xGAELjtIoj30wJQGgyRnM9JoZOTpggvlJu3zZkBgVly0Yl+/YVX7kqVDBkDKd1CqFxWsWRL4QGcwAEUglAaDynwuXoaCExZ378MdM5UuC++8TqSo6SJV2XlbKnXmbik/gSlPvlqJsnT4pVnoRNmzLtM7xjRyE2+erWxdwMgACEJgB4OFVJAhAaJcuGoI0gAKH5h+q1/fvp8H/+QxeWL78Ds7hMFBUlLhUFcpnIdflq/XpxCSujy1eFH36Yyr//PuWOjDSi3LbvE0Jj+xIjwXQEIDSYEiCAFRrXHDg9bRodfO01SklM9JgXhZs3F6smhVu0MGS+XFixQqwGXfj1V4/+g0JDqdKIEVS8WzdDxrVzpxAaO1cXuWVEAEKDeQECEBpKvnmTDr32Gp2ePt1jPhRr316ITP577zVlnlzaulWIzdk5czzGK96lC1UcMYKCc+QwJQ47DAKhsUMVkYMvBCA0vtDCsbYm4NRLTvHr1tGhoUPp2sGDrvrmqV6dKg0fTvnr17ek5pe2bKGDQ4fS1V27XOPnrlSJKg4fTgUbNbIkJtUGhdCoVjHEGygBCE2gBHG+bQg4UWjOL15Mu3v08Khhid69hcxQUJC1tU1JEVJzatIkjziqTZlCRR591NrYFBgdQqNAkRCirgQgNLriRGcqE3Ca0MQtXEh7evVylYz3geHLOvyUkUyNn7Liy2F8I3Fau3vyZApr3VqmMKWLBUIjXUkQkMEEIDQGA0b36hBwktDwY9Pb3YQgV7lyVG36dOLLOjI2vhy2u0sXuv73367wai1cSPzYOFrGBCA0mBlOIwChcVrFkW+mBJwiNLfOnqVtjzxCN44dEyxyVaxI1X/4QfoN7XiDv11du9L1Q4dE3DnLlKHaS5ZQ9vBwzOoMCEBoMC2cRgBC47SKI1/HC83ODh3o4m+/CQ7Z8uWj2osXU56qVZWYGVf37qVtjz5Kty9fFvEWeughqjF7thKxmx0khMZs4hjPagIQGqsrgPGlIeCEFZoT335LMW+/7WJ+95QpFKbYDbbpb2Su8PHHVKp/f2nmkSyBQGhkqQTiMIsAhMYs0hhHegJ2Fxq+/+SP+++nlKQkUYuyr71GZV9/Xfq6ZBTg0c8+o6MjRoiPgkJCqN7vvxPfB4T2LwEIDWaD0whAaJxWceSbKQG7C83+F1+kMzNnivzz1q5NddLtyqva1PireXO6vG2bCDu8UyeK/Oor1VIwNF4IjaF40bmEBCA0EhYFIVlDwM5Cc3X3bopu3NgFtuaCBeKdTCo3fgfUjieecKVQd/Vq4g0B0f4hAKHBTHAaAQiN0yqOfB25QrP/hRdcb83mTel4czo7NN4UkO+pEas0HTtS5NixdkhLlxwgNLpgRCcKEYDQKFQshGosAbuu0Nw6d442uT3FZKf9W9Lvp3P/3r0UWrSosRNFkd4hNIoUCmHqRgBCoxtKdKQ6AbsKzYlvvqGYd94R5eHLTHy5yU6NLzvx5SduFYYNo1LPPWen9PzOBULjNzqcqCgBCI2ihUPY+hOwq9D81aIFXf7rLwGs0uefU4l0727Sn6S5PZ6aMoUODhkiBs13zz10z4oV5gYg6WgQGkkLg7AMIwChMQwtOlaNgB2F5sbx47TlnntEKYKyZaMHYmIoW968qpUmy3hvX7lCGytUoJTbt8Vx9f/6i3KWLm2rHP1JBkLjDzWcozIBCI3K1UPsuhKwo9CcmTWL9g8YIDjZeVdd992P+fFtfozb6Q1C4/QZ4Lz8ITTOqzkyzoSAHYXmwMCBdHr6dJFxuXfeodIvv2zL+h//4gv6e9gwkVvxLl2o8ujRAed58eJFWr16NYWHh9MDDzwQcH9mdwChMZs4xrOaAITG6gpgfGkIyCA07733Hi1evJhCQ0OJ46lfvz41aNCAypcv7xcn3hmY31TNjd/ZlL9+fb/6kf2kS1u2iHc8ceM3hvPOwf60xMREWrNmjRCZlStX0q1bt+jRRx+ljz76yJ/uLD0HQmMpfgxuAQHHCI37D6tNmzbdgdr987QPMzrOghphSJMIyCA07777rhAabkFBQZSSkkK5c+cWKwQsNg899BAVKlRIM5F14eGue0uijh6lbHnyaD5X64FhYWFaDxXHxcXF+XS8loNvX71KG8qW/YdbtmzU6MwZLae5juH/11lili1bRpcuXXKx5wMgND6hxMEgYBkBRwhNellJLyoZyQykxrI5adnAsglNRiCKFCki5KZhw4bUvHnzLFldP3yYtqauyOQoWZIa7NhhCFtfhMYImUlLanPNmnTz5Enx13u3bKFcXla1du3aJSRm6dKldOrUqUzZQGgMmTboFAR0J+A4oclKZtw/87aio3sl0KHlBFQQGndIERERFBUVRc2aNfPY5j7tmAu//kq7Um+OLdioEdWcN88Sxu7CY6TQ7HjqKYpft07kWH3mTCqcgfAdOXLEJTEHUy/FeYMCofFGCJ+DgBwEbC803sQkq8+9nStHCRGFXgRUExr3vCMjI6lx48bUokUL1/02Z2fPpn3PPy8OK9a+PVX55hu9UGnuxyyZ4YD2PfccnZ0zR8TGuXLO3Fii0i4n/fnnn5pjTzsQQuMzMpwAApYQgNDcd58LvNbVG0sqhUENJ6Cy0KTdb8OQ+GZQFpsacXF09t13BbcSPXtSpZEjDWfoPkD6S1FGrs7wuAdfeYVOff+9CKHMp5/S3lKlaMWKFeImX27ujHwBAaHxhRaOBQHrCNhaaLSssHg7xtvnRpWOf5N84YUXNHdfp04dGpvFi/m4L19+O+W+uM+Mml1jy0ho+vXrR9HR0ZrrMG7cuAwv/3AH3A/3Z3TLli0b3b59m7IFB1ONK1eo1tWr1LpXLyr/4YdGD+3Rv5mrMzzw4ffeo2WTJtG2PHloW758lJScTMHBwZScnBxQ3ixCzFS1lpSU5Ao5JCREtfCVjHf9+vVKxm2XoCE0WazQcJEhNHdOdQhN5v/7yyA07tEFExH/OC+QIwc93KaNWLlxf5zXqH/IzJQZFkVeiVn+88+UcPMmpeVsVG7oFwQyI4AnY62dGxAaCE2GMxArNP+syqi4QpN2aYVXaGpfvky1rl2jVr17U/n//Me0f23MFJq0pA6//z4tnTiRtrut0Ph7mck0UBjIVgTMFprMntA1Ow5ZigihkVRoZJkgTopD5Xto3OvETz7xSkz106fp+Jtvio/MvIfGCpnhHA+++iqdmjz5n3yHDaODZcqIlZu1a9cGNI1btWpFH3zwQUB9WHEy71uU1nApxJwKmHlpL6vtRtKydZrYQGggNOb8n67AKCoLTfXq1ally5Zib5pixYoJ2lY85WT2jcDu04qf6OKcuVX5+msq1qGD+O/Y2Fj6/fffacmSJbR9+3afZyJuCvYZGU4wmIDWp3M5DCdJDYQGQmPw/3rqdK+a0Nx111308MMPC4mpUKHCHaAvrFhBuzp3Ft8v+OCDVHPuXMOLYdXqDCe2o21bik9djak+YwYVbtHijnz37Nkj5IY30zt8+LAmHhAaTZhwkEkEtNzXqeUYk8I1dRgIDYTG1Akn82AqCA2vvvDlpKZNm1Lt2rWzxHk9Joa2pl52yBERQQ22bTMUv5Uyw4ltrl2bbsbGihzv3byZcmUgee4AWGz4t9fly5fT2bNnM2UDoTF02qBzHwj4Iiq+HOtDCFIfamuhYfLeiqp16c5Jy3ZSz1gDg5NVaPLlyydWYZo0aSJeeaC5paTQ2vBwotTHlhseP07BuXJpPt2XA6281MRxJl+/TutLl/4n5OBgepDf5RQUpCkFfndTmtysWrWKrly54nEehEYTRhxkAgFvP8/cQ/DlWBNCN2UIRwkNE/Vl8zwnTghTZp2kg8gkNPy2bX4RJe/+y6824L/70/647z66duiQOLX2smWUv25df7rxeo7VqzOXoqNpW8uWIs7cFStSvQxeQOs1Cbf7bVhwNmzYQLyXC4RGCzkcYwYBX34m+XKsGbGbMYbthYYhpr8b3Ns7m7I63oyiYAxrCMggNPxD9MyZM2I1pnDhwgGD2P/ii3Rm5kzRT7n33qPSAwcG3Gf6DqyWGY7n+OjR9HfqxoHhnTpR5FdfBZwnv+mcv7jxBn2qNff9hrDCrFr17ozXH0Hx5xyVSTlCaDKSGq1Fwz8EWkmpf5wMQqM3xdPTp9OBVInhlzXySxv1br68bZvHNuIVCPwSTn4ZJ7fKo0dT8S5d9E5Tuf4gNMqVLMuA/ZETf85RmZpjhCatSFqe3edjITIqT2v/Yrej0Nw4epS2pF5mCs6Rgx44dEjX+2h8lRkjhIbvn9lYsSIl37wpCl8/Oppyli3r3ySw0VkQGhsVU8P9oBllC6Gx1xzwmo3TCu4ViIMPsKPQcDn/bNqUruzY8c/qxZdfUvGuXW1V5dM//EAHBg0SOeWtWZPqrFplq/z8TQZC4y85Oc/z52eVP+fImb22qBy3QpMei9MKrm1aOPMouwpN7JgxxK8F4GbWfjRmziD3/Wf49Q4RAwaYOby0Y0FopC2NX4FpvbqQWedOuOoAofGyD41fMw8nKUnArkJz6/Rp2lS9uqsmtZcupfz16ilZo/RBX/rjD9r2yCOub9+3cydlL1HCFrkFmgSEJlCCcp0PofFeDwgNhMb7LHHIEXYVGi7fvn796GzqTsFhbdrQ3RMn2qKqe/v0oXM//yxyKda2LVUZN84WeemRBIRGD4ry9OHP1QR/zpEnY98jgdBAaHyfNTY9w85Cc2X7dvqzWbN/V2kWLaL8DRrQ1atX6eLFixQREaFcVS9t3kzbHnvMFXedlSspb61ayuVhVMAQGqPIWtOvP3LizznWZKfPqBAaCI0+M8kGvdhZaMQqzXPP0dk5c0SlEuvUoR2tWtGFCxfoo48+UrJ6fKmJLzmJ1Zn27anKN98omYdRQUNojCJrTb/+yIk/51iTnT6jQmggNPrMJBv0YnehuXbgAC1o0oTW5stHW/LmFRWbMWMGVa5cWbnqHRs5ko588okr7nobN1JuBfMwEjyExki65vftj5z4c475mek3ouOFRj+U6El1AnYWmq1bt9LMmTNp9erVlLbnbbVr1+jz0aMzfCu1zLV0f4s4x1n+gw8o4sUXZQ7ZktggNJZgN3RQXwTFl2MNDdrEziE0JsLGUHITsKPQrFy5kmbNmkXR0dHifVCJiYmuIgw8fZoq58lDtZcs8fpmalkqx28Q39aqFSVeuCBCKtiwIdWcP1+W8KSKA0IjVTl0CcYXSfHlWF2Ck6ATCI0ERUAIchCwk9DMmzePZs+eTfv3779DZLJly0aVEhOp//HjAnyeu++m6tOnUw7Jbwy+GRtLu7p0oat79oi4sxcvTvcsXSp93FbNbgiNVeSNG9cXSfHlWOMiNrdnCI25vDGaxARUF5qbN2+Ky0o//fQTnThxglhcbt++nSHxT/v3pxxvveX6LHeVKkJqcpYpI2WFbhw7JmTm2r59rvhqzp0rNgpEy5gAhMaeM0OLqGg5xo50IDR2rCpy8ouAqkJz7tw5sRrDIhMfH59l7iw51apVo0mTJol9aXh/mrQWWqQIVRw+nIo+8YRf/Iw66dyCBXRo6FBKPH/eNQTvN8P7zqBlTgBCY9/ZkZWwOFVmuNoQGvvOeWTmIwHVhCYmJob40tLcuXOJV2e0ts8++4yaN28uDj83fz7tfeYZj1NLPfccVRg2TGt3hh4X8847dCLd49hVv/uOij75pKHj2qFzCI0dqph5Dt52DnbCqw7S04HQ2HvOIzsfCKgiNCdPnqTx48fTzz//nOVlpfSpBwcHU6VKlWj69OkeH1349VexAsKXddJavrp1qdLw4ZZtVMcbAR4cOpQuR0e7YuLLYbyCVDhVxnworSMPhdDYv+yZSY0TZQYrNPaf78jQBwIqCM2CBQvoww8/pJCQEEpKSvIhu38O/eCDD+jxxx+/47ykhAQhNWd/+snjs+LdulHJZ56hvG7vgvJ5UB9OuLJrF52cMIFOT53qcVaxdu2EzIQUKOBDb84+FELj7Po7MXus0Dix6sg5QwKyC82YMWNo6tSpHo9eay0lr86UKVNG3GeTVTv53Xd06I037jgkrHVrKtm3LxVs1EjrkD4dF79unRCZuIUL7ziv4qefCqlC840AhMY3XjhafQIQGvVriAx0IiC70MTGxlLPnj293vibGY6hQ4fS008/7ZUWX+45/J//UPzatXccW+C++6hAVJTY/4X/DApO26bPa7ceB6QkJ1PC+vUUv2EDJfDXpk13dMBPMJV//33LLnv5lpF8R0No5KsJIjKWAITGWL7oXSECsgsNo/z777+pT58+dOnSJc1kg4KCqESJEvTLL79oPocPZNHgVZO0t1mnPzkoNFSITcGoKMpesiRlDwsjflIqlP8MCxOHJ8bFiaeT+M9b/HXypJCY+PXrKcVtkz/3vou2aSNWg1iY0PwnAKHxnx3OVJMAhEbNuiFqAwioIDSc9oEDB6hfv350+fJlzRReeukl6tWrl+bj3Q+8unu3EJtTU6b4db7Wk0r06CFEJk+1alpPwXFZEIDQmDc9Vq1aJZ40bNWqlXmDYqQ7CEBoMClAIJWAKkLD4e7atYteeOEFunr1qtf6hYWF0aJFi8SNxIE0XmkRl4hSLxVd278/kO4od2SkWN0pkLrKw6s7aPoRgNDoxzKznhISEuh///sfLVy4kAoXLiw2tsyfP7/xA2OEDAlAaDAxQEBBoeGQ//rrL+KVl+vXr2dZw/79+4sVHb0bv4qABefKjh13XFpK2wSPJUVcinK7HJW3Zk0hMrK/akFvXmb3B6ExlviSJUto9OjRdPHiRddA/ATh22+/bezA6D1TAhAaTA4QUFRoOOzNmzfT4MGDM91Yj39b5Ee9zfyt8dtvv6WJEycKqny/DwsVmvkEIDTGMD9z5oxYlfn11189BmjTpo34BSNfvnzGDIxevRKA0HhFhAOcQkClS07uNdmwYQO98sorGT7O3bt3b3rxxRdNLSELzbhx48SYvDIEoTEVv2swCI3+3Hlnbl6VcV8VLVWqlBCZJk2a6D8gevSJAITGJ1w42M4EVBUarsnq1auJH8t2fxllrly5aM6cOVS8eHFTywahMRV3poNBaPSrw9GjR8WqzPr16z067dixo/iFIXv27PoNhp78JgCh8RsdTrQbAZWFhmvBS+BvvPEGpaSkiNJ07dqVhgwZYnqZIDSmI89wQAiNPnWYMWOGkJnk5GRXhxUqVBCrMt7ep6RPBOhFKwEIjVZSOM72BFQXGi4Q36j4zjvviCea+ImLcuXKmV43CI3pyCE0BiDfv3+/EJk//vjDo/fu3bvTgAEDDBgRXQZKAEITKEGcbxsCKgjNsGHDxH4yERERmXLnDfR2794tVmusaBAaK6jfOSZWaPyvw6RJk4jnsXurVq2aWJWpXbu2/x3jTEMJQGgMxYvOVSIgu9Bs3LhR/IPK98Twjbb8VEVmjTfds+ppCwiNHLMeQuN7HbZv3y5WZXifJ/f27LPPUt++fX3vEGeYSgBCYypuDCYzAdmFhh/PXpv6fqWmTZvSiBEjpMQJoZGjLBAa3+rw9ddf0/fff+9xEjPkXyKqVKniW2c42hICEBpLsGNQGQnILDR8GemDDz5wYZsyZQrxEriMDUIjR1UgNNrqsGXLFvEo9qFDhzxOYJHhG+vR1CEAoVGnVojUYAKyCg0/is33zezZs0cQ6Ny5M7366qsG0/C/ewiN/+z0PBNCkzXNpKQkcXlp1qxZHgdGRUWJVZm77rpLz3KgLxMIQGhMgIwh1CAgq9DwMjj/BsmtYMGCNHXqVCpZsqS0UCE0cpQGQpN5HfjSLf8/FRsb6zooZ86cQmTatWsnRwERhc8EIDQ+I8MJdiUgo9DExcURPyZ69uxZgX3QoEHUo0cPqUsAoZGjPBCaO+tw5coVsSrDrwNxb82aNXPdcC9H9RCFPwQgNP5Qwzm2JCCj0IwaNYqmTZsmeEdGRtKECROIdwCWuUFo5KgOhMazDsuXLxerMvxLQlrjFU9elXnsscfkKBqiCIgAhCYgfDjZTgRkE5oDBw5Qt27dXK8z4JuC+W2+sjcIjRwVgtD8UwcWGF6VWbZsmUdhWGL4tQWFChWSo2CIImACEJqAEaIDuxCQTWjeffddWrx4scDbsGFD+vLLL5VADaGRo0wQGqKff/5ZyAzvy5TWeB8nXpXhy0xo9iIAobFXPZFNAARkEppNmzZ5bK/OMsNSo0KD0MhRJScLDd/syyKzZs0aj2K0bdtWyIzsl23lmEHqRQGhUa9miNggAjIJTXx8PC1cuJAWLVok7p1x34PGoPR16xZCoxvKgDpyqtD8+OOPQmYSExNd/PgRbBYZfiQbzb4EIDT2rS0y85GATEKTFjr/o8z30si6iV4INkWVAAAgAElEQVRGiCE0Pk48gw53mtDwxngsMps3b/Yg2qVLFyEzQUFBBpFGt7IQgNDIUgnEYTkBGYXGcih+BACh8QOaAac4SWh45+yxY8d6UOTXFbDIuHMwADO6lIgAhEaiYiAUawlAaPThD6HRh2OgvThBaPit8vwoNr9U0r316dOH+vXrFyhCnK8YAQiNYgVDuMYRgNDowxZCow/HQHuxu9CMGzeOJk6c6IGpdu3aYlVGpUu0gdYZ5/9LAEKD2QACqQQgNPpMBQiNPhwD7cWuQhMdHS1WZfbv3++B6IUXXpB+F+1Aa4rzsyYAocEMAQFJhIYfNY2IiFC+HhAaOUpoN6FJSUkRN/1Onz7dAzD/IsIb5FWsWFEO8IjCMgIQGsvQY2DZCFi9QvPWW2/RrVu3xG7AjRs3lg2P5nggNJpRGXqgnYRmw4YNQmaOHDniYhYaGiouL3Xs2NFQjuhcHQIQGnVqhUgNJmC10PAeGTdu3BBZ8qOnISEhBmdsTPcQGmO4+tqrHYTm+vXrQmTmzp3rkX6TJk3EqowdVjR9rSuOz5wAhAazAwRSCVgpNPy0RtpbtMuVK0dz5sxRti4QGjlKp7rQrFy5UsjM6dOnXUDz5csnVmXatGkjB2REIRUBCI1U5UAwVhKwUmjcJYB/8+zdu7eVKAIaG0ITED7dTlZVaC5evChEJu09ZmlAWrZsKWQmLCxMN0boyF4EIDT2qieyCYCAlULDu5mmPbXBbwVW+R9tCE0Ak1DHU1UUGn7VB8sMv/ojrRUtWlSIzMMPP6wjHXRlRwIQGjtWFTn5RcAqoTl27Bg99dRTIuaCBQsSL7Wr3CA0clRPJaHhy0r8KPaqVas84D3xxBNCZvLmzSsHVEQhNQEIjdTlQXBmErBKaH766Sf673//K1Lt2rUrDRkyxMy0dR8LQqM7Ur86VEVoeP7zqkzaDfGcbOnSpYXIPPjgg37ljpOcSQBC48y6I+sMCFglNG+88QatWLFCRMRvCq5QoYLS9YHQyFE+2YWGH8HmVZmNGzd6AHv66aeFzKj6lJ8c1XdmFBAaZ9YdWUsiNHwDZOvWrV2/nfIuqKo3CI0cFZRZaH744QexKuPeKlWqJESmfv36cgBEFMoRgNAoVzIEbBQBK1ZoeGWGV2i4PfbYY/Thhx8alZ5p/UJoTEOd5UAyCs2+ffuEyKQX9549e9Lzzz8vBzhEoSwBCI2ypUPgehOwQmj43hm+h4Abv2zP/YeQ3vmZ1R+ExizSWY8jm9BMmDCBxo8f7xF0jRo1xAZ5tWrVkgMaolCaAIRG6fIheD0JmC00t2/fpmeeeYb27t1LiYmJd/zWqmduZvYFoTGTduZjySI027ZtE6syvHmke+vfv7/S+y3JUWVE4U4AQoP5AAKpBMwWmjTwV69epePHj1OVKlVsUQsIjRxllEFoxowZQ1OnTvUAUq9ePXGvTGRkpBygEIVtCEBobFNKJBIoAauEJtC4ZTsfQiNHRawUmk2bNolVmZiYGBeM4OBgITKdO3eWAxCisB0BCI3tSoqE/CUAofGXnOd5EBp9OAbaixVCw2+LZ5GZPXu2R/gNGzYUMlO2bNlA08L5IJApAQgNJgcIWHzJyW4FgNDIUVGzhWb16tVCZk6cOOECkDt3biEyaTthy0EGUdiVAITGrpVFXj4TwAqNz8gyPAFCow/HQHsxS2guX74sNsj75ZdfPEJu3ry5kJnw8PBAU8H5IKCJAIRGEyYc5AQCEBp9qgyh0YdjoL2YITRLly4VqzLnz593hVu4cGEhMq1atQo0BZwPAj4RgND4hAsH25kAhEaf6kJo9OEYaC9GCs25c+eEyCxfvtwjTN71mmWmQIECgYaP80HAZwIQGp+R4QS7EjBTaMaOHUsnT56kQoUKiTcJ854cdmkQGjkqaZTQzJ8/X8gMbzeQ1kqWLCk2yGvatKkcySMKRxKA0Diy7Eg6IwJmCs1nn30mXkTJbejQocQv5LNLg9DIUUm9hYb3SmKRWbt2rUeC7du3F6syOXLkkCNxROFYAhAax5YeiacnYKbQvPPOO7RkyRIRwscff0yPPPKIbQoCoZGjlHoKzcyZM4XM8O7Waa18+fJiVeaBBx6QI2FE4XgCEBrHTwEASCNgptAMGjSI1q9fL4b+6quv6P7777dNISA0cpRSD6E5ePCgEJktW7Z4JNWtWzchM2ggIBMBCI1M1UAslhIwU2j69OlD27dvF/lOmjSJatasaWnueg4OodGTpv99BSo0kydPpm+++cYjgKpVq4rLS3Xq1PE/MJwJAgYRgNAYBBbdqkfATKHp0KEDHT58WEDiXVV5+d4uDUIjRyX9FZqdO3eKVZkdO3Z4JMIvUuUvNBCQlQCERtbKIC7TCZgpNHzPDD/6ym3x4sW22nwMQmP61M1wQH+EhldkeGXGvd1zzz1iVebuu++WIzFEAQKZEIDQYGqAQCoBM4UmKiqKbty4IUbmp0by5MljmzpAaOQopS9Cs3XrVrEqc+DAAY/g+T4Zvl8GDQRUIAChUaFKiNEUAmYJTUpKCtWrV0/kFBoaSvxmYjs1CI0c1dQiNMnJyeK1BfwUk3vjm9R5VcZOl0LlqAqiMJIAhMZIuuhbKQJmCc2lS5fooYceEmyKFi1KvH28ao0vl3399dcZhr1v3z7av3+/+CwyMpKqVKlyx3HlypWj7t27q5a2UvF6E5p169aJVZljx4658uK9ZFhkeG8ZNBBQjQCERrWKIV7DCJglNLxD8OOPPy7y4N+A+aZgFVubNm083qzsSw58GWPw4MG+nIJjfSSQmdBcu3ZNiMy8efM8emTJZpnhXX/RQEBFAhAaFauGmA0hYJbQ8OpFly5dRA78uDY/tq1i4/1z/I191KhR9OCDD6qYtjIxZyQ0K1asEDJz9uxZVx783iW+VyZNspVJEIGCQDoCEBpMCRBIJWCW0ERHR3swd//Bo1Ix+PHeXr16+RWy3R5V9wuCwSe5zyt+ko5FJm136rSh+Y3YvCrDb8hGAwHVCUBoVK8g4teNgFlCo1vAEnTUuXPnO56M8RZW2bJlxXusQkJCvB2KzwMg4C40+fPnJ753K62Fh4eLVZkWLVoEMAJOBQG5CEBo5KoHorGQAITGd/gTJ06kMWPG+HRi165daciQIT6dg4N9J5DZyt9TTz0lVmVy587te6c4AwQkJgChkbg4CM1cAhAa33n//fffPj8Rw48J8z48aMYRmDVrFg0fPtxjgDJlygiRadSokXEDo2cQsJAAhMZC+BhaLgIQGv/q0a9fP0p/X1BmPfFus1OnTvVvIJzllUBMTAx98cUXtHHjRo9j+dIgX2LKli2b1z5wAAioSgBCo2rlELfuBCA0/iHlTdlGjBih6WR+F9Dzzz+v6Vgc5BuB77//XmySl1Gz2+aNvpHB0U4hAKFxSqWRp1cCEBqviDI8IC4ujvjdVLwDsrc2ZcoUqlatmrfD8LkPBHbv3i1WZf78889Mz4LQ+AAUhypLAEKjbOkQuN4EzBSa69ev05UrV8TXhQsXSNVHt9Nq8Morr9CaNWsylZqgoCCqX78+jR07Vu+yObo/99dMpIGoVasWvfzyy9S7d28XGwiNo6eJY5KH0Dim1EjUGwEzhYaf8mEB4Pbmm2/6fGOtt1zM/nzRokX03nvvZTnsoEGDqEePHmaHZsvx+J4lXpXZs2ePR34DBgygPn36iO95e/WBLcEgKUcTgNA4uvxI3p2AmULDO+zyTrvcOnXqRK+99prSxbh58yY1a9aMeOUpo8bvCJozZw621dehyl9++SXxpTv31qBBA7EqU7lyZde3ITQ6wEYXShGA0ChVLgRrJAEzhWbDhg00cOBAkQ7/MLLDpRheoeEdadPfS8OXmxo3bkwjR440sny275vnDK/KHD582JUrb07IK19pr9JwhwChsf2UQILpCEBoMCVAIJWAmUITGxtLTzzxhBiZd21lEVC9rV27NtMXTn7wwQd4V5CfBebVLxYZ3l3ZvTVp0kTIDO8vk1GD0PgJHKcpSwBCo2zpELjeBMwUGo7d/QcOvzTQDu/Tad68OV28eNGjNLztPgtbrly59C6Z7ftbuXKlkBl+Q3tay5s3rxCZtm3bZpk/hMb20wMJYoUGcwAEMiZgttB07NiReCM0buPGjVP+SSfOg3en5V1q3Ru/xZlXaNC0E0hISBAi8/PPP3ucxI/Hs8wUK1bMa2cQGq+IcIDNCGCFxmYFRTr+EzBbaN5//31auHChCJg3m+NN51Rvf/311x158E2sDRs2VD010+LnJ8ZYZvhx/rQWFhYmRObRRx/VHAeERjMqHGgTAhAamxQSaQROwGyhmTdvHg0bNkwEzmP7+pLHwDM2pge+N4jvEeJWvHhx4h/QaN4JnD59mlj+li9f7nHwk08+KWSGL9350iA0vtDCsXYgAKGxQxWRgy4EzBaaM2fOuH7j5sea079/R5ekTOgk+dYtSoyL++fr/HmaMG8ezdiwQYzcvkEDeq5DBwotUoRCw8LEV3D27CZEpdYQ/Eg7y8y1a9dcgUdERAiRadq0qV/JQGj8woaTFCYAoVG4eAhdXwJmCw1H734T7YIFC4h/iMnekhISKH7DBkpYv178eXX3bo+QT2XPTsNLlBDfG3zqFJW5dcvj8zzVqlHBqCgq0LCh+DOkQAHZUzYsviNHjojLS+vWrfMYg/cmYpnJHoD8QWgMKxs6lpQAhEbSwiAs8wlYITS8S3DaJQa+p6ZNmzbmJ65hxJsnTtCpyZMpft06uvTHH17P+LxECeI3O71y6pTXY/PXq0cFGzWiEr16UY5Spbweb5cDpk2bJmTGfd+eihUrCpF54IEHAk4TQhMwQnSgGAEIjWIFQ7jGEbBCaPj+En7PDq/U8OZzsjVefTk5YQKdSrczrUecQUGUWKgQJRYoQEkFC4o/N968KYQmKkcOCk1IoJD4ePFnKD/SncVLLEv06EEl+/YlXsWxa9u3b58Qma1bt3qk2KtXL3rppZd0SxtCoxtKdKQIAQiNIoVCmMYTsEJo+PHcAhJecknYsEGIzLl0jw2nVeFKlSp0uWZNulyjBl2qWZNSQkM9CnTt4kVKSU6mPEWKeLpPYiLl37GD8vHXzp2Ud9++DAtbtE0bITYFoqKML7yJI3z33Xf09ddfe4xYvXp1sSpTp04dXSOB0OiKE50pQABCo0CREKI5BKwQGnMy822UmDffpBPjx99xEgvMmccfp0u1atHtvHl96zSTo7NduUL5t2+n8F9+EZKTvpV69lmq8MknuoxlZSfbtm0TqzI7d+70COO5556jZ5991pDQIDSGYEWnEhOA0EhcHIRmLgGnC83lP/+kQ0OH0uVt2zzAX4yKorOtWwuRMbKx2BRbuJAKpT4hlTZWvtq1qeLw4ZRP5xUMI3Nx75tfQsovI3Vv9erVEy+TrFq1qmFhQGgMQ4uOJSUAoZG0MAjLfAJOFpoT335LMW+/7QH9fLNmdPrJJ+lahQqmFiN3TAwVnz+fiqxc6TFuhY8/plL9+5saSyCD8b1RvCpz8OBBj25YZLp37x5I15rOhdBowoSDbEQAQmOjYiKVwAg4VWj29utH5+bO/RdeUBAdfeEFOvvYY4EBDfDsYosWUdmxYz1uIi7ati1VHTcuwJ6NPT0pKUmIzIwZMzwG4t2SWWbKlStnbACpvUNoTMGMQSQiAKGRqBgIxVoCMglNXFwc8Xb3Rre9ffp43Ph75e67hcxcK1/e6KE19Z/78GEhNXn37HEdzzcMV504UdP5Zh+0evVqITPHjx93Dc0v5eSbfjt06GBqOBAaU3FjMAkIQGgkKAJCkIOADEKzZcsW+uWXX6hs2bKGv9tpT69eFJf6LimuwOm2bem4pO+TKv3dd1TcbRUprHVrunvyZDkmDhFduXJFiAy/zsK98eP4vCpTInWjQTMDhtCYSRtjyUAAQiNDFRCDFASsFhre/v6T1Cd6+AcgX7LIly+fIWwOv/suxbo9Pnyya1c60bWrIWPp1WmpadOo5PTpru4inn+eyn/0kV7d+93P0qVLhcycO3fO1UehQoXEqgy/adyqBqGxijzGtYoAhMYq8hhXOgJWCw2/x4dvFuXt8LkNHjyYunXrpjunk5Mm0aHXXnP1e6JbNzrZpYvu4xjRIQsNi01a46efSvbpY8RQXvvky4IsMkuWLPE4tnXr1mJVhqXGygahsZI+xraCAITGCuoYU0oCVgsNQ3HfeI1vHuVVmtB0m9YFAu/agQP0h9u2+nHNmtHfr7wSSJemn1tu5EgKc3sCqt7GjZS7cmW/47h06ZLPb7KeP3++kJnLly+7xuVVNV6VadGihd+x6HkihEZPmuhLBQIQGhWqhBhNISCD0Jw+fZq6du1K8fHxIuc33nhD15tJd3XuTBdWrBB9826/ez//3BS2eg9SdcgQ1y7DhVu0oOrpnijSOt7t27fFvUrjxo3TJI58sy+/Ffu3337zGKJ9+/ZiVYZvAJalQWhkqQTiMIsAhMYs0hhHegIyCA1DGjlyJE1PvVckMjKSJk6cSDlz5gyY36nvv6eDbqsxe//v/4ifalKx8VNPVV991RV6pZEjqUTPnj6nwi8HXbVqFTVo0IBGjx6d5fkzZ84UqzKJiYmu43gVjVdlGjVq5PPYRp8AoTGaMPqXjQCERraKIB7LCMgiNAcOHKDOnTu7OPAKwvPPPx8wl60NGtD1mBjRz6mOHSm2V6+A+7Syg4jvv6cSs2aJEHJVqED3bt7sUzh8Oe///u//XOd8+umnGV4u4o3xeFXm999/9+if73dimQkKCvJpXLMOhtCYRRrjyEIAQiNLJRCH5QRkERoGMWbMGLEywy0kJISmTJlCvFrjbzs1aRIdTL0ROLFgQdo+dSqlZMvmb3dSnBd0+zbV6t6dQlMvz1UaMYJK9O6tKTZ+pxK/3dq9FStW7I4bfPmVBfzqAvfGrytgkbn33ns1jWXVQRAaq8hjXKsIQGisIo9xpSMgk9Dwzaa8KrN3717BqWnTpjRixAi/mbmvzhzv04dOt2/vd18ynVh8zhwqnSp+Wldprl+/TnzPy8WLF+nmzZse6fC+MZ999pl4iSRfXuKXSro3fpEkv1BShQahUaFKiFFPAhAaPWmiL6UJyCQ0DHLFihXipuC0xnvUPPzwwz4zjl+zhna0ayfOS8qfn7ZNn04pwcE+9yPjCUHJyVS7SxcKuXRJhFdz7lwq+OCDWYb69ttvi/tmbt26leFxLVu2pGXLlnl8ds8994hVmRo1asiIIcOYIDTKlAqB6kQAQqMTSHSjPgHZhIaJfvjhh7RgwQLKnTs39e7dm/r4sefKgUGD6PQPP4gCnXn8cTqmw/04MlW7zNdfU/gvv4iQinftSpW//DLT8H788UexAuNLe+mll+64POXL+VYdC6GxijzGtYoAhMYq8hhXOgIyCg3fIDxs2DDxSHCdOnX8Yra+dGlKvn5dnLt35Ei6UrWqX/3IelLevXupaurTW8G5clFDt/coucccHR1N/fr105zG/fffL1ZlKlWqpPkcmQ6E0MhUDcRiBgEIjRmUMYYSBGQUmkDBJWzcSNvbtBHd3IiIoJ2Sv6na33xr9OtHOWNjxem1fv6ZCrhtHsjfS0hIEPv7nD9/PtNLTenHnjx5slKXmNLHD6HxdzbhPFUJQGhUrRzi1p2AHYXm6IgRdDT1Esu5Vq3oyEsv6c5Nhg7v+t//qGjqKwjKvv46lXV7tQPH995779HKlSvpxo0bmsPlnX8Xur28U/OJkhwIoZGkEAjDNAIQGtNQYyDZCdhRaHY89RTFr1sn0B8eOpTON2kiexn8iq/I6tVUfvhwcW7Bhg2p5vz5rn6mTZtGo0aN8qtf3g/oVbcN/PzqxKKTIDQWgcewlhGA0FiGHgPLRsCOQvN7lSqUGBcnUG+fPJluFSsmG3Zd4sl+9izVSt1XJjQsjO7ft0/0u2nTJhowYEBAY/Bb0HlHYNUahEa1iiHeQAlAaAIliPNtQ0A1oeGN9/imYd7hNqOWFB9PGytWFB8l58hB0fPm6VqrTx591NXfm4sXe+3b1+O9dpjugLpPPUXBqfvKPHDoEF1MTKS+ffvSmTNnKCkpydfuXMeXKVOG5unMzu9gfDgRQuMDLBxqCwIQGluUEUnoQUAloeEnn9J+yD766KP00Ucf3YHgcnQ0/dWypfj+tfLlaXe6HW/1YKZVUrQeF0hM1V58kXIfPiy6uGfZMvrkp59ozZo1Pt034z4+v9IgJSWFmjRpIl6RIOsrDjJjBqEJZDbhXBUJQGhUrBpiNoSASkLDjxOvX7/exeGJJ54QN766t3MLFtDevn3Fty40bEgxb71lCLc0WclslcYMmeHEKvz3v1Q4lcn23r3p+99+E0LiT6tSpQpFRUURcy1VqpQ/XVh+DoTG8hIgAJMJQGhMBo7h5CWgktAwxZ49e9KuXbtcQNu1a0dvuUnL6enT6cDAgeLzuBYt6O/Bgw2B701YvAmPXkGVGzWKwlasoF25ctEEL/cKpa2+8Nj83/y2bZaYypUriy8V75lJzxFCo9fMQj+qEIDQqFIpxGk4AdWEhoHwCkJs6v4r/PdOnTrRa6mPLJ8YP55i3nxTcDvTpg0dM/AdRJlJjTfZ0bOoZb75hkIWL6Yx4eGUEBpKyZmszuTMmZN407yaNWu6BKZw4cJ6hiJFXxAaKcqAIEwkAKExETaGkpuAikLDRBs3bkxXrlxxweUN5IYMGULHRo2iIx9/LL5/qmNHik33dmm9q5F+JcZMmeFcIiZPpqW//UZ7c+WixKAgV3oFChSgBx54QLwdO20FJpvibxrXUjsIjRZKOMZOBCA0dqomcgmIgKpCw0nzioP7yxb5ctSTt2//KzRPP02xPXsGxEfLye5SY9alprS4jn78MW2IjaWCt2/TvZGR1OyZZ6hatWpUsmRJLaHb7hgIje1KioS8EIDQYIqAQCoBlYWGU+B7aI4cOeKqZ8c6dej+1MeNjb7klDao+6pM2ve0PNId6CQ8vm0bhS9fTnXXrqWcycniBuFSPry3KdDxZTwfQiNjVRCTkQQgNEbSRd9KEVBdaBg2vxl648aNLu6PxMdTy4QEQ28KTl9ksy81pY1f7osvKGz5cvFXfuM2v3nbyQ1C4+TqOzN3CI0z646sMyBgB6HhtF5//XX69ddf6ZE6dahl6grNhUaNXDcIG1l8q1ZoOKcKn3xChVNf81B1wgQq+sQTRqYqfd8QGulLhAB1JgCh0RkoulOXgF2EJk1q3u7QgbY98ogoyLUKFWj3//5neHGsFJpqL71EuWNiRI61ly6l/PXqGZ6vzANAaGSuDmIzggCExgiq6FNJAnYSGi5A4oUL9HvlyqIWyTlzUvTcuYbWJaNLTWbeGFy3bVsKTn2b9v0HDlCoDR/F9qWAEBpfaOFYOxCA0NihishBFwJ2ExqG8ntkJCWePy/4bJ8yhW6FhenCKn0nVu9Dkz0ujmr16CHCCi1ShO7fv9+QPFXqFEKjUrUQqx4EIDR6UEQftiBgR6HZ3qYNJaTeJBzzxht04cEHDalVVisxZqzSFF67liqkvqSzQFQU1VqwwJA8VeoUQqNStRCrHgQgNHpQRB+2IGBHoTn62Wd0dMQIUZ+zjz1GRwcM0L1W3p5q8va5HgGVHTOGii1aJLoq+9prVPb11/XoVuk+IDRKlw/B+0EAQuMHNJxiTwJ2FJr49etpx5NPioJdL1OGdn3zja7F0yorWo/zN7jqzz1HuY4dE6fXnD+fCjZs6G9XtjkPQmObUiIRjQQgNBpB4TD7E7Cj0KQkJ9OGiAhKvnVLFHDPqFF0NTJSl2L6Kim+Hq81yDwHDtDdL78sDg/Onp0a8rutgoO1nm7b4yA0ti0tEsuEAIQGUwMEUgnYUWg4tf0vvkhnZs4UWZ558kk6ZrMddMuMG0fh8+eL/MI7daLIr77CnCYiCA2mgdMIQGicVnHkmykBuwrNxVWraGfHjiLvxIIFadv06baaBbW7dKHQ+HiRU40ff6RCTZvaKj9/k4HQ+EsO56lKAEKjauUQt+4E7Co0DGpLvXp0I/U9T8effZZOP/WU7vys6LD4vHlUevx4MXTOu+6i+n/8YUUYUo4JoZGyLAjKQAIQGgPhomu1CNhZaE6OH0+H3nxTFORWkSK0fepUtYqTSbS1unen7Kn77FT85BMq+eyztshLjyQgNHpQRB8qEYDQqFQtxGooATsLjVilqVuXbhw9Khie7NKFTnTrZihPozsvNW0alUy9fJazbFmqHx1t9JBK9Q+hUapcCFYHAhAaHSCiC3sQsLvQnJwwgQ657c+i5xNPZs+APPv3092DB7uGrfjZZ1Syb1+zw5B6PAiN1OVBcAYQgNAYABVdqknA7kLDVdnZvj1dXL1aFOhy9eq0b/hwJYtVZehQyrdrl4i9UJMmVGPOHCXzMDJoCI2RdNG3jAQgNDJWBTFZQsAJQnN1926KbtzYxfdcy5Z0ZNAgS3j7O+hdX35JRZctc51ed80aylOtmr/d2fY8CI1tS4vEMiEAocHUAIFUAk4QGk71xLffUszbb7vqHtu7N53q0EGJeVBi9myKmDTJFWuFjz+mUv37KxG72UFCaMwmjvGsJgChsboCGF8aAk4RGgbO99LwPTVpLbZnTzr19NPS1CKjQErMmkUR33/v+ojvmeF7Z9AyJgChwcxwGgEIjdMqjnwzJeAkoWEIu7p0oQvLl7t4nOzUiU706CHlDCk1ZQqVTN3tmAMs/PDDVN1mGwTqDR5CozdR9Cc7AQiN7BVCfKYRcJrQMNjd3brR+aVLXYwv1aol3sh9IyLCNO5ZDZQzNpb4Tdr5t293HVbkkUeo2rRpUsQncxAQGpmrg9iMIAChMYIq+lSSgBOFhgu1p2dPilu0yFWz5Bw56OgLL1BcixaW1jFsxQoqO3YsBd+86Yoj7LHH6G63y06WBij54BAayQuE8HQnAKHRHX9psawAAAr9SURBVCk6VJWAU4WG63V89Gj6+8MPPUp37pFH6HTbtqav1vCqTPG5c6mo28oRB1buvfeo9MCBqk4v0+OG0JiOHANaTABCY3EBMLw8BJwsNFyFhN9/p0NDh9LVvXs9inL+oYfobOvWdKVqVUOLlXffPir2yy9U5LffPMbJU7UqVRw+nArcf7+h49utcwiN3SqKfLwRgNB4I4TPHUPA6ULDhU5JShJSc2rKlDvqnnDvvXSmdWu6VLs2pYSG6jIvghITKf+2bRS+cCEV2Lr1jj5L9OghZCYoJESX8ZzUCYTGSdVGrkwAQoN5AAKpBCA0/06FCytXise63Z+Ccn0aFESXatakyzVq0GX+s3p1n+YQ7/Cbb8cO8ZV/506ilJQ7zuenmPix7MLNmvnUNw7+lwCEBrPBaQQgNE6rOPLNlACE5k40l6Oj6eTEiXRm1qxMuSXnzEmJhQpRYoEClFSw4D9/Figgjg9JSKDQhAQKiY8Xf4ZevEjBN25k2ld4x45CZPLVrYuZGiABCE2AAHG6cgQgNMqVDAEbRQBCkznZ6zExdGryZLq4di3x6xP0bPzagkIPPkglevWiXBUq6Nm1o/uC0Di6/I5MHkLjyLIj6YwIQGi0zYtbZ85Q/Pr1lLBhg/jz+uHD2k5MPSpX+fJUsGFDKhAVJf7MHh7u0/k4WBsBCI02TjjKPgQgNPapJTIJkACExj+ASQkJlBgXJ75unT/v+m/uLTQsTHxlL1LE9d8hqZej/BsNZ2klAKHRSgrH2YUAhMYulUQeAROA0ASMEB1IRABCI1ExEIopBCA0pmDGICoQgNCoUCXEqJUAhEYrKRxnFwIQGrtUEnkETABCEzBCdCARAQiNRMVAKKYQgNCYghmDqEAAQqNClRCjVgIQGq2kcJxdCEBo7FJJ5BEwAQhNwAjRgUQEIDQSFQOhmEIAQmMKZgyiAgEIjQpVQoxaCUBotJLCcXYhAKGxSyWRR8AEIDQBI0QHEhGA0EhUDIRiCgEIjSmYMYgKBCA0KlQJMWolAKHRSgrH2YUAhMYulUQeAROA0ASMEB1IRABCI1ExEIopBCA0pmDGICoQgNCoUCXEqJUAhEYrKRxnFwIQGrtUEnkETABCEzBCdCARAQiNRMVAKKYQgNCYghmDqEAAQqNCldSI0V0moqOjMw067bisjvE3YwiNv+RwnqoEIDSqVg5x604AQqM7Usd26C4TDCEzYYHQOHaKIHEDCEBoDICKLtUkAKFRs24yRg2hkbEqiMnuBCA0dq8w8tNMAEKjGRUO9ELAfeUlq1UYrNBgKoGAfgQgNPqxRE+KE4DQKF5AicLPSGgyuvQEoZGoaAhFeQIQGuVLiAT0IgCh0Ysk+kkvKpmJC4QGcwUE9CMAodGPJXpSnACERvECShR+RqKi9Xt6pYGnnPQiiX5UIQChUaVSiNNwAhAawxE7ZgCt8oIVGsdMCSRqAgEIjQmQMYQaBCA0atRJhSi1XmKC0KhQTcSoCgEIjSqVQpyGE4DQGI7YMQN4ExoGwXvTQGgcMyWQqAkEIDQmQMYQahCA0KhRJxWi1PqoNoRGhWoiRlUIQGhUqRTiNJwAhMZwxI4ZwJuoaN14LxBguCk4EHo4V0UCEBoVq4aYDSEAoTEEqyM7hdA4suxI2mICEBqLC4Dh5SEAoZGnFqpH4k1oOD+tL7D0lwVWaPwlh/NUJQChUbVyiFt3AhAa3ZE6tkMIjWNLj8QtJAChsRA+hpaLAIRGrnogmsAIYIUmMH44Wz0CEBr1aoaIDSIAoTEILLq1hACExhLsGNRCAhAaC+FjaLkIQGjkqgeiCYwAhCYwfjhbPQIQGvVqhogNIgChMQgsurWEAITGEuwY1EICEBoL4WNouQhAaOSqB6IJjACEJjB+OFs9AhAa9WqGiA0iAKExCCy6tYQAhMYS7BjUQgIQGgvhY2i5CEBo5KoHogmMAIQmMH44Wz0CEBr1aoaIDSIAoTEILLq1hACExhLsGNRCAhAaC+FjaLkIQGjkqgeiCYwAhCYwfjhbPQIQGvVqhogNIgChMQgsurWEAITGEuwY1EICEBoL4WNouQhAaOSqB6IJjACEJjB+OFs9AhAa9WqGiA0iAKExCCy6tYQAhMYS7BjUQgIQGgvhY2i5CEBo5KoHogmMAIQmMH44Wz0CEBr1aoaIDSIAoTEILLq1hACExhLsGNRCAhAaC+FjaLkIQGjkqgeiCYwAhCYwfjhbPQIQGvVqhogNIgChMQgsurWEAITGEuwY1EICEBoL4WNouQhAaOSqB6IJjACEJjB+OFs9AhAa9WqGiA0iAKExCCy6tYQAhMYS7BjUQgIQGgvhY2i5CLgLzebNmykkJESuABENCGgkkJSURA0aNHAdvWnTJo1n4jAQUJcAhEbd2iFynQm0bduWTp48KXqdM2cOlStXTucR0B0ImEPgyJEj1K5dOzFYyZIlae7cueYMjFFAwEICEBoL4WNouQgMHDiQtmzZIoL64osvqFGjRnIFiGhAQCOBdevW0csvvyyOrl+/Po0ePVrjmTgMBNQlAKFRt3aIXGcCo0aNolmzZoleq1WrRlOmTNF5BHQHAuYQ6NGjB+3evVsM9vTTT9PgwYPNGRijgICFBCA0FsLH0HIROHXqFHXo0IH4/gNuNWrUoH79+lHp0qXFFxoIyEzg+PHjxF/ffvst7dq1S4TK94HNnj2bSpQoIXPoiA0EdCEAodEFIzqxC4GpU6fSmDFj7kgnODiY+AsNBGQkkJycTPyVvg0YMIC6d+8uY8iICQR0JwCh0R0pOlSdAF9q4t9yb9++rXoqiN+hBLJly0b9+/cnvvSEBgJOIQChcUqlkadPBPhppxkzZtDhw4cpNjaWzpw549P5OBgEzCYQHh5OERERVL58eercubN4ugkNBJxEAELjpGojV78JpN1X43cHOBEEDCaAfZMMBozupScAoZG+RAgQBEAABEAABEDAGwEIjTdC+BwEQAAEQAAEQEB6AhAa6UuEAEEABEAABEAABLwRgNB4I4TPQQAEQAAEQAAEpCcAoZG+RAgQBEAABEAABEDAGwEIjTdC+BwEQAAEQAAEQEB6AhAa6UuEAEEABEAABEAABLwRgNB4I4TPQQAEQAAEQAAEpCcAoZG+RAgQBEAABEAABEDAGwEIjTdC+BwEQAAEQAAEQEB6AhAa6UuEAEEABEAABEAABLwRgNB4I4TPQQAEQAAEQAAEpCcAoZG+RAgQBEAABEAABEDAGwEIjTdC+BwEQAAEQAAEQEB6AhAa6UuEAEEABEAABEAABLwRgNB4I4TPQQAEQAAEQAAEpCcAoZG+RAgQBEAABEAABEDAGwEIjTdC+BwEQAAEQAAEQEB6AhAa6UuEAEEABEAABEAABLwRgNB4I4TPQQAEQAAEQAAEpCcAoZG+RAgQBEAABEAABEDAGwEIjTdC+BwEQAAEQAAEQEB6AhAa6UuEAEEABEAABEAABLwRgNB4I4TPQQAEQAAEQAAEpCcAoZG+RAgQBEAABEAABEDAGwEIjTdC+BwEQAAEQAAEQEB6AhAa6UuEAEEABEAABEAABLwRgNB4I4TPQQAEQAAEQAAEpCcAoZG+RAgQBEAABEAABEDAGwEIjTdC+BwEQAAEQAAEQEB6AhAa6UuEAEEABEAABEAABLwRgNB4I4TPQQAEQAAEQAAEpCcAoZG+RAgQBEAABEAABEDAGwEIjTdC+BwEQAAEQAAEQEB6AhAa6UuEAEEABEAABEAABLwR+H/ooVxM/Rk9gwAAAABJRU5ErkJggg=="
    }
   },
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![VAE%E6%A8%A1%E5%9E%8B%E8%A1%A8%E7%A4%BA-2.jpg](attachment:VAE%E6%A8%A1%E5%9E%8B%E8%A1%A8%E7%A4%BA-2.jpg)\n",
    "<center> VAE模型"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "&emsp;&emsp;因为$p(z|x)$的难处理（intractable），我们采用近似的方法，用一个概率分布$q_\\varphi(z|x)$去近似$p_\\theta(z|x)$，$q_\\varphi(z|x)$又叫做变分分布，用变分分布不断逼近真实的条件分布，这就是变分推断的思想。那么问题来了，变分分布怎么选择呢，如何衡量$q(z|x)$和$p(z|x)$的相似程度呢，根据通用近似定理，神经网络可以拟合一个任意函数，因此我们可以利用神经网络来拟合这个变分分布，这就是VAE的思想。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 数学推导"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "&emsp;&emsp;我们知道衡量两个分布之间的差异使用最多的是$KL$散度，$KL$散度也叫做相对熵，对于离散分布，其计算公式为：\n",
    "$$KL(p||q)=\\sum_x p(x)\\frac{p(x)}{q(x)} \\tag{3}$$ \n",
    "&emsp;&emsp;为方便推导，我们将参数写在函数里面，我们采用最大似然估计，因此我们的目标函数为：\n",
    "$$\\max_{\\theta}p(x;\\theta)=\\max_{\\theta}log\\;p(x;\\theta)\\tag{4}$$\n",
    "$$\\begin{align} \n",
    "log \\; p(x;\\theta) &= log \\; p(x,z;\\theta) - log \\; p(z|x;\\theta) \\\\\n",
    "&= log \\frac{p(x,z;\\theta)}{q(z|x;\\varphi)}-log \\frac{p(z|x;\\theta)}{q(z|x;\\varphi)}\\tag{5}\n",
    "\\end{align}$$ \\\n",
    "&emsp;&emsp;我们将等式两边同时乘$q(z|x;\\varphi)$并对$z$求积分，因为$\\int_z q(z|x;\\varphi)=1$，因此，$$等式左边=\\int_z q(z|x;\\varphi)log \\; p(x;\\theta)\\; dz=log \\; p(x;\\theta) $$\n",
    "$$\\begin{align} \n",
    "log \\; p(x;\\theta) &= \\int_z q(z|x;\\varphi)log \\frac{p(x,z;\\theta)}{q(z|x;\\varphi)} \\; dz - \n",
    "\\int_z q(z|x;\\varphi)log \\frac{p(z|x;\\theta)}{q(z|x;\\varphi)} \\; dz  \\\\\n",
    "&=E_{q(z|x;\\varphi)} \\left[log \\frac{p(x,z;\\theta)}{q(z|x;\\varphi)} \\right] + KL\\left(q(z|x;\\varphi) || p(z|x;\\theta) \\right) \\\\\n",
    "&=ELBO(q,x;\\theta,\\varphi)+KL\\left(q(z|x;\\varphi) || p(z|x;\\theta) \\right) \\tag{6}\n",
    "\\end{align} $$ \\\n",
    "&emsp;&emsp;其中，$ELBO(q,x;\\theta,\\varphi)=E_{q(z|x;\\varphi)} \\left[log \\frac{p(x,z;\\theta)}{q(z|x;\\varphi)} \\right]$称为证据下界，又叫做变分下界。$KL$散度大于等于0，$log\\;p(x;\\theta)$对数似然函数也叫做证据，因此$ELBO$叫做证据下界。固定$\\theta$，已知样本时，似然函数是一个定值，因此如果我们不断的提高证据下界，相当于间接的减小了两个分布之间的$KL$散度。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### VAE的基本思想"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "&emsp;&emsp;既然变分分布$q(z|x;\\varphi)$,和生成分布$p(x|z;\\theta)$都是事先人为假设的，也就是说需要专家知识，那么我们可不可以通过神经网络强大的拟合能力直接拟合这两个分布呢，答案是可以的，这就是VAE的基本思想。\\\n",
    "&emsp;&emsp;VAE由两部分组成，一部分是推断网络（Encoder）$f_e(x;\\varphi)$，用来来逼近变分分布$q(z|x;\\varphi)$，另一部分是生成网络（Decoder）$f_d(z;\\theta)$，用来来逼近生成分布$p(x|z;\\theta)$，因为酷似于编码器解码器，比如，你可以将推断网络看做一个概率编码器，最终将数据编码成一定维度的格式，将信息编码至隐变量$z$里，将生成网络看做一个概率解码器，用于恢复原始数据，所以这就是VAE名字的由来，变分自编码器。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## VAE的学习问题"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "&emsp;&emsp;我们会用神经网络来逼近两个分布，即用推断网络（Encoder）$f_e(x;\\varphi)$来逼近$q(z|x;\\varphi)$，用生成网络（Decoder）$f_d(z;\\theta)$来逼近$p(x|z;\\theta)$。理论上，这两个网络都可以逼近任意分布，可以是非参数化的分布，但为了模型的可解释性以及推导，我们假定两个网络都服从同一类分布，比如指数族分布，本文选择高斯分布，因为$x$是$D$维的连续变量。\\\n",
    "&emsp;&emsp;为防止混乱，我们先进行分布的假设：\n",
    "$$p(z;\\theta)=\\mathcal{N}(z;0,I)\\;,\\;q(z|x;\\varphi)=\\mathcal{N}(z;\\mu_e,\\sigma_e^2I)\\;,\\;p(x|z;\\theta)=\\mathcal{N}(x;\\mu_d,\\lambda I) \\tag{7}$$\n",
    "&emsp;&emsp;其中，$(\\mu_e,\\sigma_e)$分别为推断网络$f_e(x;\\varphi)$的输出；$\\mu_d$是生成网络$f_d(z;\\theta)$的输出；$\\lambda$是事先指定的超参数，是一个标量，用于控制方差的大小。请注意，这里的$\\sigma_e^2I$不是矩阵乘法，$\\sigma_e^2$不是标量，而是$J$维向量，$\\sigma_e^2I$是一个协方差对角矩阵，$\\sigma_e^2$是该矩阵对角元素的向量，所以$\\sigma_e^2 I = diag(\\sigma_e^2)$。因此，在我们的假定下，推断网络服从一个各向异性的高斯分布，而生成网络服从各项同性的高斯分布。\\\n",
    "&emsp;&emsp;生成网络是依据$x$的类型假设的，如果$x \\in \\{0,1\\}^D$是$D$维的二值变量，可以假设$p(x|z;\\theta)$服从多变量的伯努利分布，如果$x \\in \\mathbb R^D$是$D$维的连续变量，可以假设$p(x|z;\\theta)$是多维高斯分布。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 推断网络的目标"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "&emsp;&emsp;在公式（7）的假设下，推断网络$f_e(x;\\varphi)$的输出是参数化的，即$f_e(x;\\varphi) \\rightarrow (\\mu_e,\\sigma_e)$，推断网络的隐藏层可以是全连接层或卷积层，输出层是线性层，不用加激活函数，这是为了保证均值有正有负，而方差是非负的，因此可以让线性层学习$log\\sigma_e$，这样就保证了方差是非负的。\\\n",
    "&emsp;&emsp;推断网络的目标是使得$q(z|x;\\varphi)$尽可能的接近真实的后验$p(z|x;\\theta)$，需要找到一组参数$\\varphi^*$来最小化两个分布的$KL$散度,即\n",
    "$$\\varphi^*=\\mathop{\\arg\\min}\\limits_{\\varphi}KL\\left(q(z|x;\\varphi)||p(z|x;\\theta) \\right) \\tag{8}$$\n",
    "&emsp;&emsp;但是上面提到过，$p(z|x;\\theta)$几乎是不可解的，传统的方法是采样法或变分推断，但是采样的效率很低，速度很慢，也并没有那么准确，一般是采用变分推断，即用一个简单的分布$q$去近似复杂的分布$p(z|x;\\theta)$，但是遇到$p(z|x;\\theta)$非常复杂的时候，就很难用简单分布去近似，因此采用间接的手段，根据式（6）的推导，推断网络的目标函数可以转换为：\n",
    "$$\\begin{align} \\varphi^* &=\\mathop{\\arg\\min}\\limits_{\\varphi}KL\\left(q(z|x;\\varphi)||p(z|x;\\theta)\\right)\\\\\n",
    "&=\\mathop{\\arg\\min}\\limits_{\\varphi}log \\; p(x;\\theta) - ELBO(q,x;\\theta,\\varphi) \\\\\n",
    "&=\\mathop{\\arg\\max}\\limits_{\\varphi}ELBO(q,x;\\theta,\\varphi)\\tag{9}\\end{align}$$\n",
    "&emsp;&emsp;即推断网络的目标转换为寻找一组参数$\\varphi^*$使得证据下界$ELBO(q,x;\\theta,\\varphi)$最大化，这和变分推断的转换类似。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 生成网络的目标"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "&emsp;&emsp;生成网络的目标是找到一组参数$\\theta^*$最大化证据下界$ELBO(q,x;\\theta,\\varphi)$,即：\n",
    "$$\\theta^*=\\mathop{\\arg\\max}\\limits_{\\theta}ELBO(q,x;\\theta,\\varphi)\\tag{10}$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 整体目标的推导"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "&emsp;&emsp;我们终于去掉了$p(z|x;\\theta)$这个大麻烦，$p(z|x;\\theta)$并不在证据下界里。推断网络和生成网络的目标是一致的，都是最大化证据下界，因此我们可以保证VAE训练的收敛性，为方便我们分析和优化，我们进一步推导VAE的总目标函数（也即原文献2.3节中的SGVB估计方法的B方法）：\n",
    "$$\\begin{align}\n",
    "\\max_{\\theta,\\varphi}ELBO(q,x;\\theta,\\varphi)&=\\max_{\\theta,\\varphi}E_{q(z|x;\\varphi)} \\left[log \\frac{p(x,z;\\theta)}{q(z|x;\\varphi)}\\right]\\\\\n",
    "&=\\max_{\\theta,\\varphi}E_{q(z|x;\\varphi)} \\left[log \\frac{p(x|z;\\theta)p(z;\\theta)}{q(z|x;\\varphi)}\\right]\\\\\n",
    "&=\\max_{\\theta,\\varphi}E_{q(z|x;\\varphi)} \\left[log \\; p(x|z;\\theta) +log \\frac{p(z;\\theta)}{q(z|x;\\varphi)}\\right]\\\\\n",
    "&=\\max_{\\theta,\\varphi}E_{q(z|x;\\varphi)} \\left[log \\; p(x|z;\\theta) \\right] - KL\\left(q(z|x;\\varphi)||p(z;\\theta)\\right) \\tag{11}\n",
    "\\end{align}$$\n",
    "&emsp;&emsp;其中，$p(z;\\theta)$为隐变量$z$的先验分布。\\\n",
    "&emsp;&emsp;我们分别来看看公式中的两项。\\\n",
    "&emsp;&emsp;（1）公式中的第一项期望，通常使用采样的方式来近似计算，对于每个样本$x$，从$q(z|x;\\varphi)$中采集L个$z^{(l)}$，就有\n",
    "$$E_{q(z|x;\\varphi)} \\left[log\\;p(x|z;\\theta) \\right] \\approx \\frac{1}{L} \\sum_{l=1}^L log \\; p(x|z^{(l)};\\theta) \\tag{12}$$\n",
    "&emsp;&emsp;这样的处理是没有问题的，但是会发现这个期望与参数$\\varphi$无关了，原本是确定性关系，但近似后变成了随机采样的关系，因此我们不能使用随机梯度下降来学习这个参数了，原文献中使用了重参数化技巧，让$z$的随机性来源于另一个简单的随机变量$\\epsilon$，这样就可以使用随机梯度下降算法了，重参数化技巧后面会写。\\\n",
    "&emsp;&emsp;（2）公式第二项的$KL$散度通常可以直接计算，因为这两个分布通常比较相似，一个是隐变量$z$的先验，另一个是后验，可以假定属于同一个类型的指数族分布，比如高斯分布，我们以高斯分布为例。\\\n",
    "&emsp;&emsp;给定$J$维空间中的两个高斯分布$\\mathcal{N}(\\mu_1,\\Sigma_1)$和$\\mathcal{N}(\\mu_2,\\Sigma_2)$，其$KL$散度为\n",
    "$$\\begin{align} &KL\\left(\\mathcal{N}(\\mu_1,\\Sigma_1)||\\mathcal{N}(\\mu_2,\\Sigma_2)\\right)\\\\\n",
    "=&\\frac{1}{2}\\left(tr(\\Sigma_2^{-1}\\Sigma_1)+(\\mu_2-\\mu_1)^T\\Sigma_2^{-1}(\\mu_2-\\mu_1)-J+log \\frac{|\\Sigma_2|}{|\\Sigma_1|}\\right)\\tag{13}\\end{align}$$\n",
    "&emsp;&emsp;这样，当我们假定先验后验均为高斯分布时，我们就能直接计算出$KL$散度。具体的，我们上面假定了$p(z;\\theta)=\\mathcal{N}(z;0,I),q(z|x;\\varphi)=\\mathcal{N}(z;\\mu_e,\\sigma_e^2I)$，其中$\\mu_e,\\sigma_e$分别为推断网络$f_e(x;\\varphi)$的输出，再次强调，这里的$\\sigma_e^2I$不是矩阵乘法，$\\sigma_e^2$是$J$维向量，$\\sigma_e^2I$是一个协方差对角矩阵，$\\sigma_e^2$是该矩阵对角元素的向量。在这样的假定下，可以很容易的得到两个分布的$KL$散度，即\n",
    "$$\\begin{align}&KL\\left(q(z|x;\\varphi)||p(z;\\theta) \\right) \\\\\n",
    "=&\\frac{1}{2} \\left(tr(\\sigma_e^2I)+\\mu_e^T\\mu_e-J-log(|\\sigma_e^2I|)\\right)\\tag{14}\\end{align}$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 重参数化技巧"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "&emsp;&emsp;我们对优化目标的第一项进行采样近似，导致参数消失了，变成了随机采样的关系，而不是确定的关系了，因此我们需要将参数复原出来，这就是重参数化技巧。\\\n",
    "&emsp;&emsp;我们观察$E_{q(z|x;\\varphi)} \\left[log p(x|z;\\theta) \\right]$，这个期望的计算依赖于分布$q$的参数$\\varphi$，但是由于随机变量$z$采样自后验分布$q(z|x;\\varphi)$,因此他们之间不再是确定性的关系，我们想办法把$z$的随机性转入其他随机变量，找到一个$z$关于参数$\\varphi$的确定性关系，再去不断采样这个随机变量就大功告成了，等于变相的采样$z$。\\\n",
    "&emsp;&emsp;我们引入一个随机变量$\\epsilon$，他服从一个简单分布$p(\\epsilon)$，这样,$E_{q(z|x;\\varphi)} \\left[log p(x|z;\\theta) \\right]$可以重写为\n",
    "$$E_{q(z|x;\\varphi)} \\left[log p(x|z;\\theta) \\right]=E_{p(\\epsilon)}\\left[log \\; p(x|g(\\varphi,\\epsilon);\\theta)\\right]\\tag{15}$$\n",
    "&emsp;&emsp;其中，$z=g(\\varphi,\\epsilon)$为一个确定性的函数。\\\n",
    "&emsp;&emsp;具体的，我们假设$q(z|x;\\varphi)$为高斯分布$\\mathcal{N}(\\mu_e,\\sigma_e^2)$，其中$(\\mu_e,\\sigma_e)$是推断网络$f_e(x;\\varphi)$的输出，推断网络必须依赖于参数$\\varphi$，这样才能进行梯度下降算法，我们通过以下方式进行重参数化：\n",
    "$$z=\\mu_e+\\sigma_e \\odot \\epsilon \\tag{16}$$\n",
    "&emsp;&emsp;其中$\\epsilon \\sim \\mathcal{N}(0,I)$，$\\odot$代表逐元素点积，如$[x \\odot y]_i=x_iy_i$。非常棒，这样我们就把$z$和参数$\\varphi$从采样关系变为确定性关系了。\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## VAE的训练"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "&emsp;&emsp;我们在上一节推导了VAE的优化目标，即最大化证据下界，但是在训练过程中，我们更喜欢最小化损失函数，用损失函数来分析模型，并作为模型的衡量标准。因此在本节中，我们先制定优化算法，这一步骤对应于原文献中的2.3节。接下来，依据我们公式（7）的高斯分布假设，对优化目标进一步的推导，使其更加具体化，这一步骤对应于原文献中的第3大节。最后，我们会在原文献的基础上，将优化目标转换为最小化损失函数，并进一步推导，最终解释VAE在训练过程中的真正目标。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### AEVB优化算法"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "&emsp;&emsp;现有一个数据集$X=\\{x^{(i)}\\}_{n=1}^N$， AEVB算法的思想是，每轮迭代，选择$M$个样本训练，计算证据下界的平均值，代表每个样本计算的证据下界，最终乘以样本量$N$，代表所有样本计算的证据下界，也就是说，每一轮迭代，模型并没有按照批次训练所有的样本，而是仅仅随机抽样了$M$个样本进行学习。\\\n",
    "&emsp;&emsp;根据公式（11）和（12），对于每一个样本$x^{(i)}$，VAE的优化目标为：\n",
    "$$\\begin{align}\n",
    "\\mathcal{L}(\\theta,\\varphi|x^{(i)}) &=ELBO(q,x^{(i)};\\theta,\\varphi) \\\\\n",
    "&\\approx \\frac{1}{L}\\sum_{l=1}^L log\\;p(x^{(i)}|z^{(i,l)};\\theta) - KL\\left(q(z|x^{(i)};\\varphi)||p(z;\\theta)\\right) \\tag{17}\n",
    "\\end{align}$$\n",
    "&emsp;&emsp;对于所有样本$X$，VAE的优化目标为：\n",
    "$$\\mathcal L (\\theta,\\varphi|X) \\approx \\mathcal L(\\theta,\\varphi|X^M)=\\frac{N}{M}\\sum_{i=1}^M \\mathcal L(\\theta,\\varphi|x^{(i)}) \\tag{18}\n",
    "$$\n",
    "&emsp;&emsp;其中，$X^M=\\{x^{(i)}\\}_{i=1}^M$是从所有数据集中随机抽样$M$个。\\\n",
    "&emsp;&emsp;在原文献中，$M=100\\;,\\;L=1$，也就是说，每次迭代训练只抽样100个样本进行训练，从高斯噪声中只采样一个值计算隐变量$z$，采样一个值直接当做期望计算，而不是用平均值。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 优化目标"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "&emsp;&emsp;为方便推导，我们对一个样本$x$不加上角标，根据公式（17），VAE的优化目标为：\n",
    "$$\\begin{align}\n",
    "\\max_{\\theta,\\varphi}\\mathcal{L}(\\theta,\\varphi|x) \\approx \\frac{1}{L}\\sum_{l=1}^L log\\;p(x|z^{(l)};\\theta) - KL\\left(q(z|x;\\varphi)||p(z;\\theta)\\right) \\tag{19}\n",
    "\\end{align}$$\n",
    "&emsp;&emsp;依据公式（14）以及公式（7）的假定，有\n",
    "$$\\begin{align}\n",
    "- KL\\left(q(z|x;\\varphi)||p(z;\\theta)\\right) &= -\\frac{1}{2} \\left(tr(\\sigma^2I)+\\mu^T\\mu-J-log(|\\sigma^2I|)\\right) \\\\\n",
    "&=\\frac{1}{2}\\left(J+log(\\prod_{j=1}^J \\sigma_j^2)- \\sum_{j=1}^J\\sigma_j^2-\\sum_{j=1}^J\\mu_j^2 \\right)\\\\\n",
    "&=\\frac{1}{2}\\sum_{j=1}^J(1+log\\sigma_j^2-\\sigma_j^2-\\mu_j^2) \\tag{20}\n",
    "\\end{align}$$\n",
    "\n",
    "&emsp;&emsp;带入式（19）得\n",
    "$$\\mathcal{L}(\\theta,\\varphi|x) \\approx \\frac{1}{L}\\sum_{l=1}^L log\\;p(x|z^{(l)};\\theta) + \\frac{1}{2}\\sum_{j=1}^J(1+log\\sigma_j^2-\\sigma_j^2-\\mu_j^2))\\tag{21}$$\n",
    "&emsp;&emsp;考虑数据集$X=\\{x^{(i)}\\}_{n=1}^N$，对于一个样本$x^{(i)}$，随机采样$L$个变量$\\epsilon^{(n,l)},1 \\le l \\le L$，并通过式（16）计算$z^{(n,j)}$，所以VAE的目标函数为\n",
    "$$\\max_{\\theta,\\varphi}\\mathcal{L}(\\theta,\\varphi|x^{(i)}) \\approx \\frac{1}{L}\\sum_{l=1}^L log\\;p(x^{(i)}|z^{(i,l)};\\theta) + \\frac{1}{2}\\sum_{j=1}^J \\left( 1+log(\\sigma_j^{(i)})^2-(\\sigma_j^{(i)})^2-(\\mu_j^{(i)})^2 \\right)\\tag{22} $$\n",
    "&emsp;&emsp;其中$z^{(i,l)}=\\mu^{(i)}+ \\sigma^{(i)} \\odot \\epsilon^{(i)} ，\\epsilon^{(i)} \\sim \\mathcal{N}(0,I)$。$(\\mu_j^{(i)},\\sigma_j^{(i)})$代表$(\\mu^{(i)},\\sigma^{(i)})$的第$j$个维度，而$(\\mu^{(i)},\\sigma^{(i)})$依赖于参数$\\varphi$，是推断网络$f_e(x^{(i)};\\varphi)$的输出。式（22）对应于原文献中的公式（10）。\\\n",
    "&emsp;&emsp;在实际训练中，我们更喜欢用最小化损失函数作为我们优化的目标，而最大化似然函数等价于最小化损失函数，因此我们定义损失函数$\\mathcal{J}(\\theta,\\varphi|x^{(i)})= - \\mathcal{L}(\\theta,\\varphi|x^{(i)})$，则优化目标为：\n",
    "$$\\begin{align}\n",
    "\\min_{\\theta,\\varphi} \\mathcal{J}(\\theta,\\varphi|x^{(i)}) &= - \\mathcal{L}(\\theta,\\varphi|x^{(i)}) \\\\\n",
    "&\\approx - \\frac{1}{L}\\sum_{l=1}^L log\\;p(x^{(i)}|z^{(i,l)};\\theta) + \\frac{1}{2}\\sum_{j=1}^J \\left((\\mu_j^{(i)})^2+ (\\sigma_j^{(i)})^2  - log(\\sigma_j^{(i)})^2 - 1 \\right) \\tag{23} \n",
    "\\end{align}$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 优化目标的进一步解释"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "&emsp;&emsp;为方便推导，我们对一个样本$x$不加上角标，引用式（19），则优化目标为：\n",
    "$$\\begin{align}\n",
    "\\min_{\\theta,\\varphi} \\mathcal{J}(\\theta,\\varphi|x) &= \\min_{\\theta,\\varphi} - \\mathcal{L}(\\theta,\\varphi|x) \\\\\n",
    "&\\approx \\min_{\\theta,\\varphi} - \\frac{1}{L}\\sum_{l=1}^L log\\;p(x|z^{(l)};\\theta) + KL\\left(q(z|x;\\varphi)||p(z;\\theta)\\right) \\\\\n",
    "&= \\min_{\\theta,\\varphi} - \\frac{1}{L} \\sum_{l=1}^L log \\mathcal N (x;\\mu_d^{(l)},\\lambda I) + KL\\left(q(z|x;\\varphi)||p(z;\\theta)\\right)\n",
    "\\quad [公式（7）的高斯分布假设]\\\\\n",
    "&= \\min_{\\theta,\\varphi} - \\frac{1}{L} \\sum_{l=1}^L log\\left(\\frac{1}{\\sqrt{(2\\pi)^J|\\lambda I|}} exp \\left(-\\frac{1}{2}(x-\\mu_d^{(l)})^T (\\lambda I)^{-1} (x-\\mu_d^{(l)})\\right) \\right) + KL\\left(q(z|x;\\varphi)||p(z;\\theta)\\right) \\\\\n",
    "&= \\min_{\\theta,\\varphi} - logC - \\frac{1}{L} \\sum_{l=1}^L log\\;exp\\left(-\\frac{1}{2}(x-\\mu_d^{(l)})^T \\frac{1}{\\lambda} I (x-\\mu_d^{(l)}) \\right) + KL\\left(q(z|x;\\varphi)||p(z;\\theta)\\right) \\\\\n",
    "&= \\min_{\\theta,\\varphi} \\frac{1}{L} \\sum_{l=1}^L \\frac{1}{2\\lambda}(x-\\mu_d^{(l)})^T (x-\\mu_d^{(l)})+ KL\\left(q(z|x;\\varphi)||p(z;\\theta)\\right)\\\\\n",
    "&= \\min_{\\theta,\\varphi} \\frac{1}{L} \\sum_{l=1}^L \\frac{1}{2}\\|x-\\mu_d^{(l)}\\|_2^2+ \\lambda \\; KL\\left(q(z|x;\\varphi)||p(z;\\theta)\\right)\\\\\n",
    "&= \\min_{\\theta,\\varphi} \\frac{1}{2}\\|x-\\mu_d\\|_2^2+ \\lambda \\; KL\\left(q(z|x;\\varphi)||p(z;\\theta)\\right) \\quad if \\;L=1\n",
    "\\tag{24}\\end{align}$$\n",
    "&emsp;&emsp;太优美了！！！我们推导到这一步基本就可以看出端倪了，如果取$L$=1（原文献是这样取得），首先看第一项，$\\mu_d$是生成网络（Decoder）$f_d(z;\\theta)$的输出！也就是说，这一部分相当于VAE的重构损失，如果取$L$>1，实际上是VAE的平均重构损失。再来看第二项，如果把第二项看做一个正则化项，那会非常有趣，VAE在尽力重构数据的同时，也在考虑隐变量$z$的变分分布与真实先验分布的$KL$散度，变分分布是近似的真实后验，也就是说，这一项可以看做隐变量$z$的近似后验分布与真实先验分布的$KL$散度，还记得$\\lambda$是什么吗，我们回看一下公式（7）的假设，我们将真实的生成分布$p(x|z;\\theta)$假设为各向同性的高斯分布，$\\lambda$用于控制高斯分布的方差，经过我们的推导，这个$\\lambda$其实是损失函数的正则化系数！因此VAE的损失函数由两部分构成，一个是数据的重构损失，另一个是VAE的潜在损失，这也是VAE显著不同于普通自编码器的原因，因为普通的自编码器，是一个确定性的编码，只是在尽力的重构数据，没有正则化项，而VAE是概率编码器，并且VAE是生成模型，意味着我们随机生成噪音，经过VAE的解码器后，他会自己生成一个手写数字体！"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## VAE网络计算图"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "&emsp;&emsp;至此，我们终于搞懂了VAE的理论，用一个词来形容那就是优美，我们依据理论来搭建网络计算图，并对该经典论文进行复现。"
   ]
  },
  {
   "attachments": {
    "VAE%E8%AE%A1%E7%AE%97%E5%9B%BE%20%281%29.jpg": {
     "image/jpeg": "iVBORw0KGgoAAAANSUhEUgAAA9MAAAK6CAYAAAA6gQ9jAAAAAXNSR0IArs4c6QAAIABJREFUeF7s3QeYFdX9//HvsoAF7CBg7ATFJEbFhiUiNvwL8YeKChawgBQrCnajxooIigV7Q4yKGonYxRIrUVmMmigkFgRBFEFRQBZ29/98x9zr3bv33pm5d8o5M+95nn1IYObM97zOuet+dmbOVNXW1jYIGwIIIIAAAggggAACCCCAAAIIeBaoIkx7tmJHBBBAAAEEEEAAAQQQQAABBBwBwjQTAQEEEEAAAQQQQAABBBBAAAGfAoRpn2DsjgACCCCAAAIIIIAAAggggABhmjmAAAIIIIAAAggggAACCCCAgE8BwrRPMHZHAAEEEEAAAQQQQAABBBBAgDDNHEAAAQQQQAABBBBAAAEEEEDApwBh2icYuyOAAAIIIIAAAggggAACCCBAmGYOIIAAAggggAACCCCAAAIIIOBTgDDtE4zdEUAAAQQQQAABBBBAAAEEECBMMwcQQAABBBBAAAEEEEAAAQQQ8ClAmPYJxu4IIIAAAggggAACCCCAAAIIEKaZAwgggAACCCCAAAIIIIAAAgj4FCBM+wRjdwQQQAABBBBAAAEEEEAAAQQI08wBBBBAAAEEEEAAAQQQQAABBHwKEKZ9grE7AggggAACCCCAAAIIIIAAAoRp5gACCCCAAAIIIIAAAggggAACPgUI0z7B2B0BBBBAAAEEEEAAAQQQQAABwjRzAAEEEEAAAQQQQAABBBBAAAGfAoRpn2DsjgACCCCAAAIIIIAAAggggABhmjmAAAIIIIAAAggggAACCCCAgE8BwrRPMHZHAAEEEEAAAQQQQAABBBBAgDDNHEAAAQQQQAABBBBAAAEEEEDApwBh2icYuyOAAAIIIIAAAggggAACCCBAmGYOIIAAAggggAACCCCAAAIIIOBTgDDtE4zdEUAAAQQQQAABBBBAAAEEECBMMwcQQAABBBBAAAEEEEAAAQQQ8ClAmPYJxu4IIIAAAggggAACCCCAAAIIEKaZAwgggAACCCCAAAIIIIAAAgj4FCBM+wRjdwQQQAABBBBAAAEEEEAAAQQI08wBBBBAAAEEEEAAAQQQQAABBHwKEKZ9grE7AggggAACCCCAAAIIIIAAAoRp5gACCCCAAAIIIIAAAggggAACPgUI0z7B2B0BBBBAAAEEEEAAAQQQQAABwjRzAAEEEEAAAQQQQAABBBBAAAGfAoRpn2DsjgACCCCAAAIIIIAAAggggABhmjmAQIUCCxculMmTJ4v+mf81bdq0ClsP//Cnn35a3nrrLZk7d67MmTNHfvzxx/BPyhkQQMAogV/96ley8cYbyxZbbCH9+vWTdu3aGVUfxSCAAAIIIGCiAGHaxFGhJisEMiE6E6QLFW1ymJ41a5bccsstTpBmQwABBDICLVq0kCFDhsjRRx8NCgIIIIAAAgiUECBMMz0QKEPgzjvvzF6NLnW4qWF65cqV0qdPH1mwYEEZvecQBBBIg8Cpp55KoE7DQNNHBBBAAIGyBQjTZdNxYFoFNEjrV+7WunVrOeCAA2TLLbeUtm3bNvoy0enGG2+UBx54IFvaku7dZdm228qqNm2kbq21TCyZmhBAICSBqro6ab5wofO17nPPScs5c5wz6RXqRx99lFu+Q3KnWQQQQAAB+wUI0/aPIT2IUCA/SHfs2FF69OjhBOn27dtHWEn5p/ruu+/kwAMPzDaw8KijZOnOO5ffIEcigECiBDqMHZsN1H379pUzzjgjUf2jMwgggAACCAQlQJgOSpJ2Ei+QH6T33XdfueKKK6zr9/vvvy8nnXSSU3ftJpvI/DPPtK4PFIwAAuEJrPGvf8mG/7v7Ztddd5Vx48aFdzJaRgABBBBAwGIBwrTFg0fp0QnoYmPHHXecs1q3brYGaa39qaeekssuu8zpx9IddpCF/ftHB8mZEEDAeIEWCxbIRldf7dSpq3w/9thjxtdMgQgggAACCMQhQJiOQ51zWieQe1Vab+3Ofd7Yts7k9uX7/feX7w46yLYuUC8CCIQooM9QbzpiRPYMpi6kGCIBTSOAAAIIIOBJgDDtiYmd0iyQf1V62LBh0t/iq7m5Yfq7Aw+U73v0SPPw0ncEECggsNnw4YRpZgYCCCCAAAIuAoRppggCLgK54VNX7Z44caI1i40V6hphmimPAAJuAoRpNyH+HQEEEEAAARHCNLMAAReB0aNHZ58ZPPTQQ+Xss8+22owwbfXwUTwCkQgQpiNh5iQIIIAAApYLEKYtH0DKD1/gggsukBdffNE50YgRI6RPnz7hnzTEMxCmQ8SlaQQSIkCYTshA0g0EEEAAgVAFCNOh8tJ4EgSGDBki7733ntOVUaNGSbdu3azuFmHa6uGjeAQiESBMR8LMSRBAAAEELBcgTFs+gJQfvsDhhx8uc+bMcU50//33S6dOncI/aYhnIEyHiEvTCCREgDCdkIGkGwgggAACoQoQpkPlpfEkCOyzzz6ybNkypytTp04VXYTM5o0wbfPoUTsC0QgQpqNx5iwIIIAAAnYLEKbtHj+qj0Cga9eu2bMk4X2rhOkIJg2nQMByAcK05QNI+QgggAACkQgQpiNh5iQ2CxCmbR49akcAgXIECNPlqHEMAggggEDaBAjTaRtx+utbgDDtm4wDEEDAcgHCtOUDSPkIIIAAApEIEKYjYeYkNgsQpm0ePWpHAIFyBAjT5ahxDAIIIIBA2gQI02kbcfrrW4Aw7ZuMAxBAwHIBwrTlA0j5CCCAAAKRCBCmI2HmJDYLEKZtHj1qRwCBcgQI0+WocQwCCCCAQNoECNNpG3H661uAMO2bjAMQQMByAcK05QNI+QgggAACkQgQpiNh5iQ2CxCmbR49akcAgXIECNPlqHEMAggggEDaBAjTaRtx+utbgDDtm4wDEEDAcgHCtOUDSPkIIIAAApEIEKYjYeYkNgsQpm0ePWpHAIFyBAjT5ahxDAIIIIBA2gQI02kbcfrrW4Aw7ZuMA0IQ0HAz+7rriracCT+l9gmhLJpMqABhOqEDS7cQQAABBAIVIEwHykljSRQgTCdxVO3rU6kwnRt8yg3TbmHdPjEqrkSAMF2JHscigAACCKRFgDCdlpGmn2ULEKbLpuPAgAS8BOlMiHbbt1jYruS4SrpZbviv5Jwc6y5AmHY3Yg8EEEAAAQQI08wBBFwECNNMkbgEcgNNbg25wVn/PjeQlrrdu5LAXE4Id3PjaribUHz/TpiOz54zI4AAAgjYI0CYtmesqDQmAcJ0TPCcNitQKHS6heb8kK3/nzDNpPIqQJj2KsV+CCCAAAJpFiBMp3n06bsnAcK0JyZ2CknALUgXu3qdKSf/qnU5V5jLDeFuJFyZdhOK798J0/HZc2YEEEAAAXsECNP2jBWVxiRAmI4JntM6ApnAmfun/r2XZ43zr16XG4rdjqtkqLz0o5L2ObY8AcJ0eW4chQACCCCQLgHCdLrGm96WIUCYLgONQwITyA/TlTTsFoqDvmrtVitXpt2E4vt3wnR89pwZAQQQQMAeAcK0PWNFpTEJEKZjgue0jZ5xzg+e+bd3FwrCbsf4IS4naPtpn33NEiBMmzUeVIMAAgggYKYAYdrMcaEqgwQI0wYNRspKyQ3DhYJxoddhlTqmFF8QV63dnt8udH5u8zZzUhOmzRwXqkIAAQQQMEuAMG3WeFCNgQKEaQMHJSUlFQunGkCLheYwwnS5ITwlw5TIbhKmEzmsdAoBBBBAIGABwnTAoDSXPAHCdPLG1MYeud2y7XaVWvtc7tXnUsfy3LONs8m9ZsK0uxF7IIAAAgggQJhmDiDgIkCYZoqYIBDEbd6EaRNG0o4aCNN2jBNVIoAAAgjEK0CYjtefs1sgQJi2YJBSUGKhW77drkaXCuD5ZG5XmIv9eznPSeefm+emzZvAhGnzxoSKEEAAAQTMEyBMmzcmVGSYAGHasAFJUTmlVuwu9m9eFy0LIkxnaii2krieI/ff3AJ7iobW+K4Spo0fIgpEAAEEEDBAgDBtwCBQgtkChGmzxyct1Zl4m7fb1er8kF0qfKdlHG3pJ2HalpGiTgQQQACBOAUI03Hqc24rBAjTVgxT4ov0GqZzIcK+zbvUomil3kudf8U68YNnYQcJ0xYOGiUjgAACCEQuQJiOnJwT2iZAmLZtxJJZb1BhuhKd/Fu2c0Nxuc9O87x0JSMS3rGE6fBsaRkBBBBAIDkChOnkjCU9CUmAMB0SLM36EvDzaqzchoMKq36ucnvtGM9Qe5WKfj/CdPTmnBEBBBBAwD4BwrR9Y0bFEQsQpiMG53QIIBC7AGE69iGgAAQQQAABCwQI0xYMEiXGK0CYjtefsyOAQPQChOnozTkjAggggIB9AoRp+8aMiiMWIExHDM7pEEAgdgHCdOxDQAEIIIAAAhYIEKYtGCRKjFeAMB2vP2dHAIHoBQjT0ZtzRgQQQAAB+wQI0/aNGRVHLECYjhic0yGAQOwChOnYh4ACEEAAAQQsECBMWzBIlBivAGE6Xn/OjgAC0QsQpqM354wIIIAAAvYJEKbtGzMqjliAMB0xOKdDAIHYBQjTsQ8BBSCAAAIIWCBAmLZgkCgxXgHCdLz+nB0BBKIXIExHb84ZEUAAAQTsEyBM2zdmVByxAGE6YnBOhwACsQsQpmMfAgpAAAEEELBAgDBtwSBRYrwChOl4/Tk7AghEL0CYjt6cMyKAAAII2CdAmLZvzKg4YgHCdMTgnA4BBGIXIEzHPgQUgAACCCBggQBh2oJBosR4BQjT8fpzdgQQiF6AMB29OWdEAAEEELBPgDBt35hRccQChOmIwTkdAgjELkCYjn0IKAABBBBAwAIBwrQFg0SJ8QoQpuP15+wIIBC9AGE6enPOiAACCCBgnwBh2r4xo+KIBQjTEYNzOgQQiF2AMB37EFAAAggggIAFAoRpCwaJEuMVIEzH68/ZEUAgegHCdPTmnBEBBBBAwD4BwrR9Y0bFEQskLUw/+OCDMm7cOEfxhz32kEV9+kQsyukQQMBkgeoffpCN//Qnp8TWrVvL1KlTTS6X2hBAAAEEEIhNgDAdGz0ntkUgaWH6tddek5EjRzr8P229tSwYMsSWoaBOBBCIQGC1zz6T9jfc4Jxpm222kXvuuSeCs3IKBBBAAAEE7BMgTNs3ZlQcsUDSwvTnn38uffv2dRQbmjeXeRdcIKvWXTdiVU5nmkBVfb00NGtmWlnUE4PAelOmyNovveScef/995fLLrsshio4JQIIIIAAAuYLEKbNHyMqjFkgaWFaOY877jj5+OOPHdkfd95ZvjvoIKkjUMc80+I9fauaGlnapUu8RXD22AVavfOOtPnLX7J1/OlPf5KDDjoo9rooAAEEEEAAARMFCNMmjgo1GSWQxDD91ltvyfDhw7POeoV6RceOsrJNG6lbay2j/CkmGoFWM2bIT1tuKXXrrBPNCTmLMQJVdXXSfOFCabFwobScMydb12677SbXXXedMXVSCAIIIIAAAqYJEKZNGxHqMU4giWFaka+//np56KGHjPOmIAQQiF+gXbt2Mnr0aNlqq63iL4YKEEAAAQQQMFSAMG3owFCWOQJJDdMqrFeob7vttuwt3+aoUwkCCMQlcPTRR8uQIUOkRYsWcZXAeRFAAAEEELBCgDBtxTBRZJwCSQ7TGVddlGzOnDkyd+5cWbp0aZzcnDsGgaefflrmz58vDQ0NzvOxG220UQxVcMo4BTp06CCbbLKJbLrpprIu6yfEORScGwEEEEDAIgHCtEWDRanxCKQhTMcjy1lNEJg9e7YceeSRTilVVVXSu3dvOeecc0wojRoQQAABBBBAAAGjBQjTRg8PxZkgQJg2YRSoISyBiRMnyk033ZRtvlWrVjJ16lQnWLMhgAACCCCAAAIIFBcgTDM7EHARIEwzRZIs0L9/f5k1a1ajLo4aNUq6deuW5G7TNwQQQAABBBBAoGIBwnTFhDSQdAHCdNJHOL39++CDD2TQoEGNAPSK9P777y9//vOf0wtDzxFAAAEEEEAAAQ8ChGkPSOySbgHCdLrHP8m915Xc77nnniZdrK6ulmeffVbW4p3jSR5++oYAAggggAACFQoQpisE5PDkCxCmkz/GaexhXV2dHH744TJv3ryC3b/oooukZ8+eaaShzwgggAACCCCAgCcBwrQnJnZKswBhOs2jn9y+v/HGG3LWWWcV7KDe6r3bbrvJ2LFjkwtAzxBAAAEEEEAAgQoFCNMVAnJ48gUI08kf4zT2UIPypEmTSnb98ccfF33/MBsCCCCAAAIIIIBAUwHCNLMCARcBwjRTJGkCixcvlr59+8r3339fsmvDhw/PvoM6aQb0BwEEEEAAAQQQqFSAMF2pIMcnWuDbb7/NPjfaunVr5/27bAjYLqCLi11yySWu3fjd734nd955p+t+7IAAAggggAACCKRRgDCdxlGnz54Fpk2bJmeccYazf8eOHeWBBx7wfCw7ImCqgAbp5557ThoaGlxLvP/++6VTp06u+7EDAggggAACCCCQNgHCdNpGnP76Epg4caLcdNNNzjF77LGHjBkzxtfx7IyAaQKzZ8+WY489Vmpraz2Vpu+hPvHEEz3ty04IIIAAAggggECaBAjTaRpt+upb4IorrpApU6Y4x/Xu3VvOPfdc321wAAImCeiiY35W6d58883loYceMqkL1IIAAggggAACCBghQJg2YhgowkQBvXKnV+VmzpzplDdw4EDniw0BmwV0UbG33nrLVxduvfVW2X777X0dw84IIIAAAggggEDSBQjTSR9h+le2wIQJE2T8+PHO8Wuvvbbo/2/fvn3Z7XEgAnELfPrpp3LUUUf5LqNfv35y+umn+z6OAxBAAAEEEEAAgSQLEKaTPLqG9S3ziild1Mv0be7cudKnT59smYQJ00eM+rwIPPXUU3LZZZd52bXRPm3bts0+7uD7YA5AAAEEEEAAAQQSKkCYNmxgc99pnF+aDSG0FKdNYfqQQw6R+fPnO92prq52rkrrat5sCNgscPnll8uTTz5ZVheuvfZa2XPPPcs6loMQQAABBBBAAIEkChCmDRvVUmE6U6qtodqGMP3MM8/IlVdeKStXrszODJ6VNuxDQjllC+jdFnrXRTnbwQcfLOeff345h3IMAggggAACCCCQSAHCtGHDWipw5gZtGwO1yWF63rx5zvPRU6dObTQjCNKGfUAopyKBm2++Wb755psmbbz99tuyaNEi5+932WUXWX/99Zvso+sF6OehefPmFdXAwQgggAACCCCAQFIECNOGjaSXwOllH8O65ZRjWt0LFiwQDRH/+Mc/nD+XLFlCkDZx4lBT6ALDhg2Tmpoa5zz6S6UuXbqEfk5OgAACCCCAAAII2C5AmDZsBL0ETrcr1Pm3ihe7iu11v9wgnMtV6up4oba9XnXXc+S3nXusW/8zNWo4/v77752QnPnK/H99NdC//vWvgqPfuXNn553S+sWGQBoECNNpGGX6iAACCCCAAAJBCxCmgxatsD0vYTo33OaGTj+LlxXbt1BAdnuO22sNGRqv+xfaT//Oa5h2qzt/qAjRFU5eDrdWgDBt7dBROAIIIIAAAgjEKECYjhG/0KmDCNPFwmrm74uFUf37YleEtdZSV4szfSlWf6lz5rddaF+vATrX1E+Y1nfv6lebNm0MmxGUg0D4AoTp8I05AwIIIIAAAggkT4AwbdiYlhumvdxC7eeqrrJ4CbCFbr92u608P9SXuhruZd9iQ+gnTGsbGqQzt3cTqg37YFBOqAKE6VB5aRwBBBBAAAEEEipAmDZsYCsN06W6kxumvawG7qWWIMK0W825wd5L3Zn2Sj0z/d5778k///nPRq/AyhyXCdW6cjEbAmkQIEynYZTpIwIIIIAAAggELUCYDlq0wva8BNhC4dLLVdi0hWm3odB3Sb///vtOqNY/84M6r8VyE+TfkyJAmE7KSNIPBBBAAAEEEIhSgDAdpbaHc3kJ04X28XJcoRBeqiQvbQZxZdrL1WYvtXjgLbnLLbfcIvfdd1+jfQjUlapyvA0ChGkbRokaEUAAAQQQQMA0AcK0YSPiFhr9LORVqGtu7ece42XfJIVp7bve/j1y5Ej54YcfshQEasM+JJQTuABhOnBSGkQAAQQQQACBFAgQpg0bZC8LiWnJXhbtynQtf5XuUitum7Cat9btteawhu+QQw6R+fPnO82vu+66cvfdd8tGG20U1ulibXfhwoXy7rvvyty5c2XOnDmydOnSWOvh5NELLFq0KHvS6upqWWeddaIvgjPGKqDf3zbeeGPZbLPNZNddd421Fk6OAAIIIICALQKEacNGyuuzz4XKLnVsOe92zoTaUkSlwnex48qpxctV8iCHUoNlnz59sk3269dPTj/99CBPYURbDzzwgNx2221SW1trRD0UgQAC8QvstttuMnjwYOncuXP8xVABAggggAACBgsQpg0bHK+BuFjZhY53e1VVpq1Szy5X0q62X2rxMy9tRx2mteYJEybI+PHjHZ4WLVo4V6c7depk2Iwpv5wRI0bI66+/Xn4DHIkAAokWGD16tPzhD39IdB/pHAIIIIAAApUIEKYr0ePYRAvo1dpBgwbJzJkznX4OGDBAhg4dmog+P/HEE3LllVdm+9Jsk1ppvtVyqWqzSqrWqktEH+kEAgh4FKivkvpvmkvDwhay8q3W2YN+/etfy8SJEz02wm4IIIAAAgikT4Awnb4xp8c+BK644gqZMmWKc0SvXr3kwgsv9HG0ubv27NlTvv32W6fAlvt9Ly17fmdusVSGAAKRCdTPbSnLb2ovDSuqnHOecsopcswxx0R2fk6EAAIIIICATQKEaZtGi1ojF9CrMjfddJNzXr3V/Prrr4+8hqBPOGvWLOnfv7/TbLMNVsmaF3wp8vPPzWwIIICArPz72rJi8nqORLdu3WTUqFGoIIAAAggggEABAcI00wKBEgL6rPcZZ5zh7LHFFlvIgw8+aL3XSy+9JOeff77Tj+a/XS6rD/za+j7RAQQQCE6gfk5LWTa2g9Ngx44dRRcqZEMAAQQQQACBpgKEaWYFAiUE9FZovSVat1atWsmLL75ovVejhdW6LZHVei+2vk90AAEEghNoWN5Mlp6/idPgaqutJn//+9+Da5yWEEAAAQQQSJAAYTpBg0lXwhHIXW281Irn4Zw9+FbvvPNO0S/dWh74nbTs8X3wJ6FFBBCwWuDH4Ztl60/C9z2rB4PiEUAAAQSMFSBMGzs0FGaKAGHalJGgDgQQiEqAMB2VNOdBAAEEELBZgDBt8+hReyQChOlImDkJAggYJECYNmgwKAUBBBBAwFgBwrSxQ0NhpggQpk0ZCepAAIGoBAjTUUlzHgQQQAABmwUI0zaPHrVHIkCYjoSZkyCAgEEChGmDBoNSEEAAAQSMFSBMGzs0FGaKAGHalJGgDgQQiEqAMB2VNOdBAAEEELBZgDBt8+hReyQChOlImDkJAggYJECYNmgwKAUBBBBAwFgBwrSxQ0NhpggQpk0ZCepAAIGoBAjTUUlzHgQQQAABmwUI0zaPHrVHIkCYjoSZkyCAgEEChGmDBoNSEEAAAQSMFSBMGzs0FGaKAGHalJGgDgQQiEqAMB2VNOdBAAEEELBZgDBt8+hReyQChOlImDkJAggYJECYNmgwKAUBBBBAwFgBwrSxQ0NhpggQpk0ZCepAAIGoBAjTUUlzHgQQQAABmwUI0zaPHrVHIkCYjoSZkyCAgEEChGmDBoNSEEAAAQSMFSBMGzs0FGaKAGHalJGgDgQQiEqAMB2VNOdBAAEEELBZgDBt8+hReyQChOlImDkJAggYJECYNmgwKAUBBBBAwFgBwrSxQ0NhpggQpk0ZCepAAIGoBAjTUUlzHgQQQAABmwUI0zaPHrVHIkCYjoSZkyCAgEEChGmDBoNSEEAAAQSMFSBMGzs0FGaKAGHalJGgjnIFNBi1vm52o8ML/V257XNc8gQI08kbU3qEAAIIIBC8AGE6eFNaTJgAYTphA2pxd8oNwIRpiwc9ptIJ0zHBc1oEEEAAAasECNNWDRfFxiFAmI5DPV3nzA0uuT3PvZrsJUgXa6cczUJXsstpJ3NMfl/y28o/XyXn4tjKBQjTlRvSAgIIIIBA8gUI08kfY3pYoQBhukJADncVcLtyXCokewmhbu3nFlgstHsJ88U6mntspX1xxWSHQAQI04Ew0ggCCCCAQMIFCNMJH2C6V7kAYbpyQ1ooLZAfVPPDZ6HAnAk7QTwLXer8mcqDDNOlrnp7+eUA8yl8AcJ0+MacAQEEEEDAfgHCtP1jSA9CFiBMhwxM81IszJYKsF6uIFd627ff28y9XJkutY/+G2HajA8EYdqMcaAKBBBAAAGzBQjTZo8P1RkgQJg2YBASXkKhK9HlBGENol6uaudzernqXE49uedxC8nFrrQnfOiN7R5h2tihoTAEEEAAAYMECNMGDQalmClAmDZzXJJUVTkBuFgg9hvM8wO4tuvnGWsv4+AW1gnSXhSj3YcwHa03Z0MAAQQQsFOAMG3nuFF1hAKE6QixU3qqKMJ0qVu2w35mulDwz/87tyvXKZ0asXWbMB0bPSdGAAEEELBIgDBt0WBRajwChOl43NN0VrcFyLxaZK4y5+7v5cqzlzDvdUXuYrW6vRpLjyNQex3p8PcjTIdvzBkQQAABBOwXIEzbP4b0IGQBwnTIwDRfdAEyPzSlbs12u/LsN0z7qcvrvtzq7VUqmv0I09E4cxYEEEAAAbsFCNN2jx/VRyBAmI4AOeWncAu7XnjcwnSxNjJXg3PDU7FXcYV95ZhA7WWko9mHMB2NM2dBAAEEELBbgDBt9/hRfQQChOkIkFN+iqBu8y7EWOg278x+hW7dLhaY3RYRC2IICdNBKAbTBmE6GEdaQQABBBBItgBhOtnjS+8CECBMB4BIEyUFvF6ZLhVoM/9WbDVvt6vNbmE5N1yVO5xuV7YJ0+XKBn8cYTp4U1pEAAEEEEieAGE6eWNKjwIWIEwHDEpzTQSGyyxGAAAgAElEQVQKBdXcK8puzzQX+/fcgF2MPfc271JXpfX4YoE8/9+KBXO3XwYUOwdTJnoBwnT05pwRAQQQQMA+AcK0fWNGxRELEKYjBk/h6Yo975wJl263gbsFb7cr3/lXhN32zwxRsSvJbn9faojdrl6ncHrE0mXCdCzsnBQBBBBAwDIBwrRlA0a50QsQpqM3T9sZSy0ephZuV3TzX4mVf7XZTxgvdL5Cx2fGyO/V7GK3ixOizZr1hGmzxoNqEEAAAQTMFCBMmzkuVGWQAGHaoMFIaClBhOlMCNY//YTpXNJiz13nt1nOMBCWy1GL7xjCdHz2nBkBBBBAwB4BwrQ9Y0WlMQkQpmOCT9FpvdxW7eWKbv6z05kQ7LZ4WKnw7bYwmZdhCqINL+dhn+AECNPBWdISAggggEByBQjTyR1behaQAGE6IEiaKSoQRtgst81izzszfOkSIEyna7zpLQIIIIBAeQKE6fLcOCpFAoTpFA02XUUAAUeAMM1EQAABBBBAwF2AMO1uxB4pFyBMp3wC0H0EUihAmE7hoNNlBBBAAAHfAoRp32QckDYBwnTaRpz+IoAAYZo5gAACCCCAgLsAYdrdiD1SLkCYTvkEoPsIpFCAMJ3CQafLCCCAAAK+BQjTvsk4IG0ChOm0jTj9RQABwjRzAAEEEEAAAXcBwrS7EXukXIAwnfIJQPcRSKEAYTqFg06XEUAAAQR8CxCmfZNxQNoECNNpG3H6iwAChGnmAAIIIIAAAu4ChGl3I/ZIuQBhOuUTgO4jkEIBwnQKB50uI4AAAgj4FiBM+ybjgLQJEKbTNuL0FwEECNPMAQQQQAABBNwFCNPuRuyRcgHCdMonAN1HIIUChOkUDjpdRgABBBDwLUCY9k3GAWkTIEynbcTpLwIIEKaZAwgggAACCLgLEKbdjdgj5QKE6ZRPALqPQAoFCNMpHHS6jAACCCDgW4Aw7ZuMA9ImQJhO24jTXwQQIEwzBxBAAAEEEHAXIEy7G7FHygUI0ymfAHQfgRQKEKZTOOh0GQEEEEDAtwBh2jcZB6RNgDCdthGnvwggQJhmDiCAAAIIIOAuQJh2N2KPlAskLUzfc889cttttzmj2qL7Elnt4MUpH2G6jwACuQINK6pk6bmbOn/VvHlzef311wFCAAEEEEAAgQIChGmmBQIuAkkL0y+88IJcdNFFP/+gvO0yWf2Eb5gDCCCAQFag/suWsuzaDs7/33zzzeWhhx5CBwEEEEAAAQQI08wBBPwLJC1Mf/TRR3L88cc7EM06rJQ1z57nH4UjEEAgsQIr/9FaVjy0gdO/PffcU6699trE9pWOIYAAAgggUIkAV6Yr0ePYVAgkLUyvWrVK9t9/f1m+fLkzfi17fict9/s+FWNJJ4sL1M1aXaq3+gmilAs0fFcty67rIA1Lqh2Jk046SU444YSUq9B9BBBAAAEECgsQppkZCLgIJC1Ma3f1ts3rr78+2/PqbZZL822WS7M2q6RqrTrmRAoFal9YR1rs+qNUrc34p234G+qqpGFhc6lf2Fxqp64jsqrKIdhoo43k0UcflWbNmqWNhP4igAACCCDgSYAw7YmJndIskMQwnbni9P7776d5aOk7AgiUELj00kulR48eGCGAAAIIIIBAEQHCNFMDAReBpIbplStXOqt6T5w4kTmAAAIIZAU6d+4sgwcPlt122w0VBBBAAAEEECghQJhmeiCQ0jCd6fbMmTPlnXfekblz5zpfP/74I3MiZQLz5s2TJUuWOL3eYostZLXVVkuZAN3VW7o33nhjZ/Xugw46CBAEEEAAAQQQ8CBAmPaAxC7pFkjqlel0jyq9zwgsXrxYevbsKfX19c5f6WJTuugUGwIIIIAAAggggEBpAcI0MwQBFwHCNFMkyQJPPfWUXHbZZdku/upXv5LHHnssyV2mbwgggAACCCCAQCAChOlAGGkkyQL77LOPLFu2zOni1KlTpXXr1knuLn1LmcCZZ54p06ZNy16Z1u7feuutsv3226dMgu4igAACCCCAAAL+BAjT/rzYO4UCRxxxhHzxxRdOz++//37p1KlTChXochIFZs+eLUceeWSTrunfDR8+PIldpk8IIIAAAggggEBgAoTpwChpKKkCw4YNk5qaGqd7o0aNkm7duiW1q/QrZQKTJk2SsWPHNun1uuuuK88++2zKNOguAggggAACCCDgT4Aw7c+LvVMo8Kc//Umef/55p+cjRoyQPn36pFCBLidRYODAgfLhhx8W7Nq1114re+65ZxK7TZ8QQAABBBBAAIFABAjTgTDSSJIFrrvuOnn44YedLh5yyCFyzjnnJLm79C0lAh988IEMGjSoYG+rqqqcFb4vvPDClGjQTQQQQAABBBBAwL8AYdq/GUekTODxxx93bu/WrWXLls5z05tttlnKFOhu0gTuvPNO0a9im75rWm/1XmONNZLWdfqDAAIIIIAAAggEIkCYDoSRRpIssHLlSunfv7989tlnTjf1f+tz1GwI2CpQV1cnRx11lOgCZKW2Sy+9VHr06GFrN6kbAQQQQAABBBAIVYAwHSovjSdF4C9/+YvccMMNTnc22GADGT9+PFenkzK4KezHG2+8IWeddVbJnuut3t27d5crr7wyhUJ0GQEEEEAAAQQQcBcgTLsbsQcCsmTJEueK9FdffeVobLvttqILNK2zzjroIGCdgK7grSt5e9mmTJkibdu29bIr+yCAAAIIIIAAAqkSIEynarjpbCUC+c+Y7r777gVfK1TJOTgWgbAFFi9eLMcee6wsXLjQ06lGjhwphx12mKd92QkBBBBAAAEEEEiTAGE6TaNNXysWyA/UHTt2dJ4pPeCAA6R9+/YVt08DCIQtoIuKXXLJJZ5Ps+OOO8rNN9/seX92RAABBBBAAAEE0iJAmE7LSNPPwAQKrYLcunVrJ1BvueWWzi2xuV+BnZiGEAhAQIO0Bmo/24MPPihbbLGFn0PYFwEEEEAAAQQQSLwAYTrxQ0wHwxDQQK2BZO7cuSWbnzZtWhinp00EyhJYsGCB9OvXT5YtW+br+KFDh8qAAQN8HcPOCCCAAAIIIIBA0gUI00kfYfoXmsBPP/0kzz33nBOqZ8yYUfA8hOnQ+Gm4DIGXX35ZzjvvPN9HdurUyXm/OhsCCCCAAAIIIIDALwKEaWYDAgEIvP322zJnzpzs1xdffOH8b8J0ALg0EZiAvtJNQ3FDQ4PvNu+44w5nFXs2BBBAAAEEEEAAgZ8FCNPMBAQQQCAlAkcffbR88sknZfVWVwA/+eSTyzqWgxBAAAEEEEAAgSQKEKaTOKr0CQEEECgg0LdvX+ed6fmb/t2qVaucv15rrbWkRYsWTfbZbrvt5KqrrsIVAQQQQAABBBBA4H8ChGmmAgIIIJBygWHDhklNTY2joLeCd+nSJeUidB8BBBBAAAEEEHAXIEy7G7EHAgggkGgBwnSih5fOIYAAAggggEBIAoTpkGBpFgEEELBFgDBty0hRJwIIIIAAAgiYJECYNmk0qAUBBBCIQYAwHQM6p0QAAQQQQAAB6wUI09YPIR1AAAEEKhMgTFfmx9EIIIAAAgggkE4BwnQ6x51eI4AAAlkBwjSTAQEEEEAAAQQQ8C9AmPZvxhEIIIBAogQI04kaTjqDAAIIIIAAAhEJEKYjguY0CCCAgKkChGlTR4a6EEAAAQQQQMBkAcK0yaNDbQgggEAEAoTpCJA5BQIIIIAAAggkToAwnbghpUMIIICAPwHCtD8v9kYAAQQQQAABBFSAMM08QAABBFIuQJhO+QSg+wgggAACCCBQlgBhuiw2DkIAAQSSI0CYTs5Y0hMEEEAAAQQQiE6AMB2dNWdCAAEEjBQgTBs5LBSFAAIIIIAAAoYLEKYNHyDKQwABBMIWIEyHLUz7CCCAAAIIIJBEAcJ0EkeVPiGAAAI+BAjTPrDYFQEEEEAAAQQQ+J8AYZqpgAACCKRcgDCd8glA9xFAAAEEEECgLAHCdFlsHIQAAggkR4AwnZyxpCcIIIAAAgggEJ0AYTo6a86EAAIIGClAmDZyWCgKAQQQQAABBAwXIEwbPkCUZ6fAokWL5Msvv5S5c+c6fw4cONDOjlB1KgQI06kYZjqJAAIIIIAAAgELEKYDBqW5dAp89tln8uyzz8q0adNkzpw5smzZsiyEBmnCdDrnhS29JkzbMlLUiQACCCCAAAImCRCmTRoNarFO4O2333ZC9HPPPSd1dXUF6zclTN95552iX7mbhv9CW9euXRv9dbE+0GYyPHPDdO7AM+5N7yhhzidjzmfmOd/rfvnE83nn8577/Z/vdXyvy8yHOH5W7NChg+jXVlttJUceeaTzv03dCNOmjgx1GS9Q6D80WnSrVq1kww03lHbt2jl/6ldcV6arqqpEv3S74447nK/cTX8ZUGjbZZddGv31oEGDRL/yN9pMhufQoUNl+vTpTcaXcWfO504KPu/J+LxnxpTv87/Mbr7X8b2O73U/C5j2fb5169YydepUozMBYdro4aE4UwUKBek99thDDjjgAOnRo0dsZdfU1Ih+4+ncubM0a9YsG6S1oNtuu01uv/32RrUVClC6w4477thov5NOOkkGDx7cpF+0mQxPHd9Cc4FxZ87nfuj5vCfj854ZU77P/zK7+V7H9zq+1/0sYNr3+V69esnFF19c9O7P2H7gzjkxYdqEUaAGqwTyg3T37t2lb9++st1228Xajx9++EEOPfTQ7DfDrbfeOtZ6OLk9ArlhWn/hkv9Dtj09oVIEEEAAAQQQsF1g3rx5Mn/+fOf27o022kgaGhpk1apVRnaLMG3ksFCUqQJvvfWWDB8+PFveTjvtJNdee62svvrqsZf80EMPyfXXX+/Uoc+YPPjgg7HXRAF2CBCm7RgnqkQAAQQQQCCtAro2UX19vXHdJ0wbNyQUZLLABRdcIC+++GI2sI4ePdp5NtqETZ97nTFjhlPKmDFjZO+99zahLGqwQIAwbcEgUSICCCCAAAIpFjD16jRhOsWTkq77E3jvvfdkyJAh2YNGjRol3bp189dIiHv3799fZs2a5ZxhypQpzm0xbAh4ESBMe1FiHwQQQAABBBCIQ0DXddGvuXPnSpcuXaRnz55xlFHwnIRpY4aCQkwXuPzyy+XJJ590ytQP8vjx440qOfcVL8UWFjOqYIoxRoAwbcxQUAgCCCCAAAII5AnkLoxmyitnMyUSppmuCHgQWL58uehCY5ntoosuMuq3YloXYdrDQLJLQQHCNBMDAQQQQAABBEwVIEybOjLUhYBHAV1VMLNStr5HOvPctMfDI9mNMB0JcyJPQphO5LDSKQQQQAABBBIhQJhOxDDSiTQL/Otf/5ITTzzRIdh8881FV842bdN3TOu7pfWLVxuZNjpm10OYNnt8qA4BBBBAAIE0CxCm0zz69D0RAq+99pqMHDnS6cvOO+8sN954o5H9qq6udsI0GwJ+BAjTfrTYFwEEEEAAAQSiFMi8d1rfNa3vntYvUzaemTZlJKjDaIHJkyfL1Vdf7dSoKwjqM9MmboRpE0fF/JoI0+aPERUigAACCCCQdoGVK1caR0CYNm5IKMhEgTvvvFP0SzfTVhHM9SJMmzh7zK+JMG3+GFEhAggggAACaRcgTKd9BtB/awUI09YOHYV7ECBMe0BiFwQQQAABBBCIVYAwHSs/J0egfAHCdPl2HGm+AGHa/DGiQgQQQAABBNIuQJhO+wyg/9YK2BCmhw0bJlVVVY7x7bffbq01hUcvQJiO3pwzIoAAAggggIA/AcK0Py/2RsAYARvCNO+ZNma6WFcIYdq6IaNgBBBAAAEEUiMwZcoU0a+GhgZnIWD9MmVjATJTRoI6jBYgTBs9PBRXoQBhukJADkcAAQQQQACB0AR4z3RotDSMQDQChOlonDlLPAKE6XjcOSsCCCBQrsCTTz4pb775psyZM8f5+uGHH8ptiuMQKCqw1lprySabbOJ87b777tKrV69YtAjTsbBzUgSCE7AhTA8dOtR5Zlq/eGY6uLFPQ0uE6TSMMn1EAIEkCMycOVNuuukmJ0izIRC1gAbqU045RbbeeutIT02YjpSbkyEQvIANYVp7zXumgx/7NLRImE7DKNNHBBCwXaC2tlZ69+4tCxYssL0r1G+xQLt27WTy5MnSsmXLyHqReWa6vr7euTrOM9OR0XMiBIIRIEwH40grZgoQps0cF6pCAAEEcgXGjRsnEyZMyP7Vttsukc02WyZrr71K1lijDiwEAhdYvrxalixpLrNnrykffLB2tv3+/fvL6aefHvj53BpkNW83If4dAUMFCNOGDgxlBSJAmA6EkUYQQACB0AQWL14s++23X7b9vfZaKJ06LQ3tfDSMQL7Af/7TSl59tU32r6dOnSrrrbdepFCE6Ui5ORkCwQkQpoOzpCXzBAjT5o0JFSGAAAK5Au+9956ceOKJzl+1aVMr//d/8wFCIHKBv/2tgyxc+PPt3XfddZdsv/32kdZAmI6Um5MhEJwAYTo4S1oyT4Awbd6YUBECCCCQK6DPjF5yySXOX2255VLp3n0hQAhELvDyy23k009bOefV+fjHP/4x0hoI05FyczIEghOwIUzPmDFDmjVr5qzmveOOOwbXeVpKvABhOvFDTAcRQMBygdzVjLff/nvZccfvLO8R5dsoMH36uvLee+s4pevPDoMHD460G4TpSLk5GQLBCdgQprt27Zrt8PTp04PrPC0lXoAwnfghpoMIIGC5QG6Y7tLlO9lhh+8t7xHl2ygwY8Y6UlOzbuRhet68eTJ//nypq6uT9u3bS4cOHYzhq6qtrW0wphoKQcBQAcK0oQNDWYEIaJj+/PPPnbsarrzySu5sCESVRhAwR+CHH35willrrbXMKYpKfAkQpn1xsXNIAnGFad4zHdKA0iwCUQkQpqOS5jxhCixfvlzeffdd0TsX/v3vf4uuDpv5ypx3jTXWcFbn1C/97e9OO+3khOuOHTuGWRptI4BAAAK6SJV+vvWxn0WLFjlf+hlftWqV03rz5s2dz/b666/v/NmlSxfn8x31IkIBdDV1TRCmUzfkRnaYMN10WLgybeRUpSjTBAjTpo0I9XgVWLp0qUyaNEneeOMN5wfscrd27do5wbpXr16yyy67lNsMxyGAQMACTz31lOgravQXZcuWLSur9TXXXNP5fOurl3r27FlWGxwUrgBhOlxfWvcmQJgmTHubKeyFQJ6ADWG6pqbGWYBMv1iAjCmszxZpiNavn376yRVk3XXXFb1yvWLFCtd9dX4dccQRjd556noQOyCAQGAC+txg5vP9xRdfuLbbqtXPq+/qL9fctk033dT5fOtXdXW12+78e0QChOmIoDlNSQHCNGGajwgCZQnYEKa1Y/qDj4ZptvQK1NfXy6hRo+TRRx8tiLDNNttkb93W27gzt3tm5o3+sJ259VtvBddbRvWK1/ffN13splOnTjJs2DDZa6+90gtOzxGIWGDixIly9913F/xMbrzxxtnP9xZbbJF9ZGO11VZzqtRflmU+35999ln28z137twmvVhnnXXkhBNOkGOOOSbiHnK6QgKEaeaFCQJxhenMAmT6yIouPsYCZCbMBmpAwIcAYdoHFrvGJqDPS1599dXyn//8p1ENnTt3dq4yaejV5yTL2TRYT548WR577LEmhx9//PFyyimnlNMsxyCAgEcBff5ZP98vvvhioyM22GAD5/N9wAEHiF5VLmfTq9vPP/+8c7X722+/bdTEvvvuK+eee67zize2+AQI0/HZc+ZfBOIK05kKeDUWsxEBSwUI05YOXIrKnjBhgowbN65Rj/XZZv0hu3v37oFJfP3119nbS3NvGdVbv/UH7i233DKwc9EQAgj8LPDKK684Qfqbb77JkuiV58zt2EE6ZW4f1yvXma1t27bO53vvvfcO8lS05UOAMO0Di11DEyBMN6VlAbLQphsNJ0mAMJ2k0UxeX/SH7EceeSTbsRYtWjg/+Pbu3Tu0zi5cuND54f7ll1/OnkMXMbrlllvkd7/7XWjnpWEE0ibw17/+Va644opG3R4wYICcdtppoVLccMMNct999zU6xwUXXCCHHnpoqOel8cIChGlmhgkChGnCtAnzkBosFCBMWzhoKSl5/Pjxctddd2V7G/UV4vwr4rrq96233lr27aYpGTa6iYAnAV2l+5xzzsnuG/UV4kJXxHVNBl31my1aAcJ0tN6crbAAYZowzWcDgbIEbAjTuhBUVVWV07/bb7+9rH5ykF0CuhDRddddly26T58+ct5550XeCX1WW5+Z1tXAddOFyTRQ6wrhbAggUJ7A22+/LUOHDs0e/Pvf/17GjBkT+bPL+qz2WWedJe+//362Fr0DhVfklTeu5R5FmC5XjuOCFCBME6aDnE+0lSIBG8J0165dsyOiKzCzJVvgySeflIsvvjjbSV186Kqrroqt0/oe69zbTrt06SJ33HFHbPVwYgRsFvjkk0+clbR//PFHpxv6fLQGWL0yHcemz2prsM88R926dWtnRfGOHTvGUU4qz0mYTuWwG9fpuML0lClTRL8aGhqkZ8+ezpcpG89MmzIS1GG0AGHa6OFJXXG6muX//d//yYIFC5y+6y9Sbr755tgdnnnmGbnwwguzdZxxxhly7LHHxl4XBSBgm8DZZ5+dXbVbV+vWIB13cNWAr4E6s9q3rvJ9zTXX2EZrbb2EaWuHLlGFxxWmc+f/wIEDRb9M2QjTpowEdRgtQJg2enhSV5wGZ70qpJu+a/Hhhx+WVq1aGeHwwAMPyNixY51adEGyv/3tb5HflmoEBEUgUKbAq6++KsOHD88erf/92WGHHcpsLdjDZsyY0eiHWH3MhPfMB2tcrDXCdDTOnKW0AGG6qQ9hmk8NAh4EbAjTesVAn5nWL56Z9jColu4ye/bsRqvp6q3eBx98sFG96devn8yaNcup6fDDD3dWFmdDAAFvAsccc4x89NFHzs66crauoG3SpiuL6wrjum2zzTaiazewhS9AmA7fmDO4CxCmCdPus4Q9ECggYEOY1rKrq6ulWbNmjGGCBfQH62effdbpoa7cbeIvTvR1WSNGjMiOgr5eh9dlJXhS0rXABB566CEZPXq0095qq63m3NkR13PSxTqlz0/rYyYrVqxwdhk5cqT07ds3MAMaKixAmGZmmCAQV5jOPDNdX18vvXr14plpEyYDNSDgR4Aw7UeLfcMS0Hc79+jRI9u8/nC10047hXW6itrV1X/1tTq6HXnkkaLPgLIhgEBpgdy7Ok499VQ57rjjjCS799575cYbb3Rq22qrreTBBx80ss4kFUWYTtJo2tuXuMJ0RkzXjDFt4zZv00aEeowUIEwbOSypK+rxxx+Xyy+/3Om3vpZGFyUydXvnnXdkyJAhTnkbbbSRswonGwIIFBf4/PPP5bDDDnN2aNGihbz++uvSvHlzI8lWrVole+65p2R+sH3sscdk8803D6xWvfrEXVaNOZMYpu+6azM58cTZvuZNoWPc2tF/z2x+z+eruBI7u9UY1HnCbocw3VSYMB32rKP9RAgQphMxjNZ3Qhcl0sWJdNMrv0cddZTRfdpvv/1k8eLFTo16JWvbbbetuF59VdBLL73k3Pq62267VdweDSBgisCECRNk3LhxTjn77LNP9nZvU+rLr0Nv79bPom6nn3669O/fv+JSdYGzF154wVm8UN9dz/aLQBLDtPbOb8j0EqZzw7Oew2+Azj++0Dwsp02/x5g4/wnThGkT5yU1WSBAmLZgkBJe4k8//eRcCdJ3LOo2efJk2WSTTYzu9aWXXipPPPGEU+OJJ54ow4YNK7ve1157zfnBferUqbJs2TI56KCD5LLLLiu7PQ5EwDSBQYMGSU1NjVOWiQsL5nvpZ1s/47pV8l75r776Sp5//nlnLYiZM2c67ekv4kaNGmXaEMVaj41h2kso9YKqIdRrW5l93YJrqcDtFvDz/91rbV776mW/uPYhTBOm45p7nNdyARvCtP5GX2+L09W8dWEqtmQJ6NWazKrYtqygq+FXr17pVs5zlR9++KHzrl3t+/z58525nfllAmE6WfM77b3JXw9Bf2m03nrrGc2id51o6M1szz33nLRp08Zzzfq51i/9jOdve++9t4wZM8ZzW2nY0cYwHca4eL0y7SVMZ/YpJxy7tZ/fd7eAHoZVGG0SpgnTYcwr2kyBgA1humvXrtmRmD59egpGJV1d1OejdR7qNnjwYDnppJOMB9DnKnfddddsnW+//baz4nypbd68ec7VZ/0h+9///nfRXQnTxg8/BfoQ0M+Gvt5QN1NX6S/UHf0+lPnvjX6P0rUcSm36CzL9bD/zzDPy7bffOt8P6urqmhyid+Fkbnn3wZjoXW0P036v3hYLq1GF6VJhuZxgXM4xJk7ouMK0/mygv1TX7xft27eXDh06GMPDM9PGDAWFmCxAmDZ5dNJRm97SrLd263b++ednFyoyvfe6+rheddPt6aeflnbt2jUpWW9h1wCtV6gyz4S79Ysw7SbEv9skoJ+Niy66yClZPzNXXnmlFeXr9yK9Iq2bfo/Sz2X+pqFZA7Texv3BBx8UDdC5x+kvh2+++WYrDKIqMglh2uvV3FLBs1goz23bS3DN3cfL/l7G2e8vDPLb9OrjpZaw9okrTOfO/4EDB4p+mbIRpk0ZCeowWoAwbfTwpKK4M844Q/S5Yd3Gjh0r3bp1C7TfmUcDgr6r4eijj5aPP/7YqfX++++X3/zmN9m633zzTeeHbP1avny585iCruLrZSNMe1FiH1sE9LNx/fXXO+XqZ+bMM8+sqHT9PLt9lr3s41aEfi964IEHnN30e9Sxxx6bPeTvf/+789nW56ELXX0u1ba+8k9/eLZ90+9neoeO9l+/Cv1vr3/31ltvySOPPOKQdOnyneyww/dW8fgJrG5hOj90+r1NO/+56vxgnQubeyt4PrgN4TfoSUKYbipKmA56ltFeIgUI04kcVqs6dcwxx2cAPZsAACAASURBVMhHH33k1HzffffJ7373u0DrDytMn3baafLGG284tV533XWy4YYbZm/zXLBgga8AndthwnSgw09jMQvoZ2PixIlOFfqZGTBgQNkV5a6ZUSxQe9nHSwH6veiGG25wdtXvUfq51LtM9Eq7LizWsmVLqa2t9dJUo3223npr50q916BZyX65IddrO16Pyazx4BugwAH6C4Z3333X6jDtxyHo27wLBe5Cz0wXu2Jd6ng//bJ9X8I0Ydr2OUz9MQnYEKZ1FVi9sqdfLEAW00QJ8bQHHnigfPPNN84ZnnrqKeeZoSC3sML0n//8Z/nb3/7mlKo16w/YQWytWrWSDTbYIIimaAOB2AX0s6HPTeumn5mePXtWVFOpsBxUkM58L/rTn/7k1Lr++uvLokWLZPXVVxd9dIMtWAFdMT2z2nvar0wXknW7zTuqMF3urd62XOUmTBOmg/3ORmupEbAhTOtg6GIuGqbZkiew8847Z2+BnjZtmrRo0SLQTgYdpvUKj97ieccdd8js2bMDrZXGEEiaQOfOnbOPQ+izwrkLSpbb10KhOcggrbcd/+UvfxF9XIOtsID+97h58+bOf5v1K/O/C/2d278vWbJE9K0dutkYpoOaI+UuQOYnTOfW6vc272K3qPu9dT0or6DbiStMZxYg058tdPExFiALemRpD4GQBQjTIQPTvKvAvvvuK999952zn4ZUvQoU5BZ0mM7Upq/z0noz22qrrSYrVqwIsnTaQsB6gdyrjldffbXsv//+gfUp/04lt2ep/ZxYF0XMf9977ivs/LSVu69+n9hyyy0rCp9u4TSKfw/yl9u2LkBW7pXazHxwez5a9/NyC7afMO12+3ehc2bqJUyX+6n3dtzKlSu97RjhXjwzHSE2p7JXgDBt79glpfIjjjhCPvnkE6c7Dz74oPPe5nI3v48BVPLD94gRI+Tll192Sj3llFNEr67oqr5ff/112c9Talu6AJsueMSGQBIEbr/9dud1UbrpZ6Zfv36BdiusX5bp96Jrr73WqfWPf/yjbLfdds4z03r3jG7lBuuOHTvKpEmTAjWwvTGbw3S5tzB7uQoddJguNE+CWDyNK9PBfAIJ08E40goCkQsQpiMn54R5AsOGDZN//OMfzt/edNNNsttuu5VtFGWYPu6445zX4fz8A89dsv322zv/+5133nFeg6UBYvHixc5t637+I8kCZGUPPwcaKKCfjfHjxzuV6Wfm1FNPDazKMK9M6/eie+65x6lVv0edeOKJzv/WWzL1M653pejt4H63zTffXB577DG/hyV6/6SEabcr1cWefXY7LnfwC4V3L1em/Zwjc75CV87LmYjl/sKhnHNVckxct3lnavbzc0Il/fRzLFem/Wixb2oFCNOpHXpjOq4r2+oKubpdcsklzlWgILewrlxpnfqDtW56S+gmm2zSqGxdqEh/SaA/cGuw/vHHHz29h5YwHeTo01bcArm3S+tnRj/jlW5hPzOd+V40ZcoUp1T9HtW7d+8mZeuaCZlfnmVW9nfr28Ybb5xduNBt37T8e5LCtNeVur3cMu3lFm+dI26vzyoWwHPnl5fA6+cqdqbtco6Ja94TppvKE6bjmo2c1yoBG8K0XhXQW+p001sG2ZIlMG7cOJkwYYLTKb1d+vjjjw+0g2GFab2Cnnk1zuuvvy5rrLFG0br1CrUGa/3Sd9OWWhGYMB3o8NNYzAL62Tj99NOdKvQzo1d8K9miWs1bvxdlrjzr96g999yzZNn6qIoGaz1G+1xs08WFnnzyyUoIEncsYfrnIXW79dtLAC/WTu6kKfSKLLfQ6/bvSZiUhGnCdBLmMX2IQcCGMJ27+mslz7jGwMspPQg8/PDDcs011zh76g+s+oNrkFsYYfqf//ynnHDCCU6Z+horDchety+//NJ5VZD+wK1fuoJn7kaY9irJfjYI/Oc//5G+ffs6pepr3/QRiHI3Lyt2e9nHy/n32msvWbp0qbPrQw89JJ06dfJymLPPxx9/7ARrfb4684x15uB27dpl78Tx3GDCd0xSmC41VKVeceX23HGmXS+3eReqIfc2b7fbzcs9h+3TNK4wrXfA6Je+u11fHVjp6wODHAeuTAepSVuJFSBMJ3ZoremY3iqde2u3/rCtP3QHtYURpm+88Ua59957nRIPOeQQufDCC8sq97///a9ztfrvf/+7855V/Y8pYbosSg4yWOCwww6Tzz//3KnwhhtukD322KPsavXz7PZLVS/7lCog92p6pc84a7DWtRXee+895/VP+t7tu+++u+z+J/HAJIVpL7dL6xgWujpcamzdwnap28u1Xa+3n2dqyz0mDVeltb9xhenc+T9w4EDRL1M2wrQpI0EdRgsQpo0entQUN2DAAPnwww+d/l5xxRVy4IEHBtb3MMJ07grkY8eOdVbgrnTT12rp7d/6mrDNNtus0uY4HgFjBK6//nq5//77nXr69Okj5513njG1FSrkqquukkcffdT5p2OPPZbV9UMeraSEabcwnPvvXkN3yPQFby3PPWfmirYp9YbpQZhuqkuYDnPG0XZiBGwI00OHDnWemdYvnplOzNRr1BGdh7fccovzdxqkNVCbuumzkRqmddN3xupVrCDfuWpqv6kLgXIF9ErySSed5Bxuw23OenfIggULnHr1vzl+3xJQrlNaj7M1TKd1vJLab8I0YTqpc5t+hSxgQ5hWgurqagJLyHMhzuZnzpwpRx11lFNCy5Yt5YknnpC2bdvGWVLRc+ttqvfdd5/z7/vtt5+MGjXKyDopCgGTBA444AD59ttvnZL0/c3du3c3qbxsLfrueH0ftm5+10MwskMWFEWYtmCQUlBiXGE688x0fX299OrVi2emUzDX6GLCBAjTCRtQi7tzzDHHyEcffeT04NBDD5ULLrjAuN58+umncvjhh2frCvqWdOM6TEEIBCSgv3SaNGmS05ou5qWLepm46WJpumiabnoHyjnnnGNimYmqiTCdqOG0tjNxhekMGO+ZtnbqUHjaBQjTaZ8B5vT/tddea/Rsoi7Ss91225lToIice+658sILLzg17bLLLtlb040qkmIQMFBAXw938MEHy7Jly5zqhg8fLvoLNJO2iRMnynXXXeeUtOaaazp3yKy33nomlZjIWgjTiRxW6zpFmG46ZDwzbd00puA4BAjTcahzzmICehVo6tSpzj/vuuuuMn78eGOw8sP+XXfdJdtvv70x9VEIAqYL6CJkuhiZbrpiv4bVdddd14iydeE/DfuZ12GdccYZzuJjbOELEKbDN+YM7gKEacK0+yxhDwQKCBCmmRYmCeQu7qV16bucTz755NhLnD17tgwZMkS+/vprp5ZKXocVe2coAIEYBXJvow7jvfLldu300093FhPUzeTb0Mvtn8nHEaZNHp301EaYJkynZ7bT00AFbAjT+m5OXS1ZV/NmVdVAh9/IxsaNGycTJkzI1hb3FSK9PVWDtL4TWrc11lhD/vrXv8qGG25opB9FIWCywEsvvSQjR47MlmjCe9Uvuugiefrpp7M1jR49WvbZZx+TGRNVG2E6UcNpbWcI04RpaycvhccrYEOY7tq1axZJX7HClnyB888/X5577rlsR/WH3d69e0fe8VWrVom+mq2mpiZ77htvvFF23333yGvhhAgkReDee+8V/RxltiOPPFLOPvvsWLqnwTl3MbRTTz1VjjvuuFhqSetJCdNpHXmz+h1XmJ43b57Mnz9f6urqpH379tKhQwdjYHhm2pihoBCTBQjTJo9OumvTH2rffPPNLMKgQYOcK8RRbR9++KFcffXV2RXG9bys3h2VPudJukDuK+a0rz169HAW+Ft77bUj6fqSJUucz3fuL+0GDBggp512WiTn5yS/CBCmmQ0mCMQVpnPn/8CBA0W/TNkI06aMBHUYLUCYNnp4Ul2crvqrV4U11GY2XZRMf+DedNNNQ7V54IEHZOzYsY3OoYuj6aty2BBAIBgB/eWUPjKR2fSKjH6+9VnqMDddTFBf1aVXgzKbqa/jC9PBlLYJ06aMRLrrIEw3HX/CdLo/E/TeowBh2iMUu8UisHDhQudq8Kuvvpo9/+qrry4nnXSSE2z1+eUgt3fffVcefvhh0ec6M5s+q3/eeefJYYcdFuSpaAsBBEScV1HpK6lyN12kTD/fm222WaBGupCgvus6/x3X+ooufVUXWzwChOl43DlrYwHCNGGazwQCZQnYEKb1eVVdgEy/WICsrGG2/qB77rlHbrrppkb90PfA6g/c+tWuXbuifdRA3qZNm5IGGp71h+x33nmn0X766iu9Uqar+7IhgEA4Ai+++KJzy/WiRYsanUAXJ9PP97bbblvRiT/44APn8527yJg2uP766zuf73333bei9jm4MgHCdGV+HB2MAGGaMB3MTKKV1AnYEKZ1UKqrq50wzZZeAV18Tn/g/vTTT5sg7LDDDs4vWnbaaSfnS68mZza9CqVXunK3L7/8UvQqtLapfy5YsKBJm/379xd9XQ4bAgiEL/DNN984n+9XXnmlycl+/etfO5/rzGc897nqjz/+2Nm/c+fO2eP0eejcz3dmJf7chvfee28nSLdt2zb8znGGkgKEaSaICQJxhenMAmS64Kk+6sICZCbMBmpAwIcAYdoHFrsaIfD44487V5lmzZpVsJ4WLVo4V6L1qtN6660n//73v51nrPUXMvqaq8xXsc7oO6T1athWW21lRH8pAoE0CbzxxhvO5zvzzudCfdcArJ9t/Yzru9/1l2f6d3plWz/fGsyLbfo8tn6+99hjjzSxGt1XwrTRw5Oa4uIK0xnglStXGmfNM9PGDQkFmShAmDZxVKjJi4DeGqo/dOsVqEq3ddZZx3n1lv6Qra+mYEMAgXgFMrdmP/vss1JfX19RMXpX04EHHhjILeMVFcLBBQUI00wMEwQI001HgTBtwsykBuMFCNPGDxEFugjolajcWzo/++wzVzNdxExvGc3cNvrb3/7W9Rh2QACB6AX0ao0+jpF5JOP999/3VMTvf//77K3h+jnXO1bYzBQgTJs5LmmrijBNmE7bnKe/AQkQpgOCpBljBJYuXZq9lVufifzqq6+c2k488UQ54IADnNtDN9hgA2PqpRAEEPAuUFtbm72dW0NY5nbwP/zhD84q/5nbv1u2bOm9UfaMVeCRRx5xnpfXbautfpQ//OHbWOvh5OkUeO21DWTWrNZO5/Vnh8MPPzxSCG7zjpSbkyEQnIANYXrYsGHZBaVuv/324DpPS4kW0CtYxx9/vNNHfV66W7duMnr06ET3mc4hkBaBFStWOLdu62JjuumjGrmvtEuLQxL6OW3aNDn55JOdrrRv/5P07Nl0Qcgk9JM+mC3w1FPt5KuvVneKvPnmm6Vr166RFkyYjpSbkyEQnIANYTr3G5re6seGgBeBcePGyYQJExrtOnXqVOfKFRsCCNgtoKt+n3XWWY06oe+s3muvvezuWAqr17crHHzwwdmeH3zwV9K27YoUStDluAS++WY1eeKJX9ZLeeKJJ+RXv/pVJOVMmTJF9KuhoUF69uzpfJmy8cy0KSNBHUYLEKaNHh6KK1NAXzHxxz/+0VnpN3e74IIL5NBDDy2zVQ5DAAFTBC666CLJXZxMFxn7f//v/8mf//xnU0qkDh8Cw4cPl1dffdU5ok2bWtlxx+9k442X+2iBXREoT2Du3DVk+vR1ZeHCnx8N0V/I6S/motpy1wwYOHCg6JcpG2HalJGgDqMFCNNGDw/FlSmgz1HmvyNaf9jeeeedZfz48WW2ymEIIGCCgL76qlevXqK/NMvddJExvdV7zTXXNKFMavAh8J///Ef69u3b6IjVV6+XtddeKfonGwJBC/z0UzNZsqSF6J+520MPPSSdOnUK+nRF2yNMR0bNiRAIR8CGMD106FDnmWn94pnpcOZB0lq98sor5a9//atz21T+NnnyZNlkk02S1mX6g0BqBPQWzEsvvbRgfy+77DI56KCDUmORpI7qQmRjx44VXWSODYGoBXTRwjPPPDPyhccI01GPNOdDIGABG8K0dlkXkNIri2wIuAksWrTIeWe0rupdaDvjjDPk2GOPdWuGf0cAAUMF9Afe1157rcn7p/W/EXqL5pgxYwytnLLcBBYsWODcPaR3GCxbtsxtd/4dgYoF9E6WffbZR3Sx23bt2lXcnt8GMs9M19fXO3fc8My0X0H2RyBmAcJ0zAPA6QMXePrpp0Wfpyy2bbPNNjJx4sTAz0uDCCAQvsAnn3wiRxxxRMkTPfPMM7LhhhuGXwxnCFVA17yYM2dO0V+MhnpyGk+8QKtWrZy71Ez5XsFq3omfcnQwqQKE6aSObHr7df7558sLL7zQ5KpVroiGaQ3VbAggYJeAfnbdFgc655xzXAO3Xb2mWgQQSLoAYTrpI0z/EitAmE7s0KayY59//rnzvJPeLlVq09Uy9Vl8NgQQsEtg0KBBMmPGjILrIWhPdG2NLl26sL6GXcNKtQikXoAwnfopAICtAoRpW0eOugsJ6Cqco0ePdsXRW7t0ITI2BBCwR0BDtNfXxuhiVltuuaU9naNSBBBItQBhOtXDT+dtFrAhTOsPULqwjF5x2HHHHW3mpvaQBfR1WG+++abrlWkt49Zbb3VelcWGAAJ2COjCVHfddZenYk8++WQ54YQTPO3LTggggEDcAoTpuEeA8yNQpoANYbpr167Z3k2fPr3MnnJY0gXef/99Of744z13U99pOnLkSM/7syMCCMQnsGLFCunfv7/897//9VTEVlttJQ8++KCnfdkJAQQQiEtg3rx5Mn/+fKmrq5P27dtLhw4d4iqlyXmramtrm75g1JjyKAQBMwQI02aMA1VULpD7rkYvra2//vry3HPP8co1L1jsg0DMAq+88oqcddZZvqq4++67ZbvttvN1DDsjgAACUQrwnukotTkXAiEIEKZDQKXJyAX0N7p6Vfpf//qXr3Nfe+210r17d1/HsDMCCEQvcPXVV4s+B+1nGzBggJx22ml+DmFfBBBAIFIBwnSk3JwMgeAFCNPBm9Ji9AIffPCBHHfccb5OrM/g9+zZUy699FJfx7EzAghEK7B06VLnVVdfffWVrxNvtNFGMmXKFF/HsDMCCCAQpQBhOkptzoVACAI2hOmamhrnVlz9YgGyECZBApp8+OGH5ZprrvHdk9VXX12eeeYZWXvttX0fywEIIBCNwMyZM+Woo44q62Q33HCD7LHHHmUdy0EIIIBA2AKE6bCFaR+BkAVsCNNKUF1dzbOtIc8Fm5sfMWKEvPzyy2V14eKLL5aDDz64rGM5CAEEwhd48cUX5eyzzy7rRIcddpicf/75ZR3LQQgggEDYApkFyFatWuUsPsYCZGGL0z4CAQsQpgMGpblYBDQMf/nll77Prbd6d+vWTcaMGeP7WA5AAIFoBK666ip59NFHyzrZOuus49x9stpqq5V1PAchgAACUQjwaqwolDkHAiEIEKZDQKXJyAUmTZok3377bZPzPvXUU84rJ3Tr1auX89qJ/G2ttdaSPn36iN7yzYYAAuYJaBj+/PPPmxSmjwDpl25dunRxvvI3XZzwyCOPlLZt25rXMSpCAAEE/idAmGYqIGCpAGHa0oGjbE8CJ510kmTeTX777bfzzL0nNXZCwA6B3GcN9bM+ePBgOwqnSgQQQCBPgDDNlEDAUgHCtKUDR9meBAjTnpjYCQErBQjTVg4bRSOAQAEBwjTTAgFLBWwI08OGDRN9tlU3vbrIhoBXAcK0Vyn2Q8A+AcK0fWNGxQggUFiAMM3MQMBSARvCdNeuXbO6mVt2LeWm7IgFCNMRg3M6BCIUIExHiM2pEEAgFIEpU6aIfjU0NEjPnj2dL1O2qtra2gZTiqEOBEwVIEybOjLUFYQAYToIRdpAwEwBwrSZ40JVCCDgXYD3THu3Yk8EjBQgTBs5LBQVkABhOiBImkHAQAHCtIGDQkkIIOBLgDDti4udETBPwIYwPXToUOeZaf3imWnz5pDJFRGmTR4dakOgMgHCdGV+HI0AAvELEKbjHwMqQKAiARvCtHawurpamjVrVlFfOTh9AoTp9I05PU6PAGE6PWNNTxFIqkDmmen6+nrp1asXz0wndaDpV3IFCNPJHVt6JkKYZhYgkFwBwnRyx5aeIZA2AVbzTtuI09/ECBCmEzOUdKSAAGGaaYFAcgUI08kdW3qGQNoECNNpG3H6mxgBwnRihpKOEKaZAwikSoAwnarhprMIJFqAMJ3o4aVzSRYgTCd5dOkbV6aZAwgkV4AwndyxpWcIpE2AMJ22Eae/iRGwIUzPmDHDWXxMV/PecccdE2NPR8IXIEyHb8wZEIhLgDAdlzznRQCBoAUI00GL0h4CEQnYEKa7du2a1Zg+fXpEMpwmCQKE6SSMIn1AoLAAYZqZgQACtgvMmzdP5s+fL3V1ddK+fXvp0KGDMV2qqq2tbTCmGgpBwFABwrShA0NZgQgQpgNhpBEEjBQgTBs5LBSFAAI+BHjPtA8sdkXARAHCtImjQk1BCRCmg5KkHQTMEyBMmzcmVIQAAv4ECNP+vNgbAeMECNPGDQkFBShAmA4Qk6YQMEyAMG3YgFAOAgj4FiBM+ybjAATMErAhTNfU1DgLkOkXC5CZNX9Mr4YwbfoIUR8C5QsQpsu340gEEDBDgDBtxjhQBQJlC9gQprVz1dXVTphmQ8CPAGHajxb7ImCXAGHarvGiWgQQaCqQWYBs1apVzuJjLEDGLEHAMgHCtGUDRrm+BAjTvrjYGQGrBAjTVg0XxSKAQAkBXo3F9EDAUgHCtKUDR9meBAjTnpjYCQErBQjTVg4bRSOAQAEBwjTTAgFLBQjTlg4cZXsSIEx7YmInBKwUIExbOWwUjQAChGnmAALJESBMJ2cs6UlTAcI0swKB5AoQppM7tvQMgbQJcGU6bSNOfxMjYEOYHjZsmFRVVTnmt99+e2Ls6Uj4AoTp8I05AwJxCRCm45LnvAggELQAYTpoUdpDICIBG8J0165dsxrTp0+PSIbTJEEg91Vq+osYXq2WhFGlDwj8LECYZiYggIDtAlOmTBH9amhokJ49ezpfpmxVtbW1DaYUQx0ImCpAmDZ1ZKgrCAHCdBCKtIGAmQKEaTPHhaoQQMC7AO+Z9m7FnggYKUCYNnJYKCogAcJ0QJA0g4CBAoRpAweFkhBAwJcAYdoXFzsjYJ6ADWF66NChzjPT+sUz0+bNIZMrIkybPDrUhkBlAoTpyvw4GgEE4hcgTMc/BlSAQEUCNoRp7WB1dbU0a9asor7afvATTzwhH374ocyZM8f5WrVqle1dCr3++vr67Dkyv5AJ/aSWn2CDDTaQTTbZxPnSZ7c233xzy3tkR/lvvvmmTJs2Lfv5/vHHH+0oPMYqcz/f+ryh/neCrbRA69ats59vXY9k9913hwwBBGIUyDwzrd/PevXqxTPTMY4Fp0agLAHCdFlskR40Y8YMGT9+vNTU1ER6Xk6GgAoMHjxYdFV0tnAEvv76a7n55pvlySefDOcEtIpACQH94f3kk0+WDTfcECcEEIhRgNW8Y8Tn1AhUIkCYrkQv/GNnzpwpRx11VPgn4gwIlBA4+OCD5eKLL8YoBIHDDz9cPv300xBapkkEvAlsueWW8sgjj3jbmb0QQCAUAcJ0KKw0ikD4AoTp8I0rOcPpp58ur7/+erYJvUL4m9/8xrlNr1WrVpU0zbEIFBSYP3++c6vx5MmTG90NcdNNN8luu+2GWoACd9xxh9x6663ZFvUq4R/+8Afn873++usHeCaaQuBngUWLFjmf79dee63R3RBDhgyRQYMGwYQAAjEJEKZjgue0CFQqQJiuVDC84zVEa5jObBMmTJDf/va34Z2QlhHIE7jooovk6aefdv52hx12EP1+wRaMwNKlS2WvvfbKNnb22WfLkUceGUzjtIKAB4GHH35Yrrnmmuyer776Kr+k9eDGLgiEIUCYDkOVNhGIQMCGMK3PDOviY7qAVO7qzBHwxHqK++67T2644Qanhn79+smIESNirYeTp09Ar1Lr1VLdVl99dXnjjTfShxBSjz/44AM57rjjnNa33npr+ctf/hLSmWgWgeIC+hiRPk6k27333ivbbrstXAggEIMAYToGdE6JQBACNoRpXXE0s02fPj2IblvRxuWXXy6PP/64UytXrawYskQWud9++8nixYudvulV6nbt2iWyn1F3Si31yr9uajxq1KioS+B8CMg555wjU6dOdSQuu+wyOeigg1BBAIEIBebNmyf6i+u6ujpp3769dOjQIcKzlz5VVW1tbYMx1VAIAoYKEKYNHZj/raL87rvvOgXeeOONvMLE3KFKdGXHH3+8vP/++04f9T3vabo7JMyBzX23qBqfcsopYZ6OthEoKKBrIdxzzz3Ov+maHLp6PxsCCEQnwHumo7PmTAiEIkCYDoU1kEb1B5vMlXhCTCCkNFKGAPOwDDQPh+T+AEWI8QDGLqEIMA9DYaVRBDwLEKY9U7EjAmYKEKbNHJfMVQLCtLnjk5bKCNPhjDQhJhxXWvUnwDz058XeCAQtQJgOWpT2EIhYwIYwXVNT4yxApl9pusWUEBPxh4HTFRRgHoYzMQgx4bjSqj8B5qE/L/ZGIGgBwnTQorSHQMQCNoRpJamurnbCdJo2QkyaRtvcvjIPwxkbQkw4rrTqT4B56M+LvREIWiCzANmqVaucxcdYgCxoYdpDIGQBwnTIwBU0T4ipAI9DAxNgHgZG2aghQkw4rrTqT4B56M+LvREIS4BXY4UlS7sIhCxAmA4ZuILmCTEV4HFoYALMw8AoCdPhUNJqBQKE6QrwOBSBAAUI0wFi0hQCUQoQpqPU9ncuQow/L/YOR4B5GI4rISYcV1r1J8A89OfF3giEJUCYDkuWdhEIWYAwHTJwBc0TYirA49DABJiHgVE2aogQE44rrfoTYB7682JvBMISIEyHJUu7CIQsYEOYHjZsmFRVVTkSn38oswAAIABJREFU+r7ltGyEmLSMtNn9ZB6GMz6EmHBcadWfAPPQnxd7IxCWAGE6LFnaRSBkARvCdNeuXbMKmfcuh8xiRPOEGCOGIfVFMA/DmQKEmHBcadWfAPPQnxd7IxC0wJQpU0S/GhoapGfPns6XKVtVbW1tgynFUAcCpgoQpk0dGRFCjLljk6bKmIfhjDYhJhxXWvUnwDz058XeCAQtwHumgxalPQQiFiBMRwzu43SEGB9Y7BqaAPMwHFpCTDiutOpPgHnoz4u9EQhagDAdtCjtIRCxwCWXXCLPPvusc9azzz5bDj300IgrcD/d0KFDnWem9Ytnpt292AOBIAUI00Fq/tIWISYcV1r1J8A89OfF3ggELUCYDlqU9hCIWKB///4ya9Ys56zXXnut7LnnnhFX4O101dXV0qxZM287J2QvQkxCBtLybjAPwxlAQkw4rrTqT4B56M+LvREIWiDzzHR9fb306tWLZ6aDBqY9BMIUqKurk3333Vd++ukn5zT33nuvdO7cOcxTlt02Yfp22XHHHcv240AEyhUgTJcrV/o4Qkw4rrTqT4B56M+LvREIS4DVvMOSpV0EQhTQK9J6ZTqzPfnkk9KmTZsQz1h+04RpwnT5s4cjKxEgTFeiV/xYQkw4rrTqT4B56M+LvREIS4AwHZYs7SIQosD48eNlwoQJzhn0irRemTZ1I0ynI0zr1fdSrz9z+3dT56/NdRGmwxm9tIYYt8+w27+HMxrpbTWt8zC9I07PTRUgTJs6MtQVmUDmXcjTpk2L7JyVnGjmzJkyYMCAbBPnnnuu9O7du5ImQz2WMJ38MJ25jd0tTOtES9P7xkP9YHlonDDtAamMXdIYYviMlzFRQj4kjfMwZFKaR6AsAcJ0WWzpPigTPosp2BJKM/XbFqb79esnn332mVO+6VelZ8yY4Sw+pqt5p+m54bSFmPwrUn7H2mvAdmvXaztp+Q6etnkY1bimMcRE8Rnn8+1vBqdxHvoTYm8EohEgTEfjnKizEKbjG87cd0trFaZflc6dK2kKOmkLMYV+0PY63rnHlvph2q09bjFt+n0pbfMwqu/MaQwxQXzG+XwHO0PTOA+DFaQ1BCoTmDdvnsyfP190UeD27dtLhw4dKmswwKOramtrGwJsj6YCFrDtSq5b923ozyuvvCJjxoyRb775JtudHj16yKWXXurWvVj/nTAtzvu13a64xDpIFZ48Pwxr6PUTbIvtG9TfV9g9qw8nTIczfGkLMWF8xvl8Vz430zYPKxejBQSCFeA908F6pqo1G8KnnwExuT9ffPGFTJo0SR599NFGXerTp4+MGDHCTzdj2ZcwnZ4wnftMZVhhutRzm37OGcuHIYaTEqbDQU9biMl8toL8jBf6vPL59jdf0zYP/emwNwLhCxCmwzdO7Bn8hM/8fXPDValnq/NvJS+2b6Fbzv22W6o/XurIPd5r/0pNDl1g7O2335a33npLampqmuyqi48NHTrUivlFmE52mC72w7XfK/H5t3CXumqlE7/S/bUNLwsqWfEhK1EkYTqcEUxTiAnjMx7U5zt3dL18z3F7VCSc2RJeq2mah+Ep0jIC5QsQpsu3S/2R5YZpr8G32DPZ+SHZz7PbbvvqoOa2X2r/YnXo33sJ0xqUv//+e/nuu++cP5csWZL987///W+jW7lzJ5u2rVek99xzT2vmoP4yQBcg0y8vP+xY0zGXQtMSYvLHNPPDqp+rxH6exSzn1tBC4ZswnZRPWjz9SFOICfozXup7g9/Pd+7n2MtMIEx7UWIfBBDwKkCY9irFfk0EygmapcJqsRCb//de9tPzFAr7xX4BUCz8lmrDa1+KTZ38RcTcppi+Wuqqq66Svfbay21XI/+dV2Ml+5npzA+0uT+o+gnT+ZO20C9d3EK63ytdhGkjv1VYU1SawnRmUPz80qvUQAb5+S71OU7DnSdpnIfWfJOg0FQIZBYgW7VqlbP4GAuQpWLYg+lkOWG60K3XQdwC7nb7d+7VYi/7FgvjGTk/QT2oMD1y5Eg57LDDghm8GFohTCc7TBf6oTX/GUu3aZcblnXfYsHcy90N+cfmt5cbDor9m1u9tvx7Wu6QiHo80hZigvqMl2onNxiXGk+vn+80/MIsbfMw6s8550PAqwCvxvIqxX5ZgXJv884nLBam3d5T7eX8hZ5j9humSw15oavkbnVn2tMr0/o89PLlyxt9rVixougp9X3SvXv3dr5s2wjTyQ3Txa7+5IbpQleVc69y5f/vUmHabe4XunpGmJ7usCV9VXm3uRHkv6cpxAT5GXcL025j5OfzTZh20+TfEUAgKAHCdFCSKWrHS5jNcHhZ3CsTQr2262U/k8N0sanyww8/yDvvvCPvvvuu8+ecOXOa7Kqh+thjj5V9993XmhlHmE5+mM6djLmvxioVmvNDdqEAnvmB2Ouzjn5+2E7DbaBcmQ7n22Qaw3Sln/HML7UquV3cz+ebMB3O3KdVBBBoKkCYZlb4FvASZpMQpr1eafbj4Qf7ww8/lKlTp8rzzz8vixYtanToFVdcYU2gJkwnN0wXm8/lXpkuFJ79PH/t54dtwrSf70bsmyuQpjAd9GecMB3cZ4l5GJwlLSFQiQBhuhK9lB7rJzzafGU67jCdmV4apDVQT548WT7//PPsrLMhUA8bNkyqqqqcmvU207Rsab8iWO4z06XCtNvz0rlXxDPzrFRgJkyn5dMYfD8JMT+/Wi7zmfMiXGwRQa/t+P18Z76X6J9e72zx0g+T9mEemjQa1JJmAcJ0mke/zL6HFaa1nFKrbse9mnemvlKvxiqT1NNh+v7pM888U7799tvs/mPGjJE99tjD0/Fx7MR7ptP3rGpuSPVzm3duAC53ZfBiV6bzf6DODeZJ/UFb+5z2X+qE9T0v7SGmks94kFemSwXmNPyyLO3zMKzPN+0i4FVgypQpol8NDQ3Ss2dP58uUraq2trbBlGKoo6mA2zubvS7O5fb6qfwze30PtB7n953U+ccEtWJ50PNH31E9YsQIqa2tdZru3r2789osUzfCdDrCdP5VY7+LjhVaoTd3TnsNvKXCdLHPiNe2Tf2MlaqLMB3OqKUxxAT1GS90h4nXz2ChRz683LESziyIv9U0zsP41akAgV8EeM80s6FsgTDDdKao/HO4rcSd2xm/+3p5B3WmfS+v+Cob1uOBf/vb3xoF6FtuuUV22GEHj0dHuxthOtlh2m11XlOuTOffjpr7/73+IB/tJyeYsxGmg3HMbyVNISboz3jQV6YzY1Ms7IczA8xoNU3z0AxxqkCgsQBhmhmBgMUC+ixyTU2N04NevXrJhRdeaGRvhg4d6jwzrV88M23kEAVaVP4P3mGGaS9XuNJwq2epASRMBzq9s42lOcRU+hn3Gqa9fL7DGV17Wk3zPLRnlKg0yQKE6SSPLn1LvMArr7wi5557brafL730kqy55ppG9pvVvNOzmne5PyiHdXWYMH1SdvEl3jMd3LfHNIcY0z7jwY2qfS2leR7aN1pUnESBzDPT9fX1zoUtnplO4ijTp0QLHHTQQdlXZk2aNEk23XRTI/tLmE5PmDZtAhKmCdNhzElCTBiqtOlXgHnoV4z9VaDY2gNh/VI7Deqs5p2GUaaPiRQ4/vjj5aOPPnL6Nn78eOnSpYuR/SRME6bjmpiEacJ0GHOPEBOGKm36FWAe+hVLzv6FArHX3hGmvUp5348w7d2KPREwSuC8886Tl19+2anp0ksvlR49ehhVX6YYwjRh2siJmYKieGY6nEEmxITjSqv+BJiH/rxs2LvY6vTlvC7SbaV7Px5ub9wot63McbYvIEiY9jMD2BcBgwRuvvlmuf/++52KdEGy/v37G1TdL6UQpgnTRk7MFBRFmA5nkAkx4bjSqj8B5qE/Lxv2drtqXCoge71N2+0cuU5+9vXi6+f1lV774+W8Ye9DmA5bmPYRCEngzjvvFP3SbeDAgc6XaduMGTOkWbNmzmreQf6W1LR+5tdDiDF9hNJRH/MwnHEmxITjSqv+BJiH/rxs2LvUAn/Fbu0u9ThTObeDuy0yWE6bGftiYbrYlW9bAjVh2oZPFzUiUEDAhjDNe6aT/Z5pPphmCxCmwxkfQkw4rrTqT4B56M/Lhr2LBdlSAdbrv1V6QSMTbIMM08XGxJb1TubNmyfz58+Xuro6ad++vXTo0MGYaVZVW1vbYEw1FIKAoQKEaUMHRkQIMeaOTZoqYx6GM9qEmHBcadWfAPPQn5cNe+cG1cz/LicEFwq+XkOw237l1JNr7+Vqsy1hmvdM2/CpokYESggQps2dHoQYc8cmTZUxD8MZbUJMOK606k+AeejPy4a9C4Vpv3UXa8NLONegG+Vt3rZfmSZM+52d7I+AYQKEacMGJKccQoy5Y5OmypiH4Yw2ISYcV1r1J8A89Odlw95RhelSq4OHGaa9jIEtV6W1L4RpLyPKPggYLGBDmK6pqXEWINOvSm8NMngompRGiLFptJJbK/MwnLElxITjSqv+BJiH/rxs2NttATKvfci9zTv3GL9Xnt1W8y7n57pSt3nntufldnCvHmHtR5gOS5Z2EYhIwIYwrRS8GotXY0X0keA0eQKE6XCmBCEmHFda9SfAPPTnZcPebleFvfSh1KrfQYdpL/V43cemK9KZPmUWIFu1apWz+BgLkHkdbfZDwBABwrQhA1GgDEKMuWOTpsqYh+GMNiEmHFda9SfAPPTnZcPeUYXpYhaFrmjnXyF2W6DMr7NtV6ML9Y9XY/kddfZHwBABwrQhA0GYNncgUl4ZYTqcCUCICceVVv0JMA/9edmwd5C3eRe6BbvQlemMS/7z2vr3hW61DjJMJyFIqxNh2oZPFzUiUECAMG3utCDEmDs2aaqMeRjOaBNiwnGlVX8CzEN/Xjbs7fXKdKlA62U1b7eQ7NZ+pZZBvLO60hqCPJ4wHaQmbSEQoQBhOkJsn6cixPgEY/dQBJiHobA2WsFVjQcPHvz/27sXYLuq+n7gKyQBNIRnMAklIFBQqG0FyxBLZ7AqJQVFzYClxgdYpCEdKMijWKYiooyKBUNpQMTSWjUULDpQeYxY6sgjo1NQlBGp9k9MBJRIg0lEQnLzn3XkXE5uzr1n73PP2XuvvT9n5g6au/dev/X5rTy+d7+GM5CjEphAQJiu3/LodTa519O+e53Zbp+ZHk+uV8id6L7mbt/LEspTeNBYr5UmTPcS8n0CFRVIIUwvWbIkTJkypSV47bXXVlRy8GUJMYM3dcT8AtZhfrMsewgxWZRsM2wB63DYwsUff7ynZ8dKej08rPM90t1Ccbfvx+N2C+Dt8cZ+f6KHm3Xu05bLEr4nUk4laAvTxf9eMSKBgQikEKbnz58/OtdU/lAcRHPOO++88J//+Z+tQ330ox8NCxYsGMRhHYNALoGFCxeGlStXtvZZvnx5OOigg3Ltb+PuAjfccEO47LLLWt+MxhdeeCEqAoULxL9bbr755ta48e+ck046qfAaDDhYgSyvohrv31KdYbkdbLudyc569nqiQD42LHcL0r0Cdbez8GM1q/7vxltvvTXEry1btoTjjjuu9VWVz5SNGzduqUox6iBQVQFhuqqdCeHKK68M//Iv/9IqcPHixeF973tfdYtVWW0FDj/88DAyMtKa3z333BNe8pKX1HauRU7s3nvvDWeeeWZryGh8zTXXFDm8sQiM/t3y7W9/u/W/4985Rx55JJnEBQYRpiNBtwd7ZT0z3RmCO8+Gjz3LnCUMd2tH1QNyniXkPdN5tGxLoIICwnQFm/JCSV/+8pfDRz7ykdb/e8UrXhG++MUvVrfYCle2Zs2aMGvWrApXWN3SbrnllnDxxRe3Cpw9e3a47bbbqltsYpX95Cc/CW9729tGq77pppvC/vvvn9gslJuywP/+7/+GE088cXQK8e+cffbZJ+UpqX2cS667vZoqS0gd7+ncvULw2DPS44Xwfho2yCeB9zP+oPcRpgct6ngEChZIIUyffvrprXum41eT7pn++c9/Ht761reG5557rrUq4j96osUuu+xS8CpJe7h4Oa1LF/P38Otf/3r45Cc/GeI6jJ/3vOc9o2dS8x/NHt0EOu9Hj2cE4+/vgw8+uLZY9913X/jsZz8bHn300fDss8+GuXPntr7mzJnT9eulL31pbS3KntgPfvCDcPXVV4d4hUT8xIDSpL9fy/Yf5vjDCpv9Hneie56H6ZDKsYXpVDqlTgLjCKQQpmPpU6dODdttt13j+vj5z38+XHHFFaPz3nnnncO8efNaX/6hmW05fOMb32id2Y//YPfpLfDEE0+EVatWhdWrV49uHAPPV77ylTBt2rTeB7BFZoHvfOc74S/+4i+22v6AAw5o/f7efffdMx8nlQ1/+MMfhocffnircts/KG3fSjB2LvH78c+6GTNmbPXf+Gvxy20H+br/9NNPt35///jHP95qx/hDjle/+tX5DmZrAgQmLdC+Zzr+GfimN73JPdOTFnUAAgULCNMFg/cx3AUXXBC+9rWv9bGnXQhMXiBeCXHRRReFo446avIHc4RtBOKfwfEMoQ+BsgTiFRGnnnpqWcMblwCBEIKneVsGBBIVEKbTaFy85HbZsmXhscceS6NgVdZCIN5aEF9NF6+I8BmewNhLboc3kiMTeFGgCbcW6DeBVASE6VQ6pU4CYwSE6bSWRHyYVrxEL35t2rQpreJLqDbeA/jUU0+1Ro4PezrkkENKqCKtIffYY4/RWwmmT5+eVvGJV7tu3brR398bNmxIfDbdy49vKOi8haDfScZbDuJl3jEQHnbYYf0epnH7xcvl27cKzZw5s3HzN2ECVRUQpqvaGXUR6CEgTFsidRV46KGHwimnnNKaXrznPl6m3H6vb13nbF4Eqi5w/fXXh6uuumrSZR5//PHhne98Z4j3mPsQIEAgdQFhOvUOqr+xAimE6QcffLD18LH4IJper2NobCNNfBuBpUuXhs997nNb/fpdd90VdtttN1oECBQsEN9KEP8sv/vuu8OXvvSlvkbfYYcdwq677hoWL14cYpj2IUCAQF0EhOm6dNI8GieQQpieP3/+aF/GviuxcQ0z4UwC8RL4N7/5zaOvdWrvdOGFF4aFCxdmOoaNCBDoX6Adnh944IFWiI7/neznhBNOaJ2Njpcp+xAgQKAOAo8//niIb9HYvHlz660j8e0ZVflM2bhx45aqFKMOAlUVEKar2hl1TUbgnnvuCX/913+91SHi1Q2HH35460FuPgQIDFZg48aNrcA8yPAcK9x+++1bV5OceeaZYcGCBYMt2tEIECBQsoD3TJfcAMMTmKyAMD1ZQftXUeDSSy8NN998c9iyZdufqcb3JTuzVcWuqSklgX7C80EHHdR6WNjatWvDHXfckWm6J510Unj3u98dZs+enWl7GxEgQCAlAWE6pW6plUAXAWHasqibwNNPPx3e+ta3hvGehnzWWWeFd73rXXWbtvkQGKrAZMLzoYce2grRu+++e6vGlStXTni7RXyKfNz23HPPDa9//euHOi8HJ0CAQJkCwnSZ+sYmMACBFMJ0vGwwXqIbvzyAbABNr/khbrvttvB3f/d3487y4IMPDp///OdrrmB6BCYn0A7P7fuds9zzHM88t4NzZ3juVkl8uv769eu7FhnPRp966qkeFji5FtqbAIEEBITpBJqkRAITCaQQpmP98dVGMUz7EOgl8Ld/+7fha1/7WhgZGRl30ximY6j2IUDgNwLDDs9jnd/73veG7373u6O/3D4bHX///tEf/ZG2ECBAoBEC7QeQxQenxoePeQBZI9puknUSEKbr1E1zeeyxx8KJJ544YZCOSvGs1+mnnw6MQGMFxobneAa62zMGOoE6zzzHM9B77LFH335j3zcdz0YvWbIkzJgxo+9j2pEAAQKpCng1VqqdU3fjBYTpxi+BWgHccMMN4bLLLus5p/gAsvggMh8CTRGI4bnzku0s4fnAAw9s3escvyYbnsc6xx98vf3tbw+zZs0K8ZV1Rx55ZFNaYZ4ECBDYRkCYtigIJCogTCfaOGV3FYivw7rvvvt6npmOO19zzTWtV2X5EKijQNXCczfjz3zmM+Hkk08O8RJvHwIECDRZQJhucvfNPWkBYTrp9im+Q+Chhx4Kp5xySmaTeFnpeeedl3l7GxKoskD8h9jY9zz3umy7fea5/dCwyVy2XWUbtREgQKDqAsJ01TukPgLjCKQQpuN9dFOmTGnN4Nprr9VLAl0FOp+ImYUovnrnzjvv9GC7LFi2qZyA8Fy5liiIAAECfQsI033T2ZFAuQIphOn58+ePIv33f/93uWBGr6TA5s2bW2elH3744Vz1ffKTnwx//Md/nGsfGxMoQ0B4LkPdmAQIEBiuwK233hriV7yS6Ljjjmt9VeUzZePGjVuqUow6CFRVQJiuamfUlUfge9/7XuveyzyfeLVD/Evr4osvzrObbQkUItBveO58z7PLtgtplUEIECDQt4D3TPdNZ0cC1RBILUz/13/9V5g5c2Y18FRRGYF/+7d/C5/4xCdy17PjjjuG22+/Pey8886597UDgUEKtMNz5xO3s9zzLDwPsguORYAAgWIF4hVyy5cvbw161llnhfg8l6p8nJmuSifUUWmBFMJ0fB9w/Afm6173unDOOeeEvfbaq9Kmiite4Nxzzw133313XwNfdNFF4fjjj+9rXzsR6FdgsuE5huj4WikfAgQIEEhX4B3veEf44Q9/2JrAsmXLWq8irMpHmK5KJ9RRaYEUwvSjjz4aDjroIK9PqfRKKre4GIZ/+tOf5i4iXup91FFHhb//+7/Pva8dCOQRGBue4w8IR0ZGJjzEb//2b2/1nmfhOY+4bQkQIFB9gXi/dLzUe926deGuu+6qVMHCdKXaoZiqCqQQptt22223XZg6deoo5eOPP+4sdVUXVsF13XjjjeEXv/jFNqN+9atfDU888UTr19/0pjeFOXPmbLNNvG3ghBNOCPGSbx8CgxKI4bnzkm3heVCyjkOAAIH6CMTbeTZt2tR6tWGVzkpHYWG6PuvMTIYokFKYjgwxUMeveEax89KY+L3XvOY1XV+dFZ8Aftppp22lGF+xFbcf+4nbdT4x3DG3fRUZT2up/fvG76MX/wxph+cPf/jDoz/AyftHN09/JneuGX8fvfj2Dn8X+7u48/dGXf4dEq9Oim8jqepHmK5qZ9RVKYHUwnQbL4bpI444YivL+BO9a665Zhvf+IduvO+683P11Vd3DdOLFy9u/XSw/XFMnmP/AreWXhRp8u+jq666qvVDvXjGuf2V5bLtV7/61SHe7xyvsrngggv8ufSCQJPXkrlv+0MUfxf7d0j8o6Gu/waL8+r1gMkqhAVhugpdUEPlBVIN0/HS3fPPPz/8z//8z6hx/Adq/EfJ2E8Mx0uWLNnql8d7yEP7YWftjR2TZ+fCsZb8Pur8gV6WfwzF+5zjE1pjiO6859laspbaa8nfR9s+cMnfxQ/6t00IrR88NuXfdVUMDMJ0FbuipsoJpBqmKwepoEoKxB+itK80qNpTMisJpqhRgXgPWzzj/J3vfGf0v73OPB9wwAGt0Bz/ATg2PKMlQIAAAQIpCQjTKXVLraUJCNOl0Ru4AAFhugDkmgzRGZ7bAVp4rklzTYMAAQIEcgsI07nJ7NBEAWG6iV1vzpyF6eb0Ou9MY3juPOuc5WnbzjznVbY9AQIECKQqIEyn2jl1FyogTBfKbbCCBYTpgsErPNxkwnP70m3vea5wg5VGgAABAgMVEKYHyulgdRUQpuvaWfOKAsJ0c9eB8Nzc3ps5AQIECExeQJievKEjNEBAmG5Akxs8RWG6Oc0fG57jJdy93t+5//77jz4sLD40zJnn5qwXMyVAgACBiQWEaSuEQAYBYToDkk2SFRCmk21dz8JjUB77tG3huSebDQgQIECAQCYBYToTk42aLiBMN30F1Hv+wnR9+jvZ8Bzve95zzz3rA2ImBAgQIEBgiALC9BBxHbo+AsJ0fXppJtsKCNPprop+w3Pne56F53T7r3ICBAgQKFdAmC7X3+iJCAjTiTRKmX0JCNN9sZWyk/BcCrtBCRAgQIBAVwFh2sIgkEFAmM6AZJNkBYTp6rYuhuex73nOcs+zM8/V7anKCBAgQKA+AsJ0fXppJkMUEKaHiOvQpQsI06W3YLQA4bk6vVAJAQIECBDoJSBM9xLyfQIhBGHaMqizgDBdXncnE57bZ5/d81xe/4xMgAABAs0WEKab3X+zzyggTGeEslmSAsJ0cW3rJzzvt99+W73nWXgurl9GIkCAAAECEwkI09YHgQwCwnQGJJskKyBMD691IyMjrfc8t9/1HO9/3rRp04QDCs/D64cjEyBAgACBQQoI04PUdKzaCgjTtW2tiYUQhOnBLYN2eO58aFie8Bwv3X7Zy142uIIciQABAgQIEBiagDA9NFoHrpOAMF2nbprLWAFhuv810W947nzatvDcv789CRAgQIBAmQLCdJn6xk5GQJhOplUK7UNAmM6OJjxnt7IlAQIECBCou4AwXfcOm99ABITpgTA6SEUFhOnxGyM8V3TRKosAAQIECFRAQJiuQBOUUH0BYbr6PVJh/wLC9It2MTy373duPzQsyz3PLtvuf/3ZkwABAgQIpCogTKfaOXUXKiBMF8ptsIIFmhymO8NzO0RnDc/tAO2e54IXrOEIECBAgEBFBITpijRCGdUWEKar3R/VTU6gSWG6n/D88pe/fKv3PAvPk1tv9iZAgAABAnUREKbr0knzGKqAMD1UXgcvWaDOYVp4LnlxGZ4AAQIECNRYQJiucXNNbXACwvTgLB2pegJ1CtNbtmwJ8V7n9v3O8b+9Ltt25rl6a1JFBAgQIEAgBQFhOoUuqbF0AWG69BYoYIgCKYfpdnhu3+8c//v8889PqNUZnuN9z7Nnzx6irkMTIECAAAECdRUQpuvaWfMaqIAwPVBOB6uYQEphut/w3Pm0beG5YgtGQMEsAAAgAElEQVRQOQQIECBAIFEBYTrRxim7WAFhulhvoxUrUOUwLTwXuxaMRoAAAQIECGQXEKazW9mywQLCdIOb34CpVylMC88NWHCmSIAAAQIEaiIgTNekkaYxXAFheri+jl6uQJlhOobnzvud4wPDstzz7LLtcteM0QkQIECAAIEQhGmrgEAGAWE6A5JNkhUoMkxPJjy3A7R7npNdagonQIAAAQK1EhCma9VOkxmWgDA9LFnHrYLAMMN0P+F53333DYceemgQnquwOtRAgAABAgQIjCcgTFsbBDIICNMZkGySrMCgw/TY9zz3umxbeE526SicAAECBAg0WkCYbnT7TT6rgDCdVcp2KQpMNkzH8Ny+7znLPc+d4TmefZ4zZ06KbGomQIAAAQIEGi4gTDd8AZh+NgFhOpuTrdIUyBumO8NzDNEbN26ccOIxPHc+MEx4TnOdqJoAAQIECBDYWkCYtiIIZBAQpjMg2SRZgV5hWnhOtrUKJ0CAAAECBIYoIEwPEdeh6yMgTNenl2ayrcDYMD1lypStXlflzLNVQ4AAAQIECBDYVkCYtioIZBCoe5h+7LHHwqpVq8Lq1avDhg0bMojYpE4CDzzwQGs68UFhDz/8cBgZGZlwervsskuYO3du617n+LXTTjvViaORc4n9nDdvXthnn33Crrvu2kgDkyZAgAABAnkFhOm8YrZvpEBdw/T9998fPv3pT4dHHnmkkX01aQIEthVYtGhRWLx4cZg+fToeAgQIECBAYAIBYdryIJBBoI5h+lOf+lS44YYbMszeJgQINE1g9uzZ4bLLLgsHHXRQ06ZuvgQIECBAILOAMJ2ZyoZNFqhbmI5npM8+++wXWzptS5h6wHNhu1nPhykzNze51Y2f+5bnpoQpO2xpvEOjADZPCSNrpoWRNdPDyKrtR6f+2te+NlxxxRWNojBZAgQIECCQR0CYzqNl28YK1C1Mn3zyyaOXdk87fH3Y4dhnwpRdNzW2vyZOgMBvBDZ9e0b49RdnjXJ88IMfDMceeyweAgQIECBAoIuAMG1ZEMggUKcwHR82dtJJJ/1m1tO2hBkX/jRM2dXZ6AzLwCYEGiHw3C27hefv3rk116OPPjpccskljZi3SRIgQIAAgbwCwnReMds3UqBOYfqb3/xmOO+881p9nPqKX4eXLP5ZI3tq0gQIdBfY/P92CM9eOaf1zYMPPjhcf/31qAgQIECAAIEuAsK0ZUEgg0CdwvTy5cvD0qVLW7OefuS6sMMJT2cQsAkBAk0R2LJuatjwwb1b042vPbvrrruaMnXzJECAAAECuQSE6VxcNm6qQJ3CdOdctl+wNmx/zDNNbat5EyAwjsD6s/cd/c6KFSs4ESBAgAABAl0EhGnLgkAGAWE6A5JNCBCojYAwXZtWmggBAgQIDFFAmB4irkPXR0CYrk8vzYQAgd4CwnRvI1sQIECAAAFh2hogkEFAmM6AZBMCBGojIEzXppUmQoAAAQJDFBCmh4jr0PUREKbr00szIUCgt4Aw3dvIFgQIECBAQJi2BghkEBCmMyDZhACB2ggI07VppYkQIECAwBAFhOkh4jp0fQSE6fr00kwIEOgtIEz3NrIFAQIECBAQpq0BAhkEhOkMSDYhQKA2AsJ0bVppIgQIECAwRAFheoi4Dl0fAWG6Pr00EwIEegsI072NbEGAAAECBIRpa4BABgFhOgOSTQgQqI2AMF2bVpoIAQIECAxRQJgeIq5D10dAmK5PL82EAIHeAsJ0byNbECBAgAABYdoaIJBBQJjOgGQTAgRqIyBM16aVJkKAAAECQxQQpoeI69D1ERCm69NLMyFAoLeAMN3byBYECBAgQECYtgYIZBAQpjMg2YQAgdoICNO1aaWJECBAgMAQBYTpIeI6dH0EhOn69NJMCBDoLSBM9zayBQECBAgQEKatAQIZBITpDEg2IUCgNgLCdG1aaSIECBAgMEQBYXqIuA5dHwFhuj69NBMCBHoLCNO9jWxBgAABAgSEaWuAQAYBYToDkk0qKxCD0U5XrNyqvm6/VtkJKKxwAWG6cHIDEiBAgECCAsJ0gk1TcvECwnTx5kbcVqDfACxMW015BYTpvGK2J0CAAIEmCgjTTey6OecWEKZzk9khh0BncOncrfNscpYgPd5xcpQyumm3M9n9HKe9z9jjtX+9XfN435/MmPbtX0CY7t/OngQIECDQHAFhujm9NtNJCAjTk8Cza0+BXmeOJwrJWUJor+N3FjheaM8S5seb6Hjjj/eDg55gNhi6gDA9dGIDECBAgEANBITpGjTRFIYvIEwP37jJI4wNm53/f6JwG80GcS/0ROO3+zKoMJ3lLHyT10JV5i5MV6UT6iBAgACBKgsI01XujtoqIyBMV6YVtSxkvDA7UYDNcgZ5spd9573MfLzmjP3hQOcPAVzmXc0lLUxXsy+qIkCAAIFqCQjT1eqHaioqIExXtDE1Kavbmeh+gnAMv1nOao9ly3LWuZ96Osdxz3Rai1WYTqtfqiVAgACBcgSE6XLcjZqYgDCdWMMSK7efADxeIM4bzMcG8HjcPPdYZ6HudYa980x1luPZZvgCwvTwjY1AgAABAukLCNPp99AMChAQpgtAbvAQRYTpiS7ZHvY90xO11mXe1Vz4wnQ1+6IqAgQIEKiWgDBdrX6opqICwnRFG1OTsno9gCzrNNtnmTu3z3LmOUuY73bfc9a64nYu886jVf62wnT5PVABAQIECFRfQJiufo9UWAEBYboCTahxCVnODPea/kSXZvc6ft4w3auWPN93ZjqPVnHbCtPFWRuJAAECBNIVEKbT7Z3KCxQQpgvEbuBQvcJuFpJeYXq8Y7TPGHeGp25nkbM8pCxLnWO3Eab7URv+PsL08I2NQIAAAQLpCwjT6ffQDAoQEKYLQG7wEIO6zLsbYbfLvNvbdbt0e6LLscf73mRaJ0xPRm94+wrTw7N1ZAIECBCoj4AwXZ9emskQBYTpIeI69DZPzx7vLHCvp2KP92qsLMfrdea5M1z127LxznjH4w0jqPdbp/1+80T39mfFihVICBAgQIAAgS4CwrRlQSCDgDCdAckmfQt0C6qdwbjXPc3jfb/96xMF4c7LvPt5SFi3M8u9gnknlDPTfS+boe4oTA+V18EJECBAoCYCwnRNGmkawxUQpofr2/Sjj3e/c3Tp9TTuzsCc9cx0t8vK22PF/2a9h3u8IJwnIOfZtunrpMj5C9NFahuLAAECBFIVEKZT7Zy6CxUQpgvlbtxgEz08rFu47QTqdvZ57NnmXvdkjxfC2+OMF747A/jYpmUNyVm3a9yiKHnCwnTJDTA8AQIECCQhIEwn0SZFli0gTJfdgXqPP4gw3Q7dnQF37Fnr8cLx2F8fe9n42GP20w33RPejVt4+wnR59kYmQIAAgXQEhOl0eqXSEgWE6RLxGzB0lsuqx7vvuTOkjheCJ7pnulf4znP/83itGsQxGrAMKjVFYbpS7VAMAQIECFRUQJiuaGOUVS0BYbpa/ahbNcMIm/0e02XXdVtd/c1HmO7PzV4ECBAg0CwBYbpZ/TbbPgWE6T7h7EaAQJICwnSSbVM0AQIECBQsIEwXDG64NAWE6TT7pmoCBPoTEKb7c7MXAQIECDRLQJhuVr/Ntk8BYbpPOLsRIJCkgDCdZNsUTYAAAQIFCwjTBYMbLk0BYTrNvqmaAIH+BITp/tzsRYAAAQLNEhCmm9Vvs+1TQJjuE85uBAgkKSBMJ9k2RRMgQIBAwQLCdMHghktTQJhOs2+qJkCgPwFhuj83exEgQIBAswSE6Wb122z7FBCm+4SzGwECSQoI00m2TdEECBAgULCAMF0wuOHSFBCm0+ybqgkQ6E9AmO7PzV4ECBAg0CwBYbpZ/TbbPgWE6T7h7EaAQJICwnSSbVM0AQIECBQsIEwXDG64NAWWL18eli5d2ir+z/7sz8LZZ5+d5kRCCJ0/GNh+wdqw/THPJDsXhRMgMBwBYXo4ro5KgAABAvUSEKbr1U+zGZLA17/+9XDhhRe2jn7UUUeFj3/840MaafiHFaaHb2wEAqkLCNOpd1D9BAgQIFCEgDBdhLIxkhf4/ve/H0499dTWPA455JDwT//0T8nOSZhOtnUKJ1CYgDBdGLWBCBAgQCBhAWE64eYpvTiBn//85+H4449vDbjnnnuGW2+9tbjBBzySMD1gUIcjUEMBYbqGTTUlAgQIEBi4gDA9cFIHrKvA/PnzR6f2r//6r+HAAw9McqrCdJJtUzSBQgWE6UK5DUaAAAECiQoI04k2TtnFC1x00UXhzjvvbA2c8kPIhOni144RCaQmIEyn1jH1EiBAgEAZAsJ0GerGTFLg7rvvDh/4wAdatc+cOTN87nOfC3Pnzk1uLluF6aOfCdsfuza5OSiYAIEhCmyeEtafu8/oACtWrBjiYA5NgAABAgTSFRCm0+2dygsW2LBhQ3jb294WfvnLX7ZGjg8kaz+UrOBSJjXcV7/61XDJJZe0jjHt0A1hx3evmdTx7EyAQL0ERn42PfzqY3u1JvVbv/Vb4d///d/rNUGzIUCAAAECAxIQpgcE6TDNELj88svDjTfeODrZc845J5x44olJTf6hhx4Kp512Wqvm7eZtDC99/xNJ1a9YAgSGK7Dp4ZeEX1/3stYgRxxxRFi6dOlwB3R0AgQIECCQqIAwnWjjlF2eQLxfeuXKlaMFXHzxxeGYY44pr6CcI69duzYsWLBgdK8d37EmTDt8Q86j2LxuAiOPbx+222tj3aZlPn0I/OryuWFk1fatPU866aRw1lln9XEUuxAgQIAAgfoLCNP177EZDljgiSeeaF3u3fmJYfpP/uRPwpFHHjng0YZzuH/4h38IX/jCF0YPvv3rfxmmvupXYbtZm8KUmZuHM6ijVlpg4+27hulveCZM2X5LpetU3BAENk8JI2umtb423rnraJCePn16+NKXvhRmz549hEEdkgABAgQIpC8gTKffQzMoQeDJJ59s3S+9Zs3W9xv//u//fvjDP/zD1ruoO79mzJhRQpXjD/n888+HE044IfzsZz+rVF2KIUCgOgJnnHFGWLRoUXUKUgkBAgQIEKiYgDBdsYYoJy2BT3ziE+Hmm2+esOiqPqjs0UcfDVdffXW4//7700JXLQECQxWIZ6QXL14sSA9V2cEJECBAoA4CwnQdumgOpQrcfvvt4Z577gnf/OY3w8aN295zWtUw3Ua77bbbWoF69erVYdWqVWH9+vWlehqcAIHiBeJTu/fee++w3377hT//8z93aXfxLTAiAQIECCQoIEwn2DQlV1MgBulvfOMb4fvf/36Il4HHe6vjV3xgWYqv0KqmsqoGLbB58+Zw3HHHhfhguvhJ8Qn1gzZxPAIECBAgQIBAFgFhOouSbQgQIFBTgXvvvbcVoNuf3/3d3w2f+cxnajpb0yJAgAABAgQIDE5AmB6cpSMRIEAgOYFLLrkkxFsVRkZGRmu/4YYbwstf/vLk5qJgAgQIECBAgECRAsJ0kdrGIkCAQIUE/u///i+85S1v2eZe//jwqZNPPrlClSqFAAECBAgQIFA9AWG6ej1REQECBAoRuOOOO8KHPvShbcaKZ6Xj2WkfAgQIECBAgACB8QWEaauDAAECDRW44IILWg/N27JlyzYC1113XXjVq17VUBnTJkCAAAECBAj0FhCmexvZggABArUTWLlyZetJ8+N93vWud4W/+qu/qt28TYgAAQIECBAgMCgBYXpQko5DgACBhARuvPHGcPnll49b8Z577hluvfXWhGakVAIECBAgQIBAsQLCdLHeRiNAgEAlBE4//fTw4IMPTljLpz71qTB//vxK1KsIAgQIECBAgEDVBITpqnVEPQQIEBiywPe+973wvve9r+coCxcuDOeff37P7WxAgAABAgQIEGiigDDdxK6bMwECjRaIDxeLX70+M2bMCPGJ39OnT++1qe8TIECAAAECBBonIEw3ruUmTIBAkwU2b94c3vOe94Qf/ehHmRg+8pGPhDe+8Y2ZtrURAQIECBAgQKBJAsJ0k7ptrgQINF7g3nvvDeecc04mhylTpoSjjz46fPjDH860vY0IECBAgAABAk0SEKab1G1zJUCg8QLxCd7xSd5ZP1OnTg3/8R//EXbbbbesu9iOAAECBAgQINAIAWG6EW02SQIECISwbt26EN8f/eSTT+bi+MAHPhDe8pa35NrHxgQIECBAgACBugsI03XvsPkRIEDgBYFvfetb4cwzz8ztccQRR4SlS5fm3s8OBAgQIECAAIE6CwjTde6uuREgQKBD4J//+Z/DNddc05fJTTfdFObNm9fXvnYiQIAAAQIECNRRQJiuY1fNiQABAl0ETjvttPDQQw/1ZXPGGWeERYsW9bWvnQgQIECAAAECdRQQpuvYVXMiQIBAF4GFCxeGNWvWbPOdTZs2hZGRkdavx3dKx6d4d362bNkSDj300HDllVdyJUCAAAECBAgQeEFAmLYUCBAg0HCBJUuWhAceeKClsGzZsnDYYYc1XMT0CRAgQIAAAQK9BYTp3ka2IECAQK0FhOlat9fkCBAgQIAAgSEJCNNDgnVYAgQIpCIgTKfSKXUSIECAAAECVRIQpqvUDbUQIECgBAFhugR0QxIgQIAAAQLJCwjTybfQBAgQIDA5AWF6cn72JkCAAAECBJopIEw3s+9mTYAAgVEBYdpiIECAAAECBAjkFxCm85vZgwABArUSEKZr1U6TIUCAAAECBAoSEKYLgjYMAQIEqiogTFe1M+oiQIAAAQIEqiwgTFe5O2ojQIBAAQLCdAHIhiBAgAABAgRqJyBM166lJkSAAIF8AsJ0Pi9bEyBAgAABAgSigDBtHRAgQKDhAsJ0wxeA6RMgQIAAAQJ9CQjTfbHZiQABAvUREKbr00szIUCAAAECBIoTEKaLszYSAQIEKikgTFeyLYoiQIAAAQIEKi4gTFe8QcojQIDAsAWE6WELOz4BAgQIECBQRwFhuo5dNScCBAjkEBCmc2DZlAABAgQIECDwgoAwbSkQIECg4QLz588fFVi2bFk47LDDGi5i+gQIECBAgACB3gLCdG8jWxAgQKDWAsJ0rdtrcgQIECBAgMCQBITpIcE6LIGUBNavXx9WrVoVVq9eHTZs2JBS6WodgMAdd9wxepRDDjkk7LPPPgM4qkOkJDB37twwb968sNdee6VUtloJECBAgECpAsJ0qfwGJ1CuwCOPPBI+/elPh/vvv7/cQoxOgEAlBGbOnBlOPvnksGjRokrUowgCBAgQIFBlAWG6yt1RG4EhClx//fWtIO1DgACBsQKvfOUrw6WXXupMtaVBgAABAgQmEBCmLQ8CDRT47ne/G/7yL/9yq5k/P3dueH7WrDCy004NFDFlAg0WGBkJ09asCdPXrAlTn3lmFOINb3hD+OhHP9pgGFMnQIAAAQITCwjTVgiBBgp0vgrp2YMPDmv/9E/DxnnzGihhygQIdArsfPfdYbdbbhn9pY997GPhda97HSQCBAgQIECgi4AwbVkQaJjAT37yk/D2t799dNaP/83fhOfnzGmYgukSIDCewO433RRm3ndf69sLFiwIH/rQh2ARIECAAAECwrQ1QIDAfffdF97//ve3IH594IHhZ0uWQCFAgMCowI4//nGYfdVVrf//O7/zO+Gzn/0sHQIECBAgQECYtgYIELjxxhvD5Zdf3oJY/9rXhl90nKWmQ4AAgXjf9N4vnI3eZZddwp133gmFAAECBAgQEKatAQIErrvuuhC/4mftggXhmWOOgUKAAIGtBPY9++zR/79ixQo6BAgQIECAgDBtDRAgIExbAwQI9BIQpnsJ+T4BAgQIEAjBA8isAgINExCmG9Zw0yXQh4Aw3QeaXQgQIECgcQLCdONabsJNFxCmm74CzJ9AbwFhureRLQgQIECAgDBtDRBomIAw3bCGmy6BPgSE6T7Q7EKAAAECjRMQphvXchNuuoAw3fQVYP4EegsI072NbEGAAAECBIRpa4BAwwSE6YY13HQJ9CEgTPeBZhcCBAgQaJyAMN24lptw0wWE6aavAPMn0FtAmO5tZAsCBAgQICBMWwMEGiYgTDes4aZLoA8BYboPNLsQIECAQOMEhOnGtdyEmy4gTDd9BZg/gd4CwnRvI1sQIECAAAFh2hog0DABYbphDTddAn0ICNN9oNmFAAECBBonIEw3ruUm3HQBYbrpK8D8CfQWEKZ7G9mCAAECBAgI09YAgYYJCNMNa7jpEuhDQJjuA80uBAgQINA4AWG6cS034aYLCNNNXwHmT6C3gDDd28gWBAgQIEBAmLYGCDRMQJhuWMNNl0AfAsJ0H2h2IUCAAIHGCQjTjWu5CTddQJhu+gowfwK9BYTp3ka2IECAAAECwrQ1QKBhAsJ0wxpuugT6EBCm+0CzCwECBAg0TkCYblzLTbjpAsJ0misghpuVV1wxbvHt8DPRNmnOXNVlCAjTZagbkwABAgRSExCmU+uYeglMUkCYniRgSbtPFKY7g0+eMN0roJc0VcNWQECYrkATlECAAAEClRcQpivfIgUSGKyAMD1YzyKOliVIt0N03oCcd/ux8+0MXf1Y5An//RzfPv0JCNP9udmLAAECBJolIEw3q99mSyAI0+ksgvGCamdwjrPpDKT9XO49mUBd1r7pdDHNSoXpNPumagIECBAoVkCYLtbbaARKFxCmS29B7gK6BdaJQnPeQJ13+84JCNO525nEDsJ0Em1SJAECBAiULCBMl9wAwxMoWkCYLlp8cuP1CtK9LrPOehl1v6G43/2iymT2nZyqvXsJCNO9hHyfAAECBAiEIExbBQQaJiBMp9XwduDs/G+cQZaQnOeMc55tx56ZnoxolnlM5vj27U9AmO7PzV4ECBAg0CwBYbpZ/TZbAu6ZTmwNjA3Twyi/36eBx1omc3Z5MvsOw8ExXxQQpq0GAgQIECDQW0CY7m1kCwK1EnBmOp12dobNscFz7OXd3c7wZg2rWY41nlrWMdJRV2n7hyRtiRUrVkAhQIAAAQIEuggI05YFgYYJCNPpNLxXmO72OqyJ9skSiPOG47Hj5dV1mXdesWK2d2a6GGejECBAgEDaAsJ02v1TPYHcAsJ0brLSdhjv4WIxgI4XmvOG6bH3Sue9dzpv+C4N08C5BITpXFw2JkCAAIGGCgjTDW28aTdXQJhOs/dZL/PuJ0yPPTucJyDn2TZN+WZWLUw3s+9mTYAAAQL5BITpfF62JpC8gDCdZgu7henJXuY93lnoPGenhek011OvqoXpXkK+T4AAAQIEvBrLGiDQOAFhOs2Wd7vkexBherx7lrOG5PEuRc+j7L7pPFrFbCtMF+NsFAIECBBIW8CZ6bT7p3oCuQWE6dxkpe0w0VO2x/te1su8e5197vX9iDLRNt2+lzWglwZu4FEBYdpiIECAAAECvQWE6d5GtiBQKwFhOs12DvIy7yxBuVdYbn9/vFdyxe93uxe726+n2ZF6Vy1M17u/ZkeAAAECgxEQpgfj6CgEkhEQppNp1VaFZg3TnTt1OxOcNUi3j9Pr7HNnYO4MYBNdPi5QV38NCtPV75EKCRAgQKB8AWG6/B6ogEChAsJ0odwDG2wQYTpvkJ4oUI89Vr/3TrtfemBLZKAHEqYHyulgBAgQIFBTAWG6po01LQLjCQjTaa6NbmG6cyadDyOb6Nf7Da/dwnO/x+oM6ZM9RprdrH7VwnT1e6RCAgQIEChfQJguvwcqIFCogDBdKLfBCCQpIEwn2TZFEyBAgEDBAsJ0weCGI1C2gDBddgeMT6D6AsJ09XukQgIECBAoX0CYLr8HKiBQqIAwXSi3wQgkKSBMJ9k2RRMgQIBAwQLCdMHghiNQtoAwXXYHjE+g+gLCdPV7pEICBAgQKF9AmC6/ByogUKiAMF0ot8EIJCkgTCfZNkUTIECAQMECwnTB4IYjULaAMF12B4xPoPoCwnT1e6RCAgQIEChfQJguvwcqIFCogDBdKLfBCCQpIEwn2TZFEyBAgEDBAsJ0weCGI1C2gDBddgeMT6D6AsJ09XukQgIECBAoX0CYLr8HKiBQqIAwXSi3wQgkKSBMJ9k2RRMgQIBAwQLCdMHghiNQtoAwXXYHjE+g+gLCdPV7pEICBAgQKF9AmC6/ByogUKiAMF0ot8EIJCkgTCfZNkUTIECAQMECwnTB4IYjULaAMF12B4xPoPoCwnT1e6RCAgQIEChfQJguvwcqIFCogDBdKLfBCCQpIEwn2TZFEyBAgEDBAsJ0weCGI1C2gDBddgeMT6D6AsJ09XukQgIECBAoX0CYLr8HKiBQqIAwXSi3wQgkKSBMJ9k2RRMgQIBAwQLCdMHghiNQtoAwXXYHjE+g+gLCdPV7pEICBAgQKF9AmC6/ByogUKhAZ5h+5phjwtoFCwod32AECFRcYMuWsO/73z9a5IoVKypesPIIECBAgEA5AsJ0Oe5GJVCawC233BIuvfTS1vgbXvOasOad7yytFgMTIFA9gelPPRX2euHPiDlz5oSvfOUr1StSRQQIECBAoAICwnQFmqAEAkUKPPjgg+H0009vDfncvvuGJ886q8jhjUWAQMUFXvKDH4SXXXttq8o/+IM/CFdddVXFK1YeAQIECBAoR0CYLsfdqARKE3jqqafCm9/85tHxnzrllPCr3/u90uoxMAEC1RKY/Y//GHb80Y9aRS1cuDCcf/751SpQNQQIECBAoCICwnRFGqEMAkUKfPzjHw9f/vKXW0NumTYtrD322Fag3rTHHkWWYSwCBCoisN2zz4Zpa9aEXW+/PcQz0+3P8uXLw3777VeRKpVBgAABAgSqJSBMV6sfqiFQiMDatWvDCSecENavX7/VeFumTg0jO+1USA0GIUCgIgIjI2HqunXbFPPe9743nHbaaRUpUhkECBAgQKB6AsJ09XqiIgKFCHzrW98Ky5YtC4888kgh4xmEAIF0BBYtWhTOOOOMdApWKQECBAgQKLbYTZ8AAAMqSURBVEFAmC4B3ZAEqiTwhS98Idxzzz1h9erVId5P7UOAQPMEdthhh7D33nuH/fffP8Qg/cpXvrJ5CGZMgAABAgRyCgjTOcFsTqDOAs8++2zYsGFDnadobgQIdBGYNWsWFwIECBAgQCCngDCdE8zmBAgQIECAAAECBAgQIEBAmLYGCBAgQIAAAQIECBAgQIBATgFhOieYzQkQIECAAAECBAgQIECAgDBtDRAgQIAAAQIECBAgQIAAgZwCwnROMJsTIECAAAECBAgQIECAAAFh2hogQIAAAQIECBAgQIAAAQI5BYTpnGA2J0CAAAECBAgQIECAAAECwrQ1QIAAAQIECBAgQIAAAQIEcgoI0znBbE6AAAECBAgQIECAAAECBIRpa4AAAQIECBAgQIAAAQIECOQUEKZzgtmcAAECBAgQIECAAAECBAgI09YAAQIECBAgQIAAAQIECBDIKSBM5wSzOQECBAgQIECAAAECBAgQEKatAQIECBAgQIAAAQIECBAgkFNAmM4JZnMCBAgQIECAAAECBAgQICBMWwMECBAgQIAAAQIECBAgQCCngDCdE8zmBAgQIECAAAECBAgQIEBAmLYGCBAgQIAAAQIECBAgQIBATgFhOieYzQkQIECAAAECBAgQIECAgDBtDRAgQIAAAQIECBAgQIAAgZwCwnROMJsTIECAAAECBAgQIECAAAFh2hogQIAAAQIECBAgQIAAAQI5BYTpnGA2J0CAAAECBAgQIECAAAECwrQ1QIAAAQIECBAgQIAAAQIEcgoI0znBbE6AAAECBAgQIECAAAECBIRpa4AAAQIECBAgQIAAAQIECOQUEKZzgtmcAAECBAgQIECAAAECBAgI09YAAQIECBAgQIAAAQIECBDIKSBM5wSzOQECBAgQIECAAAECBAgQEKatAQIECBAgQIAAAQIECBAgkFNAmM4JZnMCBAgQIECAAAECBAgQICBMWwMECBAgQIAAAQIECBAgQCCngDCdE8zmBAgQIECAAAECBAgQIEBAmLYGCBAgQIAAAQIECBAgQIBATgFhOieYzQkQIECAAAECBAgQIECAwP8HxRXBYHNg/JQAAAAASUVORK5CYII="
    }
   },
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![VAE%E8%AE%A1%E7%AE%97%E5%9B%BE%20%281%29.jpg](attachment:VAE%E8%AE%A1%E7%AE%97%E5%9B%BE%20%281%29.jpg)\n",
    "<center> VAE网络计算图"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 论文复现"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 准备阶段"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-06-12T14:05:33.679832Z",
     "start_time": "2022-06-12T14:05:31.427423Z"
    }
   },
   "outputs": [],
   "source": [
    "import torch\n",
    "import torch.nn as nn\n",
    "import torch.nn.functional as F\n",
    "import torchvision\n",
    "from torchvision import transforms\n",
    "from torch.utils.data import DataLoader\n",
    "import torch.optim as optim\n",
    "from torch.autograd import Variable\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "import warnings\n",
    "import time\n",
    "import random"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "系统配置"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-06-12T14:05:33.727146Z",
     "start_time": "2022-06-12T14:05:33.679832Z"
    }
   },
   "outputs": [],
   "source": [
    "warnings.filterwarnings('ignore')   # 忽视调用文件时的警告信息\n",
    "plt.rcParams[\"font.sans-serif\"] = [\"SimHei\"]   # 支持绘图中文标签\n",
    "plt.rcParams[\"axes.unicode_minus\"] = False\n",
    "device=torch.device('cuda' if torch.cuda.is_available() else 'cpu') # 如果电脑上有GPU则使用GPU设备，否则使用CPU"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "设置随机数种子"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-06-12T14:05:33.742343Z",
     "start_time": "2022-06-12T14:05:33.731144Z"
    }
   },
   "outputs": [],
   "source": [
    "def set_seed(seed):\n",
    "    random.seed(seed)\n",
    "    np.random.seed(seed)\n",
    "    torch.manual_seed(seed)\n",
    "    torch.cuda.manual_seed(seed)\n",
    "    torch.cuda.manual_seed_all(seed)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-06-12T14:05:33.758098Z",
     "start_time": "2022-06-12T14:05:33.745098Z"
    }
   },
   "outputs": [],
   "source": [
    "set_seed(0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 数据预处理"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "定义一个转换器，将数据转换为tensor格式"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-06-12T14:05:33.774055Z",
     "start_time": "2022-06-12T14:05:33.760057Z"
    }
   },
   "outputs": [],
   "source": [
    "transform=transforms.Compose([transforms.ToTensor()])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "手写数字图片是著名的数据集，pytorch中收录了该数据集，因此我们直接下载"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-06-12T14:05:33.900684Z",
     "start_time": "2022-06-12T14:05:33.776015Z"
    }
   },
   "outputs": [],
   "source": [
    "train=torchvision.datasets.MNIST('./',download=True,transform=transform,train=True)\n",
    "test=torchvision.datasets.MNIST('./',download=True,transform=transform,train=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "我们取第一个样本看看"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-06-12T14:05:33.915644Z",
     "start_time": "2022-06-12T14:05:33.902676Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(tensor([[[0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
      "          0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
      "          0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
      "          0.0000, 0.0000, 0.0000, 0.0000],\n",
      "         [0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
      "          0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
      "          0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
      "          0.0000, 0.0000, 0.0000, 0.0000],\n",
      "         [0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
      "          0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
      "          0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
      "          0.0000, 0.0000, 0.0000, 0.0000],\n",
      "         [0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
      "          0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
      "          0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
      "          0.0000, 0.0000, 0.0000, 0.0000],\n",
      "         [0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
      "          0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
      "          0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
      "          0.0000, 0.0000, 0.0000, 0.0000],\n",
      "         [0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
      "          0.0000, 0.0000, 0.0000, 0.0000, 0.0118, 0.0706, 0.0706, 0.0706,\n",
      "          0.4941, 0.5333, 0.6863, 0.1020, 0.6510, 1.0000, 0.9686, 0.4980,\n",
      "          0.0000, 0.0000, 0.0000, 0.0000],\n",
      "         [0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
      "          0.1176, 0.1412, 0.3686, 0.6039, 0.6667, 0.9922, 0.9922, 0.9922,\n",
      "          0.9922, 0.9922, 0.8824, 0.6745, 0.9922, 0.9490, 0.7647, 0.2510,\n",
      "          0.0000, 0.0000, 0.0000, 0.0000],\n",
      "         [0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.1922,\n",
      "          0.9333, 0.9922, 0.9922, 0.9922, 0.9922, 0.9922, 0.9922, 0.9922,\n",
      "          0.9922, 0.9843, 0.3647, 0.3216, 0.3216, 0.2196, 0.1529, 0.0000,\n",
      "          0.0000, 0.0000, 0.0000, 0.0000],\n",
      "         [0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0706,\n",
      "          0.8588, 0.9922, 0.9922, 0.9922, 0.9922, 0.9922, 0.7765, 0.7137,\n",
      "          0.9686, 0.9451, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
      "          0.0000, 0.0000, 0.0000, 0.0000],\n",
      "         [0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
      "          0.3137, 0.6118, 0.4196, 0.9922, 0.9922, 0.8039, 0.0431, 0.0000,\n",
      "          0.1686, 0.6039, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
      "          0.0000, 0.0000, 0.0000, 0.0000],\n",
      "         [0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
      "          0.0000, 0.0549, 0.0039, 0.6039, 0.9922, 0.3529, 0.0000, 0.0000,\n",
      "          0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
      "          0.0000, 0.0000, 0.0000, 0.0000],\n",
      "         [0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
      "          0.0000, 0.0000, 0.0000, 0.5451, 0.9922, 0.7451, 0.0078, 0.0000,\n",
      "          0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
      "          0.0000, 0.0000, 0.0000, 0.0000],\n",
      "         [0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
      "          0.0000, 0.0000, 0.0000, 0.0431, 0.7451, 0.9922, 0.2745, 0.0000,\n",
      "          0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
      "          0.0000, 0.0000, 0.0000, 0.0000],\n",
      "         [0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
      "          0.0000, 0.0000, 0.0000, 0.0000, 0.1373, 0.9451, 0.8824, 0.6275,\n",
      "          0.4235, 0.0039, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
      "          0.0000, 0.0000, 0.0000, 0.0000],\n",
      "         [0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
      "          0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.3176, 0.9412, 0.9922,\n",
      "          0.9922, 0.4667, 0.0980, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
      "          0.0000, 0.0000, 0.0000, 0.0000],\n",
      "         [0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
      "          0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.1765, 0.7294,\n",
      "          0.9922, 0.9922, 0.5882, 0.1059, 0.0000, 0.0000, 0.0000, 0.0000,\n",
      "          0.0000, 0.0000, 0.0000, 0.0000],\n",
      "         [0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
      "          0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0627,\n",
      "          0.3647, 0.9882, 0.9922, 0.7333, 0.0000, 0.0000, 0.0000, 0.0000,\n",
      "          0.0000, 0.0000, 0.0000, 0.0000],\n",
      "         [0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
      "          0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
      "          0.0000, 0.9765, 0.9922, 0.9765, 0.2510, 0.0000, 0.0000, 0.0000,\n",
      "          0.0000, 0.0000, 0.0000, 0.0000],\n",
      "         [0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
      "          0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.1804, 0.5098,\n",
      "          0.7176, 0.9922, 0.9922, 0.8118, 0.0078, 0.0000, 0.0000, 0.0000,\n",
      "          0.0000, 0.0000, 0.0000, 0.0000],\n",
      "         [0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
      "          0.0000, 0.0000, 0.0000, 0.0000, 0.1529, 0.5804, 0.8980, 0.9922,\n",
      "          0.9922, 0.9922, 0.9804, 0.7137, 0.0000, 0.0000, 0.0000, 0.0000,\n",
      "          0.0000, 0.0000, 0.0000, 0.0000],\n",
      "         [0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
      "          0.0000, 0.0000, 0.0941, 0.4471, 0.8667, 0.9922, 0.9922, 0.9922,\n",
      "          0.9922, 0.7882, 0.3059, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
      "          0.0000, 0.0000, 0.0000, 0.0000],\n",
      "         [0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
      "          0.0902, 0.2588, 0.8353, 0.9922, 0.9922, 0.9922, 0.9922, 0.7765,\n",
      "          0.3176, 0.0078, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
      "          0.0000, 0.0000, 0.0000, 0.0000],\n",
      "         [0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0706, 0.6706,\n",
      "          0.8588, 0.9922, 0.9922, 0.9922, 0.9922, 0.7647, 0.3137, 0.0353,\n",
      "          0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
      "          0.0000, 0.0000, 0.0000, 0.0000],\n",
      "         [0.0000, 0.0000, 0.0000, 0.0000, 0.2157, 0.6745, 0.8863, 0.9922,\n",
      "          0.9922, 0.9922, 0.9922, 0.9569, 0.5216, 0.0431, 0.0000, 0.0000,\n",
      "          0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
      "          0.0000, 0.0000, 0.0000, 0.0000],\n",
      "         [0.0000, 0.0000, 0.0000, 0.0000, 0.5333, 0.9922, 0.9922, 0.9922,\n",
      "          0.8314, 0.5294, 0.5176, 0.0627, 0.0000, 0.0000, 0.0000, 0.0000,\n",
      "          0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
      "          0.0000, 0.0000, 0.0000, 0.0000],\n",
      "         [0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
      "          0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
      "          0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
      "          0.0000, 0.0000, 0.0000, 0.0000],\n",
      "         [0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
      "          0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
      "          0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
      "          0.0000, 0.0000, 0.0000, 0.0000],\n",
      "         [0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
      "          0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
      "          0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
      "          0.0000, 0.0000, 0.0000, 0.0000]]]), 5)\n",
      "<class 'tuple'>\n"
     ]
    }
   ],
   "source": [
    "print(train[0])\n",
    "print(type(train[0]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "可以看到第一个样本是一个元组数据类型，该元组包含两个数据（X,y）,X是一个tensor数据类型，代表图片的像素值，y是该图片的标签，第一个样本标签为5，代表这张图片是手写数字5"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "接下来我们看看图片的尺寸"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-06-12T14:05:33.931603Z",
     "start_time": "2022-06-12T14:05:33.918634Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "torch.Size([1, 28, 28])\n"
     ]
    }
   ],
   "source": [
    "print(train[0][0].shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "该图片是1通道，28 * 28的尺寸，常见的彩色图片格式是采用 RGB 3通道的像素值储存的手写数字数据集是黑白图片，因此是1通道"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "接下来我们看看样本量"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-06-12T14:05:33.947558Z",
     "start_time": "2022-06-12T14:05:33.934592Z"
    }
   },
   "outputs": [],
   "source": [
    "train_size=len(train)\n",
    "test_size=len(test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-06-12T14:05:33.963516Z",
     "start_time": "2022-06-12T14:05:33.952542Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "60000 10000\n"
     ]
    }
   ],
   "source": [
    "print(train_size,test_size)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "可以看到训练样本足足有60000张，测试样本为10000张"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "我们每次训练并不是训练全部样本，而是不重复地抽样一批次的样本进行训练，因此需要设置每次训练时所抽取的样本数"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-06-12T14:05:33.979471Z",
     "start_time": "2022-06-12T14:05:33.965509Z"
    }
   },
   "outputs": [],
   "source": [
    "M=100   # 每次进行训练时选择的批量大小"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-06-12T14:05:33.995428Z",
     "start_time": "2022-06-12T14:05:33.982464Z"
    }
   },
   "outputs": [],
   "source": [
    "trainloader=DataLoader(train,batch_size=M,shuffle=True,num_workers=0,drop_last=True)  \n",
    "testloader=DataLoader(test,batch_size=M,shuffle=True,num_workers=0,drop_last=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## VAE模型的搭建"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-06-12T14:05:34.011386Z",
     "start_time": "2022-06-12T14:05:33.997424Z"
    }
   },
   "outputs": [],
   "source": [
    "class VAE(nn.Module):\n",
    "    \n",
    "    def __init__(self,D_in,D_enc,D_z,D_dec,D_out):   \n",
    "        super().__init__()\n",
    "        self.linear1=nn.Linear(D_in,D_enc)\n",
    "        self.linear21=nn.Linear(D_enc,D_z)\n",
    "        self.linear22=nn.Linear(D_enc,D_z)\n",
    "        self.linear3=nn.Linear(D_z,D_dec)\n",
    "        self.linear4=nn.Linear(D_dec,D_out)\n",
    "    \n",
    "    def encode(self,x):    # 定义编码方法\n",
    "        h=F.relu(self.linear1(x))\n",
    "        mu=self.linear21(h)\n",
    "        logsigma=self.linear22(h)   # 线性层逼近log(σ)好处在于线性函数可以输出负值，也能保证σ是非负的\n",
    "        return mu,logsigma   # encoder输出μ和log(σ)的编码，只是一个线性关系，有正有负，没有非线性激活函数\n",
    "    \n",
    "    def reparameterize(self,mu,logsigma):    # 重参数化技巧\n",
    "        sigma=torch.exp(logsigma)     # 还原σ\n",
    "        mu,sigma=mu.to(device),sigma.to(device)\n",
    "        self.mu=mu    # 记录编码后的mu\n",
    "        self.sigma=sigma    # 记录编码后的sigma\n",
    "        \n",
    "        eps=torch.from_numpy(np.random.normal(0, 1, size=sigma.size())).float()   # 随机采样高斯噪声\n",
    "        eps=Variable(eps)    # 高斯噪声是可变张量\n",
    "        eps=eps.to(device)   # 把张量放在GPU上\n",
    "        z=mu + sigma * eps        \n",
    "        return z   # 返回隐变量z\n",
    "    \n",
    "    def decode(self,z):    # 定义解码方法\n",
    "        h=F.relu(self.linear3(z))     \n",
    "        return F.sigmoid(self.linear4(h))   \n",
    "    \n",
    "    def forward(self,x,L):\n",
    "        if L >1:\n",
    "            outputs=torch.zeros(x.shape[0],L,x.shape[1])   # 解码器会输出三维张量，因为每一个样本会产生L个隐变量的编码\n",
    "            for i in range(x.shape[0]):\n",
    "                mu,logvar=self.encode(x[i])  # 记录一个样本经编码器后的输出\n",
    "                for l in range(L):\n",
    "                    z=self.reparameterize(mu,logvar)    # 随机采样L次高斯噪声，并重参数化z\n",
    "                    output=self.decode(z)       # 解码\n",
    "                    outputs[i,l,:]=output   # 记录该条输出\n",
    "        else:\n",
    "            mu,logvar=self.encode(x)\n",
    "            z=self.reparameterize(mu,logvar)\n",
    "            outputs=self.decode(z)\n",
    "        return outputs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 定义损失函数"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-06-12T14:05:34.027345Z",
     "start_time": "2022-06-12T14:05:34.013381Z"
    }
   },
   "outputs": [],
   "source": [
    "class VAE_loss(nn.Module):\n",
    "    \n",
    "    def __init__(self,lmd):\n",
    "        super().__init__()\n",
    "        self.lmd = lmd    # 超参数lambda\n",
    "\n",
    "    def forward(self, outputs, x, mu, sigma):\n",
    "        if len(outputs.shape)>=3:     # 代表着 L > 1 \n",
    "            total_recons_loss=0    # 记录该批样本的总重构损失\n",
    "            for m in range(outputs.shape[0]):\n",
    "                total_recons_loss_l=0     # 记录每一个样本经隐变量采样L次后的总损失\n",
    "                for l in range(outputs.shape[1]):\n",
    "                    recons_loss=nn.MSELoss(reduction='sum')(outputs[m,l,:],x[m]) # 计算重构损失\n",
    "                    total_recons_loss_l+=recons_loss\n",
    "\n",
    "                mean_recons_loss_l=total_recons_loss_l/outputs.shape[1] # 计算每一个样本经隐变量采样L次后的平均重构损失，outputs.shape[1]=L\n",
    "                total_recons_loss+=mean_recons_loss_l\n",
    "\n",
    "            mean_recons_loss=total_recons_loss/outputs.shape[0]   # 计算该批样本的平均损失，outputs.shape[0]=M\n",
    "        else:\n",
    "            mean_recons_loss=nn.MSELoss(reduction='mean')(outputs,x)\n",
    "        \n",
    "        latent_loss = 0.5 * torch.mean(torch.square(mu) + torch.square(sigma) - torch.log(torch.square(sigma)) - 1)   # 计算潜在损失\n",
    "        return mean_recons_loss + self.lmd * latent_loss"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 进行实验"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "定义一个实验函数，方便后面更改隐变量维度继续实验"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-06-12T14:05:34.059256Z",
     "start_time": "2022-06-12T14:05:34.030338Z"
    }
   },
   "outputs": [],
   "source": [
    "def experiment(trainloader,testloader,input_dim,N,N_z,N_h,lr=0.001,epochs=int(1e6),lmd=1,M=100,L=1):\n",
    "    '''\n",
    "    :param trainloader: 训练数据集的加载器\n",
    "    :param testloader: 测试数据集的加载器\n",
    "    :param input_dim: 输入维度\n",
    "    :param N: 总样本量\n",
    "    :param N_z: VAE隐变量z的维度\n",
    "    :param N_h: 隐藏层的节点数，原文献在MNIST数据集上设置为500,Frey Face数据集上为200\n",
    "    :param lr: 学习率，默认设置为0.001\n",
    "    :param epochs: 迭代轮数，默认迭代10^6轮，训练10^8个样本\n",
    "    :param lmd: 正则化系数，也是生成网络的方差，原文献是单位矩阵，因此默认设定为1\n",
    "    :param M: AEVB算法中的每轮迭代训练样本数\n",
    "    :param L: 采样的高斯噪声的样本数，用于期望的估计，原文献选择的是1，因此默认为1\n",
    "    :return: \n",
    "    '''\n",
    "    vae=VAE(input_dim,N_h,N_z,N_h,input_dim)\n",
    "    vae=vae.to(device)    # 将模型放在GPU上\n",
    "\n",
    "    loss_fn=VAE_loss(lmd)\n",
    "    loss_fn=loss_fn.to(device)   # 将损失函数放在GPU上\n",
    "\n",
    "    optimizer=optim.SGD(vae.parameters(),lr=lr)    # 原文献推荐SGD或Adagrad优化器，这里选择SGD\n",
    "\n",
    "    train_samples = 0      \n",
    "    train_samples_lst=[]   # 记录训练的样本数\n",
    "    train_ELBO_lst=[]     # 记录训练集上的ELBO\n",
    "    test_ELBO_lst=[]      # 记录测试集上的ELBO\n",
    "    start_time=time.time()   # 记录开始时间\n",
    "    \n",
    "    for epoch in range(epochs):\n",
    "        vae.train()   # 将模型设置为训练模式\n",
    "        for data in trainloader:         # 在所有样本抽取M个样本，只抽一次，就退出循环进入下一轮迭代\n",
    "            inputs,_=data       # 获取输入数据，因为我们是无监督学习，所以不用获取标签\n",
    "            inputs=inputs.reshape(-1,28*28)      # 因为输入是一个28*28的2D图像，因此我们需要把他展平成28*28=784维的向量\n",
    "            inputs=inputs.to(device)   # 将数据放在GPU上\n",
    "\n",
    "            optimizer.zero_grad()   # 将梯度置零，每个批次样本之间是独立的，不能相互影响\n",
    "            outputs=vae(inputs,L)    # 记录vae的输出，即数据经encoder,再经decoder后的输出\n",
    "            outputs=outputs.to(device)    # 将输出数据放在GPU上\n",
    "\n",
    "            train_loss=loss_fn(outputs, inputs, vae.mu, vae.sigma)    # 计算损失\n",
    "            train_loss.backward()    # 误差反向传播\n",
    "            optimizer.step()    # 更新模型参数\n",
    "  \n",
    "            train_samples+=M\n",
    "            train_samples_lst.append(train_samples)\n",
    "        \n",
    "            train_loss=train_loss.cpu()    # 放回把loss放回CPU，绘图需在CPU上运行\n",
    "            train_loss=train_loss.detach().numpy()   # 把数据卸下来，不用有梯度信息了，并转成numpy格式\n",
    "            train_ELBO_lst.append(-train_loss*N)   # 依据式（18）和式（23），ELBO就是损失函数的负数\n",
    "            break\n",
    "\n",
    "        vae.eval()    # 将模型设置为验证模式\n",
    "        with torch.no_grad():    # 梯度置零\n",
    "            for data in testloader:\n",
    "                inputs,_=data\n",
    "                inputs=inputs.reshape(-1,28*28)\n",
    "                inputs=inputs.to(device)\n",
    "                outputs=vae(inputs,L)\n",
    "                outputs=outputs.to(device)\n",
    "                test_loss=loss_fn(outputs, inputs, vae.mu, vae.sigma)\n",
    "                \n",
    "                test_loss=test_loss.cpu()\n",
    "                test_loss=test_loss.detach().numpy()  \n",
    "                test_ELBO_lst.append(-test_loss*N)\n",
    "                break\n",
    "        \n",
    "        if train_samples%int(1e5)==0:   \n",
    "            print(f'------ train_samples = {train_samples} ------')\n",
    "            print(f'train_loss = {train_loss:.4f}')\n",
    "            print(f'test_loss = {test_loss:.4f}')\n",
    "            \n",
    "            end_time=time.time()\n",
    "            print(f'总运行时间：{end_time-start_time:.2f}s')\n",
    "            \n",
    "    return train_samples_lst,train_ELBO_lst,test_ELBO_lst,vae"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "设定数据的输入维度和样本量"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-06-12T14:05:34.075215Z",
     "start_time": "2022-06-12T14:05:34.061253Z"
    }
   },
   "outputs": [],
   "source": [
    "input_dim=28*28     # 输入维度\n",
    "N=len(train)+len(test)   # 全部的样本数"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### N_z=3"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "start_time": "2022-06-12T14:05:31.497Z"
    },
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "------ train_samples = 100000 ------\n",
      "train_loss = 0.2389\n",
      "test_loss = 0.2375\n",
      "总运行时间：46.01s\n",
      "------ train_samples = 200000 ------\n",
      "train_loss = 0.2347\n",
      "test_loss = 0.2348\n",
      "总运行时间：92.31s\n",
      "------ train_samples = 300000 ------\n",
      "train_loss = 0.2320\n",
      "test_loss = 0.2322\n",
      "总运行时间：138.74s\n",
      "------ train_samples = 400000 ------\n",
      "train_loss = 0.2294\n",
      "test_loss = 0.2293\n",
      "总运行时间：184.94s\n",
      "------ train_samples = 500000 ------\n",
      "train_loss = 0.2261\n",
      "test_loss = 0.2271\n",
      "总运行时间：231.56s\n",
      "------ train_samples = 600000 ------\n",
      "train_loss = 0.2248\n",
      "test_loss = 0.2242\n",
      "总运行时间：278.68s\n",
      "------ train_samples = 700000 ------\n",
      "train_loss = 0.2220\n",
      "test_loss = 0.2215\n",
      "总运行时间：325.14s\n",
      "------ train_samples = 800000 ------\n",
      "train_loss = 0.2196\n",
      "test_loss = 0.2191\n",
      "总运行时间：371.44s\n",
      "------ train_samples = 900000 ------\n",
      "train_loss = 0.2171\n",
      "test_loss = 0.2171\n",
      "总运行时间：417.72s\n",
      "------ train_samples = 1000000 ------\n",
      "train_loss = 0.2154\n",
      "test_loss = 0.2140\n",
      "总运行时间：464.10s\n",
      "------ train_samples = 1100000 ------\n",
      "train_loss = 0.2126\n",
      "test_loss = 0.2122\n",
      "总运行时间：510.58s\n",
      "------ train_samples = 1200000 ------\n",
      "train_loss = 0.2100\n",
      "test_loss = 0.2099\n",
      "总运行时间：557.01s\n",
      "------ train_samples = 1300000 ------\n",
      "train_loss = 0.2080\n",
      "test_loss = 0.2078\n",
      "总运行时间：603.42s\n",
      "------ train_samples = 1400000 ------\n",
      "train_loss = 0.2061\n",
      "test_loss = 0.2053\n",
      "总运行时间：649.89s\n",
      "------ train_samples = 1500000 ------\n",
      "train_loss = 0.2037\n",
      "test_loss = 0.2032\n",
      "总运行时间：696.29s\n",
      "------ train_samples = 1600000 ------\n",
      "train_loss = 0.2011\n",
      "test_loss = 0.2014\n",
      "总运行时间：742.81s\n",
      "------ train_samples = 1700000 ------\n",
      "train_loss = 0.1998\n",
      "test_loss = 0.1997\n",
      "总运行时间：789.70s\n",
      "------ train_samples = 1800000 ------\n",
      "train_loss = 0.1978\n",
      "test_loss = 0.1981\n",
      "总运行时间：836.06s\n",
      "------ train_samples = 1900000 ------\n",
      "train_loss = 0.1951\n",
      "test_loss = 0.1959\n",
      "总运行时间：882.35s\n",
      "------ train_samples = 2000000 ------\n",
      "train_loss = 0.1938\n",
      "test_loss = 0.1944\n",
      "总运行时间：928.94s\n",
      "------ train_samples = 2100000 ------\n",
      "train_loss = 0.1917\n",
      "test_loss = 0.1917\n",
      "总运行时间：976.78s\n",
      "------ train_samples = 2200000 ------\n",
      "train_loss = 0.1893\n",
      "test_loss = 0.1901\n",
      "总运行时间：1024.66s\n",
      "------ train_samples = 2300000 ------\n",
      "train_loss = 0.1880\n",
      "test_loss = 0.1879\n",
      "总运行时间：1072.49s\n",
      "------ train_samples = 2400000 ------\n",
      "train_loss = 0.1867\n",
      "test_loss = 0.1870\n",
      "总运行时间：1121.03s\n",
      "------ train_samples = 2500000 ------\n",
      "train_loss = 0.1849\n",
      "test_loss = 0.1821\n",
      "总运行时间：1168.88s\n",
      "------ train_samples = 2600000 ------\n",
      "train_loss = 0.1828\n",
      "test_loss = 0.1829\n",
      "总运行时间：1215.93s\n",
      "------ train_samples = 2700000 ------\n",
      "train_loss = 0.1805\n",
      "test_loss = 0.1816\n",
      "总运行时间：1262.71s\n",
      "------ train_samples = 2800000 ------\n",
      "train_loss = 0.1805\n",
      "test_loss = 0.1781\n",
      "总运行时间：1309.41s\n",
      "------ train_samples = 2900000 ------\n",
      "train_loss = 0.1781\n",
      "test_loss = 0.1773\n",
      "总运行时间：1355.66s\n",
      "------ train_samples = 3000000 ------\n",
      "train_loss = 0.1765\n",
      "test_loss = 0.1767\n",
      "总运行时间：1402.21s\n",
      "------ train_samples = 3100000 ------\n",
      "train_loss = 0.1764\n",
      "test_loss = 0.1736\n",
      "总运行时间：1449.30s\n",
      "------ train_samples = 3200000 ------\n",
      "train_loss = 0.1739\n",
      "test_loss = 0.1723\n",
      "总运行时间：1496.30s\n",
      "------ train_samples = 3300000 ------\n",
      "train_loss = 0.1722\n",
      "test_loss = 0.1708\n",
      "总运行时间：1543.27s\n",
      "------ train_samples = 3400000 ------\n",
      "train_loss = 0.1702\n",
      "test_loss = 0.1713\n",
      "总运行时间：1596.32s\n",
      "------ train_samples = 3500000 ------\n",
      "train_loss = 0.1698\n",
      "test_loss = 0.1695\n",
      "总运行时间：1643.80s\n",
      "------ train_samples = 3600000 ------\n",
      "train_loss = 0.1685\n",
      "test_loss = 0.1676\n",
      "总运行时间：1690.07s\n",
      "------ train_samples = 3700000 ------\n",
      "train_loss = 0.1635\n",
      "test_loss = 0.1651\n",
      "总运行时间：1737.15s\n",
      "------ train_samples = 3800000 ------\n",
      "train_loss = 0.1655\n",
      "test_loss = 0.1625\n",
      "总运行时间：1784.20s\n",
      "------ train_samples = 3900000 ------\n",
      "train_loss = 0.1633\n",
      "test_loss = 0.1640\n",
      "总运行时间：1831.06s\n",
      "------ train_samples = 4000000 ------\n",
      "train_loss = 0.1610\n",
      "test_loss = 0.1626\n",
      "总运行时间：1878.07s\n",
      "------ train_samples = 4100000 ------\n",
      "train_loss = 0.1588\n",
      "test_loss = 0.1578\n",
      "总运行时间：1924.54s\n",
      "------ train_samples = 4200000 ------\n",
      "train_loss = 0.1581\n",
      "test_loss = 0.1588\n",
      "总运行时间：1972.35s\n",
      "------ train_samples = 4300000 ------\n",
      "train_loss = 0.1568\n",
      "test_loss = 0.1589\n",
      "总运行时间：2019.14s\n",
      "------ train_samples = 4400000 ------\n",
      "train_loss = 0.1570\n",
      "test_loss = 0.1566\n",
      "总运行时间：2066.18s\n",
      "------ train_samples = 4500000 ------\n",
      "train_loss = 0.1553\n",
      "test_loss = 0.1552\n",
      "总运行时间：2114.18s\n",
      "------ train_samples = 4600000 ------\n",
      "train_loss = 0.1548\n",
      "test_loss = 0.1538\n",
      "总运行时间：2161.86s\n",
      "------ train_samples = 4700000 ------\n",
      "train_loss = 0.1519\n",
      "test_loss = 0.1520\n",
      "总运行时间：2209.44s\n",
      "------ train_samples = 4800000 ------\n",
      "train_loss = 0.1540\n",
      "test_loss = 0.1522\n",
      "总运行时间：2257.22s\n",
      "------ train_samples = 4900000 ------\n",
      "train_loss = 0.1497\n",
      "test_loss = 0.1511\n",
      "总运行时间：2304.09s\n",
      "------ train_samples = 5000000 ------\n",
      "train_loss = 0.1472\n",
      "test_loss = 0.1515\n",
      "总运行时间：2351.48s\n",
      "------ train_samples = 5100000 ------\n",
      "train_loss = 0.1482\n",
      "test_loss = 0.1476\n",
      "总运行时间：2400.21s\n",
      "------ train_samples = 5200000 ------\n",
      "train_loss = 0.1453\n",
      "test_loss = 0.1484\n",
      "总运行时间：2446.89s\n",
      "------ train_samples = 5300000 ------\n",
      "train_loss = 0.1470\n",
      "test_loss = 0.1460\n",
      "总运行时间：2494.44s\n",
      "------ train_samples = 5400000 ------\n",
      "train_loss = 0.1451\n",
      "test_loss = 0.1446\n",
      "总运行时间：2542.31s\n",
      "------ train_samples = 5500000 ------\n",
      "train_loss = 0.1449\n",
      "test_loss = 0.1439\n",
      "总运行时间：2590.59s\n",
      "------ train_samples = 5600000 ------\n",
      "train_loss = 0.1442\n",
      "test_loss = 0.1433\n",
      "总运行时间：2642.24s\n",
      "------ train_samples = 5700000 ------\n",
      "train_loss = 0.1405\n",
      "test_loss = 0.1403\n",
      "总运行时间：2691.72s\n",
      "------ train_samples = 5800000 ------\n",
      "train_loss = 0.1406\n",
      "test_loss = 0.1418\n",
      "总运行时间：2740.96s\n",
      "------ train_samples = 5900000 ------\n",
      "train_loss = 0.1386\n",
      "test_loss = 0.1386\n",
      "总运行时间：2789.40s\n",
      "------ train_samples = 6000000 ------\n",
      "train_loss = 0.1371\n",
      "test_loss = 0.1417\n",
      "总运行时间：2843.90s\n",
      "------ train_samples = 6100000 ------\n",
      "train_loss = 0.1368\n",
      "test_loss = 0.1362\n",
      "总运行时间：2899.79s\n",
      "------ train_samples = 6200000 ------\n",
      "train_loss = 0.1371\n",
      "test_loss = 0.1373\n",
      "总运行时间：2952.82s\n",
      "------ train_samples = 6300000 ------\n",
      "train_loss = 0.1375\n",
      "test_loss = 0.1380\n",
      "总运行时间：3004.20s\n",
      "------ train_samples = 6400000 ------\n",
      "train_loss = 0.1337\n",
      "test_loss = 0.1368\n",
      "总运行时间：3054.74s\n",
      "------ train_samples = 6500000 ------\n",
      "train_loss = 0.1327\n",
      "test_loss = 0.1367\n",
      "总运行时间：3104.23s\n",
      "------ train_samples = 6600000 ------\n",
      "train_loss = 0.1334\n",
      "test_loss = 0.1325\n",
      "总运行时间：3151.38s\n",
      "------ train_samples = 6700000 ------\n",
      "train_loss = 0.1294\n",
      "test_loss = 0.1320\n",
      "总运行时间：3201.04s\n",
      "------ train_samples = 6800000 ------\n",
      "train_loss = 0.1302\n",
      "test_loss = 0.1310\n",
      "总运行时间：3248.36s\n",
      "------ train_samples = 6900000 ------\n",
      "train_loss = 0.1293\n",
      "test_loss = 0.1300\n",
      "总运行时间：3300.00s\n",
      "------ train_samples = 7000000 ------\n",
      "train_loss = 0.1316\n",
      "test_loss = 0.1309\n",
      "总运行时间：3352.66s\n",
      "------ train_samples = 7100000 ------\n",
      "train_loss = 0.1288\n",
      "test_loss = 0.1297\n",
      "总运行时间：3404.02s\n",
      "------ train_samples = 7200000 ------\n",
      "train_loss = 0.1282\n",
      "test_loss = 0.1292\n",
      "总运行时间：3456.48s\n",
      "------ train_samples = 7300000 ------\n",
      "train_loss = 0.1300\n",
      "test_loss = 0.1287\n",
      "总运行时间：3506.60s\n"
     ]
    }
   ],
   "source": [
    "N_z=3\n",
    "train_samples_lst_3,train_ELBO_lst_3,test_ELBO_lst_3,vae_3=experiment(\n",
    "    trainloader,\n",
    "    testloader,\n",
    "    input_dim,\n",
    "    N,\n",
    "    N_z=N_z,\n",
    "    N_h=500,\n",
    "    lr=0.001,\n",
    "    epochs=int(1e5), # 由于训练回合数太多速度太慢，我们减少迭代轮数\n",
    "    lmd=1,\n",
    "    M=100,\n",
    "    L=1\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### N_z=5"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "start_time": "2022-06-12T14:05:31.501Z"
    }
   },
   "outputs": [],
   "source": [
    "N_z=5\n",
    "train_samples_lst_5,train_ELBO_lst_5,test_ELBO_lst_5,vae_5=experiment(trainloader,\n",
    "                                                                       testloader,\n",
    "                                                                       input_dim,\n",
    "                                                                       N,\n",
    "                                                                       N_z=N_z,\n",
    "                                                                       N_h=500,\n",
    "                                                                       lr=0.001,\n",
    "                                                                       epochs=int(1e5),\n",
    "                                                                       lmd=1,\n",
    "                                                                       M=100,\n",
    "                                                                       L=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### N_z=10"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "start_time": "2022-06-12T14:05:31.506Z"
    }
   },
   "outputs": [],
   "source": [
    "N_z=10\n",
    "train_samples_lst_10,train_ELBO_lst_10,test_ELBO_lst_10,vae_10=experiment(trainloader,\n",
    "                                                                       testloader,\n",
    "                                                                       input_dim,\n",
    "                                                                       N,\n",
    "                                                                       N_z=N_z,\n",
    "                                                                       N_h=500,\n",
    "                                                                       lr=0.001,\n",
    "                                                                       epochs=int(1e5),\n",
    "                                                                       lmd=1,\n",
    "                                                                       M=100,\n",
    "                                                                       L=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 试验结果展示"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "start_time": "2022-06-12T14:05:31.511Z"
    }
   },
   "outputs": [],
   "source": [
    "def show(x,y1,y2,N_z):\n",
    "    plt.figure(figsize=(5,5))\n",
    "    plt.plot(x,y1)\n",
    "    plt.plot(x,y2,linestyle='--')\n",
    "    plt.legend(['AEVB(train)','AEVB(test)'])\n",
    "    plt.ylabel('L')\n",
    "    plt.title(f'MNIST,Nz={N_z}')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "start_time": "2022-06-12T14:05:31.519Z"
    }
   },
   "outputs": [],
   "source": [
    "show(train_samples_lst_3,train_ELBO_lst_3,test_ELBO_lst_3,3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "start_time": "2022-06-12T14:05:31.524Z"
    },
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "show(train_samples_lst_5,train_ELBO_lst_5,test_ELBO_lst_5,5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "start_time": "2022-06-12T14:05:31.529Z"
    }
   },
   "outputs": [],
   "source": [
    "show(train_samples_lst_10,train_ELBO_lst_10,test_ELBO_lst_10,10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 可视化展示"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "start_time": "2022-06-12T14:05:31.535Z"
    }
   },
   "outputs": [],
   "source": [
    "train=train.data.float().to(device)\n",
    "test=test.data.float().to(device)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "start_time": "2022-06-12T14:05:31.539Z"
    }
   },
   "outputs": [],
   "source": [
    "def display(n_image,data_disp,vae):\n",
    "    fig=plt.figure(figsize=(n_image*1.5,3))\n",
    "    for i in range(n_image):\n",
    "        plt.subplot(2,n_image,1+i)\n",
    "        plt.imshow(data_disp.data[i].cpu(),cmap='gray')\n",
    "        plt.axis('off')\n",
    "        plt.subplot(2,n_image,1+n_image+i)\n",
    "        outputs=vae(data_disp.data[i].reshape(-1,28*28))\n",
    "        outputs=outputs.cpu()           # 把输出值放回CPU展示\n",
    "        outputs=outputs.data[0].numpy().reshape(28,28)\n",
    "        plt.imshow(outputs,cmap='gray')\n",
    "        plt.axis('off')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "start_time": "2022-06-12T14:05:31.543Z"
    }
   },
   "outputs": [],
   "source": [
    "display(5,test,vae_3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "start_time": "2022-06-12T14:05:31.547Z"
    }
   },
   "outputs": [],
   "source": [
    "display(5,test,vae_5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "start_time": "2022-06-12T14:05:31.551Z"
    }
   },
   "outputs": [],
   "source": [
    "display(5,test,vae_10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 继续阅读"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "&emsp;&emsp;尽管本篇论文非常经典，但是还是有点晦涩难懂，有两篇关于VAE的详细介绍，都是基于本篇论文的补充和教程，一篇是《Tutorial on Variational Autoencoders》，另一篇是《An Introduction to Variational Autoencoders》，更推荐后者，后者就是原作者Diederik P. Kingma和Max Welling写的，介绍的更加详细，VAE的前因后果、详细的数学推导、AVEB算法等等，都进行了详细的介绍。这两篇论文都打包在本次复现的附录里了。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 参考文献"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[1] Diederik P. Kingma, Max Welling, 2019, An Introduction to Variational Autoencoders[M]. $Foundations \\;and\\; Trends^R\\;in\\;Machine\\;Learning$.\\\n",
    "[2] Doersch C, 2016. Tutorial on Variational Autoencoders[J]. $CoRR$.\\\n",
    "[3] 邱锡鹏, 2020, 神经网络与深度学习[M]. 机械工业出版社.\\\n",
    "[4] Aurelien Geion, 2020, 机器学习实战（第二版）[M]. 机械工业出版社."
   ]
  }
 ],
 "metadata": {
  "interpreter": {
   "hash": "7758e92e9a61d7a3490898707f7eeb937c85e9d1e8d4e877cc6c187218f226d5"
  },
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.7"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {
    "height": "calc(100% - 180px)",
    "left": "10px",
    "top": "150px",
    "width": "223px"
   },
   "toc_section_display": true,
   "toc_window_display": true
  },
  "varInspector": {
   "cols": {
    "lenName": 16,
    "lenType": 16,
    "lenVar": 40
   },
   "kernels_config": {
    "python": {
     "delete_cmd_postfix": "",
     "delete_cmd_prefix": "del ",
     "library": "var_list.py",
     "varRefreshCmd": "print(var_dic_list())"
    },
    "r": {
     "delete_cmd_postfix": ") ",
     "delete_cmd_prefix": "rm(",
     "library": "var_list.r",
     "varRefreshCmd": "cat(var_dic_list()) "
    }
   },
   "position": {
    "height": "466.841px",
    "left": "231.455px",
    "right": "20px",
    "top": "111px",
    "width": "290px"
   },
   "types_to_exclude": [
    "module",
    "function",
    "builtin_function_or_method",
    "instance",
    "_Feature"
   ],
   "window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}