Introduction to Reinforcement Learning  .ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h2 align='center'>Reinforcement Learning Algorithms</h2>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Gym is a toolkit for developing and comparing reinforcement learning algorithms. It makes no assumptions about the structure of your agent, and is compatible with any numerical computation library, such as TensorFlow or Theano.\n",
    "\n",
    "The gym library is a collection of test problems — environments — that you can use to work out your reinforcement learning algorithms. These environments have a shared interface, allowing you to write general algorithms."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Installation\n",
    "To get started, you’ll need to have Python 3.5+ installed. Simply install gym using pip:"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "\n",
    "pip install gym\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And you’re good to go!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Environments**\n",
    "\n",
    "Here’s a bare minimum example of getting something running. This will run an instance of the CartPole-v0 environment for 1000 timesteps, rendering the environment at each step. You should see a window pop up rendering the classic cart-pole problem:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "import gym\n",
    "env = gym.make('CartPole-v0')\n",
    "env.reset()\n",
    "for _ in range(1000):\n",
    "    env.render()\n",
    "    env.step(env.action_space.sample()) # take a random action\n",
    "env.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Normally, we’ll end the simulation before the cart-pole is allowed to go off-screen. More on that later. For now, please ignore the warning about calling step() even though this environment has already returned done = True.\n",
    "\n",
    "If you’d like to see some other environments in action, try replacing CartPole-v0 above with something like MountainCar-v0, MsPacman-v0 (requires the Atari dependency), or Hopper-v1 (requires the MuJoCo dependencies). Environments all descend from the Env base class."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Observations**\n",
    "\n",
    "If we ever want to do better than take random actions at each step, it’d probably be good to actually know what our actions are doing to the environment.\n",
    "\n",
    "The environment’s step function returns exactly what we need. In fact, step returns four values. These are:\n",
    "\n",
    "`observation (object):` an environment-specific object representing your observation of the environment. For example, pixel data from a camera, joint angles and joint velocities of a robot, or the board state in a board game.\n",
    "\n",
    "`reward (float):` amount of reward achieved by the previous action. The scale varies between environments, but the goal is always to increase your total reward.\n",
    "\n",
    "`done (boolean):` whether it’s time to reset the environment again. Most (but not all) tasks are divided up into well-defined episodes, and done being True indicates the episode has terminated. (For example, perhaps the pole tipped too far, or you lost your last life.)\n",
    "\n",
    "`info (dict):` diagnostic information useful for debugging. It can sometimes be useful for learning (for example, it might contain the raw probabilities behind the environment’s last state change). However, official evaluations of your agent are not allowed to use this for learning.\n",
    "This is just an implementation of the classic “agent-environment loop”. Each timestep, the agent chooses an action, and the environment returns an observation and a reward.\n",
    "\n",
    "![](https://gym.openai.com/assets/docs/aeloop-138c89d44114492fd02822303e6b4b07213010bb14ca5856d2d49d6b62d88e53.svg)\n",
    "\n",
    "\n",
    "The process gets started by calling reset(), which returns an initial observation. So a more proper way of writing the previous code would be to respect the done flag:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[-0.04101546  0.01264148 -0.02200515 -0.00575931]\n",
      "[-0.04076263  0.20807199 -0.02212034 -0.3053031 ]\n",
      "[-0.03660119  0.40350207 -0.0282264  -0.60487935]\n",
      "[-0.02853115  0.20878598 -0.04032399 -0.32121903]\n",
      "[-0.02435543  0.01426079 -0.04674837 -0.04152046]\n",
      "[-0.02407021  0.21002084 -0.04757878 -0.34857876]\n",
      "[-0.0198698   0.40578607 -0.05455035 -0.65587718]\n",
      "[-0.01175407  0.21146429 -0.06766789 -0.38085793]\n",
      "[-0.00752479  0.40747859 -0.07528505 -0.69408568]\n",
      "[ 6.24784081e-04  6.03559652e-01 -8.91667671e-02 -1.00948704e+00]\n",
      "[ 0.01269598  0.79975125 -0.10935651 -1.32878562]\n",
      "[ 0.028691    0.60616588 -0.13593222 -1.0722289 ]\n",
      "[ 0.04081432  0.41307692 -0.1573768  -0.82510833]\n",
      "[ 0.04907586  0.22041708 -0.17387896 -0.58576797]\n",
      "[ 0.0534842   0.02810211 -0.18559432 -0.35250857]\n",
      "[ 0.05404624  0.22531132 -0.1926445  -0.69749433]\n",
      "[ 0.05855247  0.03330842 -0.20659438 -0.47110547]\n",
      "Episode finished after 17 timesteps\n",
      "[0.00957159 0.03880505 0.00524677 0.04973339]\n",
      "[ 0.01034769 -0.15639174  0.00624144  0.34406709]\n",
      "[ 0.00721986 -0.35160192  0.01312278  0.63871163]\n",
      "[ 1.87817606e-04 -1.56665380e-01  2.58970136e-02  3.50189984e-01]\n",
      "[-0.00294549 -0.35214588  0.03290081  0.65092528]\n",
      "[-0.00998841 -0.54771025  0.04591932  0.95378418]\n",
      "[-0.02094261 -0.74341892  0.064995    1.26053327]\n",
      "[-0.03581099 -0.93930922  0.09020567  1.57284339]\n",
      "[-0.05459718 -0.74537168  0.12166254  1.30960436]\n",
      "[-0.06950461 -0.94180652  0.14785462  1.63775922]\n",
      "[-0.08834074 -1.13832059  0.18060981  1.97262525]\n",
      "Episode finished after 11 timesteps\n",
      "[0.01760217 0.04146358 0.04129258 0.02222415]\n",
      "[ 0.01843144  0.23596977  0.04173706 -0.25714987]\n",
      "[0.02315084 0.04027756 0.03659406 0.04839994]\n",
      "[ 0.02395639 -0.15534949  0.03756206  0.35240032]\n",
      "[0.0208494  0.03921875 0.04461007 0.07179448]\n",
      "[ 0.02163378  0.2336737   0.04604596 -0.20648676]\n",
      "[0.02630725 0.03792458 0.04191622 0.10035843]\n",
      "[ 0.02706574 -0.15777225  0.04392339  0.4059656 ]\n",
      "[ 0.0239103  -0.35348865  0.0520427   0.71216638]\n",
      "[ 0.01684052 -0.54929111  0.06628603  1.02076607]\n",
      "[ 0.0058547  -0.35511202  0.08670135  0.74961092]\n",
      "[-0.00124754 -0.55131606  0.10169357  1.06826977]\n",
      "[-0.01227386 -0.74762534  0.12305896  1.3910588 ]\n",
      "[-0.02722637 -0.9440461   0.15088014  1.71954996]\n",
      "[-0.04610729 -0.75094092  0.18527114  1.4773736 ]\n",
      "Episode finished after 15 timesteps\n",
      "[ 0.02310776  0.02451955  0.03503003 -0.0076848 ]\n",
      "[ 0.02359815  0.21912207  0.03487634 -0.28911277]\n",
      "[0.02798059 0.02352059 0.02909408 0.01436259]\n",
      "[ 0.028451   -0.17200627  0.02938133  0.3160812 ]\n",
      "[ 0.02501087 -0.36753414  0.03570296  0.61788329]\n",
      "[ 0.01766019 -0.17292863  0.04806062  0.3366555 ]\n",
      "[ 0.01420162 -0.36870041  0.05479373  0.64409847]\n",
      "[ 0.00682761 -0.56454146  0.0676757   0.95352047]\n",
      "[-0.00446322 -0.76050546  0.08674611  1.26667507]\n",
      "[-0.01967333 -0.56659223  0.11207961  1.0023703 ]\n",
      "[-0.03100517 -0.37313168  0.13212702  0.74688269]\n",
      "[-0.03846781 -0.56980508  0.14706467  1.07805388]\n",
      "[-0.04986391 -0.37689891  0.16862575  0.83489945]\n",
      "[-0.05740189 -0.57387354  0.18532374  1.17551201]\n",
      "[-0.06887936 -0.77085507  0.20883398  1.52009867]\n",
      "Episode finished after 15 timesteps\n",
      "[-0.03435198 -0.04568105 -0.01506038  0.0183231 ]\n",
      "[-0.0352656   0.14965361 -0.01469392 -0.27907325]\n",
      "[-0.03227253  0.34498206 -0.02027538 -0.57635415]\n",
      "[-0.02537289  0.54038227 -0.03180247 -0.87535476]\n",
      "[-0.01456524  0.34570673 -0.04930956 -0.59283747]\n",
      "[-0.00765111  0.54148301 -0.06116631 -0.90063608]\n",
      "[ 0.00317855  0.34724082 -0.07917903 -0.62778935]\n",
      "[ 0.01012337  0.54337337 -0.09173482 -0.94432077]\n",
      "[ 0.02099083  0.34959895 -0.11062123 -0.68181351]\n",
      "[ 0.02798281  0.54606919 -0.1242575  -1.0071739 ]\n",
      "[ 0.0389042   0.74261133 -0.14440098 -1.33615186]\n",
      "[ 0.05375642  0.939227   -0.17112402 -1.67031238]\n",
      "[ 0.07254096  0.7464571  -0.20453027 -1.43544621]\n",
      "Episode finished after 13 timesteps\n",
      "[-0.01277226 -0.01490297  0.02443451  0.01889701]\n",
      "[-0.01307032 -0.21036666  0.02481245  0.31918806]\n",
      "[-0.01727766 -0.40583304  0.03119621  0.6195915 ]\n",
      "[-0.02539432 -0.21116038  0.04358804  0.33689478]\n",
      "[-0.02961752 -0.01668494  0.05032594  0.05826953]\n",
      "[-0.02995122  0.17768066  0.05149133 -0.21812017]\n",
      "[-0.02639761 -0.01813811  0.04712892  0.09035044]\n",
      "[-0.02676037 -0.21390279  0.04893593  0.39752218]\n",
      "[-0.03103843 -0.01950802  0.05688638  0.12066053]\n",
      "[-0.03142859 -0.21539691  0.05929959  0.43073413]\n",
      "[-0.03573653 -0.0211626   0.06791427  0.15731864]\n",
      "[-0.03615978 -0.21718786  0.07106064  0.47063013]\n",
      "[-0.04050354 -0.41323783  0.08047325  0.78483718]\n",
      "[-0.04876829 -0.2193084   0.09616999  0.51851803]\n",
      "[-0.05315446 -0.41564344  0.10654035  0.83989017]\n",
      "[-0.06146733 -0.61204611  0.12333815  1.16408739]\n",
      "[-0.07370825 -0.41872657  0.1466199   0.91248047]\n",
      "[-0.08208278 -0.22585989  0.16486951  0.66923645]\n",
      "[-0.08659998 -0.42284373  0.17825424  1.00895784]\n",
      "[-0.09505685 -0.61983902  0.1984334   1.35189827]\n",
      "Episode finished after 20 timesteps\n",
      "[-0.00583005  0.03326931 -0.02036465  0.03959753]\n",
      "[-0.00516467 -0.16155479 -0.01957269  0.32578628]\n",
      "[-0.00839576 -0.35639267 -0.01305697  0.61223313]\n",
      "[-1.55236149e-02 -5.51329732e-01 -8.12306903e-04  9.00775150e-01]\n",
      "[-0.02655021 -0.74644067  0.0172032   1.19320264]\n",
      "[-0.04147902 -0.94178117  0.04106725  1.49122752]\n",
      "[-0.06031465 -0.74718245  0.0708918   1.21164593]\n",
      "[-0.0752583  -0.94314429  0.09512472  1.52567512]\n",
      "[-0.09412118 -0.74929063  0.12563822  1.26413373]\n",
      "[-0.10910699 -0.9457744   0.15092089  1.59337687]\n",
      "[-0.12802248 -1.14233056  0.18278843  1.92906223]\n",
      "Episode finished after 11 timesteps\n",
      "[0.02090883 0.03534119 0.0279414  0.0212054 ]\n",
      "[ 0.02161566 -0.16017009  0.02836551  0.32257156]\n",
      "[0.01841225 0.03453669 0.03481694 0.03896721]\n",
      "[ 0.01910299  0.22914252  0.03559628 -0.24253051]\n",
      "[0.02368584 0.03353067 0.03074567 0.06116449]\n",
      "[ 0.02435645  0.22819861  0.03196896 -0.22166161]\n",
      "[ 0.02892042  0.42284937  0.02753573 -0.50409129]\n",
      "[ 0.03737741  0.61757265  0.01745391 -0.787971  ]\n",
      "[ 0.04972886  0.42221536  0.00169449 -0.48984851]\n",
      "[ 0.05817317  0.61731336 -0.00810248 -0.78199692]\n",
      "[ 0.07051944  0.81254574 -0.02374242 -1.07721797]\n",
      "[ 0.08677035  0.61774532 -0.04528678 -0.79207931]\n",
      "[ 0.09912526  0.42327341 -0.06112837 -0.51398029]\n",
      "[ 0.10759073  0.22906324 -0.07140797 -0.24116969]\n",
      "[ 0.11217199  0.42512873 -0.07623137 -0.55549453]\n",
      "[ 0.12067457  0.23115521 -0.08734126 -0.28776958]\n",
      "[ 0.12529767  0.03738034 -0.09309665 -0.02386093]\n",
      "[ 0.12604528 -0.15629176 -0.09357387  0.23805832]\n",
      "[ 0.12291944  0.04003378 -0.0888127  -0.08261223]\n",
      "[ 0.12372012 -0.15371022 -0.09046495  0.18078175]\n",
      "[ 0.12064591 -0.34742903 -0.08684931  0.44361132]\n",
      "[ 0.11369733 -0.15119248 -0.07797709  0.1248643 ]\n",
      "[ 0.11067348  0.04495498 -0.0754798  -0.19136487]\n",
      "[ 0.11157258  0.24107102 -0.0793071  -0.50687134]\n",
      "[ 0.116394    0.43721565 -0.08944452 -0.82345537]\n",
      "[ 0.12513832  0.63343988 -0.10591363 -1.14287675]\n",
      "[ 0.13780711  0.82977419 -0.12877117 -1.46680837]\n",
      "[ 0.1544026   0.63644239 -0.15810733 -1.21696589]\n",
      "[ 0.16713145  0.83320981 -0.18244665 -1.55472576]\n",
      "Episode finished after 29 timesteps\n",
      "[ 0.03053488  0.03422325 -0.00883938  0.04360086]\n",
      "[ 0.03121934  0.22947083 -0.00796736 -0.25185781]\n",
      "[ 0.03580876  0.42470564 -0.01300452 -0.54704311]\n",
      "[ 0.04430287  0.22976879 -0.02394538 -0.25848578]\n",
      "[ 0.04889825  0.03499673 -0.0291151   0.02654929]\n",
      "[ 0.04959818 -0.15969585 -0.02858411  0.30990576]\n",
      "[ 0.04640427 -0.35439913 -0.022386    0.59343879]\n",
      "[ 0.03931628 -0.54920069 -0.01051722  0.87898698]\n",
      "[ 0.02833227 -0.74417817  0.00706252  1.16834502]\n",
      "[ 0.01344871 -0.93939128  0.03042942  1.46323375]\n",
      "[-0.00533912 -0.7446551   0.05969409  1.18020958]\n",
      "[-0.02023222 -0.94049905  0.08329828  1.49099189]\n",
      "[-0.0390422  -1.13653035  0.11311812  1.80848069]\n",
      "[-0.06177281 -1.33271771  0.14928774  2.1340668 ]\n",
      "[-0.08842716 -1.13935723  0.19196907  1.89097988]\n",
      "Episode finished after 15 timesteps\n",
      "[-0.02274515 -0.03618687 -0.01484552 -0.00683635]\n",
      "[-0.02346889 -0.23109281 -0.01498225  0.28112591]\n",
      "[-0.02809074 -0.42599788 -0.00935973  0.56904609]\n",
      "[-0.0366107  -0.23074591  0.00202119  0.27342925]\n",
      "[-0.04122562 -0.03565286  0.00748978 -0.0186155 ]\n",
      "[-0.04193868  0.15936089  0.00711747 -0.30892594]\n",
      "[-0.03875146  0.35438071  0.00093895 -0.59935574]\n",
      "[-0.03166384  0.54948951 -0.01104817 -0.89174276]\n",
      "[-0.02067405  0.74475958 -0.02888302 -1.18787809]\n",
      "[-0.00577886  0.94024384 -0.05264058 -1.4894726 ]\n",
      "[ 0.01302601  0.74580094 -0.08243003 -1.21368146]\n",
      "[ 0.02794203  0.5518337  -0.10670366 -0.94792549]\n",
      "[ 0.03897871  0.748218   -0.12566217 -1.27213794]\n",
      "[ 0.05394307  0.94469951 -0.15110493 -1.60138381]\n",
      "[ 0.07283706  0.75165524 -0.18313261 -1.3593729 ]\n",
      "Episode finished after 15 timesteps\n",
      "[-0.03239147 -0.02728597 -0.02641767  0.04598116]\n",
      "[-0.03293719 -0.22201934 -0.02549805  0.3302134 ]\n",
      "[-0.03737758 -0.02654388 -0.01889378  0.02959991]\n",
      "[-0.03790845 -0.22138986 -0.01830178  0.31626227]\n",
      "[-0.04233625 -0.4162464  -0.01197654  0.60311771]\n",
      "[-5.06611793e-02 -6.11198813e-01  8.58146278e-05  8.92004337e-01]\n",
      "[-0.06288516 -0.80632193  0.0179259   1.18471424]\n",
      "[-0.07901159 -0.61143704  0.04162019  0.89770381]\n",
      "[-0.09124033 -0.41690323  0.05957426  0.61838859]\n",
      "[-0.0995784  -0.61280451  0.07194203  0.92922369]\n",
      "[-0.11183449 -0.41872358  0.09052651  0.65998806]\n",
      "[-0.12020896 -0.22497028  0.10372627  0.39712661]\n",
      "[-0.12470837 -0.03146113  0.1116688   0.13886385]\n",
      "[-0.12533759  0.16189901  0.11444608 -0.11660741]\n",
      "[-0.12209961  0.35521098  0.11211393 -0.37110471]\n",
      "[-0.11499539  0.54857643  0.10469184 -0.62643942]\n",
      "[-0.10402386  0.74209327  0.09216305 -0.88440217]\n",
      "[-0.089182    0.93585109  0.074475   -1.14674783]\n",
      "[-0.07046497  0.73983987  0.05154005 -0.83167059]\n",
      "[-0.05566818  0.93422098  0.03490664 -1.10770902]\n",
      "[-0.03698376  0.73865803  0.01275246 -0.80428283]\n",
      "[-0.0222106   0.93360284 -0.0033332  -1.09292715]\n",
      "[-0.00353854  0.73852497 -0.02519174 -0.80129192]\n",
      "[ 0.01123196  0.9339832  -0.04121758 -1.10179183]\n",
      "[ 0.02991162  0.73942707 -0.06325342 -0.82232004]\n",
      "[ 0.04470017  0.93535476 -0.07969982 -1.13420795]\n",
      "[ 0.06340726  0.74136104 -0.10238398 -0.86754803]\n",
      "[ 0.07823448  0.93771596 -0.11973494 -1.19058637]\n",
      "[ 0.0969888   0.74433153 -0.14354667 -0.93770459]\n",
      "[ 0.11187543  0.55140625 -0.16230076 -0.693351  ]\n",
      "[ 0.12290356  0.74836308 -0.17616778 -1.03241089]\n",
      "[ 0.13787082  0.55596585 -0.196816   -0.7998076 ]\n",
      "Episode finished after 32 timesteps\n",
      "[-0.00264788 -0.01783791  0.03028298  0.01114739]\n",
      "[-0.00300464  0.17683695  0.03050593 -0.27182918]\n",
      "[ 5.32101339e-04  3.71510615e-01  2.50693423e-02 -5.54736471e-01]\n",
      "[ 0.00796231  0.1760458   0.01397461 -0.25426176]\n",
      "[ 0.01148323 -0.01927287  0.00888938  0.04279604]\n",
      "[ 0.01109777 -0.21452116  0.0097453   0.33827034]\n",
      "[ 0.00680735 -0.01953923  0.01651071  0.04867641]\n",
      "[ 0.00641656  0.17534213  0.01748423 -0.23875186]\n",
      "[ 0.00992341 -0.02002518  0.0127092   0.05939441]\n",
      "[ 0.0095229   0.17491226  0.01389708 -0.22925173]\n",
      "[ 0.01302115 -0.02040549  0.00931205  0.06778223]\n",
      "[ 0.01261304  0.17458172  0.01066769 -0.22194821]\n",
      "[ 0.01610467 -0.02069107  0.00622873  0.07408055]\n",
      "[ 0.01569085  0.17434103  0.00771034 -0.2166307 ]\n",
      "[ 0.01917767  0.36935192  0.00337773 -0.50687151]\n",
      "[ 0.02656471  0.17418253 -0.0067597  -0.21312605]\n",
      "[ 0.03004836  0.36940047 -0.01102222 -0.50793361]\n",
      "[ 0.03743637  0.56467598 -0.0211809  -0.80406954]\n",
      "[ 0.04872989  0.76008183 -0.03726229 -1.10333929]\n",
      "[ 0.06393153  0.5654693  -0.05932907 -0.82257571]\n",
      "[ 0.07524091  0.37120705 -0.07578059 -0.54912755]\n",
      "[ 0.08266505  0.56730713 -0.08676314 -0.86469164]\n",
      "[ 0.0940112   0.37346655 -0.10405697 -0.60050085]\n",
      "[ 0.10148053  0.56987849 -0.11606699 -0.92406273]\n",
      "[ 0.1128781   0.37649949 -0.13454824 -0.669993  ]\n",
      "[ 0.12040809  0.18347934 -0.1479481  -0.42251749]\n",
      "[ 0.12407767 -0.00927085 -0.15639845 -0.17988894]\n",
      "[ 0.12389226 -0.20184926 -0.15999623  0.05965903]\n",
      "[ 0.11985527 -0.39435863 -0.15880305  0.29789649]\n",
      "[ 0.1119681  -0.19737144 -0.15284512 -0.04035845]\n",
      "[ 0.10802067 -0.00042573 -0.15365229 -0.3770947 ]\n",
      "[ 0.10801216  0.19650654 -0.16119418 -0.71400915]\n",
      "[ 0.11194229  0.00393964 -0.17547437 -0.47608949]\n",
      "[ 0.11202108 -0.18832695 -0.18499616 -0.24344345]\n",
      "[ 0.10825454 -0.38039104 -0.18986503 -0.01434228]\n",
      "[ 0.10064672 -0.57235477 -0.19015187  0.2129433 ]\n",
      "[ 0.08919962 -0.37509501 -0.185893   -0.13318144]\n",
      "[ 0.08169772 -0.56713517 -0.18855663  0.09557766]\n",
      "[ 0.07035502 -0.75912469 -0.18664508  0.3233499 ]\n",
      "[ 0.05517253 -0.95116668 -0.18017808  0.55185429]\n",
      "[ 0.03614919 -1.1433618  -0.169141    0.78279382]\n",
      "[ 0.01328196 -0.9463692  -0.15348512  0.44203095]\n",
      "[-0.00564543 -1.13902402 -0.1446445   0.68266831]\n",
      "[-0.02842591 -1.33187273 -0.13099114  0.92654321]\n",
      "[-0.05506336 -1.52500579 -0.11246027  1.17535957]\n",
      "[-0.08556348 -1.7185014  -0.08895308  1.43077585]\n",
      "[-0.11993351 -1.9124198  -0.06033756  1.69438567]\n",
      "[-0.1581819  -2.1067957  -0.02644985  1.96769047]\n",
      "[-0.20031782 -1.9114045   0.01290396  1.66693083]\n",
      "[-0.23854591 -1.71643498  0.04624258  1.37829456]\n",
      "[-0.27287461 -1.91210291  0.07380847  1.68507317]\n",
      "[-0.31111666 -2.10799732  0.10750993  1.99979476]\n",
      "[-0.35327661 -1.91415002  0.14750583  1.74224968]\n",
      "[-0.39155961 -2.11061111  0.18235082  2.0769508 ]\n",
      "Episode finished after 54 timesteps\n",
      "[ 0.04878568  0.02602506 -0.00758092  0.04346816]\n",
      "[ 0.04930618 -0.16898737 -0.00671155  0.33374963]\n",
      "[ 4.59264316e-02 -3.64013156e-01 -3.65605485e-05  6.24308536e-01]\n",
      "[ 0.03864617 -0.5591346   0.01244961  0.91697995]\n",
      "[ 0.02746348 -0.75442265  0.03078921  1.21354941]\n",
      "[ 0.01237502 -0.55971125  0.0550602   0.93067134]\n",
      "[ 1.18079863e-03 -7.55531353e-01  7.36736243e-02  1.24013589e+00]\n",
      "[-0.01392983 -0.56142868  0.09847634  0.97141214]\n",
      "[-0.0251584  -0.36775638  0.11790458  0.71121644]\n",
      "[-0.03251353 -0.17444739  0.13212891  0.45784978]\n",
      "[-0.03600248 -0.37116551  0.14128591  0.78908794]\n",
      "[-0.04342579 -0.17823733  0.15706767  0.54397925]\n",
      "[-0.04699053 -0.37517725  0.16794725  0.88174095]\n",
      "[-0.05449408 -0.18268564  0.18558207  0.64621077]\n",
      "[-0.05814779  0.00943204  0.19850629  0.41723101]\n",
      "[-0.05795915  0.20126893  0.20685091  0.19310487]\n",
      "Episode finished after 16 timesteps\n",
      "[-0.01725801  0.02771037 -0.04487526 -0.03971372]\n",
      "[-0.0167038  -0.1667403  -0.04566953  0.23847976]\n",
      "[-0.02003861  0.02900332 -0.04089994 -0.06825171]\n",
      "[-0.01945854 -0.16550911 -0.04226497  0.21125171]\n",
      "[-0.02276872  0.03019086 -0.03803994 -0.0944583 ]\n",
      "[-0.02216491  0.22583679 -0.0399291  -0.39889594]\n",
      "[-0.01764817  0.03130336 -0.04790702 -0.11906441]\n",
      "[-0.0170221  -0.16310062 -0.05028831  0.15812772]\n",
      "[-0.02028411 -0.35746787 -0.04712576  0.43453149]\n",
      "[-0.02743347 -0.16171154 -0.03843513  0.12737313]\n",
      "[-0.0306677  -0.3562624  -0.03588766  0.40768676]\n",
      "[-0.03779295 -0.16065046 -0.02773393  0.10390906]\n",
      "[-0.04100596 -0.3553642  -0.02565575  0.38771463]\n",
      "[-0.04811324 -0.55011276 -0.01790145  0.67219937]\n",
      "[-0.0591155  -0.3547466  -0.00445747  0.37393434]\n",
      "[-0.06621043 -0.54980495  0.00302122  0.66520847]\n",
      "[-0.07720653 -0.35472515  0.01632539  0.37347835]\n",
      "[-0.08430103 -0.15983887  0.02379496  0.08598733]\n",
      "[-0.08749781  0.03493406  0.0255147  -0.19909431]\n",
      "[-0.08679913  0.22968196  0.02153282 -0.48362057]\n",
      "[-0.08220549  0.4244935   0.01186041 -0.76943998]\n",
      "[-0.07371562  0.22921033 -0.00352839 -0.47304895]\n",
      "[-0.06913141  0.42438194 -0.01298937 -0.76684188]\n",
      "[-0.06064377  0.61968029 -0.02832621 -1.06358347]\n",
      "[-0.04825017  0.42494454 -0.04959788 -0.7799238 ]\n",
      "[-0.03975128  0.23053827 -0.06519636 -0.5032488 ]\n",
      "[-0.03514051  0.42651562 -0.07526133 -0.81574444]\n",
      "[-0.0266102   0.2325004  -0.09157622 -0.54765139]\n",
      "[-0.02196019  0.03877624 -0.10252925 -0.28516855]\n",
      "[-0.02118467  0.23519965 -0.10823262 -0.60834719]\n",
      "[-0.01648067  0.04174399 -0.12039956 -0.35161999]\n",
      "[-0.0156458  -0.15147842 -0.12743196 -0.09919657]\n",
      "[-0.01867536  0.04521769 -0.1294159  -0.42921207]\n",
      "[-0.01777101  0.2419121  -0.13800014 -0.75972854]\n",
      "[-0.01293277  0.43863829 -0.15319471 -1.09245581]\n",
      "[-0.00416     0.24582999 -0.17504382 -0.85149373]\n",
      "[ 7.56597878e-04  4.42850954e-01 -1.92073698e-01 -1.19370961e+00]\n",
      "Episode finished after 37 timesteps\n",
      "[ 0.02694327  0.02790044  0.04239838 -0.03816809]\n",
      "[ 0.02750128 -0.16780305  0.04163501  0.26758473]\n",
      "[ 0.02414522  0.02670073  0.04698671 -0.01168095]\n",
      "[ 0.02467923  0.22111844  0.04675309 -0.28917664]\n",
      "[0.0291016  0.02536206 0.04096956 0.01787747]\n",
      "[ 0.02960884 -0.17032277  0.04132711  0.32320008]\n",
      "[ 0.02620239 -0.36600808  0.04779111  0.62862414]\n",
      "[ 0.01888223 -0.17158453  0.06036359  0.35136703]\n",
      "[ 0.01545053 -0.36751064  0.06739093  0.66245705]\n",
      "[ 0.00810032 -0.56350229  0.08064007  0.97557514]\n",
      "[-0.00316972 -0.36954912  0.10015157  0.7092733 ]\n",
      "[-0.01056071 -0.56590511  0.11433704  1.03172674]\n",
      "[-0.02187881 -0.3724744   0.13497158  0.77701705]\n",
      "[-0.0293283  -0.56916876  0.15051192  1.1089365 ]\n",
      "[-0.04071167 -0.3763103   0.17269065  0.86700298]\n",
      "[-0.04823788 -0.57330831  0.19003071  1.20862385]\n",
      "Episode finished after 16 timesteps\n",
      "[-0.01800931 -0.02515828 -0.02297112  0.04859181]\n",
      "[-0.01851247  0.1702854  -0.02199928 -0.25124927]\n",
      "[-0.01510677 -0.02451561 -0.02702427  0.03441427]\n",
      "[-0.01559708 -0.21923981 -0.02633598  0.31844975]\n",
      "[-0.01998188 -0.41397697 -0.01996699  0.6027123 ]\n",
      "[-0.02826141 -0.21858151 -0.00791274  0.30380764]\n",
      "[-0.03263304 -0.02334769 -0.00183659  0.00863976]\n",
      "[-0.0331     -0.21844325 -0.00166379  0.30074265]\n",
      "[-0.03746886 -0.41354145  0.00435106  0.59290039]\n",
      "[-0.04573969 -0.60872404  0.01620907  0.88695071]\n",
      "[-0.05791417 -0.80406222  0.03394808  1.18468474]\n",
      "[-0.07399542 -0.60939669  0.05764178  0.90283351]\n",
      "[-0.08618335 -0.80525011  0.07569845  1.21306301]\n",
      "[-0.10228835 -0.61118228  0.09995971  0.94502902]\n",
      "[-0.114512   -0.80749827  0.11886029  1.26737228]\n",
      "[-0.13066196 -1.00392092  0.14420773  1.59479015]\n",
      "[-0.15074038 -0.81077385  0.17610354  1.3503271 ]\n",
      "[-0.16695586 -0.61824632  0.20311008  1.11750955]\n",
      "Episode finished after 18 timesteps\n",
      "[-0.04837696  0.03188926  0.03991736  0.02125178]\n",
      "[-0.04773918 -0.16378174  0.04034239  0.32625706]\n",
      "[-0.05101481  0.03074331  0.04686754  0.04656435]\n",
      "[-0.05039995  0.22516297  0.04779882 -0.2309709 ]\n",
      "[-0.04589669  0.02939174  0.0431794   0.07639804]\n",
      "[-0.04530885 -0.16632177  0.04470736  0.38238547]\n",
      "[-0.04863529 -0.36204905  0.05235507  0.68882263]\n",
      "[-0.05587627 -0.55785702  0.06613153  0.99751749]\n",
      "[-0.06703341 -0.36367862  0.08608188  0.72631508]\n",
      "[-0.07430698 -0.16984571  0.10060818  0.46191911]\n",
      "[-0.0777039  -0.36623494  0.10984656  0.78454226]\n",
      "[-0.0850286  -0.17277999  0.12553741  0.52833877]\n",
      "[-0.0884842  -0.36942381  0.13610418  0.8577944 ]\n",
      "[-0.09587267 -0.17639241  0.15326007  0.61081619]\n",
      "[-0.09940052 -0.37328675  0.16547639  0.94757817]\n",
      "[-0.10686625 -0.57020354  0.18442796  1.28734686]\n",
      "Episode finished after 16 timesteps\n",
      "[-3.93005844e-02 -3.15004664e-05  4.28209299e-02  4.41742850e-02]\n",
      "[-0.03930121 -0.19574049  0.04370442  0.35005418]\n",
      "[-0.04321602 -0.00126647  0.0507055   0.0714667 ]\n",
      "[-0.04324135  0.19309326  0.05213483 -0.20479716]\n",
      "[-0.03937949  0.38743239  0.04803889 -0.48058908]\n",
      "[-0.03163084  0.58184447  0.03842711 -0.75775278]\n",
      "[-0.01999395  0.77641637  0.02327205 -1.03810038]\n",
      "[-0.00446562  0.58099299  0.00251005 -0.73820332]\n",
      "[ 0.00715424  0.38583647 -0.01225402 -0.44473151]\n",
      "[ 0.01487097  0.19089002 -0.02114865 -0.15593638]\n",
      "[ 0.01868877 -0.00392285 -0.02426738  0.13000033]\n",
      "[ 0.01861031 -0.19868892 -0.02166737  0.41492951]\n",
      "[ 0.01463653 -0.00326667 -0.01336878  0.11549524]\n",
      "[ 0.0145712   0.19204426 -0.01105888 -0.18137528]\n",
      "[ 0.01841208 -0.00291772 -0.01468638  0.10779855]\n",
      "[ 0.01835373  0.19241158 -0.01253041 -0.18948143]\n",
      "[ 0.02220196 -0.00252889 -0.01632004  0.09922246]\n",
      "[ 0.02215138 -0.19741318 -0.01433559  0.3867121 ]\n",
      "[ 0.01820312 -0.39232873 -0.00660135  0.67484086]\n",
      "[ 0.01035654 -0.19711566  0.00689547  0.38008686]\n",
      "[ 0.00641423 -0.39233485  0.01449721  0.67493593]\n",
      "[-0.00143247 -0.58765524  0.02799592  0.97214775]\n",
      "[-0.01318557 -0.78314147  0.04743888  1.27349192]\n",
      "[-0.0288484  -0.97883551  0.07290872  1.58064454]\n",
      "[-0.04842511 -1.17474575  0.10452161  1.89514539]\n",
      "[-0.07192003 -0.98090072  0.14242452  1.6366381 ]\n",
      "[-0.09153804 -1.17737715  0.17515728  1.97010008]\n",
      "Episode finished after 27 timesteps\n",
      "[ 0.03787096 -0.03413097 -0.01585659 -0.01304299]\n",
      "[ 0.03718834 -0.22902197 -0.01611745  0.27459511]\n",
      "[ 0.0326079  -0.03367381 -0.01062555 -0.0231274 ]\n",
      "[ 0.03193442  0.16159889 -0.0110881  -0.31914378]\n",
      "[ 0.0351664   0.35687699 -0.01747098 -0.61530275]\n",
      "[ 0.04230394  0.55223864 -0.02977703 -0.9134367 ]\n",
      "[ 0.05334871  0.35753187 -0.04804576 -0.63025918]\n",
      "[ 0.06049935  0.55329016 -0.06065095 -0.93767778]\n",
      "[ 0.07156516  0.74917515 -0.0794045  -1.24878545]\n",
      "[ 0.08654866  0.94522016 -0.10438021 -1.56524679]\n",
      "[ 0.10545306  1.14142334 -0.13568515 -1.88858184]\n",
      "[ 0.12828153  1.33773388 -0.17345678 -2.22011033]\n",
      "Episode finished after 12 timesteps\n",
      "[-0.00602385  0.02764941  0.0125973  -0.03908473]\n",
      "[-0.00547086 -0.1676509   0.01181561  0.257546  ]\n",
      "[-0.00882388  0.02730039  0.01696653 -0.03138681]\n",
      "[-0.00827787 -0.16806071  0.01633879  0.26660058]\n",
      "[-0.01163909 -0.36341199  0.0216708   0.56439178]\n",
      "[-0.01890733 -0.55883119  0.03295864  0.86382247]\n",
      "[-0.03008395 -0.75438595  0.05023509  1.16668338]\n",
      "[-0.04517167 -0.95012434  0.07356876  1.47468348]\n",
      "[-0.06417415 -0.75597447  0.10306243  1.20585614]\n",
      "[-0.07929364 -0.95226607  0.12717955  1.52897794]\n",
      "[-0.09833897 -1.14867188  0.15775911  1.85849733]\n",
      "[-0.1213124  -0.95559436  0.19492905  1.61866659]\n",
      "Episode finished after 12 timesteps\n"
     ]
    }
   ],
   "source": [
    "import gym\n",
    "env = gym.make('CartPole-v0')\n",
    "for i_episode in range(20):\n",
    "    observation = env.reset()\n",
    "    for t in range(100):\n",
    "        env.render()\n",
    "        print(observation)\n",
    "        action = env.action_space.sample()\n",
    "        observation, reward, done, info = env.step(action)\n",
    "        if done:\n",
    "            print(\"Episode finished after {} timesteps\".format(t+1))\n",
    "            break\n",
    "env.close()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}