OpenAI_Gym_Basics.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Open AI Gym - Scratch Pad"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Basic Environment methods:**\n",
    "* reset(self): Reset and returns observation.\n",
    "* step(self, action): run one time steps and returns observation, reward, done, info\n",
    "* render(self, mode='human', close=False): render one frame (*mode E {'human','rgb_array','ansi'}*)\n",
    "\n",
    "**Check Spaces**\n",
    "* Action space: env.action_space\n",
    "* Observation space: env.observation_space\n",
    "\n",
    "** Check Environemnts**\n",
    "* Check all available environments: gym.envs.registry.env_specs or gym.envs.registry.all()\n",
    "* CartPole v0 State Representation:  [Cart Position, Cart Velocity, Pole Angle, Pole Velocity at Tip] [Cart Pole Wiki](https://github.com/openai/gym/wiki/CartPole-v0)\n",
    "* Pendulum V0 observation and actions space specs: [Pendulum Wiki](https://github.com/openai/gym/wiki/Pendulum-v0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* try better way to automatically construct tf.case DICTIONARY\n",
    "* create head specific name space\n",
    "* ow to check that for giben task only one HEAD is being updated"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-07-14T17:47:24.592760Z",
     "start_time": "2018-07-14T17:47:24.581257Z"
    }
   },
   "source": [
    "https://stackoverflow.com/questions/35678883/tensorflow-getting-variable-by-name"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- penalize instead of reward for touching grounds with legs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-08-31T18:11:51.893349Z",
     "start_time": "2018-08-31T18:11:51.613268Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[33mWARN: gym.spaces.Box autodetected dtype as <class 'numpy.float32'>. Please provide explicit dtype.\u001b[0m\n",
      "\u001b[33mWARN: gym.spaces.Box autodetected dtype as <class 'numpy.float32'>. Please provide explicit dtype.\u001b[0m\n",
      "\u001b[33mWARN: gym.spaces.Box autodetected dtype as <class 'numpy.float32'>. Please provide explicit dtype.\u001b[0m\n",
      "\u001b[33mWARN: gym.spaces.Box autodetected dtype as <class 'numpy.float32'>. Please provide explicit dtype.\u001b[0m\n",
      "\u001b[33mWARN: gym.spaces.Box autodetected dtype as <class 'numpy.float32'>. Please provide explicit dtype.\u001b[0m\n"
     ]
    }
   ],
   "source": [
    "import gym\n",
    "%load_ext autoreload\n",
    "%autoreload\n",
    "env = gym.make(\"LunarLander-v2\")\n",
    "# env = gym.make(\"LunarLanderGoal-v2\")\n",
    "\n",
    "\n",
    "env = gym.make(\"LunarLanderContinuous-v2\")\n",
    "env = gym.make(\"BipedalWalker-v2\")\n",
    "# env = gym.make('MountainCarContinuous-v0')\n",
    "# env = gym.make(\"CarRacing-v0\")\n",
    "# env = gym.make(\"LunarLanderContinuous-v2\")\n",
    "# env = gym.make(\"Pendulum-v0\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-08-23T11:23:50.855513Z",
     "start_time": "2018-08-23T11:23:49.366225Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[-0.00316385 -0.01901737 -0.00792151  0.02046022 -0.29977825 -0.61344475\n",
      "  1.48511237  0.99365536  1.          0.3249442   0.99864846  0.13513219\n",
      " -0.99992212  1.          0.45069224  0.45581052  0.47176284  0.50052053\n",
      "  0.54607153  0.61596161  0.72504026  0.90578473  1.          1.        ]\n",
      "[-0.01530648 -0.02248161 -0.01003614  0.00759794 -0.01325959 -0.14647758\n",
      "  1.06851652 -0.38180176  1.          0.35582209  0.52752727  0.23603183\n",
      " -0.09515553  1.          0.45374468  0.45889762  0.47495797  0.50391048\n",
      "  0.54976994  0.6201334   0.72995079  0.91191941  1.          1.        ]\n",
      "[-0.03500723 -0.03956782 -0.01530124  0.04476847  0.00612771  0.22362922\n",
      "  1.04670604  0.          1.          0.37172496  0.0988487   0.27593279\n",
      "  0.25152216  1.          0.45667598  0.46186221  0.4780263   0.50716585\n",
      "  0.5533216   0.62413961  0.73466641  0.91781062  1.          1.        ]\n",
      "[-0.05717643 -0.0453887  -0.02638958  0.01940268  0.02674892  0.5013873\n",
      "  0.94389576 -0.92852195  1.          0.44631964  0.99999988  0.16868591\n",
      " -1.00000016  1.          0.45823953  0.46344352  0.47966295  0.50890225\n",
      "  0.55521601  0.62627649  0.73718178  0.92095298  1.          1.        ]\n",
      "[-0.09490535 -0.07525761 -0.03916016  0.01374876  0.15796769  0.92961752\n",
      "  0.49381971 -1.          1.          0.48939073  0.64648402  0.30983168\n",
      "  1.00000032  0.          0.46005449  0.4652791   0.48156276  0.5109179\n",
      "  0.55741507  0.628757    0.74010152  0.9246006   1.          1.        ]\n",
      "[-0.11941598 -0.04907078 -0.02418613 -0.00653906  0.19347805  0.47767037\n",
      "  0.37761796 -1.00000008  1.          0.50948185  0.27115375  0.43261254\n",
      "  1.00000008  0.          0.45964849  0.46486849  0.48113781  0.51046699\n",
      "  0.55692315  0.62820214  0.73944837  0.92378467  1.          1.        ]\n",
      "[-0.13276628 -0.02688061 -0.0158398  -0.03472402  0.192432    0.02185333\n",
      "  0.26413935 -1.00000008  0.          0.58030349  0.90785313  0.31797856\n",
      " -0.99999984  0.          0.45792437  0.46312478  0.47933307  0.50855225\n",
      "  0.55483419  0.62584579  0.73667473  0.92031956  1.          1.        ]\n",
      "[-0.15843788 -0.05142832 -0.02630273 -0.04028843  0.23378474  0.54784214\n",
      "  0.14839613 -1.00000008  0.          0.59773666  0.23866524  0.44102073\n",
      "  1.00000016  0.          0.45591986  0.46109751  0.47723484  0.50632614\n",
      "  0.55240548  0.62310618  0.73345006  0.916291    1.          1.        ]\n",
      "[-0.20356305 -0.09054677 -0.03564816 -0.05098608  0.24494937  0.15372264\n",
      "  0.27187085  1.00000032  0.          0.66983271  0.94784337  0.56835318\n",
      "  1.          0.          0.45329639  0.45844427  0.47448874  0.5034126\n",
      "  0.54922682  0.61952072  0.72922963  0.91101843  1.          1.        ]\n",
      "[-0.25264367 -0.09839866 -0.04009117 -0.06919997  0.27576405  0.41118097\n",
      "  0.3958571   1.          0.          0.74752551  1.          0.66427106\n",
      "  0.76065755  0.          0.44981551  0.45492384  0.4708451   0.49954689\n",
      "  0.54500926  0.61476338  0.72362983  0.90402269  1.          1.        ]\n",
      "[-0.27373916 -0.04240036 -0.01343411 -0.08614453  0.27140951 -0.05072656\n",
      "  0.39635569 -0.00326854  0.          0.74504697 -0.01363841  0.81997088\n",
      "  1.27671138  0.          0.44565728  0.4507184   0.46649247  0.49492893\n",
      "  0.53997105  0.60908031  0.7169404   0.89566565  1.          1.        ]\n",
      "[-0.28017181 -0.01296731  0.00848236 -0.09968355  0.1899256  -1.\n",
      "  0.53353196  1.44422102  1.          0.72483128 -0.24921933  0.94028708\n",
      "  1.          0.          0.44092309  0.44593045  0.46153697  0.48967132\n",
      "  0.53423494  0.60261011  0.70932436  0.88615102  1.          1.        ]\n",
      "[-0.2974419  -0.03423595 -0.01479584 -0.11092712  0.25624418  0.60108095\n",
      "  0.37627804 -0.8038133   1.          0.73990232  0.16826446  0.95602107\n",
      "  0.23156595  1.          0.43579721  0.44074634  0.45617142  0.48397872\n",
      "  0.52802432  0.59560454  0.70107824  0.87584925  1.          1.        ]\n",
      "[-0.32372031 -0.05216488 -0.06554515 -0.08187519  0.33538824  1.0021081\n",
      "  0.26065969 -0.99973989  1.          0.83792049  1.21348715  0.82951364\n",
      " -1.00289774  1.          0.43189979  0.43680465  0.45209178  0.47965041\n",
      "  0.52330208  0.59027797  0.69480836  0.8680163   1.          1.        ]\n",
      "[-0.34171605 -0.03597973 -0.0690826  -0.07527876  0.40159807  0.84881008\n",
      "  0.14438063 -1.00000008  0.          0.8970561   0.73603702  0.76790181\n",
      " -0.51710018  1.          0.42829579  0.43315974  0.44831932  0.47564796\n",
      "  0.51893538  0.58535236  0.6890105   0.86077315  1.          1.        ]\n",
      "[-0.35845742 -0.03348348 -0.08843791 -0.05231364  0.45332778  0.66803002\n",
      "  0.02825868 -0.99999992  0.          0.93123311  0.42417017  0.75397518\n",
      " -0.1208936   1.          0.42577165  0.43060693  0.44567716  0.47284475\n",
      "  0.51587707  0.58190262  0.68494987  0.85570025  1.          1.        ]\n",
      "[-0.37534922 -0.03393168 -0.07616249 -0.06291299  0.42480269 -0.3400439\n",
      "  0.15150142  0.99999992  0.          1.01079631  0.99999934  0.63496822\n",
      " -1.00000095  1.          0.42273262  0.42753339  0.44249606  0.46946973\n",
      "  0.51219487  0.57774919  0.68006086  0.84959251  1.          1.        ]\n",
      "[-0.37095088  0.0086449  -0.05661982 -0.07183728  0.34312868 -1.00000012\n",
      "  0.27787262  1.06606849  1.          1.04779029  0.47068763  0.54881814\n",
      " -0.73510647  1.          0.41934937  0.42411172  0.43895462  0.46571243\n",
      "  0.50809562  0.5731253   0.67461818  0.84279299  1.          1.        ]\n",
      "[-0.38057819 -0.01901024 -0.06244821 -0.05744998  0.40542594  0.47123477\n",
      "  0.18238288 -0.41686527  1.          1.04454207 -0.020517    0.67264953\n",
      "  1.00000008  0.          0.41669229  0.42142445  0.43617332  0.46276158\n",
      "  0.5048762   0.56949383  0.67034364  0.83745283  1.          1.        ]\n",
      "[-0.3854     -0.00975215 -0.05926653 -0.082222    0.42830247  0.32611576\n",
      "  0.06623781 -0.99999984  1.          1.04995561  0.07960577  0.79431158\n",
      "  0.99999992  0.          0.41281122  0.4174993   0.43211082  0.45845142\n",
      "  0.50017381  0.56418955  0.66410005  0.82965285  1.          1.        ]\n",
      "[-0.37541717  0.01979517 -0.0455395  -0.09057153  0.43538985  0.11847498\n",
      " -0.04997981 -1.          0.          1.00145292 -0.60104775  0.91495848\n",
      "  1.00000016  0.          0.408582    0.41322207  0.42768386  0.4537546\n",
      "  0.49504957  0.55840951  0.65729642  0.8211531   1.          1.        ]\n",
      "[ -3.48262012e-01   5.41211128e-02  -2.70431077e-02  -1.12243805e-01\n",
      "   3.68851930e-01  -7.89800406e-01  -1.51233673e-01  -9.05435324e-01\n",
      "   0.00000000e+00   9.71513808e-01  -3.70080143e-01   9.15583253e-01\n",
      "  -9.93410746e-09   0.00000000e+00   4.03394699e-01   4.07975852e-01\n",
      "   4.22254056e-01   4.47993815e-01   4.88764495e-01   5.51320016e-01\n",
      "   6.48951471e-01   8.10727835e-01   1.00000000e+00   1.00000000e+00]\n",
      "[-0.31128111  0.07371691 -0.09316763 -0.06059681  0.28377736 -0.99948549\n",
      " -0.26174855 -0.99996662  0.          0.995709    0.31160101  0.80778736\n",
      " -0.91537301  1.          0.40066028  0.40521038  0.41939178  0.44495708\n",
      "  0.48545137  0.54758286  0.64455253  0.80523235  1.          1.        ]\n",
      "[-0.26918396  0.08392567 -0.09452208 -0.07373765  0.20059399 -0.98201382\n",
      " -0.37394524 -1.          0.          1.01653624  0.27560496  0.69092652\n",
      " -1.00000016  1.          0.39733201  0.40184432  0.41590792  0.44126084\n",
      "  0.48141876  0.54303414  0.63919824  0.79854327  1.          1.        ]\n",
      "[-0.23437642  0.0694136  -0.08453317 -0.07670771  0.09501344 -1.31802142\n",
      " -0.31755984  0.43861564  0.          0.98389524 -0.3919881   0.81320852\n",
      "  0.99999992  1.          0.39383119  0.39830375  0.41224343  0.43737298\n",
      "  0.47717708  0.53824955  0.63356644  0.79150748  1.          1.        ]\n",
      "[-0.21121654  0.04609967 -0.09023916 -0.09013225  0.01371719 -1.         -0.193753\n",
      "  0.99999984  0.          0.97155124 -0.14577371  0.93394148  1.          0.\n",
      "  0.38965812  0.39408329  0.40787527  0.43273851  0.47212085  0.53254622\n",
      "  0.62685311  0.78312057  1.          1.        ]\n",
      "[-0.2142483  -0.0062512  -0.13635195 -0.12015424  0.07063156  0.73151881\n",
      " -0.31015158 -0.99999984  0.          1.04427505  0.91399986  0.93439877\n",
      "  0.          0.          0.38398668  0.38834742  0.40193868  0.42644006\n",
      "  0.46524912  0.52479506  0.61772931  0.77172232  1.          1.        ]\n",
      "[-0.20863155  0.01081512 -0.13180223 -0.13499923  0.11812608  0.6072793\n",
      " -0.42666233 -1.          0.          1.1230123   1.00000048  0.81779742\n",
      " -0.99999928  0.          0.37763289  0.38192147  0.39528784  0.41938379\n",
      "  0.4575507   0.51611131  0.60750777  0.75895274  1.          1.        ]\n",
      "[-0.17809297  0.06094121 -0.10910614 -0.13562926  0.1171172  -0.00527796\n",
      " -0.49596596 -0.58791041  0.          1.12201691  0.          0.70014668\n",
      " -0.99999968  0.          0.37138039  0.37559798  0.38874301  0.41244\n",
      "  0.44997498  0.50756603  0.59744924  0.74638671  1.          1.        ]\n",
      "[-0.14184047  0.07235846 -0.10124087 -0.15246571  0.08011155 -0.43339428\n",
      " -0.58581614 -0.77074703  0.          1.05149221 -0.871526    0.82166946\n",
      "  0.99999984  0.          0.36435953  0.36849737  0.38139394  0.40464294\n",
      "  0.44146833  0.49797058  0.58615458  0.73227638  1.          1.        ]\n",
      "[-0.12656297  0.03025161 -0.11825139 -0.18072897  0.0878098   0.11269979\n",
      " -0.70430207 -1.          0.          1.04263484 -0.10152993  0.94249123\n",
      "  0.99999992  0.          0.35592228  0.35996431  0.37256223  0.39527288\n",
      "  0.43124551  0.48643941  0.57258135  0.71531951  0.98224002  1.        ]\n",
      "[-0.11029536  0.03237722 -0.09644561 -0.18516582  0.04971885 -0.45370078\n",
      " -0.58269131  0.99999936  0.          1.07144928  0.33317482  0.93490571\n",
      "  0.          0.          0.34728181  0.35122573  0.36351779  0.38567713\n",
      "  0.42077649  0.47463048  0.55868125  0.69795424  0.95839489  1.        ]\n",
      "[-0.10713518  0.00600454 -0.11237028 -0.20083157  0.06336597  0.22378924\n",
      " -0.45853102  1.00000024  0.          1.15063393  1.00000036  0.81695318\n",
      " -0.99999921  0.          0.33785966  0.34169656  0.35365516  0.37521327\n",
      "  0.40936032  0.46175319  0.54352355  0.67901796  0.93239254  1.        ]\n",
      "[-0.07092975  0.06926219 -0.08802891 -0.208846   -0.00144631 -0.77081776\n",
      " -0.34577739  0.91211915  0.          1.13490689  0.          0.73077434\n",
      " -0.99999968  0.          0.32829142  0.33201966  0.34363958  0.36458716\n",
      "  0.39776719  0.44867626  0.52813089  0.65978807  0.90598702  1.        ]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[-0.03208157  0.07760849 -0.08345442 -0.2181971  -0.08343862 -1.00000262\n",
      " -0.23963845 -0.37728576  1.          1.05439019 -0.99999845  0.85154182\n",
      "  1.00000223  0.          0.31819773  0.32181135  0.33307397  0.35337752\n",
      "  0.38553739  0.43488121  0.51189291  0.63950211  0.87813145  1.        ]\n",
      "[ 0.01369165  0.0914045  -0.06753192 -0.23377424 -0.16184023 -1.         -0.245754\n",
      " -0.10430223  1.          1.03228557 -0.26072794  0.73481494 -0.99999984\n",
      "  0.          0.30738038  0.31087115  0.32175091  0.3413642   0.37243077\n",
      "  0.42009711  0.49449077  0.61776179  0.84827876  1.        ]\n",
      "[ 0.06016894  0.09277615 -0.13659675 -0.22624596 -0.19029382 -0.31831637\n",
      " -0.36155307 -0.99999547  1.          1.06816614  0.46092415  0.61690974\n",
      " -1.00772667  1.          0.29693067  0.30030277  0.31081265  0.32975918\n",
      "  0.35976961  0.40581551  0.47768006  0.59676039  0.81944066  1.        ]\n",
      "[ 0.10243658  0.08416356 -0.11849531 -0.22807684 -0.2712495  -0.99999976\n",
      " -0.35623372 -0.45875327  1.          1.05052519 -0.18601218  0.7428292\n",
      "  0.99999976  1.          0.28636411  0.2896162   0.29975209  0.3180244\n",
      "  0.34696686  0.39137417  0.46068135  0.57552409  0.7902801   1.        ]\n",
      "[ 0.15814981  0.11145482 -0.05884438 -0.05536887 -0.31421494 -0.72335321\n",
      " -0.35206604 -0.11571337  1.          0.97084278 -0.98320401  0.86537719\n",
      "  1.00242654  0.          0.28393698  0.28716153  0.2972115   0.3153289\n",
      "  0.34402609  0.38805699  0.45677677  0.57064611  0.78358191  1.        ]\n",
      "[ 0.21758871  0.11879426 -0.05196624 -0.07450993 -0.39265892 -0.97196299\n",
      " -0.37487674 -0.20181914  0.          0.89024305 -1.00000024  0.98577213\n",
      "  0.99999984  0.          0.28063831  0.2838254   0.2937586   0.31166553\n",
      "  0.34002933  0.38354871  0.45147011  0.56401658  0.77447855  1.        ]\n",
      "[ 0.27667522  0.11791159 -0.05664463 -0.11078591 -0.47649425 -0.99999988\n",
      " -0.48922348 -0.99999984  0.          0.89861065  0.11801612  0.86821604\n",
      " -1.          0.          0.27561846  0.27874854  0.28850406  0.30609071\n",
      "  0.33394715  0.37668809  0.44339454  0.5539279   0.7606253   1.        ]\n",
      "[ 0.34660062  0.13960029 -0.03257476 -0.050393   -0.5484162  -0.87220138\n",
      " -0.59780574 -0.88548088  1.          0.8644833  -0.40634575  0.75183606\n",
      " -0.99904521  0.          0.27348199  0.27658781  0.28626773  0.30371803\n",
      "  0.33135852  0.37376818  0.43995756  0.5496341   0.75472927  1.        ]\n",
      "[ 0.42065176  0.14801616 -0.02164764 -0.03755814 -0.62694758 -0.93809772\n",
      " -0.71126318 -0.99999865  1.          0.8165434  -0.57951856  0.63464338\n",
      " -1.0005451   0.          0.27193812  0.27502638  0.28465167  0.30200347\n",
      "  0.32948792  0.37165815  0.43747386  0.54653126  0.75046861  1.        ]\n",
      "[  4.95398104e-01   1.46776733e-01  -2.13389039e-02  -2.46380568e-02\n",
      "  -7.04833329e-01  -8.03021073e-01  -6.34907007e-01  -2.12589900e-06\n",
      "   1.00000000e+00   7.30419874e-01  -9.99996781e-01   5.24295807e-01\n",
      "  -9.99994755e-01   0.00000000e+00   2.71186292e-01   2.74266034e-01\n",
      "   2.83864707e-01   3.01168501e-01   3.28576982e-01   3.70630622e-01\n",
      "   4.36264396e-01   5.45020282e-01   7.48393834e-01   1.00000000e+00]\n",
      "[ 0.56293011  0.13499914 -0.03619256 -0.02199142 -0.77393031 -0.82614547\n",
      " -0.52524328  0.88721379  1.          0.61113185 -1.47946465  0.64594358\n",
      "  0.99999984  1.          0.27032754  0.27339751  0.28296578  0.30021483\n",
      "  0.32753649  0.36945698  0.43488291  0.54329437  0.74602389  1.        ]\n",
      "[ 0.62864345  0.13134088 -0.03128746 -0.04348182 -0.85584134 -0.99999976\n",
      " -0.40285277  1.00000008  1.          0.50452244 -1.32076311  0.76809359\n",
      "  1.          1.          0.26843962  0.27148816  0.28098962  0.29811817\n",
      "  0.32524905  0.36687675  0.43184575  0.53950012  0.74081379  1.        ]\n",
      "[ 0.62885892  0.00899365 -0.08977491 -0.00271453 -0.83512861 -0.02910234\n",
      " -0.31019235  1.01382248  1.          0.48799688 -0.2635577   0.88916492\n",
      "  1.00040785  0.          0.2683765   0.27142432  0.28092355  0.29804808\n",
      "  0.32517257  0.36679047  0.43174422  0.53937322  0.74063963  1.        ]\n",
      "[ 0.64935154  0.0415755  -0.09646796  0.00407105 -0.83506495 -0.00343767\n",
      " -0.42485929 -0.94782035  1.          0.48345816 -0.04579985  0.77205837\n",
      " -0.99972097  0.          0.26867726  0.27172852  0.28123838  0.2983821\n",
      "  0.325537    0.36720154  0.43222806  0.53997773  0.74146962  1.        ]\n",
      "[ 0.66572744  0.0326984  -0.09022309  0.00150189 -0.83520144  0.0150312\n",
      " -0.50824213 -0.70107683  0.          0.43029594 -0.66114736  0.89252055\n",
      "  1.00019137  0.          0.26878151  0.27183393  0.28134748  0.29849786\n",
      "  0.3256633   0.36734402  0.43239576  0.54018724  0.74175733  1.        ]\n",
      "[  6.84789360e-01   3.87136364e-02  -8.30594158e-02  -2.69496173e-03\n",
      "  -8.35079491e-01  -8.46639276e-04  -6.28315449e-01  -9.99579589e-01\n",
      "   0.00000000e+00   3.78350616e-01  -6.50391817e-01   1.01257813e+00\n",
      "   1.00006763e+00   0.00000000e+00   2.68741548e-01   2.71793514e-01\n",
      "   2.81305641e-01   2.98453480e-01   3.25614870e-01   3.67289394e-01\n",
      "   4.32331473e-01   5.40106893e-01   7.41647065e-01   1.00000000e+00]\n"
     ]
    }
   ],
   "source": [
    "# %load_ext autoreload\n",
    "%autoreload\n",
    "\n",
    "num_episodes = 1\n",
    "num_time_steps = 50\n",
    "\n",
    "# chunk = 5\n",
    "# chunk_x1 = env.env.sky_polys[chunk-1][0][0]\n",
    "# chunk_x2 = env.env.sky_polys[chunk][1][0]\n",
    "# chunk_y1 = env.env.sky_polys[chunk-1][0][1]\n",
    "# chunk_y2 = env.env.sky_polys[chunk][1][1]\n",
    "\n",
    "for episode in range(num_episodes):\n",
    "#     print(\"\\nEpisode No.: {}\".format(episode))\n",
    "#     env.env.seed(12) # Set seed before reseting env to allow repeatability\n",
    "    observation = env.reset()\n",
    "#     env.env.helipad_x1 = chunk_x1\n",
    "#     env.env.helipad_x2 = chunk_x2\n",
    "#     env.env.helipad_y = min(chunk_y1, chunk_y2)\n",
    "    \n",
    "    for t in range(num_time_steps):\n",
    "#         print(observation)\n",
    "        env.render()\n",
    "        action = env.action_space.sample() # Uniform selection across\n",
    "#         print(action.shape)\n",
    "        observation, reward, done, _ = env.step(action)\n",
    "        print(observation)\n",
    "#         print(reward)\n",
    "        \n",
    "        \n",
    "        if done:\n",
    "#             print(\"Episode finished after {} timesteps\\n\\n\".format(t+1))\n",
    "            print(t)\n",
    "            break\n",
    "    \n",
    "env.close() # Close windows (and also destroys environment :( )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Basic AI Gym Setup**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-08-23T11:23:11.021924Z",
     "start_time": "2018-08-23T11:23:04.406081Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[33mWARN: gym.spaces.Box autodetected dtype as <class 'numpy.float32'>. Please provide explicit dtype.\u001b[0m\n",
      "\u001b[33mWARN: gym.spaces.Box autodetected dtype as <class 'numpy.float32'>. Please provide explicit dtype.\u001b[0m\n",
      "\n",
      "Episode No.: 0\n",
      "-0.354682439242\n",
      "-0.315427576107\n",
      "-0.315273771011\n",
      "-0.354407770122\n",
      "-0.448146412914\n",
      "-0.631061174631\n",
      "-0.864754340164\n",
      "-1.26622147669\n",
      "-1.67208877028\n",
      "-2.17833086487\n",
      "-3.13369996565\n",
      "-4.17925874359\n",
      "-5.68054233044\n",
      "-7.5093517057\n",
      "-9.32591694967\n",
      "-12.063794102\n",
      "-14.9950282975\n",
      "-13.8222377564\n",
      "-11.2489370167\n",
      "-8.84379911044\n",
      "-6.64501174762\n",
      "-4.77230132604\n",
      "-3.408275377\n",
      "-2.50720952427\n",
      "-1.77934858464\n",
      "-1.406374862\n",
      "-1.09902215911\n",
      "-0.972314388928\n",
      "-0.965897290261\n",
      "-1.13212602147\n",
      "-1.43390040843\n",
      "-1.85275224257\n",
      "-2.6437189158\n",
      "-3.61617886728\n",
      "-4.96667661154\n",
      "-6.27944156548\n",
      "-8.25680765767\n",
      "-10.5272772685\n",
      "-12.9808882591\n",
      "-14.1859951856\n",
      "-12.176369339\n",
      "-9.90090931068\n",
      "-7.81756456711\n",
      "-6.12032433839\n",
      "-4.35825538773\n",
      "-3.25510701168\n",
      "-2.40132183059\n",
      "-1.64503455061\n",
      "-1.14049939495\n",
      "-0.884967838329\n",
      "-0.796373208939\n",
      "-0.835563585775\n",
      "-1.01284659506\n",
      "-1.21186506361\n",
      "-1.74838431089\n",
      "-2.55634532478\n",
      "-3.74169221506\n",
      "-5.03492530766\n",
      "-6.94661380107\n",
      "-9.0999846441\n",
      "-11.745325814\n",
      "-14.6752823928\n",
      "-14.3991496823\n",
      "-11.9339559673\n",
      "-9.89968706153\n",
      "-7.91491562618\n",
      "-6.04902469235\n",
      "-4.26175764814\n",
      "-3.26611262256\n",
      "-2.19149308998\n",
      "-1.71668241401\n",
      "-1.08433075429\n",
      "-0.810336968538\n",
      "-0.499263713905\n",
      "-0.37764991536\n",
      "-0.283196779591\n",
      "-0.371436933441\n",
      "-0.465672200248\n",
      "-0.676752829373\n",
      "-0.949964976915\n",
      "-1.44195825033\n",
      "-2.07947370466\n",
      "-2.95314069697\n",
      "-4.42035752069\n",
      "-5.92789181602\n",
      "-7.96066476384\n",
      "-10.7133142331\n",
      "-13.5182795917\n",
      "-15.9903687279\n",
      "-14.150957964\n",
      "-11.4054396452\n",
      "-9.25419716122\n",
      "-7.03296594929\n",
      "-5.47695222615\n",
      "-3.88372033253\n",
      "-2.87372468045\n",
      "-2.16214367443\n",
      "-1.5042987265\n",
      "-1.22898575395\n",
      "-0.813054562486\n",
      "-0.763019893147\n",
      "-0.596812958744\n",
      "-0.629846702241\n",
      "-0.603898537084\n",
      "-0.57849370839\n",
      "-0.837183283838\n",
      "-1.10842980871\n",
      "-1.50931502757\n",
      "-2.10052392559\n",
      "-3.13555648867\n",
      "-4.08078538176\n",
      "-5.68803431632\n",
      "-7.39112286702\n",
      "-9.5261089613\n",
      "-12.3066564285\n",
      "-14.6981152471\n",
      "-14.3273652591\n",
      "-11.7287263941\n",
      "-9.44612034378\n",
      "-7.15027478935\n",
      "-5.32225386625\n",
      "-3.83788042426\n",
      "-2.7934114421\n",
      "-1.89383819862\n",
      "-1.37422635827\n",
      "-1.07949707335\n",
      "-0.920355570461\n",
      "-0.878514677957\n",
      "-0.912801764126\n",
      "-1.10432555186\n",
      "-1.38884089827\n",
      "-1.91810966477\n",
      "-2.62200807895\n",
      "-3.42639186993\n",
      "-4.54103053773\n",
      "-6.11753006677\n",
      "-8.06241609023\n",
      "-10.2958936293\n",
      "-12.7879638506\n",
      "-14.2870336144\n",
      "-12.1490101202\n",
      "-10.3053252552\n",
      "-8.13462022141\n",
      "-6.24688371968\n",
      "-4.88364557534\n",
      "-3.70962472634\n",
      "-2.74066450654\n",
      "-1.80943993006\n",
      "-1.37117889323\n",
      "-1.00148364961\n",
      "-0.783979353375\n",
      "-0.491075640855\n",
      "-0.385887851193\n",
      "-0.258011945028\n",
      "-0.22102570892\n",
      "-0.228072171613\n",
      "-0.25357670216\n",
      "-0.29456411201\n",
      "-0.387494308078\n",
      "-0.567785949646\n",
      "-0.934609452284\n",
      "-1.28987412753\n",
      "-1.88271144758\n",
      "-2.58162909874\n",
      "-3.43411099316\n",
      "-4.45787954992\n",
      "-5.80541167583\n",
      "-8.0910064422\n",
      "-10.5342000412\n",
      "-12.8386841475\n",
      "-15.1229529963\n",
      "-12.9031796353\n",
      "-10.9779087368\n",
      "-8.94761181144\n",
      "-7.19327180877\n",
      "-5.18335961551\n",
      "-3.91570430653\n",
      "-2.88733643598\n",
      "-1.88618501261\n",
      "-1.27180020594\n",
      "-1.04950175192\n",
      "-0.886716406917\n",
      "-0.67983084556\n",
      "-0.579448552364\n",
      "-0.655342818133\n",
      "-0.59618893463\n",
      "-0.705565153606\n",
      "-0.926917369541\n",
      "-1.2462047094\n",
      "-1.6534194101\n",
      "-2.44633561678\n",
      "-3.51555058539\n",
      "-5.11220441844\n",
      "-7.25320937692\n",
      "-9.20384824614\n",
      "-11.5932967182\n",
      "-14.3779685483\n",
      "-15.3496096255\n",
      "-13.1095117872\n",
      "-11.019542901\n"
     ]
    }
   ],
   "source": [
    "import gym\n",
    "\n",
    "env = gym.make(\"Pendulum-v0\")\n",
    "\n",
    "num_episodes = 1\n",
    "num_time_steps = 200\n",
    "\n",
    "for episode in range(num_episodes):\n",
    "    print(\"\\nEpisode No.: {}\".format(episode))\n",
    "    observation = env.reset()\n",
    "    \n",
    "    for t in range(num_time_steps):\n",
    "#         print(observation)\n",
    "        env.render()\n",
    "        action = env.action_space.sample() # Uniform selection across\n",
    "        observation, reward, done, _ = env.step(action)\n",
    "        print(reward)\n",
    "        \n",
    "        if done:\n",
    "#             print(\"Episode finished after {} timesteps\\n\\n\".format(t+1))\n",
    "            break\n",
    "    \n",
    "env.close() # Close windows (and also destroys environment :( )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true
   },
   "source": [
    "## CEM (Binary Policy - Cart Pole) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-06-12T23:44:29.913392Z",
     "start_time": "2018-06-12T23:44:29.575268Z"
    },
    "code_folding": [
     4,
     16,
     51,
     69
    ],
    "hidden": true
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "\n",
    "# Simple Policy\n",
    "class BinaryActionLinearPolicy(object):\n",
    "    # Policy --> Linear weighted combination of the observations space\n",
    "    # thresholded at zero\n",
    "    def __init__(self, theta):\n",
    "        self.w = theta[:-1]\n",
    "        self.b = theta[-1]\n",
    "        \n",
    "    def act(self, ob):\n",
    "        y = ob.dot(self.w) + self.b\n",
    "        a = int(y < 0)\n",
    "        return a\n",
    "\n",
    "def cem(f, num_steps, th_mean, batch_size, n_iter, elite_frac, initial_std=1.0):\n",
    "    \"\"\"\n",
    "    Generic implementation of the cross-entropy method for maximizing a black-box function\n",
    "    f: a function mapping from vector -> scalar (i.e. from thetas --> total Reward)\n",
    "    num_steps: number of time steps that 'f' will use to evaluate each parameter\n",
    "    th_mean: initial mean of parameters (thetas)\n",
    "    batch_size: number of samples of theta to evaluate per iteration\n",
    "    n_iter: number of iterations\n",
    "    elite_frac: each iteration, select this fraction of the top-performing samples\n",
    "    initial_std: initial standard deviation over parameter (theta) vectors\n",
    "    \"\"\"\n",
    "    \n",
    "    n_elite = int(np.round(batch_size*elite_frac))\n",
    "    th_std = np.ones_like(th_mean) * initial_std # each individual theta std.\n",
    "\n",
    "    for _ in range(n_iter):\n",
    "        \n",
    "        # ths: batch_size x th_mean.size matrix containing in each row a set of\n",
    "        # parameters to be evaluated \n",
    "        ths = np.array([th_mean + dth for dth in  th_std[None,:]*np.random.randn(batch_size, th_mean.size)])\n",
    "        \n",
    "        # Evaluate each sample\n",
    "        ys = np.array([f(th, num_steps) for th in ths])\n",
    "        \n",
    "        # Select top best samples\n",
    "        elite_inds = ys.argsort()[::-1][:n_elite]\n",
    "        elite_ths = ths[elite_inds]\n",
    "#         print(\"Num. Elite: {}\".format(elite_ths.shape[0]))\n",
    "\n",
    "        # Compute new mean and std. by obtaining the avg. of the top (elite) samples\n",
    "        th_mean = elite_ths.mean(axis=0)\n",
    "        th_std = elite_ths.std(axis=0)\n",
    "        yield {'ys' : ys, 'theta_mean' : th_mean, 'y_mean' : ys.mean()}\n",
    "        \n",
    "def do_rollout(agent, env, num_steps, render=False):\n",
    "    \"\"\"\n",
    "    Simulate current agent/policy for 'num_steps' and return the total (undisccounted)\n",
    "    reward and elapsed time steps before episode ended\n",
    "    \"\"\"\n",
    "    \n",
    "    total_rew = 0\n",
    "    ob = env.reset()\n",
    "    for t in range(num_steps):\n",
    "        a = agent.act(ob)\n",
    "        (ob, reward, done, _info) = env.step(a)\n",
    "        total_rew += reward\n",
    "        if render and t%2==0: env.render(mode='human')#rgb_array\n",
    "        if done: break\n",
    "            \n",
    "    return total_rew, t+1\n",
    "\n",
    "def agent_eval(theta, num_steps):\n",
    "    \"\"\"\n",
    "    Given a policy (theta), evaluate it by simulating a single episode runned by\n",
    "    'num_steps' time steps of the simulation,and obtaining the total reward\n",
    "    \"\"\"\n",
    "    agent = BinaryActionLinearPolicy(theta)\n",
    "    rew, T = do_rollout(agent, env, num_steps)\n",
    "    \n",
    "    return rew\n",
    "\n",
    "# def writefile(fname, s):\n",
    "#     with open(path.join(outdir, fname), 'w') as fh: fh.write(s)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-06-12T23:46:38.157468Z",
     "start_time": "2018-06-12T23:46:36.702839Z"
    },
    "hidden": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[33mWARN: gym.spaces.Box autodetected dtype as <class 'numpy.float32'>. Please provide explicit dtype.\u001b[0m\n"
     ]
    },
    {
     "ename": "Error",
     "evalue": "Tried to reset environment which is not done. While the monitor is active for CartPole-v0, you cannot call reset() unless the episode is over.",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mError\u001b[0m                                     Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-4-7a9ce38f362a>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[1;32m     19\u001b[0m \u001b[0;31m# Train the agent\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     20\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 21\u001b[0;31m \u001b[0;32mfor\u001b[0m \u001b[0;34m(\u001b[0m\u001b[0mi\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0miterdata\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;32min\u001b[0m \u001b[0menumerate\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mcem\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0magent_eval\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mnum_steps\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mnp\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mzeros\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mnum_thetas\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mparams\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     22\u001b[0m     \u001b[0mprint\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m'Iteration: %2i; Iteration mean reward: %7.3f'\u001b[0m\u001b[0;34m%\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mi\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0miterdata\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m'y_mean'\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     23\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m<ipython-input-1-04e522745ec2>\u001b[0m in \u001b[0;36mcem\u001b[0;34m(f, num_steps, th_mean, batch_size, n_iter, elite_frac, initial_std)\u001b[0m\n\u001b[1;32m     36\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     37\u001b[0m         \u001b[0;31m# Evaluate each sample\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 38\u001b[0;31m         \u001b[0mys\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mnp\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0marray\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mf\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mth\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mnum_steps\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;32mfor\u001b[0m \u001b[0mth\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mths\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     39\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     40\u001b[0m         \u001b[0;31m# Select top best samples\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m<ipython-input-1-04e522745ec2>\u001b[0m in \u001b[0;36m<listcomp>\u001b[0;34m(.0)\u001b[0m\n\u001b[1;32m     36\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     37\u001b[0m         \u001b[0;31m# Evaluate each sample\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 38\u001b[0;31m         \u001b[0mys\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mnp\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0marray\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mf\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mth\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mnum_steps\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;32mfor\u001b[0m \u001b[0mth\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mths\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     39\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     40\u001b[0m         \u001b[0;31m# Select top best samples\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m<ipython-input-1-04e522745ec2>\u001b[0m in \u001b[0;36magent_eval\u001b[0;34m(theta, num_steps)\u001b[0m\n\u001b[1;32m     71\u001b[0m     \"\"\"\n\u001b[1;32m     72\u001b[0m     \u001b[0magent\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mBinaryActionLinearPolicy\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mtheta\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 73\u001b[0;31m     \u001b[0mrew\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mT\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mdo_rollout\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0magent\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0menv\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mnum_steps\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     74\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     75\u001b[0m     \u001b[0;32mreturn\u001b[0m \u001b[0mrew\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m<ipython-input-1-04e522745ec2>\u001b[0m in \u001b[0;36mdo_rollout\u001b[0;34m(agent, env, num_steps, render)\u001b[0m\n\u001b[1;32m     55\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     56\u001b[0m     \u001b[0mtotal_rew\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;36m0\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 57\u001b[0;31m     \u001b[0mob\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0menv\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mreset\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     58\u001b[0m     \u001b[0;32mfor\u001b[0m \u001b[0mt\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mrange\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mnum_steps\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     59\u001b[0m         \u001b[0ma\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0magent\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mact\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mob\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/Downloads/[/home/david/Programs/anaconda3]/envs/mtrl/lib/python3.6/site-packages/gym/wrappers/monitor.py\u001b[0m in \u001b[0;36mreset\u001b[0;34m(self, **kwargs)\u001b[0m\n\u001b[1;32m     35\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     36\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0mreset\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 37\u001b[0;31m         \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_before_reset\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     38\u001b[0m         \u001b[0mobservation\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0menv\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mreset\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     39\u001b[0m         \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_after_reset\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mobservation\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/Downloads/[/home/david/Programs/anaconda3]/envs/mtrl/lib/python3.6/site-packages/gym/wrappers/monitor.py\u001b[0m in \u001b[0;36m_before_reset\u001b[0;34m(self)\u001b[0m\n\u001b[1;32m    183\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0m_before_reset\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    184\u001b[0m         \u001b[0;32mif\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0menabled\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0;32mreturn\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 185\u001b[0;31m         \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mstats_recorder\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mbefore_reset\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    186\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    187\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0m_after_reset\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mobservation\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/Downloads/[/home/david/Programs/anaconda3]/envs/mtrl/lib/python3.6/site-packages/gym/wrappers/monitoring/stats_recorder.py\u001b[0m in \u001b[0;36mbefore_reset\u001b[0;34m(self)\u001b[0m\n\u001b[1;32m     66\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     67\u001b[0m         \u001b[0;32mif\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mdone\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0;32mNone\u001b[0m \u001b[0;32mand\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mdone\u001b[0m \u001b[0;32mand\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msteps\u001b[0m \u001b[0;34m>\u001b[0m \u001b[0;36m0\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 68\u001b[0;31m             \u001b[0;32mraise\u001b[0m \u001b[0merror\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mError\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"Tried to reset environment which is not done. While the monitor is active for {}, you cannot call reset() unless the episode is over.\"\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mformat\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0menv_id\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     69\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     70\u001b[0m         \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mdone\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;32mFalse\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mError\u001b[0m: Tried to reset environment which is not done. While the monitor is active for CartPole-v0, you cannot call reset() unless the episode is over."
     ]
    }
   ],
   "source": [
    "import gym\n",
    "from gym import wrappers\n",
    "from os import path\n",
    "\n",
    "# Environment Config.\n",
    "display = False\n",
    "env_id = \"CartPole-v0\"\n",
    "env = gym.make(env_id) # create environment\n",
    "env.seed(0)\n",
    "outdir = './results/cem-binary-cartPole'\n",
    "# env = wrappers.Monitor(env, outdir, force=True) #Used to save log data and video\n",
    "\n",
    "# Learning Algorithm Config.\n",
    "np.random.seed(0)\n",
    "params = dict(n_iter=20, batch_size=30, elite_frac = 0.2)\n",
    "num_thetas = env.observation_space.shape[0]+1 # plus 1 to create bias term\n",
    "num_steps = 100 # number of time steps per episode\n",
    "\n",
    "# Train the agent\n",
    "\n",
    "for (i, iterdata) in enumerate(cem(agent_eval, num_steps, np.zeros(num_thetas), **params)):\n",
    "    print('Iteration: %2i; Iteration mean reward: %7.3f'%(i, iterdata['y_mean']))\n",
    "    \n",
    "    if display and i%3==0: \n",
    "        agent = BinaryActionLinearPolicy(iterdata['theta_mean'])\n",
    "        do_rollout(agent, env, num_steps, render=True)\n",
    "\n",
    "env.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true
   },
   "source": [
    "## CEM (Continuous Policy - Pendulum & MountainCar) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-06-07T13:26:48.535287Z",
     "start_time": "2018-06-07T13:26:48.132093Z"
    },
    "code_folding": [
     2,
     13
    ],
    "hidden": true
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "\n",
    "class ContinuousActionLinearPolicy(object):\n",
    "    def __init__(self, theta, n_in, n_out):\n",
    "        assert len(theta) == (n_in + 1) * n_out\n",
    "        self.W = theta[0 : n_in * n_out].reshape(n_in, n_out)\n",
    "        self.b = theta[n_in * n_out : None].reshape(1, n_out)\n",
    "        \n",
    "    def act(self, ob):\n",
    "        a = ob.dot(self.W) + self.b\n",
    "        return a\n",
    "\n",
    "    \n",
    "def cem(f,n_in, n_out, num_steps, th_mean, batch_size, n_iter, elite_frac, initial_std=1.0):\n",
    "    \"\"\"\n",
    "    Generic implementation of the cross-entropy method for maximizing a black-box function\n",
    "    f: a function mapping from vector -> scalar (i.e. from thetas --> total Reward)\n",
    "    num_steps: number of time steps that 'f' will use to evaluate each parameter\n",
    "    th_mean: initial mean of parameters (thetas)\n",
    "    batch_size: number of samples of theta to evaluate per iteration\n",
    "    n_iter: number of iterations\n",
    "    elite_frac: each iteration, select this fraction of the top-performing samples\n",
    "    initial_std: initial standard deviation over parameter (theta) vectors\n",
    "    \"\"\"\n",
    "    \n",
    "    n_elite = int(np.round(batch_size*elite_frac))\n",
    "    th_std = np.ones_like(th_mean) * initial_std # each individual theta std.\n",
    "\n",
    "    for _ in range(n_iter):\n",
    "        \n",
    "        # ths: batch_size x th_mean.size matrix containing in each row a set of\n",
    "        # parameters to be evaluated \n",
    "        ths = np.array([th_mean + dth for dth in  th_std[None,:]*np.random.randn(batch_size, th_mean.size)])\n",
    "        \n",
    "        # Evaluate each sample\n",
    "        ys = np.array([f(th, n_in, n_out, num_steps) for th in ths])\n",
    "        \n",
    "        # Select top best samples\n",
    "        elite_inds = ys.argsort()[::-1][:n_elite]\n",
    "        elite_ths = ths[elite_inds]\n",
    "\n",
    "        # Compute new mean and std. by obtaining the avg. of the top (elite) samples\n",
    "        th_mean = elite_ths.mean(axis=0)\n",
    "        th_std = elite_ths.std(axis=0)\n",
    "        yield {'ys' : ys, 'theta_mean' : th_mean, 'y_mean' : ys.mean()}\n",
    "        \n",
    "        \n",
    "def do_rollout(agent, env, num_steps, render=False):\n",
    "    \"\"\"\n",
    "    Simulate current agent/policy for 'num_steps' and return the total (undisccounted)\n",
    "    reward and elapsed time steps before episode ended\n",
    "    \"\"\"\n",
    "    \n",
    "    total_rew = 0\n",
    "    ob = env.reset()\n",
    "    for t in range(num_steps):\n",
    "        a = agent.act(ob)[0]\n",
    "        ob, reward, done, _info = env.step(a)\n",
    "        total_rew += reward\n",
    "        if render and t%2==0: env.render(mode='human')#rgb_array\n",
    "        if done: break\n",
    "            \n",
    "    return total_rew, t+1       \n",
    "\n",
    "\n",
    "def agent_eval_cont(theta, n_in, n_out, num_steps):\n",
    "    \"\"\"\n",
    "    Given a policy (theta), evaluate it by simulating a single episode runned by\n",
    "    'num_steps' time steps of the simulation,and obtaining the total reward\n",
    "    \"\"\"\n",
    "    agent = ContinuousActionLinearPolicy(theta, n_in, n_out)\n",
    "    rew, T = do_rollout(agent, env, num_steps)\n",
    "    \n",
    "    return rew"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-06-07T13:15:27.204992Z",
     "start_time": "2018-06-07T13:14:49.655831Z"
    },
    "collapsed": true,
    "hidden": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[33mWARN: gym.spaces.Box autodetected dtype as <class 'numpy.float32'>. Please provide explicit dtype.\u001b[0m\n",
      "\u001b[33mWARN: gym.spaces.Box autodetected dtype as <class 'numpy.float32'>. Please provide explicit dtype.\u001b[0m\n",
      "Iteration:  0; Iteration mean reward: -1484.572\n",
      "Iteration:  1; Iteration mean reward: -1586.916\n",
      "Iteration:  2; Iteration mean reward: -1485.845\n",
      "Iteration:  3; Iteration mean reward: -1400.281\n",
      "Iteration:  4; Iteration mean reward: -1370.339\n",
      "Iteration:  5; Iteration mean reward: -1279.676\n",
      "Iteration:  6; Iteration mean reward: -1266.759\n",
      "Iteration:  7; Iteration mean reward: -1186.064\n",
      "Iteration:  8; Iteration mean reward: -1269.237\n",
      "Iteration:  9; Iteration mean reward: -1265.276\n",
      "Iteration: 10; Iteration mean reward: -1173.775\n",
      "Iteration: 11; Iteration mean reward: -1223.677\n",
      "Iteration: 12; Iteration mean reward: -1205.208\n",
      "Iteration: 13; Iteration mean reward: -1215.291\n",
      "Iteration: 14; Iteration mean reward: -1177.453\n",
      "Iteration: 15; Iteration mean reward: -1227.330\n",
      "Iteration: 16; Iteration mean reward: -1126.090\n",
      "Iteration: 17; Iteration mean reward: -1132.339\n",
      "Iteration: 18; Iteration mean reward: -1078.917\n",
      "Iteration: 19; Iteration mean reward: -1137.810\n"
     ]
    },
    {
     "ename": "KeyboardInterrupt",
     "evalue": "",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mKeyboardInterrupt\u001b[0m                         Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-36-4728c0049ab3>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[1;32m     20\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     21\u001b[0m \u001b[0;31m# Train the agent\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 22\u001b[0;31m \u001b[0;32mfor\u001b[0m \u001b[0;34m(\u001b[0m\u001b[0mi\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0miterdata\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;32min\u001b[0m \u001b[0menumerate\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mcem\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0magent_eval_cont\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mn_in\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mn_out\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mnum_steps\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mnp\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mzeros\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mnum_thetas\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mparams\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     23\u001b[0m     \u001b[0mprint\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m'Iteration: %2i; Iteration mean reward: %7.3f'\u001b[0m\u001b[0;34m%\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mi\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0miterdata\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m'y_mean'\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     24\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m<ipython-input-35-2b9cce491bf7>\u001b[0m in \u001b[0;36mcem\u001b[0;34m(f, n_in, n_out, num_steps, th_mean, batch_size, n_iter, elite_frac, initial_std)\u001b[0m\n\u001b[1;32m     34\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     35\u001b[0m         \u001b[0;31m# Evaluate each sample\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 36\u001b[0;31m         \u001b[0mys\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mnp\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0marray\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mf\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mth\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mn_in\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mn_out\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mnum_steps\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;32mfor\u001b[0m \u001b[0mth\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mths\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     37\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     38\u001b[0m         \u001b[0;31m# Select top best samples\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m<ipython-input-35-2b9cce491bf7>\u001b[0m in \u001b[0;36m<listcomp>\u001b[0;34m(.0)\u001b[0m\n\u001b[1;32m     34\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     35\u001b[0m         \u001b[0;31m# Evaluate each sample\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 36\u001b[0;31m         \u001b[0mys\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mnp\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0marray\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mf\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mth\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mn_in\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mn_out\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mnum_steps\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;32mfor\u001b[0m \u001b[0mth\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mths\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     37\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     38\u001b[0m         \u001b[0;31m# Select top best samples\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m<ipython-input-35-2b9cce491bf7>\u001b[0m in \u001b[0;36magent_eval_cont\u001b[0;34m(theta, n_in, n_out, num_steps)\u001b[0m\n\u001b[1;32m     70\u001b[0m     \"\"\"\n\u001b[1;32m     71\u001b[0m     \u001b[0magent\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mContinuousActionLinearPolicy\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mtheta\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mn_in\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mn_out\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 72\u001b[0;31m     \u001b[0mrew\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mT\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mdo_rollout\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0magent\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0menv\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mnum_steps\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     73\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     74\u001b[0m     \u001b[0;32mreturn\u001b[0m \u001b[0mrew\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m<ipython-input-35-2b9cce491bf7>\u001b[0m in \u001b[0;36mdo_rollout\u001b[0;34m(agent, env, num_steps, render)\u001b[0m\n\u001b[1;32m     56\u001b[0m     \u001b[0;32mfor\u001b[0m \u001b[0mt\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mrange\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mnum_steps\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     57\u001b[0m         \u001b[0ma\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0magent\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mact\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mob\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;36m0\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 58\u001b[0;31m         \u001b[0mob\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mreward\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mdone\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0m_info\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0menv\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mstep\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0ma\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     59\u001b[0m         \u001b[0mtotal_rew\u001b[0m \u001b[0;34m+=\u001b[0m \u001b[0mreward\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     60\u001b[0m         \u001b[0;32mif\u001b[0m \u001b[0mrender\u001b[0m \u001b[0;32mand\u001b[0m \u001b[0mt\u001b[0m\u001b[0;34m%\u001b[0m\u001b[0;36m2\u001b[0m\u001b[0;34m==\u001b[0m\u001b[0;36m0\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0menv\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mrender\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mmode\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;34m'human'\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;31m#rgb_array\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/Downloads/[/home/david/Programs/anaconda3]/envs/mlp/lib/python3.6/site-packages/gym/wrappers/monitor.py\u001b[0m in \u001b[0;36mstep\u001b[0;34m(self, action)\u001b[0m\n\u001b[1;32m     30\u001b[0m         \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_before_step\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0maction\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     31\u001b[0m         \u001b[0mobservation\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mreward\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mdone\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0minfo\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0menv\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mstep\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0maction\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 32\u001b[0;31m         \u001b[0mdone\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_after_step\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mobservation\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mreward\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mdone\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0minfo\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     33\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     34\u001b[0m         \u001b[0;32mreturn\u001b[0m \u001b[0mobservation\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mreward\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mdone\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0minfo\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/Downloads/[/home/david/Programs/anaconda3]/envs/mlp/lib/python3.6/site-packages/gym/wrappers/monitor.py\u001b[0m in \u001b[0;36m_after_step\u001b[0;34m(self, observation, reward, done, info)\u001b[0m\n\u001b[1;32m    177\u001b[0m         \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mstats_recorder\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mafter_step\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mobservation\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mreward\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mdone\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0minfo\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    178\u001b[0m         \u001b[0;31m# Record video\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 179\u001b[0;31m         \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mvideo_recorder\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mcapture_frame\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    180\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    181\u001b[0m         \u001b[0;32mreturn\u001b[0m \u001b[0mdone\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/Downloads/[/home/david/Programs/anaconda3]/envs/mlp/lib/python3.6/site-packages/gym/wrappers/monitoring/video_recorder.py\u001b[0m in \u001b[0;36mcapture_frame\u001b[0;34m(self)\u001b[0m\n\u001b[1;32m     99\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    100\u001b[0m         \u001b[0mrender_mode\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m'ansi'\u001b[0m \u001b[0;32mif\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mansi_mode\u001b[0m \u001b[0;32melse\u001b[0m \u001b[0;34m'rgb_array'\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 101\u001b[0;31m         \u001b[0mframe\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0menv\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mrender\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mmode\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mrender_mode\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    102\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    103\u001b[0m         \u001b[0;32mif\u001b[0m \u001b[0mframe\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/Downloads/[/home/david/Programs/anaconda3]/envs/mlp/lib/python3.6/site-packages/gym/core.py\u001b[0m in \u001b[0;36mrender\u001b[0;34m(self, mode)\u001b[0m\n\u001b[1;32m    282\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    283\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0mrender\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mmode\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;34m'human'\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 284\u001b[0;31m         \u001b[0;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0menv\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mrender\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mmode\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    285\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    286\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0mclose\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/Downloads/[/home/david/Programs/anaconda3]/envs/mlp/lib/python3.6/site-packages/gym/envs/classic_control/pendulum.py\u001b[0m in \u001b[0;36mrender\u001b[0;34m(self, mode)\u001b[0m\n\u001b[1;32m     80\u001b[0m             \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mimgtrans\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mscale\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m(\u001b[0m\u001b[0;34m-\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mlast_u\u001b[0m\u001b[0;34m/\u001b[0m\u001b[0;36m2\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mnp\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mabs\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mlast_u\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m/\u001b[0m\u001b[0;36m2\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     81\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 82\u001b[0;31m         \u001b[0;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mviewer\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mrender\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mreturn_rgb_array\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mmode\u001b[0m\u001b[0;34m==\u001b[0m\u001b[0;34m'rgb_array'\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     83\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     84\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0mclose\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/Downloads/[/home/david/Programs/anaconda3]/envs/mlp/lib/python3.6/site-packages/gym/envs/classic_control/rendering.py\u001b[0m in \u001b[0;36mrender\u001b[0;34m(self, return_rgb_array)\u001b[0m\n\u001b[1;32m     81\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0mrender\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mreturn_rgb_array\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mFalse\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     82\u001b[0m         \u001b[0mglClearColor\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;36m1\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;36m1\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;36m1\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;36m1\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 83\u001b[0;31m         \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mwindow\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mclear\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     84\u001b[0m         \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mwindow\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mswitch_to\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     85\u001b[0m         \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mwindow\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mdispatch_events\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/Downloads/[/home/david/Programs/anaconda3]/envs/mlp/lib/python3.6/site-packages/pyglet/window/__init__.py\u001b[0m in \u001b[0;36mclear\u001b[0;34m(self)\u001b[0m\n\u001b[1;32m   1226\u001b[0m         \u001b[0mbuffer\u001b[0m\u001b[0;34m.\u001b[0m  \u001b[0mThe\u001b[0m \u001b[0mwindow\u001b[0m \u001b[0mmust\u001b[0m \u001b[0mbe\u001b[0m \u001b[0mthe\u001b[0m \u001b[0mactive\u001b[0m \u001b[0mcontext\u001b[0m \u001b[0;34m(\u001b[0m\u001b[0msee\u001b[0m\u001b[0;31m \u001b[0m\u001b[0;31m`\u001b[0m\u001b[0mswitch_to\u001b[0m\u001b[0;31m`\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1227\u001b[0m         \"\"\"\n\u001b[0;32m-> 1228\u001b[0;31m         \u001b[0mgl\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mglClear\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mgl\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mGL_COLOR_BUFFER_BIT\u001b[0m \u001b[0;34m|\u001b[0m \u001b[0mgl\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mGL_DEPTH_BUFFER_BIT\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   1229\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1230\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0mdispatch_event\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/Downloads/[/home/david/Programs/anaconda3]/envs/mlp/lib/python3.6/site-packages/pyglet/gl/lib.py\u001b[0m in \u001b[0;36merrcheck\u001b[0;34m(result, func, arguments)\u001b[0m\n\u001b[1;32m     83\u001b[0m     \u001b[0;32mpass\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     84\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 85\u001b[0;31m \u001b[0;32mdef\u001b[0m \u001b[0merrcheck\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mresult\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mfunc\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0marguments\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     86\u001b[0m     \u001b[0;32mif\u001b[0m \u001b[0m_debug_gl_trace\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     87\u001b[0m         \u001b[0;32mtry\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mKeyboardInterrupt\u001b[0m: "
     ]
    }
   ],
   "source": [
    "import gym\n",
    "from gym import wrappers\n",
    "from os import path\n",
    "\n",
    "# Environment Config.\n",
    "display = False\n",
    "env_id = \"Pendulum-v0\"\n",
    "env = gym.make(env_id) # create environment\n",
    "env.seed(0)\n",
    "outdir = './results/cem-cont-pendulum'\n",
    "env = wrappers.Monitor(env, outdir, force=True) #Used to save log data and video\n",
    "\n",
    "# Learning Algorithm Config.\n",
    "np.random.seed(0)\n",
    "params = dict(n_iter=500, batch_size=25, elite_frac = 0.3)\n",
    "num_steps = 300 # number of time steps per episode\n",
    "n_in = env.observation_space.shape[0]\n",
    "n_out = env.action_space.shape[0]\n",
    "num_thetas = (n_in + 1) * n_out # plus 1 to create bias term\n",
    "\n",
    "# Train the agent\n",
    "for (i, iterdata) in enumerate(cem(agent_eval_cont, n_in, n_out, num_steps, np.zeros(num_thetas), **params)):\n",
    "    print('Iteration: %2i; Iteration mean reward: %7.3f'%(i, iterdata['y_mean']))\n",
    "    \n",
    "    if display and i%10==0: \n",
    "        agent = ContinuousActionLinearPolicy(iterdata['theta_mean'], n_in, n_out)\n",
    "        do_rollout(agent, env, num_steps, render=True)\n",
    "\n",
    "env.close()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-06-10T10:34:23.794755Z",
     "start_time": "2018-06-10T10:34:23.730876Z"
    },
    "hidden": true
   },
   "outputs": [
    {
     "ename": "NameError",
     "evalue": "name 'wrappers' is not defined",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mNameError\u001b[0m                                 Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-40-40c8e98c600e>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[1;32m      1\u001b[0m \u001b[0mnum_simu\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;36m5\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 2\u001b[0;31m \u001b[0menv\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mwrappers\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mMonitor\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0menv\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0moutdir\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mforce\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mTrue\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;31m#Used to save log data and video\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m      3\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      4\u001b[0m \u001b[0;32mfor\u001b[0m \u001b[0mi\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mrange\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mnum_simu\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      5\u001b[0m     \u001b[0magent\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mContinuousActionLinearPolicy\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0miterdata\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m'theta_mean'\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mn_in\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mn_out\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mNameError\u001b[0m: name 'wrappers' is not defined"
     ]
    }
   ],
   "source": [
    "num_simu = 5\n",
    "env = wrappers.Monitor(env, outdir, force=True) #Used to save log data and video\n",
    "\n",
    "for i in range(num_simu):\n",
    "    agent = ContinuousActionLinearPolicy(iterdata['theta_mean'], n_in, n_out)\n",
    "    do_rollout(agent, env, num_steps, render=True)\n",
    "# env.close()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-06-07T13:26:29.932533Z",
     "start_time": "2018-06-07T13:25:29.880822Z"
    },
    "collapsed": true,
    "hidden": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[33mWARN: gym.spaces.Box autodetected dtype as <class 'numpy.float32'>. Please provide explicit dtype.\u001b[0m\n",
      "\u001b[33mWARN: gym.spaces.Box autodetected dtype as <class 'numpy.float32'>. Please provide explicit dtype.\u001b[0m\n",
      "Iteration:  0; Iteration mean reward: -26.275\n"
     ]
    },
    {
     "ename": "KeyboardInterrupt",
     "evalue": "",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mKeyboardInterrupt\u001b[0m                         Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-52-33d82916e542>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[1;32m     20\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     21\u001b[0m \u001b[0;31m# Train the agent\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 22\u001b[0;31m \u001b[0;32mfor\u001b[0m \u001b[0;34m(\u001b[0m\u001b[0mi\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0miterdata\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;32min\u001b[0m \u001b[0menumerate\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mcem\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0magent_eval_cont\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mn_in\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mn_out\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mnum_steps\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mnp\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mzeros\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mnum_thetas\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mparams\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     23\u001b[0m     \u001b[0mprint\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m'Iteration: %2i; Iteration mean reward: %7.3f'\u001b[0m\u001b[0;34m%\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mi\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0miterdata\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m'y_mean'\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     24\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m<ipython-input-51-dc5cf6c25e95>\u001b[0m in \u001b[0;36mcem\u001b[0;34m(f, n_in, n_out, num_steps, th_mean, batch_size, n_iter, elite_frac, initial_std)\u001b[0m\n\u001b[1;32m     34\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     35\u001b[0m         \u001b[0;31m# Evaluate each sample\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 36\u001b[0;31m         \u001b[0mys\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mnp\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0marray\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mf\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mth\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mn_in\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mn_out\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mnum_steps\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;32mfor\u001b[0m \u001b[0mth\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mths\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     37\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     38\u001b[0m         \u001b[0;31m# Select top best samples\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m<ipython-input-51-dc5cf6c25e95>\u001b[0m in \u001b[0;36m<listcomp>\u001b[0;34m(.0)\u001b[0m\n\u001b[1;32m     34\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     35\u001b[0m         \u001b[0;31m# Evaluate each sample\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 36\u001b[0;31m         \u001b[0mys\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mnp\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0marray\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mf\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mth\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mn_in\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mn_out\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mnum_steps\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;32mfor\u001b[0m \u001b[0mth\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mths\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     37\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     38\u001b[0m         \u001b[0;31m# Select top best samples\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m<ipython-input-51-dc5cf6c25e95>\u001b[0m in \u001b[0;36magent_eval_cont\u001b[0;34m(theta, n_in, n_out, num_steps)\u001b[0m\n\u001b[1;32m     71\u001b[0m     \"\"\"\n\u001b[1;32m     72\u001b[0m     \u001b[0magent\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mContinuousActionLinearPolicy\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mtheta\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mn_in\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mn_out\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 73\u001b[0;31m     \u001b[0mrew\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mT\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mdo_rollout\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0magent\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0menv\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mnum_steps\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     74\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     75\u001b[0m     \u001b[0;32mreturn\u001b[0m \u001b[0mrew\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m<ipython-input-51-dc5cf6c25e95>\u001b[0m in \u001b[0;36mdo_rollout\u001b[0;34m(agent, env, num_steps, render)\u001b[0m\n\u001b[1;32m     57\u001b[0m     \u001b[0;32mfor\u001b[0m \u001b[0mt\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mrange\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mnum_steps\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     58\u001b[0m         \u001b[0ma\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0magent\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mact\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mob\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;36m0\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 59\u001b[0;31m         \u001b[0mob\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mreward\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mdone\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0m_info\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0menv\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mstep\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0ma\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     60\u001b[0m         \u001b[0mtotal_rew\u001b[0m \u001b[0;34m+=\u001b[0m \u001b[0mreward\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     61\u001b[0m         \u001b[0;32mif\u001b[0m \u001b[0mrender\u001b[0m \u001b[0;32mand\u001b[0m \u001b[0mt\u001b[0m\u001b[0;34m%\u001b[0m\u001b[0;36m2\u001b[0m\u001b[0;34m==\u001b[0m\u001b[0;36m0\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0menv\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mrender\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mmode\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;34m'human'\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;31m#rgb_array\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/Downloads/[/home/david/Programs/anaconda3]/envs/mlp/lib/python3.6/site-packages/gym/wrappers/monitor.py\u001b[0m in \u001b[0;36mstep\u001b[0;34m(self, action)\u001b[0m\n\u001b[1;32m     30\u001b[0m         \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_before_step\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0maction\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     31\u001b[0m         \u001b[0mobservation\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mreward\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mdone\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0minfo\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0menv\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mstep\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0maction\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 32\u001b[0;31m         \u001b[0mdone\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_after_step\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mobservation\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mreward\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mdone\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0minfo\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     33\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     34\u001b[0m         \u001b[0;32mreturn\u001b[0m \u001b[0mobservation\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mreward\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mdone\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0minfo\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/Downloads/[/home/david/Programs/anaconda3]/envs/mlp/lib/python3.6/site-packages/gym/wrappers/monitor.py\u001b[0m in \u001b[0;36m_after_step\u001b[0;34m(self, observation, reward, done, info)\u001b[0m\n\u001b[1;32m    177\u001b[0m         \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mstats_recorder\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mafter_step\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mobservation\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mreward\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mdone\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0minfo\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    178\u001b[0m         \u001b[0;31m# Record video\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 179\u001b[0;31m         \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mvideo_recorder\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mcapture_frame\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    180\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    181\u001b[0m         \u001b[0;32mreturn\u001b[0m \u001b[0mdone\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/Downloads/[/home/david/Programs/anaconda3]/envs/mlp/lib/python3.6/site-packages/gym/wrappers/monitoring/video_recorder.py\u001b[0m in \u001b[0;36mcapture_frame\u001b[0;34m(self)\u001b[0m\n\u001b[1;32m    114\u001b[0m                 \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_encode_ansi_frame\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mframe\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    115\u001b[0m             \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 116\u001b[0;31m                 \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_encode_image_frame\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mframe\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    117\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    118\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0mclose\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/Downloads/[/home/david/Programs/anaconda3]/envs/mlp/lib/python3.6/site-packages/gym/wrappers/monitoring/video_recorder.py\u001b[0m in \u001b[0;36m_encode_image_frame\u001b[0;34m(self, frame)\u001b[0m\n\u001b[1;32m    164\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    165\u001b[0m         \u001b[0;32mtry\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 166\u001b[0;31m             \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mencoder\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mcapture_frame\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mframe\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    167\u001b[0m         \u001b[0;32mexcept\u001b[0m \u001b[0merror\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mInvalidFrame\u001b[0m \u001b[0;32mas\u001b[0m \u001b[0me\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    168\u001b[0m             \u001b[0mlogger\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mwarn\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m'Tried to pass invalid video frame, marking as broken: %s'\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0me\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/Downloads/[/home/david/Programs/anaconda3]/envs/mlp/lib/python3.6/site-packages/gym/wrappers/monitoring/video_recorder.py\u001b[0m in \u001b[0;36mcapture_frame\u001b[0;34m(self, frame)\u001b[0m\n\u001b[1;32m    299\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    300\u001b[0m         \u001b[0;32mif\u001b[0m \u001b[0mdistutils\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mversion\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mLooseVersion\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mnp\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m__version__\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;34m>=\u001b[0m \u001b[0mdistutils\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mversion\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mLooseVersion\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m'1.9.0'\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 301\u001b[0;31m             \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mproc\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mstdin\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mwrite\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mframe\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mtobytes\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    302\u001b[0m         \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    303\u001b[0m             \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mproc\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mstdin\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mwrite\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mframe\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mtostring\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mKeyboardInterrupt\u001b[0m: "
     ]
    }
   ],
   "source": [
    "import gym\n",
    "from gym import wrappers\n",
    "from os import path\n",
    "\n",
    "# Environment Config.\n",
    "display = False\n",
    "env_id = \"MountainCarContinuous-v0\"\n",
    "env = gym.make(env_id) # create environment\n",
    "env.seed(0)\n",
    "outdir = './results/cem-cont-mountainCar'\n",
    "env = wrappers.Monitor(env, outdir, force=True) #Used to save log data and video\n",
    "\n",
    "# Learning Algorithm Config.\n",
    "np.random.seed(0)\n",
    "params = dict(n_iter=500, batch_size=100, elite_frac = 0.5)\n",
    "num_steps = 300 # number of time steps per episode\n",
    "n_in = env.observation_space.shape[0]\n",
    "n_out = env.action_space.shape[0]\n",
    "num_thetas = (n_in + 1) * n_out # plus 1 to create bias term\n",
    "\n",
    "# Train the agent\n",
    "for (i, iterdata) in enumerate(cem(agent_eval_cont, n_in, n_out, num_steps, np.zeros(num_thetas), **params)):\n",
    "    print('Iteration: %2i; Iteration mean reward: %7.3f'%(i, iterdata['y_mean']))\n",
    "    \n",
    "    if display and i%10==0: \n",
    "        agent = ContinuousActionLinearPolicy(iterdata['theta_mean'], n_in, n_out)\n",
    "        do_rollout(agent, env, num_steps, render=True)\n",
    "\n",
    "env.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true
   },
   "source": [
    "## Policy Gradient w/ Pong (Based on Andrej Kaparthy's Code) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-06-12T08:03:20.878660Z",
     "start_time": "2018-06-12T08:03:19.153467Z"
    },
    "hidden": true
   },
   "outputs": [],
   "source": [
    "\"\"\" \n",
    "Trains an agent with (stochastic) Policy Gradients on Pong. Uses OpenAI Gym. \n",
    " - Orignal Code by Andrek Kaparthy: https://gist.github.com/karpathy/a4166c7fe253700972fcbc77e4ea32c5\n",
    " - Blog explainning Code: http://karpathy.github.io/2016/05/31/rl/\n",
    " - Code in TF: https://github.com/gameofdimension/policy-gradient-pong/blob/master/policy_gradient_pong.py\n",
    " - Code multiple actions w/ Softmax: https://gist.github.com/etienne87/6803a65653975114e6c6f08bb25e1522\n",
    " - Actor Mimic version: https://github.com/schinger/pong_actor-critic\n",
    "\"\"\"\n",
    "\n",
    "import numpy as np\n",
    "import pickle\n",
    "import gym\n",
    "from gym import wrappers\n",
    "import os\n",
    "\n",
    "# General settings\n",
    "log_level = 0 # 0:don't print anything ; 1:print fundamental things; 2: print everything\n",
    "\n",
    "load_agent_dir = \"./results/pong_agents\"\n",
    "# agent_to_load = \"pongAgent_ep_100_epTotR_-20.0_runningAvgR_-20.999.p\" # Basic (shit) Agent\n",
    "# agent_to_load = \"pongAgent_ep_1900_meanR_-14_runningAvgR_-11.5.p\" # Intermediate Agent\n",
    "agent_to_load = \"pongAgent_ep_5100_epTotR_-6.0_runningAvgR_-0.3960004310609734.p\" # Further trainned agent\n",
    "\n",
    "resume = True # resume from previous checkpoint\n",
    "simulate = False # Test trainned agent in simulation \n",
    "render = False # Show video window of environment simulation\n",
    "save_video = False # save video when simulating agent\n",
    "num_episodes = 1 # number of episodes to simulate agent for\n",
    "\n",
    "\n",
    "# Policy Network Hyperparameters\n",
    "H = 200 # number of hidden layer neurons\n",
    "batch_size = 10 # every how many episodes to do a param update\n",
    "# learning_rate = 1e-3 # For initial 4000 episodes\n",
    "learning_rate = 3e-4 # For further episodes\n",
    "gamma = 0.99 # discount factor for reward\n",
    "decay_rate = 0.99 # decay factor for RMSProp leaky sum of grad^2\n",
    "\n",
    "\n",
    "# Model Initialization\n",
    "input_D = 80 * 80 # input dimensionality: 80x80 grid\n",
    "if resume: # Load trainned agent\n",
    "    model = pickle.load(open(load_agent_dir + \"/\" + agent_to_load, 'rb'))\n",
    "else:\n",
    "    model = {} # Dictionary storing Policy Neural Network parameters\n",
    "    model['W1'] = np.random.randn(H,input_D) / np.sqrt(input_D) # \"Xavier\" initialization\n",
    "    model['W2'] = np.random.randn(H) / np.sqrt(H)\n",
    "    \n",
    "# model.items() returns a touple of Keys ('W1') and values (initial weight values)\n",
    "grad_buffer = { key : np.zeros_like(val) for key,val in model.items() } # update buffers that add up gradients over a batch\n",
    "rmsprop_cache = { key : np.zeros_like(val) for key,val in model.items() } # rmsprop memory"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-06-12T08:03:24.937770Z",
     "start_time": "2018-06-12T08:03:24.365656Z"
    },
    "code_folding": [
     0,
     4,
     14,
     25,
     36,
     55,
     78
    ],
    "hidden": true
   },
   "outputs": [],
   "source": [
    "def sigmoid(x): \n",
    "    return 1.0 / (1.0 + np.exp(-x)) # sigmoid \"squashing\" function to probability --> interval [0,1]\n",
    "\n",
    "\n",
    "def prepro(I):\n",
    "    \"\"\" prepro 210x160x3 uint8 frame into 6400 (80x80) 1D float vector \"\"\"\n",
    "    I = I[35:195] # crop\n",
    "    I = I[::2,::2,0] # downsample by factor of 2 and only use Red channel (could had choosen any channel since in the following lines we binarize everything)\n",
    "    I[I == 144] = 0 # erase background (background type 1)\n",
    "    I[I == 109] = 0 # erase background (background type 2)\n",
    "    I[I != 0] = 1 # everything else (paddles, ball) just set to 1 (binarize)\n",
    "    return I.astype(np.float).ravel() # ravel() == flatten\n",
    "\n",
    "\n",
    "def discount_rewards(r):\n",
    "    \"\"\" take 1D float array of rewards and compute discounted reward \"\"\"\n",
    "    discounted_r = np.zeros_like(r)\n",
    "    running_add = 0\n",
    "    for t in reversed(range(0, r.size)):\n",
    "        if r[t] != 0: running_add = 0 # reset the sum, since this was a game boundary (pong specific!)\n",
    "        running_add = running_add * gamma + r[t]\n",
    "        discounted_r[t] = running_add\n",
    "    return discounted_r\n",
    "\n",
    "\n",
    "def policy_forward(x):\n",
    "    # Neural Network performing logistic regresion, hence, having a sigmoid activation function at the\n",
    "    # output layer establishing P(y=1 | x;w,b)\n",
    "    act_hid = np.dot(model['W1'], x)\n",
    "    h = act_hid*(act_hid>0) + 0*(act_hid<=0) # ReLU nonlinearity\n",
    "    logp = np.dot(model['W2'], h)\n",
    "    p = sigmoid(logp) # P(y=1 | x;w,b)\n",
    "    \n",
    "    return p, h # return probability of taking action 2, and hidden state\n",
    "   \n",
    "    \n",
    "def policy_backward(eph, epdlogp, epx):  # ONLY THING PEDING TO FULLY UNDERSTAND --> DIMENSIONS OF DW2 AND DW1 ?\n",
    "    \"\"\" \n",
    "    backward pass: \n",
    "    eph --> Episodes Hidden --> array of intermediate hidden states\n",
    "    epdlogp --> Episodes gradient --> array containing the gradients dloss/dlogp = dL/dactivation_of_sigmoid)\n",
    "    epx --> Episodes input features\n",
    "    \"\"\"\n",
    "    # Computing each gradient through Reverse Mode Differiantation (a.k.a. Backpropagation)\n",
    "    # Note that each WEight gradient is computed based on the sum of epdlogp gradients across all time steps in episode\n",
    "    dW2 = np.dot(eph.T, epdlogp).ravel() # dL/dW2 = dL/dlogp * dlogp/dW2 = (y-p)*h = epdlogp*eph\n",
    "    dh = np.outer(epdlogp, model['W2']) # dL/dh = dL/dlogp * dlogp/dh = (y-p)*W2 = epdlogp*W2\n",
    "    dact_hid = dh*(eph > 0) + 0*(eph <= 0) # backprop. ReLu\n",
    "    dW1 = np.dot(dact_hid.T, epx) # dL/dW1 = dL/dact_hid * dact_hid/dW1 = (...)*x = dact_hid*x\n",
    "    \n",
    "    return {'W1':dW1, 'W2':dW2}\n",
    "\n",
    "\n",
    "\n",
    "# Methods to test/simulate agent\n",
    "def act (observation, prev_x):\n",
    "    \"\"\"\n",
    "    Selects action taken by trainned agent based on (preproccessed difference) of observations\n",
    "    Input:\n",
    "        * observation (np array): Matrix containing the raw RGB image of the current state of the game\n",
    "        * prev_x (np array): previous preproccessed game frame\n",
    "    \n",
    "    Output:\n",
    "        * action (int): action selected by policy network\n",
    "        * prev_x (np array): updated previous preproccessed game frame\n",
    "    \"\"\"\n",
    "    # Get preprocesses image + compute difference between frames\n",
    "    cur_x = prepro(observation)\n",
    "    x = cur_x - prev_x if prev_x is not None else np.zeros(input_D)\n",
    "    prev_x = cur_x # Save current image for next environment step\n",
    "    \n",
    "    # Select action\n",
    "    a_prob, _ = policy_forward(x)\n",
    "    action = 2 if np.random.uniform() < a_prob else 3 # roll the dice! to select action\n",
    "    \n",
    "    return action, prev_x\n",
    "    \n",
    "    \n",
    "def sim_agent(env, num_episodes, render=False, save_video=False):\n",
    "    \"\"\"\n",
    "    Simulates trainned agent (i.e. model with NN's weights) in given environment (env)\n",
    "    Input:\n",
    "        * env (Open AI gym object): Open Ai Gym Environment\n",
    "        * num_episodes (int): number of episodes (in Pong, one episoe= 21 games) to simulate\n",
    "        * render (bool): determines if video should be rendered in window\n",
    "        * save_video (bool): enables saving video and other stats of simulated episodes\n",
    "        \n",
    "    Returns:\n",
    "        * mean_reward_episodes (double): Mean reward obtained across all episodes\n",
    "        * mean_won_episodes (double): Mean number of episodes won \n",
    "        * if save_video=True, stores videos and stats in folder determined by 'outdir'\n",
    "    \"\"\" \n",
    "    \n",
    "    # General Config\n",
    "    episodes_tot_reward = []\n",
    "    episodes_perc_won = []\n",
    "    \n",
    "    # Monitoring Config\n",
    "    if save_video: \n",
    "        outdir = './results/pg-binary-Pong'\n",
    "        if not os.path.exists(outdir):  os.makedirs(outdir) # create directory if it doesn't exist\n",
    "        env = wrappers.Monitor(env, outdir, force=True) # Used to save log data and video\n",
    "        \n",
    "    # Simulate each Episode\n",
    "    for episode in range(num_episodes):\n",
    "        print(\"\\nEpisode No.: {}\".format(episode))\n",
    "        observation = env.reset()\n",
    "        prev_x = None # variable storing previous (preprocessed) observation\n",
    "        reward_sum = 0\n",
    "        reward_list = []\n",
    "        \n",
    "        while True:\n",
    "            if render: env.render()\n",
    "            action, prev_x = act(observation, prev_x) # Select action based on  PMF parametrized by agent/NN weights\n",
    "            \n",
    "            observation, reward, done, _ = env.step(action)\n",
    "            reward_sum += reward\n",
    "            if reward !=0: reward_list.append(reward)\n",
    "            \n",
    "            if done:\n",
    "                perc_won = 100*np.sum(np.array(reward_list)>0)/len(reward_list)\n",
    "                episodes_tot_reward.append(reward_sum)\n",
    "                episodes_perc_won.append(perc_won)\n",
    "                \n",
    "                print(\"Episode {} finished ; Total Reward: {} ; Perc. Matches Won: {}\\n\\n\"\\\n",
    "                      .format(episode, reward_sum,  perc_won ))\n",
    "\n",
    "                break\n",
    "\n",
    "    env.close() # Close windows (and also destroys environment :( )\n",
    "    mean_reward_episodes = np.mean(episodes_tot_reward)\n",
    "    mean_won_episodes = np.mean(episodes_perc_won)\n",
    "    \n",
    "    return mean_reward_episodes, mean_won_episodes\n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-06-12T08:08:42.982691Z",
     "start_time": "2018-06-12T08:07:55.700773Z"
    },
    "code_folding": [
     10,
     24
    ],
    "hidden": true,
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "env = gym.make(\"Pong-v0\")\n",
    "\n",
    "# Variable already stablished ontop --> just having them here for quick changes\n",
    "log_level = 1\n",
    "# render = False\n",
    "# simulate = False # Test trainned agent in simulation \n",
    "# save_video = True # save video when simulating agent\n",
    "# num_episodes = 1 # number of episodes to simulate agent for\n",
    "\n",
    "# SIMULATE TRAINED AGENT\n",
    "if simulate: # simulate pre-trainned loaded agent\n",
    "    mean_reward_episodes, mean_won_episodes = sim_agent(env, num_episodes, render, save_video)\n",
    "    print(\"Mean Reward: {} ; Mean Perc. Won: {}\".format(mean_reward_episodes, mean_won_episodes))\n",
    "\n",
    "# TRAIN AGENT \n",
    "else: # Train agent (Policy Network)       \n",
    "    \n",
    "    observation = env.reset()\n",
    "    prev_x = None # variable storing previous observation (image)\n",
    "    xs,hs,dlogps,drs = [],[],[],[] # Stores episode trajectory: observations, NN Hidden state, gradients, rewards \n",
    "    running_reward = None\n",
    "    reward_sum = 0\n",
    "    episode_number = 0\n",
    "\n",
    "    while True:\n",
    "        if render: env.render()\n",
    "\n",
    "        # PREPROCESS: Set input to network to be difference between images\n",
    "        cur_x = prepro(observation)\n",
    "        x = cur_x - prev_x if prev_x is not None else np.zeros(input_D)\n",
    "        prev_x = cur_x # Save current image for next environment step\n",
    "\n",
    "        # FORWARD POLICY NN: forward the policy network and sample an action from the returned probability\n",
    "        a_prob, h = policy_forward(x) # Get probability of action 2 (this assumption is established by the programmer, and could be\n",
    "                                      # switched to action 3) given current state/observation/image\n",
    "        action = 2 if np.random.uniform() < a_prob else 3 # roll the dice! to select action\n",
    "\n",
    "        # CACHING FOR BACKPROP: Store time step variables needed for backprop ... here we are basically\n",
    "        # appending data points and whe done==True, we will update the params for the COMPLETE BATCH\n",
    "        xs.append(x) # observation stored\n",
    "        hs.append(h) # hidden state stored\n",
    "\n",
    "        # Create \"fake\" label pushing the NN to make the action it just choose more probable\n",
    "        # see EC1 at the End of Code for detailed explenation\n",
    "        y = 1 if action == 2 else 0 \n",
    "\n",
    "        # Gradient of Negative Log Likelihood Loss function w.r.t activation of sigmoid (i.e. logp)\n",
    "        # see EC2 at the End of Code for detailed explenation\n",
    "        dlogps.append(y - a_prob) \n",
    "\n",
    "\n",
    "        # STEP ENV: get new measurements\n",
    "        observation, reward, done, info = env.step(action)\n",
    "        reward_sum += reward\n",
    "        drs.append(reward) # record reward \n",
    "\n",
    "        if done: # an episode finished\n",
    "            episode_number += 1\n",
    "\n",
    "            # STACK EPISODE VARIABLES: Stack inputs, hidden states,  gradients, and rewards \n",
    "            epx = np.vstack(xs) # Episode Inputs\n",
    "            eph = np.vstack(hs)\n",
    "            epdlogp = np.vstack(dlogps)\n",
    "            epr = np.vstack(drs) # Episode Rewards\n",
    "            xs,hs,dlogps,drs = [],[],[],[] # RESET array memory\n",
    "\n",
    "            # COMPUTE EPISODE'S DISCCOUNTED REWARD: returning a list, done for each time step in episode\n",
    "            discounted_epr = discount_rewards(epr)\n",
    "            discounted_epr -= np.mean(discounted_epr) # Standardize (mu=0, std=1)\n",
    "            discounted_epr /= np.std(discounted_epr)\n",
    "\n",
    "            # SCORE FUNCCTION GRADIENT ESTIMATOR: the group of the episodes gradients stored in 'epdlogp' pushes the\n",
    "            # parameters towards increasing the prob. of selecting the actions it selected\n",
    "            # during the Episode. However, we modulate this gradients movements by how good it was to take\n",
    "            # each action; based on the advantage (disccounted reward) at each time step\n",
    "            epdlogp *= discounted_epr # Policy Gradient (PG) magic happens right here !\n",
    "            grad = policy_backward(eph, epdlogp, epx)# Compute gradient w.r.t  Weights for current Episode\n",
    "            for key in model: grad_buffer[key] += grad[key] # Accumulate grad. over all batches\n",
    "\n",
    "            # RMSPROP: perform update every \"batch_size\" episodes\n",
    "            if episode_number % batch_size == 0:\n",
    "                for key,val in model.items():\n",
    "                    g = grad_buffer[key] # gradient accumulated thorugh all batches\n",
    "                    rmsprop_cache[key] = decay_rate * rmsprop_cache[key] + (1 - decay_rate) * g**2\n",
    "                    model[key] += learning_rate * g / (np.sqrt(rmsprop_cache[key]) + 1e-5) # UPDATE model params.  \n",
    "                    grad_buffer[key] = np.zeros_like(val) # reset batch gradient buffer\n",
    "\n",
    "            # Information Variables and Storing Model\n",
    "            running_reward =  reward_sum if running_reward is None else running_reward * 0.99 + reward_sum * 0.01\n",
    "            if log_level>=1: print ('Resetting Env --> Episode: %i ; Tot. Reward: %f ; Current Running mean: %f' % (episode_number, reward_sum, running_reward))\n",
    "            if episode_number % 100 == 0: \n",
    "                pickle.dump(model, open(load_agent_dir + \"/\" + 'pongAgent_ep_{}_epTotR_{}_runningAvgR_{}.p'.format(episode_number, reward_sum, running_reward), 'wb'))\n",
    "                if log_level>=1: print (\"Saved Model at Episode %i \\n\\n\" % (episode_number))\n",
    "            \n",
    "            # RESET VARIABLES\n",
    "            reward_sum = 0\n",
    "            observation = env.reset() # reset env\n",
    "            prev_x = None\n",
    "\n",
    "        if reward != 0 and log_level>=2: # Print info when the game ends.\n",
    "            print (\"Episode: {} - Outcome: {} \".format(episode_number, \"LOST\" if reward == -1 else \"WON\"))\n",
    "\n",
    "        \n",
    "\"\"\"\n",
    "********* Extended Comments (EC) **********\n",
    "\n",
    "* EC1: Fake label for Score Function Gradient Estimator\n",
    "Create \"fake\" target label, in which with out mattering if the chossen 'action' gave us a high/low\n",
    "reward, we want the NN to make the that action more likely (i.e. action=2, make it more likely by\n",
    "p--> 1; if action=3, make it more likely by pushing p-->0) ...its naive to believe that making any\n",
    "action choosen by the non-optimal NN more probable will in turn improve the agent, HOWEVER, we will\n",
    "multiply the gradient by the score function (advantage function, which is the disccounted reward \n",
    "for each time step), and multiplying the gradient by this will indeed move our NN's parameters\n",
    "in the direction maximizing the Expected value of the score/advange function, which is what we want!\n",
    "(see details in \"score function gradient estimator\" section of karpathy.github.io/2016/05/31/rl/)\n",
    "\n",
    "* EC2: Gradient for Maximizing (log) Likelihood of the model / Minimizing Negative Log Likelihood:\n",
    "Compute gradient --> this is the gradient of the Loss function w.r.t activation of sigmoid(logp = W2.h).\n",
    "The loss function we are minimizing is L = -log(likelihood) i.e. Negative Log Likelihood, which \n",
    "in turn is equivalent to maximizing the (log) likelihood of the data (see more on L and derivation of \n",
    "dL/dp at MLPR Uni. of Edinburgh notes or at  http://cs231n.github.io/neural-networks-2/#losses)\n",
    "\n",
    "********* Extended Comments (EC) **********\n",
    "\"\"\"\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true
   },
   "source": [
    "## Proximal Policy Optimization (PPO, based on Patrick Coady's Code) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-06-13T22:44:46.733454Z",
     "start_time": "2018-06-13T22:44:26.529382Z"
    },
    "hidden": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[33mWARN: gym.spaces.Box autodetected dtype as <class 'numpy.float32'>. Please provide explicit dtype.\u001b[0m\n",
      "\u001b[33mWARN: gym.spaces.Box autodetected dtype as <class 'numpy.float32'>. Please provide explicit dtype.\u001b[0m\n"
     ]
    },
    {
     "ename": "KeyboardInterrupt",
     "evalue": "",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mKeyboardInterrupt\u001b[0m                         Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-8-91e34c825480>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[1;32m      6\u001b[0m \u001b[0menv\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mreset\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      7\u001b[0m \u001b[0;32mfor\u001b[0m \u001b[0mt\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mrange\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;36m1000\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 8\u001b[0;31m     \u001b[0menv\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mrender\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m      9\u001b[0m     \u001b[0ma\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0menv\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0maction_space\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msample\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     10\u001b[0m     \u001b[0mobs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mrew\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mdone\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0minfo\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0menv\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mstep\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0ma\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/Downloads/[/home/david/Programs/anaconda3]/envs/mtrl/lib/python3.6/site-packages/gym/core.py\u001b[0m in \u001b[0;36mrender\u001b[0;34m(self, mode)\u001b[0m\n\u001b[1;32m    282\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    283\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0mrender\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mmode\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;34m'human'\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 284\u001b[0;31m         \u001b[0;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0menv\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mrender\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mmode\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    285\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    286\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0mclose\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/Downloads/[/home/david/Programs/anaconda3]/envs/mtrl/lib/python3.6/site-packages/gym/envs/box2d/bipedal_walker.py\u001b[0m in \u001b[0;36mrender\u001b[0;34m(self, mode)\u001b[0m\n\u001b[1;32m    479\u001b[0m         \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mviewer\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mdraw_polyline\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mf\u001b[0m \u001b[0;34m+\u001b[0m \u001b[0;34m[\u001b[0m\u001b[0mf\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;36m0\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mcolor\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;36m0\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;36m0\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;36m0\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mlinewidth\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;36m2\u001b[0m \u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    480\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 481\u001b[0;31m         \u001b[0;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mviewer\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mrender\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mreturn_rgb_array\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mmode\u001b[0m\u001b[0;34m==\u001b[0m\u001b[0;34m'rgb_array'\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    482\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    483\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0mclose\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/Downloads/[/home/david/Programs/anaconda3]/envs/mtrl/lib/python3.6/site-packages/gym/envs/classic_control/rendering.py\u001b[0m in \u001b[0;36mrender\u001b[0;34m(self, return_rgb_array)\u001b[0m\n\u001b[1;32m     88\u001b[0m             \u001b[0mgeom\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mrender\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     89\u001b[0m         \u001b[0;32mfor\u001b[0m \u001b[0mgeom\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0monetime_geoms\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 90\u001b[0;31m             \u001b[0mgeom\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mrender\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     91\u001b[0m         \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mtransform\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mdisable\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     92\u001b[0m         \u001b[0marr\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/Downloads/[/home/david/Programs/anaconda3]/envs/mtrl/lib/python3.6/site-packages/gym/envs/classic_control/rendering.py\u001b[0m in \u001b[0;36mrender\u001b[0;34m(self)\u001b[0m\n\u001b[1;32m    156\u001b[0m         \u001b[0;32mfor\u001b[0m \u001b[0mattr\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mreversed\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mattrs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    157\u001b[0m             \u001b[0mattr\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0menable\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 158\u001b[0;31m         \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mrender1\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    159\u001b[0m         \u001b[0;32mfor\u001b[0m \u001b[0mattr\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mattrs\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    160\u001b[0m             \u001b[0mattr\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mdisable\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/Downloads/[/home/david/Programs/anaconda3]/envs/mtrl/lib/python3.6/site-packages/gym/envs/classic_control/rendering.py\u001b[0m in \u001b[0;36mrender1\u001b[0;34m(self)\u001b[0m\n\u001b[1;32m    278\u001b[0m         \u001b[0mglBegin\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mGL_LINE_LOOP\u001b[0m \u001b[0;32mif\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mclose\u001b[0m \u001b[0;32melse\u001b[0m \u001b[0mGL_LINE_STRIP\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    279\u001b[0m         \u001b[0;32mfor\u001b[0m \u001b[0mp\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mv\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 280\u001b[0;31m             \u001b[0mglVertex3f\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mp\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;36m0\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mp\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;36m1\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;36m0\u001b[0m\u001b[0;34m)\u001b[0m  \u001b[0;31m# draw each vertex\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    281\u001b[0m         \u001b[0mglEnd\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    282\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0mset_linewidth\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mx\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mKeyboardInterrupt\u001b[0m: "
     ]
    }
   ],
   "source": [
    "env.env.world.gravity = (0,-200)\n",
    "# env = gym.make(\"CarRacing-v0\")\n",
    "\n",
    "env.reset()\n",
    "for t in range(1000):\n",
    "    env.render()\n",
    "    a = env.action_space.sample()\n",
    "    obs, rew, done, info = env.step(a)\n",
    "    if done: break\n",
    "env.close()"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}