framestack

[merv801]

Hello.
Thanks for sharing this repo.
It seems that you have not used the frame stacking(using 3 or 4 consecutive frames as state) in the Atari environment. Is that correct?
I also was wondering that have you test it in other Atari games?

[zplizzi]

That's right - it should be pretty easy to add though if you wanted to try it! I think I briefly tried some other games, but unfortunately didn't record the details.

[merv801]

Thanks for the reply. I am planning to rewrite your implementation to learn about ppo so I would try adding frame-stacking, however I was wondering is there any reason that you didn't add frame-stacking? Does it help with the learning in a specific way? Does is hinder the learning process?

[zplizzi]

Frame-stacking is generally understood to be important for making games more markovian - meaning that the true state of the game is fully represented by the input to the model. Eg in Pong from a single frame, you can't tell the direction the ball is moving - but from a stack of 4 frames you can. Honestly I think I just forgot to add it here - and in Pong it's probably not too important because a policy of "keep the paddle in front of the ball" works decently even if you don't know which way the ball is moving.