[Feature Request] Core Improvement

[Trinkle23897]

Road Map:

1. Install and run envpool examples successfully;
2. Setup bazel build environment successfully: refer to the online documentation or CI scripts under .github/;
3. Read code, typically envpool/core/, make sure you understand the current system structure; in short:

• array.h contains a C++-based numpy-style array, paired with spec.h; for data transfer
• dict.h contains a C++-based dict; for declaring env attributes such as observation space and action space;
• env_spec.h contains the common field of environment attributes;
• env.h is the single env abstraction in envpool; other C++ env MUST inherits this class and overwrite Reset, Step, and IsDone;
• state_buffer_queue.h and action_buffer_queue.h are (almost) lock-free queue implementation for AsyncEnvPool;
• envpool.h is the base class for various of envpool implementation, e.g., PyEnvPool is the python binding of envpool, AsyncEnvPool is the C++ async env execution;
• async_envpool.h contains the main logic of concurrent execution; the thread pool is inside its constructor.

Things to do:

• [Feature Request] Support elementwise bounds for array spec #27
• [Feature Request] Py3.10, Windows and MacOS support #168
• Find out the cause of the following issue, and maybe Merge Circular Buffer's threadpool into Async EnvPool's threadpool to improve performance #3, or lock-free thread pool?

However, the curve shows some strange things: suppose we use N CPU cores with
a single machine, when we set the number of thread (M) to M<N/2 , the performance
is linearly scaled; when M>=N/2 , the performance gain is limited.

For Win/Mac:

• https://docs.bazel.build/versions/main/platforms-intro.html
• https://docs.bazel.build/versions/main/configurable-attributes.html

[alek5k]

Hi there, do you have an estimation for when Windows will be supported? Thanks

[Trinkle23897]

I'm curious about your usecase. Do you want to use it on windows or develop your own environment on windows?

[alek5k]

Hi there. My typical workflow is developing on Windows and running experiments on Linux computers. I work in a corporation and we are issued Windows computers, so a lot of our tooling for development is only on Windows machines. Some of our customers also run dedicated Windows servers, so if we wanted to deploy it would make it easier to just do it natively.

For the dev use case, it might be good enough to just be able to install it and run it, regardless of whether I enjoy the performance benefits or not.

Hope this helps. What are your plans with Windows support going into the future?

[Trinkle23897]

Hope this helps. What are your plans with Windows support going into the future?

Thanks for sharing this valuable information with us! For long-term planning, we are going to support windows/macos, but the priority is not higher than new environment integration. Also, some people working on this project (typically CMU master students) are going for their summer internship, so we probably won't have progress in this issue. However, if you are interested in making this thing work, I can offer my help and please do not hesitate to submit the pull request!