How can I use OpenAI's Gym environments "step" function or something equivalent without actually "stepping" the environment?
I just want to know what is the next state given current state and action but not to actually do the "step".
Related
I'm working on applying RL techniques in a real-time task and I've created a custom environment following OpenAI Gym interfaces. I'm trying to utilize a method of dynamically setting the time interval of interactions (the same as setting the time of calling step function) but I have no idea of how to implement this with Ray RLlib.
I've checked the rollout worker and custom evaluation in RLlib but it seems the further modification to RLlib is needed. So is there a simple way to do this? Or is there another RL framework which could be helpful? Thank you very much!
I want to integrate my environment into the openAI gym and then use stable baselines library for training it.
link to stable baseline:
https://stable-baselines.readthedocs.io/
The learning method in the stable baseline is with one-line learning and you don't have access to the actions that are taken during the training.
model.learn(total_timesteps=10000)
More specifically you don't do the line where you sample from the environment in your code:
action = space.sample()
However, I like to add some logic to the place where I choose the action of the next state and reject some actions that the logic doesn't apply to them (like chess board illegal moves), something like:
for _ in range(1000):
action = env.space.sample()
if some_logic(action):
continue
one way to do it is to write a wrapper for the action_space sample() function and only choose legal actions. Like the one here class DiscreteWrapper(spaces.Discrete) in the following link:
https://github.com/rockingdingo/gym-gomoku/blob/master/gym_gomoku/envs/gomoku.py
But the problem is that stable baseline only accepts certain data types and doesn't allow that either.
How can I do it in a way that is integrable into the stable baseline framework and doesn't violate the stable baselines criteria? If that is not possible at all, does anyone know a reinforcement learning framework other that unlike stable baseline allows access to the actions?
Anybody knows any OpenAI Gym environments where we can set the initial state of the game? For example, I found the MountainCarContinuous-v0 can do such thing so that we can select at which point the car starts. However, I am looking for another more complex environment. Thanks in advance for your help!
You have to redefine the reset function of the class (for example, this). You may want to define it such that it gets as input your desired state, something like
def reset(self, state):
self.state = state
return np.array(self.state)
This should work for all OpenAI gym environments. If you want to do it for other simulators, things may be different. For instance, MuJoCo allows to do something like
saved_state = env.sim.get_state()
env.sim.set_state(saved_state)
I want to modify the agents structure (for example the length of the legs) in the bipedal_walker environment.
Is there anyway to do that?
The easiest way would be to download and modify the OpenAI gym source code, and then use your locally modified version of gym in place of the pip package you installed.
For example, if you want to modify the length of the legs in the bipedal_walker environment, you'd probably want to play around with the LEG_H variable on this line of code.
There are some things that I would like to modify in the OpenAI environments. If we use the Cartpole example then we can edit things that are in the class init function but with environments that use Box2D it doesn't seem to be as straightforward.
For example, consider the BipedalWalker environment.
In this case, how would I edit things like the SPEED_HIP or SPEED_KNEE variables?
Yes, you can modify or create new environments in gym. The simplest (but not recommended) way is to modify the constants in your local gym installation directly, but of course that's not really nice.
A nicer way is to download the bipedal walker environment file (from here) and save it to a file (say, my_bipedal_walker.py)
Then you modify the constants in the my_bipedal_walker.py file, and then just import it in your code (assuming you put the file in a path that is importable, or the same folder as your other code files):
import gym
from my_bipedal_walker import BipedalWalker
env = BipedalWalker()
Then you have the env variable being an instance of the environment, with your defined constants for the physics computation, which you can use with any RL algorithm.
An even nicer way would be making your custom environment available in the OpenAI gym registry, which you can do by following the instructions here
You can edit the bipedal walker environment just like you can modify the cartpole environment.
All you have to do is modify the constants for SPEED_HIP and SPEED_KNEE.
If you want to change how those constants are used in the locomotion of the agent, you might also want to tweak the step method.
After making changes to the code, when you instantiate the environment, the new instance will be using the modifications you made.