Improve this page

Training your first model

Your goal is to construct a controller, i.e. a function from the state space (current positions, velocities and accelerations of joints) to action space (muscle excitations), that will enable the model to perform a certain task like walking, reaching, throwing a ball, etc. Suppose you trained a neural network mapping observations (the current state of the model) to actions (muscle excitations), i.e. you have a function action = my_controller(observation), then

# ...
total_reward = 0.0
for i in range(200):
    # make a step given by the controller and record the state and the reward
    observation, reward, done, info = env.step(my_controller(observation))
    total_reward += reward
    if done:
        break

# Your reward is
print("Total reward %f" % total_reward)

There are many ways to construct the function my_controller(observation). We will show how to do it with a DDPG (Deep Deterministic Policy Gradients) algorithm, using keras-rl.

Your first controller

Below we present how to train a basic controller using keras-rl. First you need to install extra packages:

conda install keras -c conda-forge
pip install tensorflow git+https://github.com/matthiasplappert/keras-rl.git
git clone https://github.com/stanfordnmbl/osim-rl.git

keras-rl is an excellent package compatible with OpenAI Gym, which allows you to quickly build your first models!

cd osim-rl/examples

To train the model using DDPG algorithm you can simply run the scirpt ddpg.keras-rl.py as follows:

Training

python ddpg.keras-rl.py --visualize --train --model sample

Testing

and for the gait example (walk as far as possible):

python ddpg.keras-rl.py --visualize --test --model sample

Back

Next