Training loop structure

[etiennemontenegro]

@noi01
This is your current training loop structure :

    for e in range(EPISODES):
        state = env.reset()
        state = np.reshape(state, [1, state_size])
        
        for time in range(500):
            # env.render() #no preview of environment
            action = agent.act(state)
            next_state, reward, done, _ = env.step(action)
            reward = reward if not done else -10
            next_state = np.reshape(next_state, [1, state_size])
            agent.remember(state, action, reward, next_state, done)
            state = next_state

1. Is the second loop to limit the maximum steps the agent can take before the episode is over? If yes, this should be moved inside the environment step function. This would allow truncating the episode and give a reward based on the fact the agent did not reach the goal in time.

2. The reward assignment here set reward to -10 when the environment is done. Is that really what you want to do? From my understanding, solving the environment should be positive for the agent. What is the goal of this line?

[noi01]

This is legacy of testing in virtual environment to check if agent works so outdated.

Truncating episodes may be important just because it will be continous real time learning. The logic would be :

• after x episodes if battery_state improved - rewards, if it didn't - negative reward
• if battery_state reaches 0 - end learning

Let me know if this makes sense, if not I'll sit down and rewrite it to be conmprehesible.

Not sure what to do because we still haven't settled with my supervisors of what the robot will be... haha. So the loop structure may change.

[etiennemontenegro]

so for this, we should buffer the battery state after each episode somewhere and compare it with the current one.

I don't see any mention of monitoring a battery in the code.. is this something you'll add later?

[noi01]

I don't see any mention of monitoring a battery in the code.. is this something you'll add later?

it's not there? Well, I guess I deleted it at some point. I'll look for what was there and get back to you.

[noi01]

Ok, did check - it was a placeholder code so we can put it as to be added later.

Since I don't have the hardware for that narrowed down yet, I think that's the best course of action.

Later it will be as you say, comparing 'battery state at this timestep' to 'battery state at timestep-1'