Is learning and cumulative reward a good metrics to evaluate a RL model? - reinforcement-learning

Question

i am new to reinforcement learning.
I have a problem here that i am using DQN on. I have plotted a cumulative reward curve while learning and taking actions. After 100 episodes and it shows a lot of fluctuations that does not show me whether it has learnt anything.
However, instead of using learning and cumulative reward, I put the model through the whole simulation without learning method after each episode and it shows me that the model is actually learning well. This extended the program runtime by quite a bit.
In addition, i have to extract the best model along the way because the final model seems to be performing badly at times.
Any advice or explanation for this?

Answer 1

Try to use the average mean return it's usually a good metric to know if the agent is improving or not.
If you're using tf_agent you can do something like this :
...
checkpoint_dir = os.path.join('./', 'checkpoint')
train_checkpointer = common.Checkpointer(
ckpt_dir=checkpoint_dir,
max_to_keep=1,
agent=agent,
policy=agent.policy,
replay_buffer=replay_buffer,
global_step=train_step
)
policy_dir = os.path.join('./', 'policy')
tf_policy_saver = policy_saver.PolicySaver(agent.policy)
def train_agent(n_iterations):
best_AverageReturn = 0
time_step = None
policy_state = agent.collect_policy.get_initial_state(tf_env.batch_size)
iterator = iter(dataset)
for iteration in range(n_iterations):
time_step, policy_state = collect_driver.run(time_step, policy_state)
trajectories, buffer_info = next(iterator)
train_loss = agent.train(trajectories)
if iteration % 10 == 0:
print("\r{} loss:{:.5f}".format(iteration, train_loss.loss.numpy()), end="")
if iteration % 1000 == 0 and averageReturnMetric.result() > best_AverageReturn:
best_AverageReturn = averageReturnMetric.result()
train_checkpointer.save(train_step)
tf_policy_saver.save(policy_dir)
After 1000 steps the train function evaluates the average return and create a checkpoint if there are any improvements