Reinforcement learning cost function - reinforcement-learning

Question

Newb question
I am writing a OpenAI Gym pong player with TensorFlow and thus far have been able to create the network based on a random initialization so that it would randomly return to move the player paddle up or down.
After the epoch is over (21 games played where the computer won) I collected a set of observations, moves and scores. The final observation of a game receives a score and each preceding observation can be scored based on Bellman equation.
Now my questions what I do not understand yet:
How do I calculate the cost function so that it would be propagated as a start gradient for backward propagation? I totally get it with supervised learning, but here we do not have any labels to score agains.
How would I start optimizing the network?
Maybe a pointer to existing code or some literature would help.
Here's where I compute the rewards:
def compute_observation_rewards(self, gamma, up_score_probabilities):
"""
Applies Bellman equation and determines reward for each stored observation
:param gamma: Learning decay
:param up_score_probabilities: Probabilities for up score
:returns: List of scores for each move
"""
score_sum = 0
discounted_rewards = []
# go backwards through all observations
for i, p in enumerate(reversed(self._states_score_action)):
o = p[0]
s = p[1]
if s != 0:
score_sum = 0
score_sum = score_sum * gamma + s
discounted_rewards.append(score_sum)
# # normalize scores
discounted_rewards = np.array(discounted_rewards)
discounted_rewards -= np.mean(discounted_rewards)
discounted_rewards /= np.std(discounted_rewards)
return discounted_rewards
Below is my network:
with tf.variable_scope('NN_Model', reuse=tf.AUTO_REUSE):
layer1 = tf.layers.conv2d(inputs,
3,
3,
strides=(1, 1),
padding='valid',
data_format='channels_last',
dilation_rate=(1, 1),
activation= tf.nn.relu,
use_bias=True,
bias_initializer=tf.zeros_initializer(),
trainable=True,
name='layer1'
)
# (N - F + 1) x (N - F + 1)
# => layer1 should be
# (80 - 3 + 1) * (80 - 3 + 1) = 78 x 78
pool1 = tf.layers.max_pooling2d(layer1,
pool_size=5,
strides=2,
name='pool1')
# int((N - f) / s +1)
# (78 - 5) / 2 + 1 = 73/2 + 1 = 37
layer2 = tf.layers.conv2d(pool1,
5,
5,
strides=(2, 2),
padding='valid',
data_format='channels_last',
dilation_rate=(1, 1),
activation= tf.nn.relu,
use_bias=True,
kernel_initializer=tf.random_normal_initializer(),
bias_initializer=tf.zeros_initializer(),
trainable=True,
name='layer2',
reuse=None
)
# ((N + 2xpadding - F) / stride + 1) x ((N + 2xpadding - F) / stride + 1)
# => layer1 should be
# int((37 + 0 - 5) / 2) + 1
# 16 + 1 = 17
pool2 = tf.layers.max_pooling2d(layer2,
pool_size=3,
strides=2,
name='pool2')
# int((N - f) / s +1)
# (17 - 3) / 2 + 1 = 7 + 1 = 8
flat1 = tf.layers.flatten(pool2, 'flat1')
# Kx64
full1 = tf.contrib.layers.fully_connected(flat1,
num_outputs=1,
activation_fn=tf.nn.sigmoid,
weights_initializer=tf.contrib.layers.xavier_initializer(),
biases_initializer=tf.zeros_initializer(),
trainable=True,
scope=None
)

Answer 1

The algorithm you're looking for is called REINFORCE.
I would suggest reading chapter 13 of Sutton and Barto's RL book.
Here's pseudocode from the book.
Here, theta is the set of weights of your neural net. If you're unfamiliar with some of the rest of the notation, I'd suggest reading Chapter 3 of the above-mentioned book. It covers the basic problem formulation.