Applying REINFORCE to easy21 - reinforcement-learning

Question

I'm trying to apply the REINFORCE algorithm (with SoftMax policy, undiscounted Gt with baseline) on David Silver's easy21, and I am having problems with the actual implementation. When compared to a pure MC approach, the produced result does not converge to Q*. Here is the related code:
hit = True
stick = False
actions = [hit, stick]
alpha = 0.1
theta = np.random.randn(420).reshape((420,1))
def psi(state, action):
if state.player < 1 or state.player > 21:
return np.zeros((420, 1))
dealers = [int(state.dealer == x + 1) for x in range(0, 10)]
players = [int(state.player == x + 1) for x in range(0, 21)]
actions = [int(action == hit), int(action == stick)]
psi = [1 if (i == 1 and j == 1 and k == 1) else 0
for i in dealers for j in players for k in actions]
return np.array(psi).reshape((420, 1))
def Q(state, action, weight):
return np.matmul(psi(state, action).T, weight)
def softmax(state, weight):
allQ = [Q(state, a, weight) for a in actions]
probs = np.exp(allQ) / np.sum(np.exp(allQ))
return probs.reshape((2,))
def score_function(state, action, weight):
probs = softmax(state, weight)
expected_score = (probs[0] * psi(state, hit)) + (probs[1] * psi(state, stick))
return psi(state, action) - expected_score
def softmax_policy(state, weight):
probs = softmax(state, weight)
if np.random.random() < probs[1]:
return stick
else:
return hit
if __name__ == "__main__":
Q_star = np.load('Q_star.npy')
for k in range(1, ITERATIONS):
terminal = False
state = game.initialise_state()
action = softmax_policy(state, theta)
history = [state, action]
while not terminal:
state, reward = game.step(state, action)
action = softmax_policy(state, theta)
terminal = state.terminal
if terminal:
state_action_pairs = zip(history[0::3], history[1::3])
history.append(reward)
history.append(state)
Gt = sum(history[2::3])
for s, a in state_action_pairs:
advantage = Gt - Q(s, a, prev_theta)
theta += alpha * score_function(s, a, theta) * advantage
else:
history.append(reward)
history.append(state)
history.append(action)
if k % 10000 == 0:
print("MSE: " + str(round(np.sum((Q_star - generate_Q(theta)) ** 2),2)))
Output:
python reinforce.py
MSE: 288.18
MSE: 248.45
MSE: 227.08
MSE: 215.46
MSE: 207.3
MSE: 202.61
MSE: 197.82
MSE: 195.96
MSE: 194.01
The table below shows the value function created using this algorithm:
Update:
Fixed the code by using a different theta initialisation:
theta = np.zeros((420,1))
Current value function:
But the current value function still does not match Q*
(missing peak at player sum = 11)
The entire code is available at:
https://github.com/Soundpulse/easy21-rl/blob/main/reinforce.py