In Keras(Deep learning library), RepeatVector + TimeDistributed = Error? - deep-learning

Question

Hi I'm a beginner keras.
I'm making some model.
step 1. Input batch and word list, (BATCH_SIZE, WORD_INDEX_LIST)
step 2. Get word embeddings each words (BATCH_SIZE, WORD_LENGTH, EMBEDDING_SIZE)
step 3. Average each each word embeddings in each batch. (BATCH_SIZE, EMBEDDING_SIZE)
step 4. Repeat vector N, (BATCH_SIZE, N, EMBEDDING_SIZE)
step 5. Apply Dense Layer each time step
So, I write code.
MAX_LEN = 20 ( = WORD_INDEX_LIST)
step 1
layer_target_input = Input(shape=(MAX_LEN,), dtype="int32", name="layer_target_input")
# step2
layer_embedding = Embedding(input_dim = n_symbols+1, output_dim=vector_dim,input_length=MAX_LEN,
name="embedding", weights= [embedding_weights],trainable = False)
encoded_target = layer_embedding(layer_target_input)
# step 3
encoded_target_agg = KL.core.Lambda( lambda x: K.sum(x, axis=1) )(encoded_target)
#step 4
encoded_target_agg_repeat = KL.RepeatVector( MAX_LEN)(encoded_target_agg)
# step 5
layer_annotated_tahn = KL.Dense(output_dim=50, name="layer_tahn")
layer_annotated_tahn_td = KL.TimeDistributed(layer_annotated_tahn) (encoded_target_agg_repeat)
model = KM.Model(input=[layer_target_input], output=[ layer_annotated_tahn_td])
r = model.predict({ "layer_target_input":dev_targ}) # dev_targ = (2, 20, 300)
But, when i run this code,
result is bellow.
Traceback (most recent call last):
File "Main.py", line 127, in <module>
r = model.predict({ "layer_target_input":dev_targ})
File "/usr/local/anaconda/lib/python2.7/site-packages/Keras-1.0.7-py2.7.egg/keras/engine/training.py", line 1180, in predict
batch_size=batch_size, verbose=verbose)
File "/usr/local/anaconda/lib/python2.7/site-packages/Keras-1.0.7-py2.7.egg/keras/engine/training.py", line 888, in _predict_loop
outs[i][batch_start:batch_end] = batch_out
ValueError: could not broadcast input array from shape (30,20,50) into shape (2,20,50)
why batch size is changed?
What I have wrong?

Answer 1

The problem is in Lambda operator. In your case it takes a tensor of shape (batch_size, max_len, embedding_size) and is expected to produce a tensor of shape (batch_size, embedding_size). However, the Lambda op doesn't know what transformation you apply internally, and therefore during the graph compilation mistakenly assumes that the shape doesn't change, therefore assuming that the output shape is (batch_size, max_len, embedding_size). The RepeastVector that follows expects the input to be two-dimensional, but never asserts that it is the case. The way it produces the expected shape is (batch_size, num_repetitions, in_shape[1]). Since Lambda mistakenly reported its shape as (batch_size, max_len, embedding_size), RepeatVector now reports its shape as (batch_size, num_repetitions, max_len) instead of expected (batch_size, num_repetitions, embedding_size). num_repetitions in your case is the same as max_len, so RepeastVector reports its shape as (batch_size, max_len, max_len). The way TimeDistributed(Dense) works is:
Reshape((-1, input_shape[2]))
Dense()
Reshape((-1, input_shape[1], num_outputs))
By now input_shape[2] is mistakenly assumed to be max_len instead of embedding_size, but the actual tensor that is given has correct shape of (batch_size, max_len, embedding_size), so what ends up happening is:
Reshape((batch_size * embedding_size, max_len))
Dense()
Reshape((batch_size * embedding_size / max_len, max_len, num_outputs))
In your case batch_size * embedding_size / max_len happens to be 2 * 300 / 20 = 30, that's where your wrong shape comes from.
To fix it, you need to explicitly tell Lambda the shape you want it to produce:
encoded_target_agg = KL.core.Lambda( lambda x: K.sum(x, axis=1), output_shape=(vector_dim,))(encoded_target)