Can I use pad_sequence with transformer in Pytorch？ - deep-learning

Question

I'm trying to use transformer to process some image data (not NLP data), e.g. 480 x 640 images with different sequence length, an example would be [6, 480, 640], [7, 480, 640], [8, 480, 640]. And I would like to put these three sequences into one batch.
However, most tutorials I saw use torchtext to deal with the non-fixed length problem. But since I run the transformer with my own dataset, torchtext is not applicable(is it?). After searching I find pad_sequence can be used to deal with this problem.
However I didn't find any tutorials about using pad_sequence with transformer. Is it applicable？Has anyone try it before?

Answer 1

Let's say we have 03 images with different dimensions. Applying pad_sequence function on them will result as follow:
import torch
from torch.nn.utils.rnn import pad_sequence
image_1 = torch.ones(25, 30)
image_2 = torch.ones(32, 30)
image_3 = torch.ones(29, 30)
images = pad_sequence([image_1, image_2, image_3])
print(images.size())
# torch.Size([32, 3, 30])
This remains the same if you are working with 3D images
import torch
from torch.nn.utils.rnn import pad_sequence
image_1 = torch.ones(25, 30, 50)
image_2 = torch.ones(32, 30, 50)
image_3 = torch.ones(29, 30, 50)
images = pad_sequence([image_1, image_2, image_3])
print(images.size())
# torch.Size([32, 3, 30, 50])
One thing you should be aware of with this function is that it only works when the images share the n - 1 dimensions. In other words, if you have something like this:
import torch
from torch.nn.utils.rnn import pad_sequence
image_1 = torch.ones(25, 30, 50)
image_2 = torch.ones(32, 50, 30)
image_3 = torch.ones(29, 31, 50)
images = pad_sequence([image_1, image_2, image_3])
# RuntimeError: The size of tensor a (50) must match the size of tensor b (30) at non-singleton dimension 2
print(images.size())
It won't work!
But anyways, since you're working with images, I suggest you to use the Pad transformation from torchvision. It works the same as the pad_sequence function but with more options. Just follow the doc.