Create lmdb for multi label classification - deep-learning

Question

I am trying to create 2 lmdbs. One for my images one for my labels. I want to determine the angle of a picture, to do so I am trying to estimate the horizontal and the vertical angle. I have classes like: 0-10 degree horizontal 10-20 degree horiozntal and so on. And the same for vertical angles. Now I do not know how to create the label db as in how the labels have to be formatted in the lmdb.
I have a .txt list file with: /path/pic.png 1 32 entries where 1 means 10-20 degree and 32 means 320-330 degrees.
My code looks like this:
for line in fileinput.input(data):
entries = re.split(' ', line.strip())
images.append(entries[0])
labels.append(entries[1:])
....
for in_idx, (image, label) in enumerate(inputs):
im = cv2.imread(image)
im = im[:,:,::-1]
im = im.transpose((2,0,1))
im_dat = caffe.io.array_to_datum(im)
print im_dat
images_txn.put('{:0>10d}'.format(in_idx), im_dat.SerializeToString())
label = np.array(label).astype(int).reshape(1, 1, len(label))
label_dat = caffe.io.array_to_datum(label)
labels_txn.put('{:0>10d}'.format(in_idx), label_dat.SerializeToString())
This however seems not to be correct since I got following error when trying to train the network:
Check failed: outer_num_ * inner_num_ == bottom[1]->count() (10 vs. 20) Number of labels must match number of predictions; e.g., if softmax axis == 1 and prediction shape is (N, C, H, W), label count (number of labels) must be NHW, with integer values in {0, 1, ..., C-1}.
My data layer looks like this:
name: "LearnAngle"
layer {
name: "data"
type: "Data"
top: "images"
include {
phase: TRAIN
}
transform_param {
mirror: true
crop_size: 256
mean_file: "/path/mean_train.binaryproto"
}
data_param {
source: "path/train2_images_lmdb"
batch_size: 10
backend: LMDB
}
}
layer {
name: "data_label"
type: "Data"
top: "labels"
include {
phase: TRAIN
}
data_param {
source: "path/train2_labels_lmdb"
batch_size: 10
backend: LMDB
}
}
My last layers looks like
layer {
name: "fc8"
type: "InnerProduct"
bottom: "fc7"
top: "fc8"
param {
lr_mult: 1
decay_mult: 1
}
param {
lr_mult: 2
decay_mult: 0
}
inner_product_param {
num_output: 36
weight_filler {
type: "gaussian"
std: 0.01
}
bias_filler {
type: "constant"
value: 0
}
}
}
layer {
name: "loss"
type: "SoftmaxWithLoss"
bottom: "fc8"
bottom: "labels"
top: "loss"
}

Answer 1

The problem is: your top: "labels" in the 2nd data layer contains 2 kind of labels for the horizontal and vertical angle, while you just used 1 SoftmaxWithLoss layer for the labels.
In fact, to train 2 classification tasks in one network, you can create 2 lmdb databases for the 2 tasks' labels respectivly, and use 2 data layers to parse them to 2 SoftmaxWithLoss layers. Like what's below:
Code for creating lmdb for 2-task classifications:
for line in fileinput.input(data):
entries = re.split(' ', line.strip())
images.append(entries[0])
horizontal_labels.append(entries[1])
vertical_labels.append(entries[2])
...
for in_idx, (image, label) in enumerate(inputs):
im = cv2.imread(image)
im = im[:,:,::-1]
im = im.transpose((2,0,1))
im_dat = caffe.io.array_to_datum(im)
print im_dat
images_txn.put('{:0>10d}'.format(in_idx), im_dat.SerializeToString())
horizontal_label = [label[0]]
vertical_label = [label[1]]
horizontal_label = np.array(horizontal_label).astype(int).reshape(1, 1, 1)
vertical_label = np.array(vertical_label).astype(int).reshape(1, 1, 1)
horizontal_label_dat = caffe.io.array_to_datum(horizontal_label)
vertical_label_dat = caffe.io.array_to_datum(vertical_label)
horizontal_labels_txn.put('{:0>10d}'.format(in_idx), horizontal_label_dat.SerializeToString())
vertical_labels_txn.put('{:0>10d}'.format(in_idx), vertical_label_dat.SerializeToString())
train_val.prototxt:
name: "LearnAngle"
layer {
name: "data"
type: "Data"
top: "images"
include {
phase: TRAIN
}
...
}
layer {
name: "horizontal_label"
type: "Data"
top: "horizontal_label"
include {
phase: TRAIN
}
data_param {
source: "path/horizontal_labels_lmdb" #created using above python code
...
}
}
layer {
name: "vertical_label"
type: "Data"
top: "vertical_label"
include {
phase: TRAIN
}
data_param {
source: "path/vertical_labels_lmdb" #created using above python code
...
}
}
... #follow layers
# branch for horizontal label classification
layer {
name: "fc_horizon"
type: "InnerProduct"
bottom: "fc7"
top: "fc_horizon"
...
inner_product_param {
num_output: 36
...
}
}
layer {
name: "loss_horizon"
type: "SoftmaxWithLoss"
bottom: "fc_horizon"
bottom: "horizontal_label"
top: "loss_horizon"
}
# branch for vertical label classification
layer {
name: "fc_vertical"
type: "InnerProduct"
bottom: "fc7"
top: "fc_vertical"
...
inner_product_param {
num_output: 36
...
}
}
layer {
name: "loss_vertical"
type: "SoftmaxWithLoss"
bottom: "fc_vertical"
bottom: "vertical_label"
top: "loss_vertical"
}