I am quite new in caffe and in deep learning. I want to train my model using the dataset that downloaded from here.
My train data has hdf5 format. It also has following parameters.
{
"debug": false,
"git_revision": "60c477dae59f3d1378568e2ebea054a135683e2f",
"height": 128,
"no_train_mirrors": false,
"output_dir": "/tmp/parse27k_crops_64x128",
"output_mode": "hdf5",
"padding": 32,
"padding_mode": "edge",
"parse_path": "/fast_work/sudowe/parse27k",
"single_threaded": false,
"verbose": false,
"width": 64
}
I have following data layer in my train model.
layer {
name: "data"
type: "HDF5Data"
top: "data"
top: "label"
hdf5_data_param {
source: "path_to_caffe/caffe/examples/hdf5_classification/data/train.txt"
batch_size: 10
}
include {
phase: TRAIN
}
}
I am getting following error message when i am trying to train my train.hdf5 data which is mentioned in train.txt file.
I1031 11:52:10.185920 8670 layer_factory.hpp:77] Creating layer data
I1031 11:52:10.185933 8670 net.cpp:100] Creating Layer data
I1031 11:52:10.185940 8670 net.cpp:408] data -> data
I1031 11:52:10.185957 8670 net.cpp:408] data -> label
I1031 11:52:10.185971 8670 hdf5_data_layer.cpp:79] Loading list of HDF5 filenames from: path_to_caffe/caffe/examples/hdf5_classification/data/train.txt
I1031 11:52:10.186003 8670 hdf5_data_layer.cpp:93] Number of HDF5 files: 2
F1031 11:52:10.186825 8670 hdf5.cpp:14] Check failed: H5LTfind_dataset(file_id, dataset_name_) Failed to find HDF5 dataset data
*** Check failure stack trace: ***
# 0x7f231a6a1daa (unknown)
# 0x7f231a6a1ce4 (unknown)
# 0x7f231a6a16e6 (unknown)
# 0x7f231a6a4687 (unknown)
# 0x7f231acca607 caffe::hdf5_load_nd_dataset_helper<>()
# 0x7f231acc93d5 caffe::hdf5_load_nd_dataset<>()
# 0x7f231ad5172e caffe::HDF5DataLayer<>::LoadHDF5FileData()
# 0x7f231ad50548 caffe::HDF5DataLayer<>::LayerSetUp()
# 0x7f231acaf3ac caffe::Net<>::Init()
# 0x7f231acb0235 caffe::Net<>::Net()
# 0x7f231ae0332a caffe::Solver<>::InitTrainNet()
# 0x7f231ae0442c caffe::Solver<>::Init()
# 0x7f231ae0475a caffe::Solver<>::Solver()
# 0x7f231adf8453 caffe::Creator_SGDSolver<>()
# 0x40f0fe caffe::SolverRegistry<>::CreateSolver()
# 0x408134 train()
# 0x405b3c main
# 0x7f23196adf45 (unknown)
# 0x4063ab (unknown)
# (nil) (unknown)
Any kind of help or suggestion will be really appreciated.
In caffe input data layer output blob can be only named after the names of datasets inside of the .hdf5 file.
My dataset has following structure
crops Dataset {27482, 3, 128, 192}
labels Dataset {27482, 12}
mean Dataset {3, 128, 192}
pids Dataset {27482}
By the help of #Shai I solve it like this :
layer {
name: "data"
type: "HDF5Data"
top: "crops"
top: "labels"
include {
phase: TRAIN
}
hdf5_data_param {
source: "path_to_caffe/examples/hdf5_classification/data/train.txt"
batch_size: 64
}
}
Related
I am trying to run UNet on my data, which is grayscale images with 256x256 resolution. UNet is downsampling the image to 1-by-5-by-84-by-84 (5 is number of classes). and I am getting the following error:
0501 02:16:17.345309 2433 net.cpp:400] loss -> loss
I0501 02:16:17.345317 2433 layer_factory.hpp:77] Creating layer loss
F0501 02:16:17.345377 2433 softmax_loss_layer.cpp:47] Check failed: outer_num_ * inner_num_ == bottom[1]->count() (7056 vs. 65536) Number of labels must match number of predictions; e.g., if softmax axis == 1 and prediction shape is (N, C, H, W), label count (number of labels) must be N*H*W, with integer values in {0, 1, ..., C-1}.
*** Check failure stack trace: ***
# 0x7f7d2c9575cd google::LogMessage::Fail()
# 0x7f7d2c959433 google::LogMessage::SendToLog()
# 0x7f7d2c95715b google::LogMessage::Flush()
# 0x7f7d2c959e1e google::LogMessageFatal::~LogMessageFatal()
# 0x7f7d2d02d4be caffe::SoftmaxWithLossLayer<>::Reshape()
# 0x7f7d2d0c61df caffe::Net<>::Init()
# 0x7f7d2d0c7a91 caffe::Net<>::Net()
# 0x7f7d2d0e1a4a caffe::Solver<>::InitTrainNet()
# 0x7f7d2d0e2db7 caffe::Solver<>::Init()
# 0x7f7d2d0e315a caffe::Solver<>::Solver()
# 0x7f7d2cf7b9f3 caffe::Creator_SGDSolver<>()
# 0x40a6d8 train()
# 0x4075a8 main
# 0x7f7d2b40b830 __libc_start_main
# 0x407d19 _start
# (nil) (unknown)
Could someone please let me know how should I set the padding values to get the exactly the input size in the output prediction? I do not know how and which layers should I change.
I am applying CNNs for semantic segmentation. I am hoping someone here can recommend. What I am doing now is that I scaled ground truth images. I have 5 classes that I scaled them to (0-1) in Data Layer based on this:
transform_param {
scale: 0.00390625
}
I am wondering whether I am wrong or right? Is this scaling value correct?
whenever I am not adding `
it is showing the following error:
[...]
I0510 23:21:28.086776 9072 solver.cpp:397] Test net output #0: accuracy = 0
I0510 23:21:28.086812 9072 solver.cpp:397] Test net output #1: loss = 1.9416 (* 1 = 1.9416 loss)
F0510 23:21:28.150539 9072 math_functions.cu:141] Check failed: status == CUBLAS_STATUS_SUCCESS (11 vs. 0) CUBLAS_STATUS_MAPPING_ERROR
*** Check failure stack trace: ***
# 0x7fb9d4e7f5cd google::LogMessage::Fail()
# 0x7fb9d4e81433 google::LogMessage::SendToLog()
# 0x7fb9d4e7f15b google::LogMessage::Flush()
# 0x7fb9d4e81e1e google::LogMessageFatal::~LogMessageFatal()
# 0x7fb9d56665ea caffe::caffe_gpu_asum<>()
# 0x7fb9d5633a38 caffe::SoftmaxWithLossLayer<>::Forward_gpu()
# 0x7fb9d54bde41 caffe::Net<>::ForwardFromTo()
# 0x7fb9d54bdf47 caffe::Net<>::Forward()
# 0x7fb9d54e8d28 caffe::Solver<>::Step()
# 0x7fb9d54e98ca caffe::Solver<>::Solve()
# 0x40acd4 train()
# 0x407418 main
# 0x7fb9d360f830 __libc_start_main
# 0x407ce9 _start
# (nil) (unknown)
When I am trying to train FCN32 for semantic segmentation on my own data, I am getting this error:
I0106 12:57:53.273977 19825 net.cpp:100] Creating Layer upscore_sign
I0106 12:57:53.273982 19825 net.cpp:434] upscore_sign <- score_fr_sign
I0106 12:57:53.274001 19825 net.cpp:408] upscore_sign -> upscore_sign
F0106 12:57:53.274119 19825 base_conv_layer.cpp:122] Check failed: channels_ % group_ == 0 (1 vs. 0)
*** Check failure stack trace: ***
# 0x7f2602e525cd google::LogMessage::Fail()
# 0x7f2602e54433 google::LogMessage::SendToLog()
# 0x7f2602e5215b google::LogMessage::Flush()
# 0x7f2602e54e1e google::LogMessageFatal::~LogMessageFatal()
# 0x7f260350701b caffe::BaseConvolutionLayer<>::LayerSetUp()
# 0x7f26033ee557 caffe::Net<>::Init()
# 0x7f26033efde1 caffe::Net<>::Net()
# 0x7f26033c5d4a caffe::Solver<>::InitTrainNet()
# 0x7f26033c7157 caffe::Solver<>::Init()
# 0x7f26033c74fa caffe::Solver<>::Solver()
# 0x7f2603400353 caffe::Creator_SGDSolver<>()
# 0x40c07a train()
# 0x408748 main
# 0x7f26014f3830 __libc_start_main
# 0x409019 _start
# (nil) (unknown)
I have not included the creation of previous layers. But it seems net creates previous layers successfully, and it reaches to the Creating Layer upscore_sign, the error comes. I changed solver as follows:
net: "train_val.prototxt"
#test_net: "val.prototxt"
test_iter: 200 #3000 #5105
# make test net, but don't invoke it from the solver itself
test_interval: 1000
display: 20
average_loss: 20
lr_policy: "step" #"fixed"
# lr for unnormalized softmax
base_lr: 1e-10
# high momentum
momentum: 0.99
# no gradient accumulation
iter_size: 1
max_iter: 300000
weight_decay: 0.0005
snapshot: 2000 #10000
snapshot_prefix: "snapshot/FCN32s_train"
test_initialization: false
solver_mode: GPU #+
and I changed the number of outputs from 60 to 5 (based on the number of classes in my data): convolution_param {num_output: 5 }
Can someone suggest any solution or idea about this? What I have set/changed wrongly? What/where is my mistake?
Your help is appreciated.
Check failed: channels_ % group_ == 0 (1 vs. 0)
This line really matters! You should check your num_output and group and find out if channels_ % group_ == 0.
I know this may be better asked in caffe user group but I cannot access the user group and don't know where to pose the question as I'm not sure whether this needs to be raised as an issue in git. In any case, what I'm doing is this:
I have a set of grayscale images that I want to use to train a CNN using caffe. I'm using a modified version of the provided caffenet model definitions with minor modifications (ie: channel = 1 instead of 3 as I have grayscale images). So far, I used the imagenet provided mean image to train the CNN and it trained and generated results. Now I wanted to compute the image mean of my own train/test dataset and use that as the mean image so I used the file in build/tools/ to do this. It needed the images to be in lmdb so I first converted images to lmdb using convert_imageset and then used compute_mean to compute the mean. I ensured that I use --gray flag when using convert_imageset as my images are grayscale. When I rerun caffe, I get the following error. From what I can understand, it's a channel mismatch but I have no idea how to fix this. Any help on this is very much appreciated.
I0829 20:41:50.429733 17065 layer_factory.hpp:77] Creating layer data
I0829 20:41:50.429764 17065 net.cpp:100] Creating Layer data
I0829 20:41:50.429769 17065 net.cpp:408] data -> data
I0829 20:41:50.429790 17065 net.cpp:408] data -> label
I0829 20:41:50.429805 17065 data_transformer.cpp:25] Loading mean file from: data/flickr_style/train_mean.binaryproto
I0829 20:41:50.438251 17065 image_data_layer.cpp:38] Opening file data/flickr_style/train.txt
I0829 20:41:50.446666 17065 image_data_layer.cpp:58] A total of 31740 images.
I0829 20:41:50.451941 17065 image_data_layer.cpp:85] output data size: 10,3,227,227
I0829 20:41:50.459661 17065 net.cpp:150] Setting up data
I0829 20:41:50.459692 17065 net.cpp:157] Top shape: 10 3 227 227 (1545870)
I0829 20:41:50.459697 17065 net.cpp:157] Top shape: 10 (10)
I0829 20:41:50.459699 17065 net.cpp:165] Memory required for data: 6183520
I0829 20:41:50.459707 17065 layer_factory.hpp:77] Creating layer conv1
I0829 20:41:50.459728 17065 net.cpp:100] Creating Layer conv1
I0829 20:41:50.459733 17065 net.cpp:434] conv1 <- data
I0829 20:41:50.459744 17065 net.cpp:408] conv1 -> conv1
F0829 20:41:50.463794 17106 data_transformer.cpp:257] Check failed: img_channels == data_mean_.channels() (3 vs. 1)
*** Check failure stack trace: ***
# 0x7f0712106daa (unknown)
# 0x7f0712106ce4 (unknown)
# 0x7f07121066e6 (unknown)
# 0x7f0712109687 (unknown)
# 0x7f071287d6cd caffe::DataTransformer<>::Transform()
# 0x7f07127fde60 caffe::ImageDataLayer<>::load_batch()
# 0x7f0712839539 caffe::BasePrefetchingDataLayer<>::InternalThreadEntry()
# 0x7f0712886020 caffe::InternalThread::entry()
# 0x7f070a762a4a (unknown)
# 0x7f070603e184 start_thread
# 0x7f07111eb37d (unknown)
# (nil) (unknown)
I have the following in train_val.prototxt
name: "FlickrStyleCaffeNet"
layer {
name: "data"
type: "ImageData"
top: "data"
top: "label"
include {
phase: TRAIN
}
transform_param {
mirror: true
crop_size: 227
mean_file: "data/ilsvrc12/imagenet_mean.binaryproto"
}
image_data_param {
source: "data/flickr_style/mri_train.txt"
batch_size: 10
new_height: 256
new_width: 256
}
}
and this in deploy.prototxt
name: "FlickrStyleCaffeNet"
layer {
name: "data"
type: "Input"
top: "data"
input_param { shape: { dim: 10 dim: 3 dim: 227 dim: 227 } }
}
You (or the interface) have failed to adjust the input for gray scale. Gray has only 1 input channel (value); the model expects 3 channels (RGB). That 3 in the layer's top shape should be 1.
Look in your *.prototxt files for something like this near the top (input layer):
shape {
dim: 10
dim: 3
dim: 227
dim: 227
}
These dimensions are batch_size, channels, rows, and columns. Wherever you have something on this order (there should be only one, and only in input files), change the 3 to 1.
I figured out how to do this. In the train_val.prototxt, there's an image_data_param section under data layer. I had to add is_color: false in it and that fixed the issue.
Thanks everyone for comments and replies, appreciate it.
I am trying to implement a pixel-wise binary classification for images using caffe. For each image having dimension 3x256x256, I have a 256x256 label array in which each entry is marked as either 0 or 1. Also, when I read my HDF5 file using the below code,
dirname = "examples/hdf5_classification/data"
f = h5py.File(os.path.join(dirname, 'train.h5'), "r")
ks = f.keys()
data = np.array(f[ks[0]])
label = np.array(f[ks[1]])
print "Data dimension from HDF5", np.shape(data)
print "Label dimension from HDF5", np.shape(label)
I get the data and label dimension as
Data dimension from HDF5 (402, 3, 256, 256)
Label dimension from HDF5 (402, 256, 256)
I am trying to feed this data into the given hdf5 classification network and while training, I have the following output(using the default solver, but in GPU mode).
!cd /home/unni/MTPMain/caffe-master/ && ./build/tools/caffe train -solver examples/hdf5_classification/solver.prototxt
gives
I1119 01:29:02.222512 11910 caffe.cpp:184] Using GPUs 0
I1119 01:29:02.509752 11910 solver.cpp:47] Initializing solver from parameters:
train_net: "examples/hdf5_classification/train_val.prototxt"
test_net: "examples/hdf5_classification/train_val.prototxt"
test_iter: 250
test_interval: 1000
base_lr: 0.01
display: 1000
max_iter: 10000
lr_policy: "step"
gamma: 0.1
momentum: 0.9
weight_decay: 0.0005
stepsize: 5000
snapshot: 10000
snapshot_prefix: "examples/hdf5_classification/data/train"
solver_mode: GPU
device_id: 0
I1119 01:29:02.519805 11910 solver.cpp:80] Creating training net from train_net file: examples/hdf5_classification/train_val.prototxt
I1119 01:29:02.520031 11910 net.cpp:322] The NetState phase (0) differed from the phase (1) specified by a rule in layer data
I1119 01:29:02.520053 11910 net.cpp:322] The NetState phase (0) differed from the phase (1) specified by a rule in layer accuracy
I1119 01:29:02.520104 11910 net.cpp:49] Initializing net from parameters:
name: "LogisticRegressionNet"
state {
phase: TRAIN
}
layer {
name: "data"
type: "HDF5Data"
top: "data"
top: "label"
include {
phase: TRAIN
}
hdf5_data_param {
source: "examples/hdf5_classification/data/train.txt"
batch_size: 10
}
}
layer {
name: "fc1"
type: "InnerProduct"
bottom: "data"
top: "fc1"
param {
lr_mult: 1
decay_mult: 1
}
param {
lr_mult: 2
decay_mult: 0
}
inner_product_param {
num_output: 2
weight_filler {
type: "xavier"
}
bias_filler {
type: "constant"
value: 0
}
}
}
layer {
name: "loss"
type: "SoftmaxWithLoss"
bottom: "fc1"
bottom: "label"
top: "loss"
}
I1119 01:29:02.520256 11910 layer_factory.hpp:76] Creating layer data
I1119 01:29:02.520277 11910 net.cpp:106] Creating Layer data
I1119 01:29:02.520290 11910 net.cpp:411] data -> data
I1119 01:29:02.520331 11910 net.cpp:411] data -> label
I1119 01:29:02.520352 11910 hdf5_data_layer.cpp:80] Loading list of HDF5 filenames from: examples/hdf5_classification/data/train.txt
I1119 01:29:02.529341 11910 hdf5_data_layer.cpp:94] Number of HDF5 files: 1
I1119 01:29:02.542645 11910 hdf5.cpp:32] Datatype class: H5T_FLOAT
I1119 01:29:10.601307 11910 net.cpp:150] Setting up data
I1119 01:29:10.612926 11910 net.cpp:157] Top shape: 10 3 256 256 (1966080)
I1119 01:29:10.612963 11910 net.cpp:157] Top shape: 10 256 256 (655360)
I1119 01:29:10.612969 11910 net.cpp:165] Memory required for data: 10485760
I1119 01:29:10.612983 11910 layer_factory.hpp:76] Creating layer fc1
I1119 01:29:10.624948 11910 net.cpp:106] Creating Layer fc1
I1119 01:29:10.625015 11910 net.cpp:454] fc1 <- data
I1119 01:29:10.625039 11910 net.cpp:411] fc1 -> fc1
I1119 01:29:10.645814 11910 net.cpp:150] Setting up fc1
I1119 01:29:10.645864 11910 net.cpp:157] Top shape: 10 2 (20)
I1119 01:29:10.645875 11910 net.cpp:165] Memory required for data: 10485840
I1119 01:29:10.645912 11910 layer_factory.hpp:76] Creating layer loss
I1119 01:29:10.657094 11910 net.cpp:106] Creating Layer loss
I1119 01:29:10.657133 11910 net.cpp:454] loss <- fc1
I1119 01:29:10.657147 11910 net.cpp:454] loss <- label
I1119 01:29:10.657163 11910 net.cpp:411] loss -> loss
I1119 01:29:10.657189 11910 layer_factory.hpp:76] Creating layer loss
F1119 01:29:14.883095 11910 softmax_loss_layer.cpp:42] Check failed: outer_num_ * inner_num_ == bottom[1]->count() (10 vs. 655360) Number of labels must match number of predictions; e.g., if softmax axis == 1 and prediction shape is (N, C, H, W), label count (number of labels) must be N*H*W, with integer values in {0, 1, ..., C-1}.
*** Check failure stack trace: ***
# 0x7f0652e1adaa (unknown)
# 0x7f0652e1ace4 (unknown)
# 0x7f0652e1a6e6 (unknown)
# 0x7f0652e1d687 (unknown)
# 0x7f0653494219 caffe::SoftmaxWithLossLayer<>::Reshape()
# 0x7f065353f50f caffe::Net<>::Init()
# 0x7f0653541f05 caffe::Net<>::Net()
# 0x7f06535776cf caffe::Solver<>::InitTrainNet()
# 0x7f0653577beb caffe::Solver<>::Init()
# 0x7f0653578007 caffe::Solver<>::Solver()
# 0x7f06535278b3 caffe::Creator_SGDSolver<>()
# 0x410831 caffe::SolverRegistry<>::CreateSolver()
# 0x40a16b train()
# 0x406908 main
# 0x7f065232cec5 (unknown)
# 0x406e28 (unknown)
# (nil) (unknown)
Aborted
Basically the error is
softmax_loss_layer.cpp:42] Check failed:
outer_num_ * inner_num_ == bottom[1]->count() (10 vs. 655360)
Number of labels must match number of predictions;
e.g., if softmax axis == 1 and prediction shape is (N, C, H, W),
label count (number of labels) must be N*H*W,
with integer values in {0, 1, ..., C-1}.
I am not able to understand why the number of labels expected is just same as my batch size. How exactly should I tackle this problem ? Is this a problem with my labeling method ?
Your problem is that "SoftmaxWithLoss" layer tries to compare a prediction vector of 2 elements per input image to a label of size 256-by-256 per image.
This makes no sense.
Root cause of the error: I guess what you tired to do is to have a binary classifier applied to each pixel in the image. To that end you defined "fc1" as an "InnerProduct" layer with num_output=2. However, the way caffe sees this is that you have a single binary classifier applied to the entire image. Thus caffe gives you a single binary prediction to the entire image.
How to solve: when working on pixel-wise predictions you no longer need to use "InnerProduct" layers and you have a "fully convolutional net". If you replace "fc1" with a conv layer (for instance a kernel that examine the 5-by-5 environment of each pixel and makes a decision according to this patch):
layer {
name: "bin_class"
type: "Convolution"
bottom: "data"
top: "bin_class"
convolution_param {
num_output: 2 # binary class output
kernel_size: 5 # 5-by-5 patch for prediciton
pad: 2 # make sure spatial output size equals size of label
}
}
Now applying "SoftmaxWithLoss" to bottom: bin_class and bottom: label should work.