I have an LMDB train data file for the VPGNet CNN model pre- trained on Caltech Lane data set.
I would like to test it on new data set different from the training data set. How to create LMDB for the new test data.
Do I need to modify prototxt files for testing with pre-trained net. For testing do I need a prototxt file or there is a specific command.
Thanks
Lightning Memory-Mapped Databases (LMDB) formats can be efficiently process as input data.
We create the native format (lmdb) for training and validating the model.
Once the trained model converged and the loss is calculated on training and validation data,
we use separate data (Unknown data/ Data which is not used for training) for inference the model.
In case if we are running a classification inference on a single image or set of images,
We need not convert those in to lmdb. Instead we can just run a forward pass on the stacked topology with the image/images converted into the desired format (numpy arrays).
For More info :
https://software.intel.com/en-us/articles/training-and-deploying-deep-learning-networks-with-caffe-optimized-for-intel-architecture
Related
I want to verify that the predicted results from the exported file are consistent with those predicted directly.
I use the output Python file with the model description of catclassifier to predict result:
But the result which is predicted directly is 2.175615211102761. It is verified that this is true for multiple data. I want to know why and how to solve it.
float_sample and cat_sample look like
Supplementary question: the results predicted by using the model file described in Python language provided by the catboost tutorial are different from those predicted directly by the model
I have category information in the training input data, I am wondering what's the best way to normalize it.
The category information is like "city", "gender" and etc.
I'd like to use Keras to handle the process.
Scikitlearn has a preprocessing library with functions to normalize or scale your data.
This video gives an example for how to preprocess data that will be used for training a model with Keras. The preprocessing here is done with the library mentioned above.
As shown in the video, with the use of Scikitlearn's MinMaxScaler class, you can specify a range that you want your data to be transformed into, and then fit your data to that range using the MinMaxScaler.fit_transform() function.
I have a problem to train for FCN with caffe.
I prepared my images(original image data, segmented image data).
eg).jpg.
Then, I want to convert my data to lmdb using convert_imageset.exe, but its format is image(array)_label(int). But my data is image(array)_label(array).
How to convert own images for FCN?
Edit convert_imageset.exe sources to handle your desired format - it's in caffe/tools/convert_imageset.cpp
I want to use the dumped weights and model architecture in other framework for testing.
I know that:
model.get_config() can give the configuration of the model
model.to_json returns a representation of the model as a JSON string, but that the representation does not include the weights, only the architecture
model.save_weights(filepath) saves the weights of the model as a HDF5 file
I want to save the architecture as well as weights in a json file.
Keras does not have any built-in way to export the weights to JSON.
Solution 1:
For now you can easily do it by iterating over the weights and saving it to the JSON file.
weights_list = model.get_weights()
will return a list of all weight tensors in the model, as Numpy arrays.
Then, all you have to do next is to iterate over this list and write to the file:
for i, weights in enumerate(weights_list):
writeJSON(weights)
Solution 2:
import json
weights_list = model.get_weights()
print json.dumps(weights_list.tolist())
I am using #shai's code to convert data to hdf5, but after converting the size of data is too large, larger than limit size of caffe (2GB)
so , my question is how we should split the data?
we only need convert data separately depends on what we want ?
It would be helpful to have more information to give a better answer. Please give the actual error that Caffe gives you. Also, what exactly is larger than 2GB? Is it the txt file?
HDF5 is not very efficient. I would recommend either using the database layer (http://caffe.help/manual/layers/data.html) or writing your own customized Python layer (http://caffe.help/manual/layers/python.html).