I have a problem to train for FCN with caffe.
I prepared my images(original image data, segmented image data).
eg).jpg.
Then, I want to convert my data to lmdb using convert_imageset.exe, but its format is image(array)_label(int). But my data is image(array)_label(array).
How to convert own images for FCN?
Edit convert_imageset.exe sources to handle your desired format - it's in caffe/tools/convert_imageset.cpp
Related
I want to verify that the predicted results from the exported file are consistent with those predicted directly.
I use the output Python file with the model description of catclassifier to predict result:
But the result which is predicted directly is 2.175615211102761. It is verified that this is true for multiple data. I want to know why and how to solve it.
float_sample and cat_sample look like
Supplementary question: the results predicted by using the model file described in Python language provided by the catboost tutorial are different from those predicted directly by the model
I have an LMDB train data file for the VPGNet CNN model pre- trained on Caltech Lane data set.
I would like to test it on new data set different from the training data set. How to create LMDB for the new test data.
Do I need to modify prototxt files for testing with pre-trained net. For testing do I need a prototxt file or there is a specific command.
Thanks
Lightning Memory-Mapped Databases (LMDB) formats can be efficiently process as input data.
We create the native format (lmdb) for training and validating the model.
Once the trained model converged and the loss is calculated on training and validation data,
we use separate data (Unknown data/ Data which is not used for training) for inference the model.
In case if we are running a classification inference on a single image or set of images,
We need not convert those in to lmdb. Instead we can just run a forward pass on the stacked topology with the image/images converted into the desired format (numpy arrays).
For More info :
https://software.intel.com/en-us/articles/training-and-deploying-deep-learning-networks-with-caffe-optimized-for-intel-architecture
I have category information in the training input data, I am wondering what's the best way to normalize it.
The category information is like "city", "gender" and etc.
I'd like to use Keras to handle the process.
Scikitlearn has a preprocessing library with functions to normalize or scale your data.
This video gives an example for how to preprocess data that will be used for training a model with Keras. The preprocessing here is done with the library mentioned above.
As shown in the video, with the use of Scikitlearn's MinMaxScaler class, you can specify a range that you want your data to be transformed into, and then fit your data to that range using the MinMaxScaler.fit_transform() function.
I am using #shai's code to convert data to hdf5, but after converting the size of data is too large, larger than limit size of caffe (2GB)
so , my question is how we should split the data?
we only need convert data separately depends on what we want ?
It would be helpful to have more information to give a better answer. Please give the actual error that Caffe gives you. Also, what exactly is larger than 2GB? Is it the txt file?
HDF5 is not very efficient. I would recommend either using the database layer (http://caffe.help/manual/layers/data.html) or writing your own customized Python layer (http://caffe.help/manual/layers/python.html).
I want to run some images through a neural network, and I want to create a .csv file for the data. How can I create a csv that will represent the images and keep each image separate?
One way to approach is to use numpy to convert image to array, which can then be converted to a CSV file or simply a comma separated list.
The csv data can be manipulated or original image can be retrieved when needed.
Here is a basic code that demonstrates above concept.
import Image
import numpy as np
#Function to convert image to array or list
def loadImage (inFileName, outType ) :
img = Image.open( inFileName )
img.load()
data = np.asarray( img, dtype="int32" )
if outType == "anArray":
return data
if outType == "aList":
return list(data)
#Load image to array
myArray1 = loadImage("bug.png", "anArray")
#Load image to a list
myList1 = loadImage("bug.png", "aList")
You can encode your image in Base64 and still use CSV, since commas are not part of characters in Base64.
See: Best way to separate two base64 strings
If possible, create a storage location just for images. If your images have unique filenames, then all you need to track is the filename. If they do not have a unique filename, you can assign one using a timestamp+randomizer function to name the photo. Once named, it must be stored in the proper location so that all you need is the filename in order to reference the appropriate image.
Due to size constraints, I would not recommend storing the actual images in the csv.
Cheers!
I guess that depends a lot on what algorithm and what implementation you select. It is not even clear that CSV is the correct choice.
For your stated requirements, Netpbm format comes to mind; if you want to have one line per image, just squish all the numbers into one line. Note that the naive neural network will ignore the topology of the image, you'd need a bit advanced setup to include it.