I'm using caffe to train a model. I am sure I have connected the data layer with the train.txt in the source of image_data_param.But when I try ./train.sh. It always prompt: can not find image.
Ubuntu 18.04 openCV3 python2
layer {
name: "data"
type: "ImageLabelmapData"
top: "data"
top: "label"
include {
phase: TRAIN
}
transform_param {
mirror: false
mean_value: 104.00699
mean_value: 116.66877
mean_value: 122.67892
}
image_data_param {
root_folder: "/home/yogang/Desktop/rawdata/train/"
source: "/home/yogang/Desktop/rawdata/train.txt"
batch_size: 1
shuffle: true
new_height:0
new_width: 0
}
}
I0828 19:29:13.834946 14079 layer_factory.hpp:77] Creating layer data
I0828 19:29:13.835011 14079 net.cpp:101] Creating Layer data
I0828 19:29:13.835031 14079 net.cpp:409] data -> data
I0828 19:29:13.835059 14079 net.cpp:409] data -> label
I0828 19:29:13.835124 14079 image_labelmap_data_layer.cpp:42] Opening file /home/yogang/Desktop/rawdata/train.txt
I0828 19:29:13.835505 14079 image_labelmap_data_layer.cpp:52] Shuffling data
I0828 19:29:13.835677 14079 image_labelmap_data_layer.cpp:57] A total of 242 images.
E0828 19:29:13.836748 14079 io.cpp:80] Could not open or find file /home/yogang/Desktop/rawdata/train//home/yogang/Desktop/rawdata/train/satellite144.jpg
E0828 19:29:13.836797 14079 io.cpp:80] Could not open or find file /home/yogang/Desktop/rawdata/train//home/yogang/Desktop/rawdata/train/400.jpg
F0828 19:29:13.836818 14079 image_labelmap_data_layer.cpp:86] Check failed: cv_img.data Could not load /home/yogang/Desktop/rawdata/train/satellite144.jpg
*** Check failure stack trace: ***
./train.sh: line 8: 14079 Aborted (core dumped) ./solve.py
i think image path is wrong
check your image path
Related
This is my json log file. I'm trying to store the file to my elastic-Search through my logstash.
{"message":"IM: Orchestration","level":"info"}
{"message":"Investment Management","level":"info"}
Here is my filebeat.yml
filebeat.inputs:
- type: log
enabled: true
paths:
- D:/Development_Avecto/test-log/tn-logs/im.log
json.keys_under_root: true
json.add_error_key: true
processors:
- decode_json_fields:
fields: ["message"]
output.logstash:
hosts: ["localhost:5044"]
input{
beats {
port => "5044"
}
}
filter {
json {
source => "message"
}
}
output{
elasticsearch{
hosts => ["localhost:9200"]
index => "data"
}
}
No able to view out put in elasticserach. Not able to find whats the error.
filebeat log
2019-06-18T11:30:03.448+0530 INFO registrar/registrar.go:134 Loading registrar data from D:\Development_Avecto\filebeat-6.6.2-windows-x86_64\data\registry
2019-06-18T11:30:03.448+0530 INFO registrar/registrar.go:141 States Loaded from registrar: 10
2019-06-18T11:30:03.448+0530 WARN beater/filebeat.go:367 Filebeat is unable to load the Ingest Node pipelines for the configured modules because the Elasticsearch output is not configured/enabled. If you have already loaded the Ingest Node pipelines or are using Logstash pipelines, you can ignore this warning.
2019-06-18T11:30:03.448+0530 INFO crawler/crawler.go:72 Loading Inputs: 1
2019-06-18T11:30:03.448+0530 INFO log/input.go:138 Configured paths: [D:\Development_Avecto\test-log\tn-logs\im.log]
2019-06-18T11:30:03.448+0530 INFO input/input.go:114 Starting input of type: log; ID: 16965758110699470044
2019-06-18T11:30:03.449+0530 INFO crawler/crawler.go:106 Loading and starting Inputs completed. Enabled inputs: 1
2019-06-18T11:30:34.842+0530 INFO [monitoring] log/log.go:144 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":312,"time":{"ms":312}},"total":{"ticks":390,"time":{"ms":390},"value":390},"user":{"ticks":78,"time":{"ms":78}}},"handles":{"open":213},"info":{"ephemeral_id":"66983518-39e6-461c-886d-a1f99da6631d","uptime":{"ms":30522}},"memstats":{"gc_next":4194304,"memory_alloc":2963720,"memory_total":4359488,"rss":22421504}},"filebeat":{"events":{"added":1,"done":1},"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":0}},"output":{"type":"logstash"},"pipeline":{"clients":1,"events":{"active":0,"filtered":1,"total":1}}},"registrar":{"states":{"current":10,"update":1},"writes":{"success":1,"total":1}},"system":{"cpu":{"cores":4}}}}}
2
https://www.elastic.co/guide/en/ecs-logging/dotnet/master/setup.html
Check step 3 at the bottom of the page for the config you need to put in your filebeat.yaml file:
filebeat.inputs:
- type: log
paths: /path/to/logs.json
json.keys_under_root: true
json.overwrite_keys: true
json.add_error_key: true
json.expand_keys: true
I am testing Faster Rcnn. Installation is fine.
During the installation, I had one issue with cudnn5.1 and I followed the suggestion here and now installation is fine.
Now I test the demo code as
./tools/demo.py
Then I have error as
I1117 09:48:41.011925 12503 net.cpp:51] Initializing net from parameters:
name: "VGG_ILSVRC_16_layers"
state {
phase: TEST
level: 0
}
.
.
.
layer {
name: "cls_prob"
type: "Softmax"
bottom: "cls_score"
top: "cls_prob"
}
I1117 09:48:41.012234 12503 layer_factory.hpp:77] Creating layer input
I1117 09:48:41.012251 12503 net.cpp:84] Creating Layer input
I1117 09:48:41.012259 12503 net.cpp:380] input -> data
I1117 09:48:41.012271 12503 net.cpp:380] input -> im_info
I1117 09:48:41.328574 12503 net.cpp:122] Setting up input
I1117 09:48:41.328608 12503 net.cpp:129] Top shape: 1 3 224 224 (150528)
I1117 09:48:41.328614 12503 net.cpp:129] Top shape: 1 3 (3)
I1117 09:48:41.328618 12503 net.cpp:137] Memory required for data: 602124
I1117 09:48:41.328624 12503 layer_factory.hpp:77] Creating layer conv1_1
I1117 09:48:41.328655 12503 net.cpp:84] Creating Layer conv1_1
I1117 09:48:41.328660 12503 net.cpp:406] conv1_1 <- data
I1117 09:48:41.328670 12503 net.cpp:380] conv1_1 -> conv1_1
F1117 09:48:41.676553 12503 cudnn.hpp:128] Check failed: status == CUDNN_STATUS_SUCCESS (3 vs. 0) CUDNN_STATUS_BAD_PARAM
*** Check failure stack trace: ***
Aborted (core dumped)
What is wrong with my installation for this faster rcnn?
I have cuda8.0 and libcudnn5_5.1.10-1+cuda8.0 is installed on Ubuntu16.04.
I have Qurdo K4200 graphic card.
Now it works for me. Since libcudnn5_5.1 is for CUDA7.5. Can check in cudnn's user guide at GPU and driver requirements. So I changed to cudnnv6.0 for CUDA8.0.
Then you may face the issue of
Check failed: error == cudaSuccess (8 vs. 0) invalid device function
For that you will need to py-faster-rcnn/lib/fast_rcnn/config.py and change
__C.USE_GPU_NMS = True
to
__C.USE_GPU_NMS = False
Then now it works.
When I try to convert a model from Caffe to Core ML model with coremltools, I get the following:
================= Starting Conversion from Caffe to CoreML ======================
Layer 0: Type: 'Data', Name: 'data'. Output(s): 'data', 'label'.
WARNING: Skipping Data Layer 'data' of type 'Data'. It is recommended to use Input layer for deployment.
Layer 1: Type: 'Split', Name: 'label_data_1_split'. Input(s): 'label'. Output(s): 'label_data_1_split_0', 'label_data_1_split_1'.
Layer 2: Type: 'Convolution', Name: 'conv1'. Input(s): 'data'. Output(s): 'conv1'.
Layer 3: Type: 'Slice', Name: 'slice1'. Input(s): 'conv1'. Output(s): 'slice1_1', 'slice1_2'.
Layer 4: Type: 'Eltwise', Name: 'etlwise1'. Input(s): 'slice1_1', 'slice1_2'. Output(s): 'eltwise1'.
Traceback (most recent call last):
File "test.py", line 2, in <module>
coreml_model = coremltools.converters.caffe.convert('_iter_3560000.caffemodel')
File "/Users/zfh/Desktop/face_verification_experiment/model/python27/lib/python2.7/site-packages/coremltools/converters/caffe/_caffe_converter.py", line 142, in convert
predicted_feature_name)
File "/Users/zfh/Desktop/face_verification_experiment/model/python27/lib/python2.7/site-packages/coremltools/converters/caffe/_caffe_converter.py", line 187, in _export
predicted_feature_name
RuntimeError: Unsupported option 'Max' for the parameter 'operation' in layer 'etlwise1' of type 'Elementwise' during caffe conversion.
This is the code I am using:
import coremltools
coreml_model = coremltools.converters.caffe.convert(('_iter_3560000.caffemodel', 'LCNN_deploy.prototxt'))
coreml_model.save('_iter_3560000.mlmodel')
Any ideas what the problem is? Thank you very much!
As the error message says, the problem is that the Max operation in an Eltwise layer is not supported by the coremltools. Core ML only supports a limited number of layers.
However... it seems like maybe you're trying to convert the .prototxt that was used for training (even though the filename is LCNN_deploy.prototxt). Are you sure this is the correct deploy.prototxt?
Recently, I extract the convert tools of caffe2mlmodel from the coremltools, its the c++ implemention.
First of all，you need to know this tool supported caffe layer, defined in caffe.proto(include in caffeconverter directory)
And, then, open the caffe.proto, you can locate at the message LayerParameter, like below:,. You can find the supported caffe layer.
caffe.proto in caffeconverter of coremltools
Finally, if you want the custom caffe layer, just add adapt the caffe.proto , and learn the Core ML mode protobuf specification(https://apple.github.io/coremltools/coremlspecification/#)
I am trying to train and validate a network on Imagenet. The validation process works without any problems (with the pretrained weights). However, when I try to perform the training, there appears an error that the imagenet_mean.binaryproto file is not found; the very same file that has worked for the valiudation process. What is wrong?
...
I0222 15:29:15.108032 15823 net.cpp:399] data -> label
I0222 15:29:15.108057 15823 data_transformer.cpp:25] Loading mean file from: /home/myuser/learning/caffe/data/ilsvrc12/imagenet_mean.binaryproto
F0222 15:29:15.108577 15830 db_lmdb.hpp:14] Check failed: mdb_status == 0 (2 vs. 0) No such file or directory
*** Check failure stack trace: ***
# 0x7fc82857edaa (unknown)
# 0x7fc82857ece4 (unknown)
# 0x7fc82857e6e6 (unknown)
# 0x7fc828581687 (unknown)
# 0x7fc828ba115e caffe::db::LMDB::Open()
# 0x7fc828b75644 caffe::DataReader::Body::InternalThreadEntry()
# 0x7fc828cc1470 caffe::InternalThread::entry()
# 0x7fc81f4a8a4a (unknown)
# 0x7fc826a98184 start_thread
# 0x7fc8271b437d (unknown)
# (nil) (unknown)
Aborted (core dumped)
Here is the prototxt I am using:
name: "CaffeNet"
layer {
name: "data"
type: "Data"
top: "data"
top: "label"
include {
phase: TRAIN
}
transform_param {
mirror: true
crop_size: 227
mean_file: "/home/myuser/learning/caffe/data/ilsvrc12/imagenet_mean.binaryproto"
#mean_file: "data/ilsvrc12/imagenet_mean.binaryproto"
}
# mean pixel / channel-wise mean instead of mean image
# transform_param {
# crop_size: 227
# mean_value: 104
# mean_value: 117
# mean_value: 123
# mirror: true
# }
data_param {
source: "examples/imagenet/ilsvrc12_train_lmdb"
batch_size: 256
backend: LMDB
}
}
layer {
name: "data"
type: "Data"
top: "data"
top: "label"
include {
phase: TEST
}
transform_param {
mirror: false
crop_size: 227
mean_file: "/home/myuser/learning/caffe/data/ilsvrc12/imagenet_mean.binaryproto"
#mean_file: "data/ilsvrc12/imagenet_mean.binaryproto"
}
# mean pixel / channel-wise mean instead of mean image
# transform_param {
# crop_size: 227
# mean_value: 104
# mean_value: 117
# mean_value: 123
# mirror: false
# }
data_param {
source: "/sdc/repository/myuser/Imagenet2012/Imagenet2012trainLMDB"
#source: "examples/imagenet/ilsvrc12_val_lmdb"
batch_size: 50
backend: LMDB
}
}
layer {
name: "conv1"
…
I trained an FC network with HDF5 data layer, then used surgery for transplantation to a convolutional network, then changed the data layer to a probe-suitable data layer, i.e.:
from:
layer {
name: "layer_data_left"
type: "HDF5Data"
top: "data_left"
top: "labels_left"
include {
phase: TRAIN
}
hdf5_data_param {
source: "/home/me/Desktop/trainLeftPatches.txt"
batch_size: 128
}
}
to
layer {
name: "data_left"
type: "Input"
top: "data_right"
input_param { shape: { dim: 1 dim: 1 dim: 1241 dim: 367 } }
}
is there any reason this would go out of memory?:
>>> fc_net.forward()
F0729 20:02:02.205382 6821 syncedmem.cpp:56] Check failed: error == cudaSuccess (2 vs. 0) out of memory
*** Check failure stack trace: ***
Aborted (core dumped)
Or, is it more likely that I made a mistake somewhere in surgery & exchanging data layers?
Thank you.