Is it possible to translate the info in a .caffemodel file such that it could be read by (for example) Matlab. That is, is there a way to write your model using something else that prototxt and import the weights trained using Caffe?
If the answer is "Nope, it's a binary file and will always remain that way", is there some documentation regarding the structure of the file so that one could extract the important information somehow?
As you know, .caffemodel consists of weights and biases.
A simple way to read weights and biases for a caffemodel given the prototxt would be to just load the network in Python and read the weights.
You can use:
import caffe
net = caffe.Net(<prototxt-file>,<model-file>,<phase>);
and access the params from net.params
source
I'll take VGG as an example
from caffe.proto import caffe_pb2
net = caffe_pb2.NetParameter()
caffemodel = sys.argv[1]
with open(caffemodel, 'rb') as f:
net.ParseFromString(f.read())
for i in net.layer:
print i.ListFields()[0][-1]
#conv1
#relu1
#norm1
#pool1
#conv2
#relu2
#norm2
#pool2
#conv3
#relu3
#conv4
#relu4
#conv5
#relu5
#pool5
#fc6
#relu6
#drop6
#fc7
#relu7
#drop7
#fc8
#prob
Related
Reproducible on google colab (transformers 4.24.0).
from transformers import TFAutoModelForSequenceClassification
model = TFRobertaForSequenceClassification.from_pretrained("cardiffnlp/twitter-roberta-base-emotion", num_labels=1)
I would like to set num_labels=1 because I would like to use it as a regression model. But the above code will give ValueError:
ValueError: cannot reshape array of size 3072 into shape (768,1)
I remembered doing this for the distillbert model worked. Is this the right way of calling this? Or this is not supported by HF for anything than the Bert families?
I trained a model in TensorFlow, and saved it on disk.
Now I want to load it from checkpoint and print the trained parameters.
Something like:
classifier = tf.estimator.DNNClassifier(
feature_columns=feature_columns,
hidden_units=hidden_units,
warm_start_from=checkpoint_path)
print(parameters(classifier))
How do I do that?
I'm using tf version 1.14.
I think you can use these two methods get_variable_names() and get_variable_value() to retrieve the parameters in your classifier.
params = classifier.get_variable_names()
for p in params:
print(p, classifier.get_variable_value(p))
I have a 42 GB jsonl file. Every element of this file is a json object. I create training samples from every json object. But the number of training samples from every json object that I extract can vary between 0 to 5 samples. What is the best way to create a custom PyTorch dataset without reading the entire jsonl file in memory?
This is the dataset I am talking about - Google Natural Questions.
You have a couple of options.
The simplest option, if having lots of small files is not a problem, is to preprocess each json object into a single file. Then you can just read each one depending on the index requested. E.g
class SingleFileDataset(Dataset):
def __init__(self, list_of_file_paths):
self.list_of_file_paths = list_of_file_paths
def __getitem__(self, index):
return np.load(self.list_of_file_paths[index]) # Or equivalent reading code for single file
You can also split the data into a constant number of files, and then calculate, given the index, which file the sample resides in. Then you need to open that file into memory and read the appropriate index. This gives a trade-off between disk access and memory usage. Assume you have n samples, and we split the samples into c files evenly during preprocessing. Now, to read the sample at index i we would do
class SplitIntoFilesDataset(Dataset):
def __init__(self, list_of_file_paths, n_splits):
self.list_of_file_paths = list_of_file_paths
self.n_splits = n_splits
def __getitem__(self, index):
# index // n_splits is the relevant file, and
# index % len(self) is the index in in that file
file_to_load = self.list_of_file_paths[index // self.n_splits]
# Load file
file = np.load(file)
datapoint = file[index % len(self)]
Finally, you could use a HDF5 file that allows access to rows on disk. This is possibly the best solution if you have a lot of data, since the data will be close on disk. There's an implementation here which I have copy pasted below:
import h5py
import torch
import torch.utils.data as data
class H5Dataset(data.Dataset):
def __init__(self, file_path):
super(H5Dataset, self).__init__()
h5_file = h5py.File(file_path)
self.data = h5_file.get('data')
self.target = h5_file.get('label')
def __getitem__(self, index):
return (torch.from_numpy(self.data[index,:,:,:]).float(),
torch.from_numpy(self.target[index,:,:,:]).float())
def __len__(self):
return self.data.shape[0]
These days, I have tried a model which implemented by cntk. But I can't find a way to predict new pic with trained model.
The trained model saved as a checkpoint:
trainer.save_checkpoint(os.path.join(output_model_folder, "model_{}".format(best_epoch)))
Then I have gotten some files like:
So, I tried to load this model checkpoint like:
model = ct.load_model('../data/models/VGG13_majority/model_94')
the code above can run successfully. Then I tried
model.eval(image_data)
but I got an error:
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ update ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
this time I have tried the method below:
model = ct.load_model('../data/models/VGG13_majority/model_94')
model.eval({model.arguments[0]: [final_image]})
then a new error raised:
For any C.Function.eval() you need to pass a dictionary as the argument.
So it will go something like this, assuming that you only have one input_variable into the model:
model = C.load_model()
model.eval({model.arguments[0]: image_data})
Anyhow, i noticed that you saved the model from the checkpoint. By doing so, you actually saved the "ground_truth" input_variable to the loss function too.
I would recommend next time that you saved the model directly. Usually the files from save_checkpoint is meant to be used in restore_from_checkpoint()
import cntk as C
from cntk.layers import Dense
model = Dense(10)(C.input_variable(1))
loss = C.binary_cross_entropy(model, C.input_variable(10))
trainer = C.Trainer(model, (loss,), [C.adam(model.parameters, 0.9, 0.9)])
trainer.save_checkpoint("hello")
model.save() # used this to save the model directly
# to recover model from checkpoint use below
trainer.restore_from_checkpoint("hello")
original_model = trainer.model
print(trainer)
for i in trainer.model.arguments:
print(i)
I am having trouble to access the coefficients of a support vector regression model (SVR) in scikit learn when the model is embedded in a pipeline and a grid search.
Consider the following example:
from sklearn.datasets import load_iris
import numpy as np
from sklearn.grid_search import GridSearchCV
from sklearn.svm import SVR
from sklearn.feature_selection import SelectKBest
from sklearn.pipeline import Pipeline
iris = load_iris()
X_train = iris.data
y_train = iris.target
clf = SVR(kernel='linear')
select = SelectKBest(k=2)
steps = [('feature_selection', select), ('svr', clf)]
pipeline = Pipeline(steps)
grid = GridSearchCV(pipeline, param_grid={"svr__C":[10,10,100],"svr__gamma": np.logspace(-2, 2)})
grid.fit(X_train, y_train)
This seems to work fine but when I try to access the coefficient of the best fitting model
grid.best_estimator_.coef_
I get an error message: AttributeError: 'Pipeline' object has no attribute 'coef_'.
I also tried to access the individual steps of the pipeline:
pipeline.named_steps['svr']
but could not find the coefficients there.
Just happened to come across the same problem and this post
had the answer:
grid.best_estimator_ contains an instance of the pipeline, which consists of steps. The last step should always be the estimator, so you should always find the coefficients at:
grid.best_estimator_.steps[-1][1].coef_