I am trying to use the lme4 package for a glmm and am getting a convergence code of 0 and a statement: Model failed to converge with max|grad| = 0.00791467 (tol = 0.001, component 1). I am interested in using the lme4 package because I would like to have AIC values to determine the appropriate model as I add in additional covariates.
Two weeks ago when I tried the same approach I got a warning message that the model failed to converge because of the max|grad| issue, but am not getting the warning message this time, just the statement at the end of the summary output.
Does this mean that the model is not converging? I also used the glmmPQL method. The coefficient parameter estimates are similar between the two model types.
Here is glmer (lme4) model code. I increased the maxfun to deal with other issues I had when I ran the model last time.
l1<-glmer(Meat_Weight~logsh+SAMS_region_2015+(1|StationID),
family="Gamma"(link="log"),data=datad,control=glmerControl(optCtrl=list(maxfun=100000)))
Here is the glmmPQL code.
m1<-glmmPQL(fixed=Meat_Weight~logsh+SAMS_region_2015,random=~1|StationID,
family=Gamma(link="log"),data=datad)
I am sure this is not information to diagnosis the problem, but if anyone has suggestions I can provide more data.
Thanks
Try to change the optimizer
l1<-glmer(Meat_Weight~logsh+SAMS_region_2015+(1|StationID),
family="Gamma"(link="log"),data=datad, control = glmerControl(optimizer="bobyqa"))
Related
I am trying to create a BentoML service for a CatBoostClassifier model that was trained using a column as a categorical feature. If i save the model and I try to make some predictions with the saved model (not as a BentoML service) all works as expected, but when I create the service using BentML I get an error
_catboost.CatBoostError: Bad value for num_feature[non_default_doc_idx=0,feature_idx=2]="Tertiary": Cannot convert 'b'Tertiary'' to float
The value is found in a column named 'road_type' and the model was trained using 'object' as the data type for the column.
If I try to give a float or an integer for the 'road_type' column I get the following error
_catboost.CatBoostError: catboost/libs/data/model_dataset_compatibility.cpp:53: Feature road_type is Categorical in model but marked different in the dataset
If someone has encountered the same issue and found a solution I would appreciate it. Thanks!
I have tried different approaches for saving the model or loading the model but unfortunately it did not worked.
You can try to explicitly pass the cat_features to the bentoml runner.
It would be something like this:
from catboost import Pool
runner = bentoml.catboost.get("bentoml_catboost_model:latest").to_runner()
cat_features = [2] # specify your cat_features indexes
prediction = runner.predict.run(Pool(input_data, cat_features=cat_features))
I'm pickling a very large (both in terms of properties and in terms raw size) class. I've been picking it no problem with pickle using pickle.dump, until I hit just under 4GB and now I consistently get 'Memory Error'. I've also tried using json.dump (and I get 'is not JSON serializable' error). I've also tried Hickle but I get the same error with Hickle as I do with Pickle.
I can't post all the code here (it's very long) but in essence It's a class that holds a dictionary of values from another class - something like this:
class one:
def __init__(self):
self.somedict = {}
def addItem(self,name,item)
self.somedict[name] = item
class two:
def __init__(self):
self.values = [0]*100
Where name is a string and item is an instance of the class two object.
There's a lot more code to it, but this is where the vast majority of things are held. Is there a reliable and ideally fast solution to saving this object to file and then being able to reload it at a later time. I save it every few thousand iterations (as a backup incase something goes wrong, so I need it to be reasonably quick).
Thanks!
Edit #1:
I've just thought that it might be useful to include some details on my system. I have 64Gb of ram - so I don't think pickling a 3-4GB file should cause this type of issue (although I could be wrong on this!).
You probably checked this one first but just in case: Did you make sure your Python installation 64 bit? The 3-4GB immediately reminded me of the memory limit of 32bit applications.
I found this resource quite useful for analyzing and resolving some of the more common memory related issues with Python.
In the Dymola translation log of one large model I see several times the warning:
Warning: Can only compute non-scalar gradients of functions specifying derivatives and not for:
noEvent
with no indication of where it comes from. Could someone explain what the warning means, and how to find and fix it?
Any message of that kind should only occur if you have set the flag Advanced.PrintFailureToDifferentiate = true; if you set this flag to false it should not occur.
The likely cause is that you have a non-scalar expression involving noEvent that for some reason needs to be solved using a system of equations; and it just means that Dymola does not yet generate analytic Jacobians for such systems.
The exact cause is difficult to say without the model; you could send that through the normal support-channels.
In the given example of MNIST in the Caffe installation.
For any given test image, how to get the softmax scores for each category and do some processing on them? Say compute the mean and variance of them.
I am newbie so a detail would help me a lot. I am able to train the model and use the testing feature to get the prediction but I am not sure which files are to be edited in order to get the above results.
You can use python interface
import caffe
net = caffe.Net('/path/to/deploy.prototxt', '/path/to/weights.caffemodel', caffe.TEST)
in_ = read_data(...) # this is up to you to read a sample and convert it to numpy array
out_ = net.forward(data=in_) # assuming your net expects "data" in blob
Now you have the output of your net in a dictionary out (keys are names of output blobs). You can run it in a loop on several examples etc.
I can try to answer your question. Assuming in your deploying net, the softmax layer is like below:
layer {
name: "prob"
type : "Softmax"
bottom: "fc6"
top: "prob"
}
In your python code that processes data, combining with the code #Shai provided, you can get the probability of each category by adding code based on #Shai's code:
predicted_prob = net.blobs['prob'].data
predicted_prob will be returned an array that contains the probabilities with all categories.
For example, if you only have two categories, predicted_prob[0][0] will be the probability that this testing data belongs to one category and predicted_prob[0][1] will be the probability of the other one.
PS:
If you don't want to write any additional python script, according to https://github.com/BVLC/caffe/tree/master/examples/mnist
it says this example will automatically do the testing every 500 iterations. "500" is defined in solver, such as https://github.com/BVLC/caffe/blob/master/examples/mnist/lenet_solver.prototxt
So you need to trace back the caffe source code that processes the solver file. I guess it should be https://github.com/BVLC/caffe/blob/master/src/caffe/solver.cpp
I am not sure solver.cpp is the correct file you need to look at. But in this file, you can see it has functions of testing and calculation of some values. I hope it can give you some ideas if no one else can answer your question.
I am trying to run the seq2seq translate example present on Tensorflow website and getting following error. I am using tensorflow-gpu==1.1.0
ValueError: Attempt to reuse RNNCell
with a different variable scope than its first use.
First use of cell was with scope
'embedding_attention_seq2seq/embedding_attention_decoder/attention_decoder/gru_cell',
this attempt is with scope 'embedding_attention_seq2seq/rnn/gru_cell'.
Please create a new instance of the cell if you would like it to use a
different set of weights. If before you were using:
MultiRNNCell([GRUCell(...)] * num_layers), change to:
MultiRNNCell([GRUCell(...) for _ in range(num_layers)]). If before
you were using the same cell instance as both the forward and reverse
cell of a bidirectional RNN, simply create two instances (one for
forward, one for reverse). In May 2017, we will start transitioning
this cell's behavior to use existing stored weights, if any, when it
is called with scope=None (which can lead to silent model degradation,
so this error will remain until then.)
On github people were suggesting to change the add the reuse argument to cell as follows :
def single_cell():
return tf.contrib.rnn.GRUCell(size, reuse = tf.get_variable_scope().reuse)
if use_lstm:
def single_cell():
return tf.contrib.rnn.BasicLSTMCell(size, reuse = tf.get_variable_scope().reuse)
cell = single_cell()
if num_layers > 1:
cell = tf.contrib.rnn.MultiRNNCell([single_cell() for _ in range(num_layers)])
But still I am getting the same error. What's the issue and how to resolve it ?
Any help is highly appreciated.
P.S: A similar post was there on stackoverflow,but that solution didn't work for me and since the version of TF is different, I created a new post.