I get the following error while loading the BertEmbedding:
Code:
name = "microsoft/codebert-base"
from transformers import BertModel
from transformers import BertTokenizer
print("[ Using pretrained BERT embeddings ]")
self.bert_tokenizer = BertTokenizer.from_pretrained(name, do_lower_case=lower_case)
self.bert_model = BertModel.from_pretrained(name)
if fix_emb:
print("[ Fix BERT layers ]")
self.bert_model.eval()
for param in self.bert_model.parameters():
param.requires_grad = False
else:
print("[ Finetune BERT layers ]")
self.bert_model.train()
[ Using pretrained BERT embeddings ]
The tokenizer class you load from this checkpoint is not the same type as the class this function is called from. It may result in unexpected tokenization.
The tokenizer class you load from this checkpoint is 'RobertaTokenizer'.
The class this function is called from is 'BertTokenizer'.
The name codebert-base is a bit misleading, as the model is actually a Roberta. The architecture of Bert and Roberta is similar and shows only minor differences but the tokenizer is totally different (and the pertaining approach but that is not relevant here).
You should load microsoft/codebert-base like this:
from transformers import RobertaModel
from transformers import RobertaTokenizer
name = "microsoft/codebert-base"
tokenizer = RobertaTokenizer.from_pretrained(name)
model = RobertaModel.from_pretrained(name)
Or you can use the Auto Classes, which will select the proper classes for you:
from transformers import AutoTokenizer, AutoModel
name = "microsoft/codebert-base"
tokenizer = AutoTokenizer.from_pretrained(name)
model = AutoModel.from_pretrained(name)
Related
Suppose I have a simple SQLAlchemy class and a simple Flask o FastAPI implementation like this:
from sqlalchemy.ext.declarative import declarative_base
from pydantic import BaseModel
Base = declarative_base()
class A(Base):
__tablename__ = 'as'
my_id = Column(String)
class AModel(BaseModel):
myId:str = None
And a simple endpoint like this:
#app_router.get('/a')
def get_all_a(session:Session = Depends(get_session)):
return session.query(A).all()
How could I ensure that the returned list of this endpoint yields in camelCase like this:
[{'myId': 'id1'},{'myId': 'id2'}, ...]
Note: My application is rather complex, as I also have pagination implemented and some post-processings require a little bit more that just snake_case to camelCase conversion, so the simplest solution would be the best.
I've tried overriding dict() methods and similar stuff with no luck, simply cannot understand how FastAPI processes the results to obtain a JSON.
require a little bit more that just snake_case to camelCase conversion
Well, if you don't use response_model you don't have a lot of choices.
The solution is returning your dict with a conversion from snake_case to camelCase. You have functions that do it recursively.
Why it is the best solution ?
using regex it's super fast, faster than any lib that convert dict to obj like pydantic
If you definitely don't want to do this, well your only solution is using pydantic models, attrs or dataclasses and convert your db query output to one of those models type with camelCase variable name (dirty).
Since you are using fastapi you should use all the power of it.
I would suggest this:
from typing import List
from sqlalchemy.ext.declarative import declarative_base
from pydantic import BaseModel, Field
from pydantic import parse_obj_as
Base = declarative_base()
class A(Base):
__tablename__ = 'as'
my_id = Column(String)
class AModel(BaseModel):
myId: str = Field(alias="my_id", default=None)
#app_router.get('/a', response_model=List[AModel])
def get_all_a(session:Session = Depends(get_session)):
return parse_obj_as(List[AModel], session.query(A).all())
Keep in mind that having classes variables in CamelCase is not a good practice.
The gold answer would be to not return camel but snake and let your client do the work of conversion if needed.
I wish to use Allen NLP Interpret for integrated visualization and Saliency mapping.on custom transformer model, can you please tell me how to do that?
It can be done by having AllenNLP wrappers around your custom model. The interpret modules require a Predictor object, so you can write your own, or use an existing one.
Here's an example for a classification model:
from allennlp.data.vocabulary import Vocabulary
from allennlp.predictors.text_classifier import TextClassifierPredictor
from allennlp.data.dataset_readers import TextClassificationJsonReader
import torch
class ModelWrapper(Model):
def __init__(self, vocab, your_model):
super().__init__(vocab)
self.your_model = your_model
self.logits_to_probs = torch.nn.Softmax()
self.loss = torch.nn.CrossEntropyLoss()
def forward(self, tokens, label=None):
if label is not None:
outputs = self.your_model(tokens, label=label)
else:
outputs = self.your_model(tokens)
probs = self.logits_to_probs(outputs["logits"])
if label is not None:
loss = self.loss(outputs["logits"], label)
outputs["loss"] = loss
outputs["probs"] = probs
return outputs
Your custom transformer model may not have an identifiable TextFieldEmbedder. This is the initial embedding layer of your model, against which gradients are calculated for the saliency interpreters. These can be specified by overriding the following methods in the Predictor.
class PredictorWrapper(TextClassifierPredictor):
def get_interpretable_layer(self):
return self._model.model.bert.embeddings.word_embeddings # This is the initial layer for huggingface's `bert-base-uncased`; change according to your custom model.
def get_interpretable_text_field_embedder(self):
return self._model.model.bert.embeddings.word_embeddings
predictor = PredictorWrapper(model=ModelWrapper(vocab, your_model),
dataset_reader=TextClassificationJsonReader())
Now you have an AllenNLP predictor, which can be used with the interpret module as follows:
from allennlp.interpret.saliency_interpreters import SimpleGradient
interpreter = SimpleGradient(predictor)
interpreter.saliency_interpret_from_json({"sentence": "This is a good movie."})
This should give you the gradients with respect to each input token.
I am new to GluonTS and deep learning in general. I have a GluonTS DeepAR model which has files like -
myPrefix-xxxx.params
myPrefix-network.json
What is the way to load this model for scoring? I tried below but this looks for prefix-symbol.json
import mxnet as mx
model = mx.mod.Module.load(myPrefix,epochs)
Thanks.
The way to load a GluonTS model is to use the Predictor class and deserialize it -
from gluonts.model.predictor import Predictor
from pathlib import Path
model = Predictor.deserialize(Path(path_to_the_model))
I'm following Pytorch seq2seq tutorial and below is how they define the encoder function.
class EncoderRNN(nn.Module):
def __init__(self, input_size, hidden_size):
super(EncoderRNN, self).__init__()
self.hidden_size = hidden_size
self.embedding = nn.Embedding(input_size, hidden_size)
self.gru = nn.GRU(hidden_size, hidden_size)
def forward(self, input, hidden):
embedded = self.embedding(input).view(1, 1, -1)
output = embedded
output, hidden = self.gru(output, hidden)
return output, hidden
def initHidden(self):
return torch.zeros(1, 1, self.hidden_size, device=device)
However, it seems like forward method is never really being called during the training.
Here is how the encoder forward method is being used in the tutorial:
for ei in range(input_length):
encoder_output, encoder_hidden = encoder(input_tensor[ei], encoder_hidden)
encoder_outputs[ei] = encoder_output[0, 0]
isn't it supposed to be encoder.forward instead of just encoder?
Is there some automatic 'forward' mechanism in Pytorch that I am not aware of?
In PyTorch, you write your own class by extending torch.nn.Module and define the forward method to express your desired computational steps that serve as the "paperwork" (e.g. calling hooks) in the model.__call__(...) method (which is what model(x) will call by python special name specifications).
If you are curious you can look at what model(x) does behind the scenes beyond calling model.forward(x) here: https://github.com/pytorch/pytorch/blob/master/torch/nn/modules/module.py#L462
Also, you can see what is the difference between explicitly calling the .foward(x) method and just simply using model(x) here: https://github.com/pytorch/pytorch/blob/master/torch/nn/modules/module.py#L72
I'm attempting basic serialization/hydration with lift-json, but without success. As near as I can tell from the package readme, this should work. Help?
I'm using Scala 2.8.0 and Lift 2.2 cross-built for 2.8 with sbt ("net.liftweb" %% "lift-json" % "2.2").
import net.liftweb.json._
import net.liftweb.json.Serialization.{read, write}
implicit val formats = Serialization.formats(NoTypeHints)
case class Route(title: String)
val rt = new Route("x277a1")
val ser = write(rt)
// ser: String = {} ...
val deser = read[Route]("""{"title":"Some Title"}""")
// net.liftweb.json.MappingException: Parsed JSON values do not match with class constructor
Lift JSON's serialization does not work for case classes defined in REPL (paranamer can't find the bytecode to read the type metadata). Compile Route with scalac and then the above example works.
The same problem applies every time when the (de)serialuzed class is not on the classpath. In such case, paranamer can't read the parameter names. It is necessary to provide a custom ParameterNameReader.
Such problem applies for e.g.:
REPL (as mentioned) - unless you define the class outside the REPL and add via classpath.
Play Framework - unless you provide a simple custom ParameterNameReader (see below) or load the (de)serialized class as a Maven/Play/... dependency
Feel free to add another situation (you can edit this post).
The PlayParameterNameReader:
import net.liftweb.json.ParameterNameReader
import java.lang.reflect.Constructor
import play.classloading.enhancers.LocalvariablesNamesEnhancer
import scala.collection.JavaConversions._
object PlayParameterReader extends ParameterNameReader{
def lookupParameterNames(constructor: Constructor[_]) = LocalvariablesNamesEnhancer.lookupParameterNames(constructor)
}