I wish to use Allen NLP Interpret for integrated visualization and Saliency mapping.on custom transformer model, can you please tell me how to do that?
It can be done by having AllenNLP wrappers around your custom model. The interpret modules require a Predictor object, so you can write your own, or use an existing one.
Here's an example for a classification model:
from allennlp.data.vocabulary import Vocabulary
from allennlp.predictors.text_classifier import TextClassifierPredictor
from allennlp.data.dataset_readers import TextClassificationJsonReader
import torch
class ModelWrapper(Model):
def __init__(self, vocab, your_model):
super().__init__(vocab)
self.your_model = your_model
self.logits_to_probs = torch.nn.Softmax()
self.loss = torch.nn.CrossEntropyLoss()
def forward(self, tokens, label=None):
if label is not None:
outputs = self.your_model(tokens, label=label)
else:
outputs = self.your_model(tokens)
probs = self.logits_to_probs(outputs["logits"])
if label is not None:
loss = self.loss(outputs["logits"], label)
outputs["loss"] = loss
outputs["probs"] = probs
return outputs
Your custom transformer model may not have an identifiable TextFieldEmbedder. This is the initial embedding layer of your model, against which gradients are calculated for the saliency interpreters. These can be specified by overriding the following methods in the Predictor.
class PredictorWrapper(TextClassifierPredictor):
def get_interpretable_layer(self):
return self._model.model.bert.embeddings.word_embeddings # This is the initial layer for huggingface's `bert-base-uncased`; change according to your custom model.
def get_interpretable_text_field_embedder(self):
return self._model.model.bert.embeddings.word_embeddings
predictor = PredictorWrapper(model=ModelWrapper(vocab, your_model),
dataset_reader=TextClassificationJsonReader())
Now you have an AllenNLP predictor, which can be used with the interpret module as follows:
from allennlp.interpret.saliency_interpreters import SimpleGradient
interpreter = SimpleGradient(predictor)
interpreter.saliency_interpret_from_json({"sentence": "This is a good movie."})
This should give you the gradients with respect to each input token.
Related
I get the following error while loading the BertEmbedding:
Code:
name = "microsoft/codebert-base"
from transformers import BertModel
from transformers import BertTokenizer
print("[ Using pretrained BERT embeddings ]")
self.bert_tokenizer = BertTokenizer.from_pretrained(name, do_lower_case=lower_case)
self.bert_model = BertModel.from_pretrained(name)
if fix_emb:
print("[ Fix BERT layers ]")
self.bert_model.eval()
for param in self.bert_model.parameters():
param.requires_grad = False
else:
print("[ Finetune BERT layers ]")
self.bert_model.train()
[ Using pretrained BERT embeddings ]
The tokenizer class you load from this checkpoint is not the same type as the class this function is called from. It may result in unexpected tokenization.
The tokenizer class you load from this checkpoint is 'RobertaTokenizer'.
The class this function is called from is 'BertTokenizer'.
The name codebert-base is a bit misleading, as the model is actually a Roberta. The architecture of Bert and Roberta is similar and shows only minor differences but the tokenizer is totally different (and the pertaining approach but that is not relevant here).
You should load microsoft/codebert-base like this:
from transformers import RobertaModel
from transformers import RobertaTokenizer
name = "microsoft/codebert-base"
tokenizer = RobertaTokenizer.from_pretrained(name)
model = RobertaModel.from_pretrained(name)
Or you can use the Auto Classes, which will select the proper classes for you:
from transformers import AutoTokenizer, AutoModel
name = "microsoft/codebert-base"
tokenizer = AutoTokenizer.from_pretrained(name)
model = AutoModel.from_pretrained(name)
I wish disable serializing all model/state weights in standard AllenNLP model training via the use of jsonnet config files.
The reason for this is that I am running automatic hyperparameter optimization using Optuna.
Testing dozens of models fills up a drive pretty quickly.
I already have disabled the checkpointer by setting num_serialized_models_to_keep to 0:
trainer +: {
checkpointer +: {
num_serialized_models_to_keep: 0,
},
I do not wish to set serialization_dir to None as I still want the default behaviour regarding logging of intermediate metrics, etc. I only want to disable the default model state, training state, and best model weights writing.
Besides the option I set above, are there any default trainer or checkpointer options to disable all serialization of model weights? I checked the API docs and webpage but could not find any.
If I need to define the functionality for such an option myself, which base function(s) from the AllenNLP should I override in my Model subclass?
Alternatively, is their any utility for cleaning up intermediate model state when training is concluded?
EDIT: #petew's answer shows the solution for a custom checkpointer, but I am not clear on how to make this code findable to allennlp train for my use-case.
I wish to make the custom_checkpointer callable from a config file as below:
trainer +: {
checkpointer +: {
type: empty,
},
What would be best practice to load the checkpointer when calling allennlp train --include-package <$my_package>?
I have my_package with submodules in subdirectories such as my_package/modelss and my_package/training.
I would like to place the custom checkpointer code in my_package/training/custom_checkpointer.py
My main model is located in my_package/models/main_model.py.
Do I have to edit or import any code/functions in my main_model class to use the custom checkpointer?
You could create and register a custom Checkpointer that basically just does nothing:
#Checkpointer.register("empty")
class EmptyCheckpointer(Registrable):
def maybe_save_checkpoint(
self, trainer: "allennlp.training.trainer.Trainer", epoch: int, batches_this_epoch: int
) -> None:
pass
def save_checkpoint(
self,
epoch: Union[int, str],
trainer: "allennlp.training.trainer.Trainer",
is_best_so_far: bool = False,
save_model_only=False,
) -> None:
pass
def find_latest_checkpoint(self) -> Optional[Tuple[str, str]]:
pass
def restore_checkpoint(self) -> Tuple[Dict[str, Any], Dict[str, Any]]:
return {}, {}
def best_model_state(self) -> Dict[str, Any]:
return {}
I'm developing a simple window that performs some operations at closure. This is my code extract:
from javax.swing import *
from java.awt import *
from java.awt.event import *
from java.io import *
import javax.swing.table.DefaultTableModel as DefaultTableModel
class registro(JFrame):
def __init__(self):
super(registro, self).__init__()
self.setDefaultCloseOperation(JFrame.EXIT_ON_CLOSE)
self.setExtendedState(JFrame.MAXIMIZED_BOTH)
#[...]
headers = ('Data e orario',
'Personale UO Q&A',
'Tipologia di attività '.decode('utf-8'),
'Personale incontrato con strutture di appartenenza',
'Note')
self.model = DefaultTableModel([["","","","",""]], headers)
self.table = JTable(self.model)
#[...]
self.addWindowListener(self.onClose())
#[...]
def onClose(self):
class saver(WindowAdapter):
tableModel = self.model
def windowClosing(self, event):
print tableModel #HERE IS THE ERROR!!!!!!!!!
return saver()
The error reported on the highlighted line is the following:
NameError: global name 'tableModel' is not defined
Although I have declared the variable inside the listener (to avoid misunderstanding between the two self), I don't understand why it has never been recognized. I'm almost a novice with object-oriented programming and Swing windows on Jython, and I hope this is not my (very) serious shortcoming!
Many thanks in advance.
There's a fairly subtle scope issue here, which is mostly about Python syntax, but also about what code you want to have access to the tableModel. The tableModel variable is not visible by default because you are inside the onClose() function. A defensive solution to this is to explicitly pass the needed variable into the new saver object. I personally prefer this as it more explicitly declares the inputs for saver objects.
class WindowAdapter:
None
class App:
def __init__(self):
self.model = 'DUMMYMODEL'
def onClose(self):
class Saver(WindowAdapter):
def __init__(self,tableModel):
WindowAdapter.__init__(self)
self.tableModel = tableModel
def windowClosing(self,event):
print (self.tableModel)
return Saver(self.model)
if __name__ == '__main__':
app = App()
sv = app.onClose()
sv.windowClosing(event=None)
(This code is cut down and in pure Python to show it is largely scoping related.)
An alternative would be using the Python global keyword to expose the tableModel variable to all lower scopes.
class WindowAdapter:
None
class App:
def __init__(self):
self.model = 'DUMMYMODEL'
def onClose(self):
global tableModel
tableModel = self.model
class Saver(WindowAdapter):
def windowClosing(self,event):
print (tableModel)
return Saver()
if __name__ == '__main__':
app = App()
sv = app.onClose()
sv.windowClosing(event=None)
I'd like to know how to store data from my app so I can review the data when I re-run the app.
e.g. I type some info in a TextInput and then when I click the submit button, the info is pasted in a label, so I close the app and when I reopen it the info should be appearing in the label. I know that there are sqlite3 and mysql but I don't know how to apply it into my python/kivy code.
Please anyone suggest me how that can be done.
If possible show with an example, it would be perfect.
Thanks in advance,
My py code:
from kivy.app import App
from kivy.uix.screenmanager import ScreenManager, Screen
from kivy.lang import Builder
from random import shuffle
from kivy.core.window import Window
Window.clearcolor = [1, 1, 1, 1]
Window.size = (550, 650)
Builder.load_file('builder.kv')
class MainScreen(ScreenManager):
pass
class Menu(Screen):
pass
class Levels(Screen):
def __init__(self, **kwargs):
super().__init__(**kwargs)
def on_pre_enter(self):
Window.bind(on_keyboard=self.voltar)
def voltar(self, window, key, *args):
if key == 27:
App.get_running_app().root.current = 'menu'
return True
def on_pre_leave(self):
Window.unbind(on_keyboard=self.voltar)
class LvLogos(Screen):
def on_pre_enter(self):
Window.bind(on_keyboard=self.voltar)
def voltar(self, window, key, *args):
if key == 27:
App.get_running_app().root.current = 'menu'
return True
def on_pre_leave(self):
Window.unbind(on_keyboard=self.voltar)
class Logo(Screen):
def __init__(self, **kwargs):
super().__init__(**kwargs)
def shuffle(self):
letter = self.letters
self.s = shuffle(letter)
return letter
def on_pre_enter(self):
Window.bind(on_keyboard=self.voltar)
def voltar(self, window, key, *args):
if key == 27:
App.get_running_app().root.current = 'menu'
return True
def on_pre_leave(self):
Window.unbind(on_keyboard=self.voltar)
class LvShields(Screen):
pass
class Shield(Screen):
def __init__(self, **kwargs):
super().__init__(**kwargs)
def shuffle(self):
letter = self.letters
self.s = shuffle(letter)
return letter
class QuizZApp(App):
def build(self):
self.icon = 'C:\\Users\\gusta\\PycharmProjects\\QuizzApp\\Images\\QuizzLogo.png'
return MainScreen()
if __name__ == '__main__':
QuizZApp().run()
Of course you can use sqlite3, but the simplest way to store basic data for kivy app like your exaple would be to use json file with kivy's own JsonStore class.
It has the benefit of allocating your file in the right place depending on the platform it is deployed on, you won't need to care where exactly.
Here is a simple example using get(), put() and exists() methods to store typed text from TextInput and load it on a Label. (You won't need to create the file itself, just initialize the object and write (put()) in it).
from kivy.uix.boxlayout import BoxLayout
from kivy.storage.jsonstore import JsonStore
from kivy.base import runTouchApp
from kivy.lang import Builder
from kivy.properties import ObjectProperty
kv='''
RootWidget:
orientation: 'vertical'
BoxLayout:
TextInput:
id: txtinpt
Label:
id: lbl
text: root.stored_data.get('mydata')['text'] if root.stored_data.exists('mydata') else ''
Button:
size_hint_y: .3
text: 'Submit'
on_press:
root.stored_data.put('mydata', text=txtinpt.text)
lbl.text = txtinpt.text
'''
class RootWidget(BoxLayout):
stored_data = ObjectProperty(None)
def __init__(self, *args, **kwargs):
super(BoxLayout, self).__init__(*args, **kwargs)
self.stored_data = JsonStore('data.json')
runTouchApp(Builder.load_string(kv))
If you are new to Json, it is a file containing list of pairs, which's value by its own may be a new list of pair. pretty much like python's dict.
For kivy's JsonStore class, it assume you are working with a minimum of two levels, hence {"mydata": {"text": "What you have written last run"}}. It doesn't make much sense in this simple example to have a nested dict, but in general it will be exactly what you want for real applications, like if you wanted to take contacts data for multiple contacts, or you want to store various configurations for multiple widget for the app itself (in this case you may want to read (get()) the data prior of loading the widgets, probably in the App-class's build() method).
reference: https://kivy.org/docs/api-kivy.storage.html
I'm following Pytorch seq2seq tutorial and below is how they define the encoder function.
class EncoderRNN(nn.Module):
def __init__(self, input_size, hidden_size):
super(EncoderRNN, self).__init__()
self.hidden_size = hidden_size
self.embedding = nn.Embedding(input_size, hidden_size)
self.gru = nn.GRU(hidden_size, hidden_size)
def forward(self, input, hidden):
embedded = self.embedding(input).view(1, 1, -1)
output = embedded
output, hidden = self.gru(output, hidden)
return output, hidden
def initHidden(self):
return torch.zeros(1, 1, self.hidden_size, device=device)
However, it seems like forward method is never really being called during the training.
Here is how the encoder forward method is being used in the tutorial:
for ei in range(input_length):
encoder_output, encoder_hidden = encoder(input_tensor[ei], encoder_hidden)
encoder_outputs[ei] = encoder_output[0, 0]
isn't it supposed to be encoder.forward instead of just encoder?
Is there some automatic 'forward' mechanism in Pytorch that I am not aware of?
In PyTorch, you write your own class by extending torch.nn.Module and define the forward method to express your desired computational steps that serve as the "paperwork" (e.g. calling hooks) in the model.__call__(...) method (which is what model(x) will call by python special name specifications).
If you are curious you can look at what model(x) does behind the scenes beyond calling model.forward(x) here: https://github.com/pytorch/pytorch/blob/master/torch/nn/modules/module.py#L462
Also, you can see what is the difference between explicitly calling the .foward(x) method and just simply using model(x) here: https://github.com/pytorch/pytorch/blob/master/torch/nn/modules/module.py#L72