Try to run an NLP model with an Electra instead of a BERT model - deep-learning

Question

I want to run the wl-coref model with an Electra model instead of a Bert model. However, I get an error message with the Electra model and can't find a hint in the Huggingface documentation on how to fix it.
I try different BERT models such like roberta-base, bert-base-german-cased or SpanBERT/spanbert-base-cased. All works.
But if I try an Electra model, like google/electra-base-discriminator or german-nlp-group/electra-base-german-uncased then it doesn't work.
The error that is displayed:
out, _ = self.bert(subwords_batches_tensor, attention_mask=torch.tensor(attention_mask, device=self.config.device))
ValueError: not enough values to unpack (expected 2, got 1)
And this is the method where the error comes from:_bertify in line 349.

Answer 1

Just remove the underscore _. ELECTRA does not return a pooling output like BERT or RoBerta:
from transformers import AutoTokenizer, AutoModel
def bla(model_id:str):
t = AutoTokenizer.from_pretrained(model_id)
m = AutoModel.from_pretrained(model_id)
print(m(**t("this is a test", return_tensors="pt")).keys())
bla("google/electra-base-discriminator")
bla("roberta-base")
Output:
odict_keys(['last_hidden_state'])
odict_keys(['last_hidden_state', 'pooler_output'])