I have tried different ways to pass the language but none worked. Here is the code:
dataframe_final=[]
for i in range(len(boxes_list)):
for j in range(len(boxes_list[i])):
s=''
if(len(boxes_list[i][j])==0):
dataframe_final.append(' ')
else:
for k in range(len(boxes_list[i][j])):
y,x,w,h = boxes_list[i][j][k][0],boxes_list[i][j][k][1],
boxes_list[i][j][k][2],boxes_list[i][j][k][3]
roi = bitnot[x:x+h, y:y+w]
kernel = cv2.getStructuringElement(cv2.MORPH_RECT, (2, 1))
border = cv2.copyMakeBorder(roi,2,2,2,2,
cv2.BORDER_CONSTANT,value=[255,255])
resizing = cv2.resize(border, None, fx=2, fy=2,
interpolation=cv2.INTER_CUBIC)
dilation = cv2.dilate(resizing, kernel,iterations=1)
erosion = cv2.erode(dilation, kernel,iterations=2)
plotting = plt.imshow(erosion,cmap='gray')
plt.title("After Erosion")
plt.show()
out = pytesseract.image_to_string(erosion)
if(len(out)==0):
out = pytesseract.image_to_string(erosion)
s = s +" "+ out
dataframe_final.append(s)
print(dataframe_final)
Now if I pass pytesseract.image_to_string(erosion, lang='fra') it doesn't work. Is there a way to pre define the language or another solution??
Here is the Image produced after erosion:
You still did not explain what does it mean: "it doesn't work"?
Why do you want to use language for the recognition of numbers??? It does not make sense
Based on your image: are you trying to OCR 120px hight letters? Please read the tesseract doc first.
Related
I've already trained several models for a binary classification problem, basing my election on F-Score and AUC. The code used has been the following:
svm = StandardScaler()
svm.fit(feat_train)
feat_train_std = svm.transform(feat_train)
feat_test_std = svm.transform(feat_test)
model_10= BalancedBaggingClassifier(base_estimator=SVC(C=1.0, random_state=1, kernel='linear'),
sampling_strategy='auto',
replacement=False,
random_state=0)
model_10.fit(feat_train_std, target_train)
pred_target_10 = model_10.predict(feat_test)
mostrar_resultados(target_test, pred_target_10)
pred_target_10 = model_10.predict_proba(feat_test)[:, 1]
average_precision_10 = average_precision_score(target_test, pred_target_10)
precision_10, recall_10, thresholds = precision_recall_curve(target_test, pred_target_10)
auc_precision_recall_10 = auc(recall_10, precision_10)
disp_10 = plot_precision_recall_curve(model_10, feat_test, target_test)
disp_10.ax_.set_title('Binary class Precision-Recall curve: '
'AUC={0:0.2f}'.format(auc_precision_recall_10))
Afterwards, I load the model as follows:
modelo_pickle = 'modelo_pickle.pkl'
joblib.dump(model_10,modelo_pickle)
loaded_model = joblib.load(modelo_pickle)
Then, the aim is to load a new dataset, which columns are the same as the model's variables, and make a prediction for each line:
lista_x=x.to_numpy().tolist()
resultados=[]
for i in lista_x:
pred = loaded_model.predict([i])
resultados.append(pred)
print(resultados)
However, every single result is equal to 1, which does not make any sense. Would anyone tell me what am I missing, please?
Thank you in advance.
Regards,
Previously described.
My goal is to create a pre-written code - or function - that I can apply to multiple questions, where I only need to fill in the information shown on the lower half. The top half code must remain general, because I do not want to replace pieces of the code every time I use it, and would rather have it do so automatically, making it Reusable.
At the moment my code keeps repeating itself even when I type in the correct answers ('Fully', 'Partially', 'None')
def enter_preset_option():
global subj_desc
subj_desc = input(question)
i = 0
while i < len(options):
if str(subj_desc.lower()) == options[i]:
print("Great!")
break
else:
print("Invalid input. Please try again!")
i += 1
enter_preset_option()
question = "Is the property fully fenced or partially fenced? (Answer 'Fully', 'Partially' or 'None'): "
options = ['fully', 'partially', 'none']
attribute = "Fenced"
enter_preset_option()
fenced = subj_desc
print(fenced)
I'm trying to figure out how sequence to sequence loss is calculated. I am using the huggingface transformers library in this case, but this might actually be relevant to other DL libraries.
So to get the required data we can do:
from transformers import EncoderDecoderModel, BertTokenizer
import torch
import torch.nn.functional as F
torch.manual_seed(42)
tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')
MAX_LEN = 128
tokenize = lambda x: tokenizer(x, max_length=MAX_LEN, truncation=True, padding=True, return_tensors="pt")
model = EncoderDecoderModel.from_encoder_decoder_pretrained('bert-base-uncased', 'bert-base-uncased') # initialize Bert2Bert from pre-trained checkpoints
input_seq = ["Hello, my dog is cute", "my cat cute"]
output_seq = ["Yes it is", "ok"]
input_tokens = tokenize(input_seq)
output_tokens = tokenize(output_seq)
outputs = model(
input_ids=input_tokens["input_ids"],
attention_mask=input_tokens["attention_mask"],
decoder_input_ids=output_tokens["input_ids"],
decoder_attention_mask=output_tokens["attention_mask"],
labels=output_tokens["input_ids"],
return_dict=True)
idx = output_tokens["input_ids"]
logits = F.log_softmax(outputs["logits"], dim=-1)
mask = output_tokens["attention_mask"]
Edit 1
Thanks to #cronoik I was able to replicate the loss calculated by huggingface as being:
output_logits = logits[:,:-1,:]
output_mask = mask[:,:-1]
label_tokens = output_tokens["input_ids"][:, 1:].unsqueeze(-1)
select_logits = torch.gather(output_logits, -1, label_tokens).squeeze()
huggingface_loss = -select_logits.mean()
However, since the last two tokens of the second input is just padding, shouldn't we calculate the loss to be:
seq_loss = (select_logits * output_mask).sum(dim=-1, keepdims=True) / output_mask.sum(dim=-1, keepdims=True)
seq_loss = -seq_loss.mean()
^This takes into account the length of the sequence of each row of outputs, and the padding by masking it out. Think this is especially useful when we have batches of varying length outputs.
ok I found out where I was making the mistakes. This is all thanks to this thread in the HuggingFace forum.
The output labels need to have -100 for the masked version. The transoformers library does not do it for you.
One silly mistake I made was with the mask. It should have been output_mask = mask[:, 1:] instead of :-1.
1. Using Model
We need to set the masks of output to -100. It is important to use clone as shown below:
labels = output_tokens["input_ids"].clone()
labels[output_tokens["attention_mask"]==0] = -100
outputs = model(
input_ids=input_tokens["input_ids"],
attention_mask=input_tokens["attention_mask"],
decoder_input_ids=output_tokens["input_ids"],
decoder_attention_mask=output_tokens["attention_mask"],
labels=labels,
return_dict=True)
2. Calculating Loss
So the final way to replicate it is as follows:
idx = output_tokens["input_ids"]
logits = F.log_softmax(outputs["logits"], dim=-1)
mask = output_tokens["attention_mask"]
# shift things
output_logits = logits[:,:-1,:]
label_tokens = idx[:, 1:].unsqueeze(-1)
output_mask = mask[:,1:]
# gather the logits and mask
select_logits = torch.gather(output_logits, -1, label_tokens).squeeze()
-select_logits[output_mask==1].mean(), outputs["loss"]
The above however ignores the fact that this comes from two different lines. So an alternate way of calculating loss could be:
seq_loss = (select_logits * output_mask).sum(dim=-1, keepdims=True) / output_mask.sum(dim=-1, keepdims=True)
seq_loss.mean()
thanks for sharing. However, the new version of transformers as of today actually does not "shift" anymore. The following is not needed.
#shift things
output_logits = logits[:,:-1,:]
label_tokens = idx[:, 1:].unsqueeze(-1)
output_mask = mask[:,1:
I have the following mixed effects model:
p1 <- lmer(log(price) ~ year*loca + (1|author), data = df)
'year' is continuous
'loca' is categorical variable with 2 levels
I am trying to plot the significant interaction from this model.
The following code (using the visreg package) plots the lines from each of the two 'loca' but it does not produce a 95% confidence band:
visreg(p1, "year", by = "loca", overlay = T,
line=list(lty = 1, col = c("grey", "black")), points=list(cex=1, pch=19,
col = c("grey", "black")), type="conditional", axes = T)
Then, I tried the following code which allows me to plot the lines, but with no data points on top and no CIs:
visreg(p1, "year", by = "loca", overlay = T,
line=list(lty = 1, col = c("grey60", "black")), points=list(cex=1,
pch=19, col = c("grey", "black")),
type="conditional", trans = exp, fill.par = list(col = c("grey80",
"grey70")))
I get CI bands when I use type = 'contrast' rather than 'conditional'. However, this doesn't work when I try to backtransform the price as above using trans = exp.
Overall I need to be able to plot the interaction with the following attributes:
Confidence bands
backtransformed points
predicted line (one for each level of 'loca')
More than happy to try other methods....but I can't seem to find any that work so far.
Help much appreciated!
one possibility is with the use of the effects package:
library(effects)
eff.p1 <- effect("year*loca", p1, KR=T)
then you could either directly plot it with what the package provides and customize it from there:
plot(eff.p1)
or take what effect produces and plot it with ggplot in a nicer plot:
eff.p1 <- as.data.frame(eff.p1)
ggplot(eff.p1, aes(year, linetype=factor(loca),
color = factor(loca))) +
geom_line(aes(y = fit, group=factor(loca)), size=1.2) +
geom_line(aes(y = lower,
group=factor(loca)), linetype =3) +
geom_line(aes(y = upper,
group=factor(loca)), linetype =3) +
xlab("year") +
ylab("Marginal Effects on Log Price") +
scale_colour_discrete("") +
scale_linetype_discrete("") +
labs(color='loca') + theme_minimal()
I can't really try the code without the data, but I think it should work.
This should do the trick:
install.packages(sjPlot)
library(sjPlot)
plot_model(p1, type = "int", terms = c(year,loca), ci.lvl = 0.95)
Although it comes out with some warnings about labels, testing on my data, it does the back transformation automatically and seems to work fine. Customising should be easy, because I believe sjPlot uses ggplot.
EDIT: #Daniel points out that alternative options which allow more customization would be plot_model(type = "pred", ...) or plot_model(type = "eff", ...)
I have some simple code that uses the minmax algoritm to locate birds. Everything works but I find my programming not good and I believe there is a better solution. I'm not that experienced in RoR but if somebody knows a better way to achieve the same solution then I'm greatful ;).
There are two parts I hate, the 4 lists I had to create to determine the max or min value for the different combinations (the core of the min-max algorithm) and the very ugly SQL hack.
Thanks!
def index
# fetch all our birds
#birds = Bird.all
# Loop over the birds
#birds.each do |bird|
#fixed = Node.where("d7type = 'f'")
xminmax = []
xmaxmin = []
yminmax = []
ymaxmin = []
#fixed.each do |fixed|
rss = Log.find_by_sql("SELECT logs.fixed_mac, AVG(logs.blinker_rss) AS avg_rss FROM logs
WHERE logs.blinker_mac = '#{bird.d7_mac}' AND logs.fixed_mac = '#{fixed.d7_mac}' ORDER BY logs.id DESC LIMIT 30")
converted_rss = calculate_distance_rss(rss[0].attributes["avg_rss"])
xminmax.push(fixed.xpos + converted_rss)
xmaxmin.push(fixed.xpos - converted_rss)
yminmax.push(fixed.ypos + converted_rss)
ymaxmin.push(fixed.ypos - converted_rss)
end
pos = {x: (xminmax.min + xmaxmin.max) / 2, y: (yminmax.min + ymaxmin.max) / 2}
puts pos
end
end
2 things you could do to start with is (assuming Birds could be a large table) Change Bird.all to
Bird.find_each do |bird|
... code ...
end
It's a more efficient way to loop over many table records.
2nd: take #fixed = Node.where("d7type = 'f'") out of the each loop since it doesn't need any variables for its query. Put it above the loop so it doesn't execute each time.
3rd (Not so much of an optimization but just safer code): Your Log.find_by_sql looks simple enough to use active_record, you can change it to:
Log.select('fixed_mac, AVG(logs.blinker_rss) AS avg_rss, blinker_mac').
where(blinker_mac: bird.d7_mac, fixed_mac: fixed.d7_mac).
order('id DESC').limit(30)
converted_rss = calculate_distance_rss(rss.first.avg_rss)
Everything else looks fine.