In my script I save python 2D list(all are float values) in a json file. When I run the script in my machine saved json file contains the correct data. But when I run the python script in xampp server, json file only contains integer values.
I checked the code but no issue found. And the same code outputs two different json files.
When I run script in my machine;
{"arr": [[5.7, 7.14, 5.72, 10.96, 15.82, 10.96, 15.78, 7.08], [5.76, 7.2, 15.72, 7.14, 15.76, 10.88, 5.8, 10.9], [4.98, 1.46, 4.98, 11.7, 14.38, 11.7, 14.44, 15.2, 29.14, 15.22, 29.12, 5.7, 21.42, 5.68, 21.34, 1.44], [17.52, 8.76, 23.18, 8.8, 23.16, 15.14, 17.46, 15.1], [21.4, 6.14, 28.74, 6.08, 28.78, 7.54, 21.44, 7.56], [5.02, 6.4, 15.82, 6.36, 15.88, 1.5, 21.28, 1.5, 21.36, 7.62, 29.06, 7.64, 29.06, 15.12, 23.28, 15.14, 23.24, 8.72, 17.48, 8.7, 17.4, 15.12, 14.52, 15.14, 14.46, 11.64, 5.06, 11.64], [5.02, 1.56, 15.72, 1.5, 15.76, 6.28, 5.06, 6.28]]}
When I run script in xampp server;
{"arr": [[5, 7, 5, 10, 15, 10, 15, 7], [5, 7, 15, 7, 15, 10, 5, 10], [4, 1, 4, 11, 14, 11, 14, 15, 29, 15, 29, 5, 21, 5, 21, 1], [17, 8, 23, 8, 23, 15, 17, 15], [21, 6, 28, 6, 28, 7, 21, 7], [5, 6, 15, 6, 15, 1, 21, 1, 21, 7, 29, 7, 29, 15, 23, 15, 23, 8, 17, 8, 17, 15, 14, 15, 14, 11, 5, 11], [5, 1, 15, 1, 15, 6, 5, 6]]}
This is my code.
#!/usr/bin/env python
import sys;
import cv2
import codecs, json
dataset=[]
image = cv2.imread("C:/xampp/htdocs/"+sys.argv[1])
gray = cv2.cvtColor(image, cv2.COLOR_BGR2GRAY)
gray = cv2.GaussianBlur(gray, (3, 3), 0)
edged = cv2.Canny(gray, 10, 250)
kernel = cv2.getStructuringElement(cv2.MORPH_RECT, (20, 20))
closed = cv2.morphologyEx(edged, cv2.MORPH_CLOSE, kernel)
cv2.waitKey(0)
(_, cnts, _) = cv2.findContours(closed.copy(), cv2.RETR_CCOMP,
cv2.CHAIN_APPROX_SIMPLE)
total = 0
lengths = 0;
i = 0
for c in cnts:
peri = cv2.arcLength(c, True)
approx = cv2.approxPolyDP(c, 0.02 * peri, True)
converted_cnts_rooms = approx.squeeze()
dataset.append([])
for iterate in converted_cnts_rooms:
x1, y1 = iterate.ravel()
dataset[i].append(x1/50)
dataset[i].append(y1/50)
total += 1
i += 1
cv2.waitKey(0)
converted_cnts = approx.squeeze()
for iterate in converted_cnts:
x1,y1 = iterate.ravel()
data = {}
data.update({"arr":dataset})
with open('dataset.json', 'w') as outfile:
json.dump(data, outfile)
Appreciate your help...!!!
Related
Can I sample an array a = [1, 2, 3, 4] based on the specified probabilities p = [0.1, 0.1, 0.3, 0.5]?
For example, in python I can use np.random.choice(a=[1, 2, 3, 4], size=100, p=[0.1, 0.1, 0.3, 0.5])
For me I will form a new random choice data list by percentages/probabilities, for example do random choice on [1, 2, 3, 3, 3, 4, 4, 4, 4, 4] will be equivalent with your data and probabilities.
You can use take (https://www.dolphindb.com/help/FunctionsandCommands/FunctionReferences/t/take.html) function for helping the data forming:
v = take(1, 1) <- take(2, 1) <- take(3, 3) <- take(4, 5)
rand(v, 100)
I am trying to implement something like this for 2 nodes (each node with 2 GPUs):
#### Parallel process initiated with torch.distributed.init_process_group()
### All GPUs work in parallel, and generate lists like :
[20, 0, 1, 17] for GPU0 of node A
[1, 2, 3, 4] for GPU1 of node A
[5, 6, 7, 8] for GPU0 of node B
[0, 2, 4, 6] for GPU1 of node B
I tried
torch.distributed.reduce()
to get a sum of these 4:
[26, 10, 15, 35]
But what I want is a concatenated version like this
[[20, 0, 1, 17], [1, 2, 3, 4] , [5, 6, 7, 8] , [0, 2, 4, 6]]
Or
[20, 0, 1, 17, 1, 2, 3, 4, 5, 6, 7, 8, 0, 2, 4, 6]
is also OK with me.
Is it possible to achieve this from torch.distributed?
You can use dist.all_gather to do this:
import torch
import torch.distributed as dist
q = torch.tensor([20, 0, 1, 17]) # generated on each gpu (with different values) as you mentioned
all_q = [torch.zeros_like(q) for _ in range(world_size)] # world_size is the total number of gpu processes you are running. 4 in your case.
all_q = dist.all_gather(all_q, q)
all_q would then have the following:
[torch.tensor([20, 0, 1, 17]), torch.tensor([1, 2, 3, 4]), torch.tensor([5, 6, 7, 8]), torch.tensor([0, 2, 4, 6])]
You can then use torch.cat to collapse all elements into one array if you like.
You can use dist.all_gather_multigpu if you list of lists of tensors.
When I try to train yolo3 with command
python mmdetection/tools/train.py config/yolo-darknet53.py
and everything goes ok until first epoch begin and error happens that
it says "ValueError: need at least one array to concatenate" I am searching for a long time on net. But no use. Please help or try to give some ideas how to achieve this.
here is the yolo3 configuration file
#!user/bin/env python3
# -*- coding: utf-8 -*-
_base_ = '../mmdetection/configs/_base_/default_runtime.py'
classes = ('hand',)
# model settings
model = dict(
type='YOLOV3',
pretrained='D:/workplace/srtp/handdetection/checkpoints/darknet53-a628ea1b.pth',
backbone=dict(type='Darknet', depth=53, out_indices=(3, 4, 5)),
neck=dict(
type='YOLOV3Neck',
num_scales=3,
in_channels=[1024, 512, 256],
out_channels=[512, 256, 128]),
bbox_head=dict(
type='YOLOV3Head',
num_classes=1,
in_channels=[512, 256, 128],
out_channels=[1024, 512, 256],
anchor_generator=dict(
type='YOLOAnchorGenerator',
base_sizes=[[(116, 90), (156, 198), (373, 326)],
[(30, 61), (62, 45), (59, 119)],
[(10, 13), (16, 30), (33, 23)]],
strides=[32, 16, 8]),
bbox_coder=dict(type='YOLOBBoxCoder'),
featmap_strides=[32, 16, 8],
loss_cls=dict(
type='CrossEntropyLoss',
use_sigmoid=True,
loss_weight=1.0,
reduction='sum'),
loss_conf=dict(
type='CrossEntropyLoss',
use_sigmoid=True,
loss_weight=1.0,
reduction='sum'),
loss_xy=dict(
type='CrossEntropyLoss',
use_sigmoid=True,
loss_weight=2.0,
reduction='sum'),
loss_wh=dict(type='MSELoss', loss_weight=2.0, reduction='sum')))
# training and testing settings
train_cfg = dict(
assigner=dict(
type='GridAssigner', pos_iou_thr=0.5, neg_iou_thr=0.5, min_pos_iou=0))
test_cfg = dict(
nms_pre=1000,
min_bbox_size=0,
score_thr=0.05,
conf_thr=0.005,
nms=dict(type='nms', iou_threshold=0.45),
max_per_img=100)
# dataset settings
dataset_type = 'VOCDataset'
dataset_root = 'D:/workplace/srtp/handdetection/VOC_HandDataSetVOC2007'
img_norm_cfg = dict(mean=[0, 0, 0], std=[255., 255., 255.], to_rgb=True)
train_pipeline = [
dict(type='LoadImageFromFile', to_float32=True),
dict(type='LoadAnnotations', with_bbox=True),
dict(type='PhotoMetricDistortion'),
dict(
type='Expand',
mean=img_norm_cfg['mean'],
to_rgb=img_norm_cfg['to_rgb'],
ratio_range=(1, 2)),
dict(
type='MinIoURandomCrop',
min_ious=(0.4, 0.5, 0.6, 0.7, 0.8, 0.9),
min_crop_size=0.3),
dict(type='Resize', img_scale=[(320, 320), (608, 608)], keep_ratio=True),
dict(type='RandomFlip', flip_ratio=0.5),
dict(type='Normalize', **img_norm_cfg),
dict(type='Pad', size_divisor=32),
dict(type='DefaultFormatBundle'),
dict(type='Collect', keys=['img', 'gt_bboxes', 'gt_labels'])
]
test_pipeline = [
dict(type='LoadImageFromFile'),
dict(
type='MultiScaleFlipAug',
img_scale=(608, 608),
flip=False,
transforms=[
dict(type='Resize', keep_ratio=True),
dict(type='RandomFlip'),
dict(type='Normalize', **img_norm_cfg),
dict(type='Pad', size_divisor=32),
dict(type='ImageToTensor', keys=['img']),
dict(type='Collect', keys=['img'])
])
]
data = dict(
samples_per_gpu=1,
workers_per_gpu=1,
train=dict(
type=dataset_type,
ann_file=f'{dataset_root}/ImageSets/Main/hand_train.txt',
img_prefix=f'{dataset_root}',
pipeline=train_pipeline),
val=dict(
type=dataset_type,
ann_file=f'{dataset_root}/ImageSets/Main/hand_val.txt',
img_prefix=f'{dataset_root}',
pipeline=test_pipeline),
test=dict(
type=dataset_type,
ann_file=f'{dataset_root}/ImageSets/Main/hand_test.txt',
img_prefix=f'{dataset_root}',
pipeline=test_pipeline))
# optimizer
optimizer = dict(type='SGD', lr=0.001, momentum=0.9, weight_decay=0.0005)
optimizer_config = dict(grad_clip=dict(max_norm=35, norm_type=2))
# learning policy
lr_config = dict(
policy='step',
warmup='linear',
warmup_iters=2000, # same as burn-in in darknet
warmup_ratio=0.1,
step=[218, 246])
# runtime settings
total_epochs = 273
evaluation = dict(interval=1, metric=['bbox'])
Shall we lower case input data for (pre) training a BERT uncased model using huggingface? I looked into this response from Thomas Wolf (https://github.com/huggingface/transformers/issues/92#issuecomment-444677920) but not entirely sure if he meant that.
What happens if we lowercase the text ?
Tokenizer will take care of that.
A simple example:
import torch
from transformers import BertTokenizer
tokenizer = BertTokenizer.from_pretrained('bert-base-uncased', max_length = 10, padding_side = 'right')
input_ids = torch.tensor(tokenizer.encode('this is a cat', add_special_tokens=True, max_length = 10, pad_to_max_length = True)).unsqueeze(0)
print(input_ids)
input_ids = torch.tensor(tokenizer.encode('This is a Cat', add_special_tokens=True, max_length = 10, pad_to_max_length = True)).unsqueeze(0)
print(input_ids)
Out:
tensor([[ 101, 2023, 2003, 1037, 4937, 102, 0, 0, 0, 0]])
tensor([[ 101, 2023, 2003, 1037, 4937, 102, 0, 0, 0, 0]])
But in case of cased,
tokenizer = BertTokenizer.from_pretrained('bert-base-cased', max_length = 10, padding_side = 'right')
input_ids = torch.tensor(tokenizer.encode('this is a cat', add_special_tokens=True, max_length = 10, pad_to_max_length = True)).unsqueeze(0)
print(input_ids)
input_ids = torch.tensor(tokenizer.encode('This is a Cat', add_special_tokens=True, max_length = 10, pad_to_max_length = True)).unsqueeze(0)
print(input_ids)
tensor([[ 101, 1142, 1110, 170, 5855, 102, 0, 0, 0, 0]])
tensor([[ 101, 1188, 1110, 170, 8572, 102, 0, 0, 0, 0]])
I think the bert-base-uncased model will lower case the text irrespective of what you pass to the model. You can also try playing with a toy dataset and print the tokens using the BERT tokenizer so as to just confirm.
I am using the embed .html example given on the bokeh site: http://docs.bokeh.org/en/latest/docs/user_guide/embed.html. Note I am using bokeh 12.3. The plots are displaying fine but the text is rendering as the exact output from the script function - including '{' and '\n' characters.
scatter function:
from bokeh.plotting import figure
from bokeh.models import Range1d
from bokeh.embed import components
def scatter():
# create some data
x1 = [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
y1 = [0, 8, 2, 4, 6, 9, 5, 6, 25, 28, 4, 7]
x2 = [2, 5, 7, 15, 18, 19, 25, 28, 9, 10, 4]
y2 = [2, 4, 6, 9, 15, 18, 0, 8, 2, 25, 28]
x3 = [0, 1, 0, 8, 2, 4, 6, 9, 7, 8, 9]
y3 = [0, 8, 4, 6, 9, 15, 18, 19, 19, 25, 28]
# select the tools we want
TOOLS="pan,wheel_zoom,box_zoom,reset,save"
# the red and blue graphs will share this data range
xr1 = Range1d(start=0, end=30)
yr1 = Range1d(start=0, end=30)
# only the green will use this data range
xr2 = Range1d(start=0, end=30)
yr2 = Range1d(start=0, end=30)
# build our figures
p1 = figure(x_range=xr1, y_range=yr1, tools=TOOLS, plot_width=300, plot_height=300)
p1.scatter(x1, y1, size=12, color="red", alpha=0.5)
p2 = figure(x_range=xr1, y_range=yr1, tools=TOOLS, plot_width=300, plot_height=300)
p2.scatter(x2, y2, size=12, color="blue", alpha=0.5)
p3 = figure(x_range=xr2, y_range=yr2, tools=TOOLS, plot_width=300, plot_height=300)
p3.scatter(x3, y3, size=12, color="green", alpha=0.5)
# plots can be a single Bokeh Model, a list/tuple, or even a dictionary
plots = {'Red': p1, 'Blue': p2, 'Green': p3}
script, div = components(plots)
return script, div
My flask code is:
script, div = scatter()
return self.render_template('bokeh_example.html', script=script, div=div)
bokeh_example.html:
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="utf-8">
<link rel="stylesheet" href="http://cdn.bokeh.org/bokeh/release/bokeh-0.12.3.min.css" type="text/css" />
<script type="text/javascript" src="http://cdn.bokeh*emphasized text*.org/bokeh/release/bokeh-0.12.3.min.js"></script>
{{ script | safe }}
</head>
<body>
<div class='bokeh'>
<h1>Scatter Example</h1>
{{ div | safe }}
</div>
</body>
</html>
The plots display fine but the div text renders as literals:
{'Red': '\n #this text displays instead of just the string 'Red'
\n #this displays on next line in smaller font
#plot displays fine here
\n #this text displays after the plot instead of creating a blank line.
Any clues?
You are passing a dictionary of plots to components:
plots = {'Red': p1, 'Blue': p2, 'Green': p3}
script, div = components(plots)
return script, div
This means (per the documentation) that the result is not a single script and a single div. Rather, it's a single script and a dictionary mapping your original names to multiple divs:
components({"Red": p1, "Blue": p2, "Green": p3})
#=> (script, {"Red": p1_div, "Blue": p2_div, "Green": p3_div})
Right now you are trying to template the dict itself into your HTML. Presumably Jinja just calls str on the dict to turn it into a string, and the browser doesn't know what to do with that. You need to template each one of the divs in the dict returned by components, individually.
For a suitably updated template, that might look like:
script, divs = scatter() # notice plural: divS
return self.render_template(
'bokeh_example.html',
script=script,
div_red=divs['Red'],
div_blue=divs['Blue'],
div_green=divs['Green'],
)
Or alternatively you might update the template to iterate over divs directly using some of Jinja2's capabilities for iterating over template arguments that are collections.