I'm looking for a effective way to add the bounding boxes to the live video displayed on a webpage.
The project is to detect the cars in the video stream (1080p 30fps) captured by a camera using built-in rtsp function and display the results at the webpage.
My current solution:
The frame image is extracted using OpenCV. I run the detection algorithms frame by frame in real time. The frame image is first compressed into JPEG and then encoded into a Base64 string. Then, the Base64 image string along with the bounding boxes is encapsulated into JSON. I transmit the JSON string to the frontend using WebSocket.
At the frontend, I use canvas to draw the frame image each time a message is received.
Now the problems are:
The compression progress JPEG(OpenCV imencode) + Base64 (base64.b64encode) takes too much time, the throughput is about 20FPS (without detection).
The images displaying on the browser is slow and not smooth, the framerate can not even achieve 20FPS (without detection). And the webpage takes much memory and may become not responding.
I wonder if there is a solution to my problems (achieve 30FPS) or there are any other ways to achieve my purpose (synchronize the backend detection results and the frontend live video).
P.S.
The bounding boxes are not rendered to the image at the backend because there may be some operations need to be done to the bounding boxes at the frontend, e.g., determing whether to show the bounding boxes or not.
EDIT: The codes in python (without detection) at the backend are like:
Main.py
from multiprocessing import Process, Queue
import base64
import json
import cv2
import WSServer
def capture(path, outq):
cap = cv2.VideoCapture(path)
while True:
ret, image = cap.read()
outq.put(image)
def conv2jpg(inq, outq):
while True:
image = inq.get()
imgEnc = cv2.imencode('.jpg', image, [cv2.IMWRITE_JPEG_QUALITY, 75])[1].tostring()
outq.put(imgEnc)
def conv2b64(inq, outq):
while True:
imgEnc = inq.get()
strEnc = str(base64.b64encode(imgEnc), 'utf-8')
outq.put(strEnc)
if __name__ == '__main__':
# After arg parsing
ws = WSServer.WSServer(args.ip, args.port)
capQ = Queue(2)
jpgQ = Queue(2)
b64Q = Queue(2)
capP = Process(target=capture, args=(args.input, capQ, ))
# args.input is the rtsp address of the camera
jpgP = Process(target=conv2jpg, args=(capQ, jpgQ, ))
b64P = Process(target=conv2b64, args=(jpgQ, b64Q, ))
b64P.start()
jpgP.start()
capP.start()
while True:
strEnc = b64Q.get()
msg = {'image': strEnc}
msgJson = json.dumps(msg)
# Send the json string over Websocket
USERS = ws.getUSERS()
for user in USERS:
user.put(msgJson)
WSServer.py
import threading
from queue import Queue
import asyncio
import websockets
class WSServer:
def __init__(self, ip="localhost", port="11000", interval=1.0/60):
self.ip = ip
self.port = port
self.interval = interval
self.USERS = set()
self.USERS_Lock = threading.Lock()
self.P = threading.Thread(target=self.proc)
self.P.setDaemon(True)
self.P.start()
async def get(self, q):
if q.empty():
return None
return q.get()
async def register(self, q):
with self.USERS_Lock:
self.USERS.add(q)
async def unregister(self, q):
with self.USERS_Lock:
self.USERS.remove(q)
async def serve(self, websocket, path):
q = Queue()
await self.register(q)
try:
while True:
while True:
msg = await self.get(q)
if type(msg) == type(None):
break
await websocket.send(msg)
await asyncio.sleep(self.interval)
finally:
await self.unregister(q)
def getUSERS(self):
with self.USERS_Lock:
return self.USERS.copy()
def proc(self):
print('Listening...')
loop = asyncio.new_event_loop()
asyncio.set_event_loop(loop)
loop.run_until_complete(websockets.serve(self.serve, self.ip, self.port))
loop.run_forever()
EDIT2: The simplified codes at the frontend:
<!DOCTYPE html>
<html>
<head>
<title>demo</title>
</head>
<body>
<img id="myImg" style="display: None;"></img>
<canvas id="myCanvas">
<script>
var img = document.getElementById('myImg')
var cvs = document.getElementById('myCanvas')
cvs.width = 1920
cvs.height = 1080
var ctx = cvs.getContext('2d')
var ws = new WebSocket("ws://127.0.0.1:11000/");
ws.onmessage = function (event) {
var msg = JSON.parse(event.data)
img.src = 'data:image/jpg;base64,' + msg.image
};
function render () {
ctx.drawImage(img, 0, 0)
window.requestAnimationFrame(render)
}
window.requestAnimationFrame(render)
</script>
</body>
</html>
Related
I am trying to display both an image and a box with an Entry widget. I can do that, but the window is so large that the widget at the bottom is mostly out of view. I have tried several calls to set the window's size or unmaximize it, but they seem to have no effect. I determined that the problem only occurs when the image is large, but still wonder how to display a large image in a resizable window or, for that matter, to make any changes to the window's geometry from code. All the function call I tried seem to have no effect.
Here is my code:
import gi
gi.require_version("Gtk", "3.0")
from gi.repository import Gtk
from gi.repository import GdkPixbuf
from urllib.request import urlopen
class Display(object):
def __init__(self):
self.window = Gtk.Window()
self.window.connect('destroy', self.destroy)
self.window.set_border_width(10)
# a box underneath would be added every time you do
# vbox.pack_start(new_widget)
vbox = Gtk.VBox()
self.image = Gtk.Image()
response = urlopen('http://1.bp.blogspot.com/-e-rzcjuCpk8/T3H-mSry7PI/AAAAAAAAOrc/Z3XrqSQNrSA/s1600/rubberDuck.jpg').read()
pbuf = GdkPixbuf.PixbufLoader()
pbuf.write(response)
pbuf.close()
self.image.set_from_pixbuf(pbuf.get_pixbuf())
self.window.add(vbox)
vbox.pack_start(self.image, False, False, 0)
self.entry = Gtk.Entry()
vbox.pack_start(self.entry, True,True, 0)
self.image.show()
self.window.show_all()
def main(self):
Gtk.main()
def destroy(self, widget, data=None):
Gtk.main_quit()
a=Display()
a.main()
Most of the posted information seems to pertain to Gtk2 rather than Gtk3, but there is a solution: to use a pix buf loader and set the size:
from gi.repository import Gtk, Gdk, GdkPixbuf
#more stuff
path = config['DEFAULT']['datasets']+'working.png'
with open(path,'rb') as f:
pixels = f.read()
loader = GdkPixbuf.PixbufLoader()
loader.set_size(400,400)
loader.write(pixels)
pb = GdkPixbuf.Pixbuf.new_from_file(path)
self.main_image.set_from_pixbuf(loader.get_pixbuf())
loader.close()
As of right now I'm attempting to make a simple music player app that streams music or video directly from a Youtube URL, and in order to do that I need the full download of the search page that's used to search for videos to stream. But I'm having some problems with the urlopen module in python 3, which is what I'm using to make the command application. It won't load the ytd-app tag on Youtube, which is what a good deal of the video and playlist references are put on when you first load the search. Anyone know what's going on, or know some type of workaround for it? Thanks!
My code so far:
BASICURL = "https://www.youtube.com/results?"
query = query.split()
ret = ""
stufffound = {}
for x in query:
ret = ret + x + "+"
ret = (ret[:len(ret)-1])
# URL BUILDER
if filtercriteria:
URL = BASICURL + "sp={0}".format(filtercriteria) + "&search_query={0}".format(ret)
else:
URL = BASICURL + "search_query={0}".format(ret)
query = urlopen(str(URL))
passdict = {}
def findvideosonpage(query,dictToAddTo):
for x in (BS(urlopen(query)).read()).findAll(attrs={'class':'yt-simple-endpoint style-scope ytd-video-renderer'})
dictToAddTo[query.index(x)] = x[href]
print(x)
return list([x for _,x in sorted(zip(dictToAddTo.values(), dictToAddTo.keys()))])
# Dictionary is meant to be converted into a list later to order the results
I am using jupyter notebook to do some mapping visualisation stuff with google maps (via the http://jupyter-gmaps.readthedocs.io/en/latest/gmaps.html library)
I want to be able to use the jupyter dropdown widget to pass it an updated string and get a new map based on this to appear on the page.
All of this works well (code below) but when I run the it, the clear_output() does not work. It works fine if I am not using google maps, like just printing something in the cell once when I change the dropdown.
Does anyone have any idea about this?
import ipywidgets as widgets
import gmaps
import pandas as pd
gmaps.configure(api_key="my api key...")
from IPython.display import display
from IPython.display import clear_output
# global scaling for circle size on map
global SCALING_NORMALIZATION
SCALING_NORMALIZATION = 30
#MAP FUNCTIONS
#function to change circle sizes on map
def create_circle_sizes(values):
scaling_values = []
for i in range(0, len(values)):
a = np.asscalar(values[i])
a = int(a / SCALING_NORMALIZATION)
if a == 0:
scaling_values.append(2)
else:
scaling_values.append(a)
return scaling_values
# function to create hover info on map
def create_hover_info(service_types, names):
hover_info = []
for i in range(0, len(service_types)):
hover_info.append(names[i])
return hover_info
#function to draw a map in gmaps
def create_map(value):
map_df = data[data['lga'] == value][['lat', 'long', 'lic_places', 'sta_name', 'distinct_se_type']]
scaling = map_df['lic_places'].tolist()
names = map_df['sta_name'].tolist()
service_types = map_df['distinct_se_type'].tolist()
map_layer = gmaps.symbol_layer(map_df[['lat', 'long']],
fill_color="blue",
stroke_color="blue",
scale=create_circle_sizes(scaling),
hover_text =
create_hover_info(service_types, names))
fig = gmaps.figure()
fig.add_layer(map_layer)
display(fig)
# WIDGET FUNCTIONS
# function to update the map when dropdown choice is changed
def update_map(args):
# clear_output() doesn't seem to work with map rendering
clear_output()
user_choice = args['new']
create_map(user_choice)
# dropdown widget
dd = widgets.Dropdown(
options=['auburn (c)', 'fairfield (c)', 'penrith (c)'],
value='fairfield (c)',
description='Number:',
disabled=False,
button_style=''
)
#display dropdown widget, with function it will call on change
dd.observe(update_map, 'value')
I am a new pyqtgraph users,try to "Embedding widgets inside PyQt applications"following the instructions in http://www.pyqtgraph.org/documentation/how_to_use.html. in my example I promote Graphics view to PlotWidget, then save as "test2.ui", also follow the "crosshair/mouse interaction" example,my code:
import sys
import numpy
from PyQt5 import QtCore, QtGui,uic,QtWidgets
from PyQt5.QtWidgets import *
import pyqtgraph as pg
import os
hw,QtBaseClass=uic.loadUiType("test.ui")
def gaussian(A, B, x):
return A * numpy.exp(-(x / (2. * B)) ** 2.)
class MyApp(QtWidgets.QMainWindow, hw):
def __init__(self):
super().__init__()
self.setupUi(self)
winSize=self.size()
self.view.resize(winSize.width(),winSize.height())
x = numpy.linspace(-5., 5., 10000)
y =gaussian(5.,0.2, x)
self.p=self.view.plot(x,y)
proxy = pg.SignalProxy(self.view.scene().sigMouseMoved, rateLimit=60, slot=self.mouseMoved)
self.view.enableAutoRange("xy", True)
def mouseMoved(evt):
print("mouseTest")
mousePoint = self.p.vb.mapSceneToView(evt[0])
label.setText(
"<span style='font-size: 14pt; color: white'> x = %0.2f, <span style='color: white'> y = %0.2f</span>" % (
mousePoint.x(), mousePoint.y()))
if __name__ == "__main__":
app = QtWidgets.QApplication(sys.argv)
window = MyApp()
window.show()
sys.exit(app.exec_())
it seems not get the mouse move event;
after change
proxy = pg.SignalProxy(self.view.scene().sigMouseMoved, rateLimit=60, slot=self.mouseMoved)
to
self.view.scene().sigMouseMoved.connect(self.mouseMoved),
output"MouseTest",but program imediatly crash.
can any one give me some help
Two things:
Re: Crashing
It seems as if you haven't placed a label in the GUI to modify, perhaps your code is seeing this and kicks it back to you. If you're using qtDesigner, it is likely defined as self.label, and in my GUI, I was required to use self.label to reference it.
Re: mouseMoved function
I was just struggling with a similar issue of it not working. I was able to get mine to work by changing the evt[0] to simply evt, something I think they moved to from pyqt4 to pyqt5.
Here's an example of what I was able to get to work:
..........setup code above... IN THE setupUi function:
..........setup code above...
Plotted = self.plot
vLine = pg.InfiniteLine(angle=90, movable=False)
hLine = pg.InfiniteLine(angle=0, movable=False)
Plotted.addItem(vLine, ignoreBounds=True)
Plotted.addItem(hLine, ignoreBounds=True)
Plotted.setMouseTracking(True)
Plotted.scene().sigMouseMoved.connect(self.mouseMoved)
def mouseMoved(self,evt):
pos = evt
if self.plot.sceneBoundingRect().contains(pos):
mousePoint = self.plot.plotItem.vb.mapSceneToView(pos)
self.label.setText("<span style='font-size: 15pt'>X=%0.1f, <span style='color: black'>Y=%0.1f</span>" % (mousePoint.x(),mousePoint.y()))
self.plot.plotItem.vLine.setPos(mousePoint.x())
self.plot.plotItem.hLine.setPos(mousePoint.y()
...the if__name__ =="__main__": function .....
In my case, I did not pass the proxy statement, and instead just went for the sigMouseMoved since it already passes the information the proxy would. I think this was in the example in pyqt5 (and commented out) because it was the change. However, the comment didn't specifically state this.
In a script that sends email in HTML format I add an image that is stored in a public shared folder.
I get the blob using UrlFetchApp.fetch(photoLink) but the image has not necessarily the right size so in the html code I use width and height attributes (for now with fixed values, see code below) but I'd like it to be automatically resized with the right ratio.
To achieve that I need to know how to get the original size of the image (height and width) but I just don't know how to get it without inserting the image in an intermediate document (which would work but I find this approach a bit weird and unnecessarily complicated... moreover I don't feel like having a useless doc appearing each time I change the image file).
Here is the relevant part of the code that creates the email message :
function sendMail(test,rowData,genTitle,day,title,stHour,endHour){
var photoLink = sh.getRange('H1').getValue();
var image = UrlFetchApp.fetch(photoLink);
//************* find the pixel size of the image to get its ratio
var msgTemplate = '<IMG SRC="'+photoLink+'" BORDER=0 ALT="logo" HEIGHT=200 WIDTH=300><BR><BR>'+
'Résumé de vos réservations au nom de <NOM><BR><BR><BR><table style="background-color:lightblue;border-collapse:collapse;" border = 1 cellpadding = 5><th></th><th><TABLEHEADER></th><EVENTS></table><BR><CONCLUSION><BR>Cordialement,<BR><BR>';
var mailTitle = 'Confirmation de réservation - '+getTextFromHtml(genTitle);
var descr = '';
for(var d = 0;d<day.length;++d){
Logger.log(Number(rowData[(d+5)]));
var content = '<tr bgcolor="#ffffbb" width="100%"><td><NUMBER> </td><td > <DESCRIPTION></td></tr>'
if(Number(rowData[(d+5)])>1){var pl = ' places'}else{var pl = ' place'};
content = content.replace('<NUMBER>',rowData[(d+5)]+pl);
content = content.replace('<DESCRIPTION>',title[d]+' de '+stHour[d]+' heures à '+endHour[d]+' heures');
if(Number(rowData[(d+5)])>0){
descr += content;
}
}
msgTemplate = msgTemplate.replace('<NOM>',rowData[1]).replace('<EVENTS>',descr).replace('<TABLEHEADER>',genTitle);
var textVersion = getTextFromHtml(msgTemplate.replace(/<br>/gi,'\n').replace(/<td>/gi,'\n'));
// Logger.log(textVersion)
if(test){
MailApp.sendEmail(Session.getEffectiveUser().getEmail(),mailTitle, textVersion,{'htmlBody':msgTemplate,"replyTo" : retour});
}
else
{
MailApp.sendEmail(rowData[2],mailTitle, textVersion,{'htmlBody':msgTemplate,"replyTo" : retour});
}
}
There is no easy way within Apps Script to figure out what an image size would be. There are some other projects that might be able to analyze the bitmap data and give you dimensions.
The last time I had to solve this problem. I just wrote a simple App Engine app to do the image math for me -
import webapp2
from google.appengine.api import urlfetch
from google.appengine.api import images
from django.utils import simplejson
class MainHandler(webapp2.RequestHandler):
def get(self):
url = self.request.get('url')
imgResp = urlfetch.fetch(url) #eg. querystring - url=http://xyz.com/img.jpg
if imgResp.status_code == 200:
img = images.Image(imgResp.content);
jsonResp = {"url":url, "h":img.height, "w":img.width, "format":img.format}
self.response.headers['Content-Type'] = 'application/json'
self.response.out.write(simplejson.dumps(jsonResp))
app = webapp2.WSGIApplication([('/imageinfo', MainHandler)], debug=True)
And then I call it from Apps Script like this -
function checkImageSizes() {
var imageUrls = ['http://developers.google.com/apps-script/images/carousel0.png','http://www.w3.org/MarkUp/Test/xhtml-print/20050519/tests/jpeg420exif.jpg'];
for(var i in imageUrls){
var resp = JSON.parse(UrlFetchApp.fetch('http://arunimageinfo.appspot.com/imageinfo?url='+imageUrls[i]).getContentText());
Logger.log('Image at %s is %s x %s',resp.url,resp.w,resp.h);
}
}
You are welcome to use my App Engine instance if your volume is a couple of times a week :)
I doubt you can do this in apps script. Certainly not natively but you might be able to find or adapt a jpg library that looks at the binary blob header and extracts the image size.