Fixed web socket endpoint for Chrome CDP (Chrome DevTools Protocol) - google-chrome

I use Chrome CDP for some tasks automation.
One have to first start the chrome with CDP:
chromium-browser --remote-debugging-port=9222
and it reports something like
DevTools listening on ws://127.0.0.1:9222/devtools/browser/3e3152c6-20fc-4cea-a9d2-60e4e6b8ad70
I have to copy the ws://... URL to my config file manually to be able to proceed with my task. I probably can work around this using python's subprocess.Popen to do this instead and extract the URL but isn't there a way how to make this URL configurable or at least fixed?

Thanks to wOxxOm! It really can be read from http://127.0.0.1:9222/json/version (Documentation)
As an alternative, I wrote Python script to launch it and get the endpoint as well:
from subprocess import Popen, PIPE
class Browser:
BANNER = "DevTools listening on "
def __init__(self, path="/usr/bin/chromium-browser",
port=9222, ignore_tls_errors=False):
cmd = [path, f"--remote-debugging-port={port}"]
if ignore_tls_errors:
cmd.append("--ignore-certificate-errors")
self.process = Popen(cmd, stdout=PIPE, stderr=PIPE, universal_newlines=True)
output = ""
for line in self.process.stderr:
output += line
if self.BANNER in output:
start_pos = output.find(self.BANNER) + len(self.BANNER)
end_pos = output.find("\n", start_pos)
self.url = output[start_pos:end_pos]
break
def close(self):
self.process.terminate()
if __name__ == "__main__":
try:
b = Browser()
print("URL:", b.url)
finally:
b.close()

Related

Creating an exe - file for data exchange with server in Tcl

I am completely lost and I do not know how to approach the following problem which my boss assigned to me.
I have to create an exe - file containing a code which works as follows when I run it: It sends a certain file, say file_A, to a server. When the server receives this file it sends back a json-file, say file_B, which contains an url. More precisely, the attribute of the json-file contains the url. The code should then open the url in a browser.
And here are the details:
The above code (one version in tcl) must accept three parameters and a fourth optional parameter (so, it is not necessary to pass a fourth parameter). The three parameters are: server, type and file.
server: this is the path to the server. For example, https://localhost:0008.
type: this is the type of the file (file_A) to be send to the server: xdt / png
file: this is the path to the file (file_A) to be send to the server.
The fourth optional parameter is:
wksName: if this paramater is given, then the url should be opened with it in the browser.
I got an example code for the above procedure written in python. It should serve as an orientation. I do not know anything about this language but to a large extend I understand the code. It looks as follows:
#!/usr/bin/env python
# -*- coding: utf-8 -*-
import platform
import sys
import webbrowser
import requests
args_dict = {}
for arg in sys.argv[1:]:
if '=' in arg:
sep = arg.find('=')
key, value = arg[:sep], arg[sep + 1:]
args_dict[key] = value
server = args_dict.get('server', 'http://localhost:0008')
request_url = server + '/NAME_List_number'
type = args_dict.get('type', 'xdt')
file = args_dict.get('file', 'xdtExamples/xdtExample.gdt')
wksName = args_dict.get('wksName', platform.node())
try:
with open(file, 'rb') as f:
try:
r = requests.post(request_url, data={'type': type}, files={'file': f})
request_url = r.json()['url'] + '?wksName=' + wksName
webbrowser.open(url = request_url, new = 2)
except Exception as e:
print('Error:', e)
print('Request url:', request_url)
except:
print('File \'' + file + '\' not found')
As you can see, the crucial part of the above code is this:
try:
with open(file, 'rb') as f:
try:
r = requests.post(request_url, data={'type': type}, files={'file': f})
request_url = r.json()['url'] + '?wksName=' + wksName
webbrowser.open(url = request_url, new = 2)
except Exception as e:
print('Error:', e)
print('Request url:', request_url)
except:
print('File \'' + file + '\' not found')
Everything else above it are just definitions. If possible, I would like to translate the above code into tcl. Could you please help me with this issue?
It doesn't have to be a 1-1 "tcl-translation" as long as it works as described above, and hopefully as simple as the above one.
I am not familiar with concepts such as sending/receiving data to/from servers, reading json-files etc.
Any help is welcome.

User to download file to their local directory

Backend - I wrote a python script that creates a csv file after some aggregation.
Frontend - Once the method finished running and the .csv file is generated and saved to a directory in the server, I want to be able to prompt the user to save the .csv file on their local computer (just like the windows prompt you get when you press "save as..." on a webpage).
This is an example of what I've done so far from what I learned in Return Excel file in Flask app and Download a file when button is pressed on web application? :
Sample code:
with open(save_path + unique_filename + ".csv", 'w', encoding = 'utf8') as g:
writer = csv.writer(g, lineterminator = '\n')
writer.writerow(['name', 'place', 'location'])
HTML:
#app.route('/login', method='POST')
def do_login():
category = request.forms.get('category')
return '''
<html><body>
Hello. Save Results
</body></html>
'''
#app.route("/getCSV", methods = ['GET', 'POST'])
def getPlotCSV():
return send_from_directory(save_path + unique_filename + ".csv", as_attachment=True)
if __name__ == "__main__":
run(app, host = 'localhost', port = 8000)
My questions are:
1) send_from_directory is from flask, what is the bottle equivalent?
2) Where in the code do I place the csv I created so the user can download it to their local machine?
3) What else is wrong with my code?
Bottle example: From https://bottlepy.org/docs/dev/tutorial.html
#route('/download/<filename:path>')
def download(filename):
return static_file(filename, root='/path/to/static/files', download=filename)

Working with coroutines in Python Tornado Web Server

I am working on an autonomous car implementation for a web browser game with Python 2x. I use Tornado Web Server to run game on localhost and I post and receive data from game with JSON data format in the function called "FrameHandler" and also I determine what the act of car should be in "to_dict_faster()" function.
Here, my problem is that I can write data to text file which is hold in speed_data variable in specific time interval with help of a coroutine. However, I can't dump JSON data to function in this specific time interval because "FrameHandler" acts like While True and it always requests data to dump. What I am trying to do is sending desired acts as writing text file in specific time interval while not changing flow frame handler because it affects FPS of the game.
I am trying to figure out How can I do that for a long time any help would be great here:
#gen.coroutine
def sampler():
io_loop = tornado.ioloop.IOLoop.current()
start = time.time()
while True:
with open("Sampled_Speed.txt", "a") as text_file:
text_file.write("%d,%.2f\n" % (speed_data, ((time.time() - start))))
yield gen.Task(io_loop.add_timeout, io_loop.time() + period)
class MainHandler(tornado.web.RequestHandler):
def get(self):
self.redirect("/static/v2.curves.html")
class FrameHandler(tornado.web.RequestHandler):
def post(self):
global speed_data
data = json.loads(self.get_arguments("telemetry")[0])
ar = np.fromstring(base64.decodestring(self.request.body), dtype=np.uint8)
image = ar.reshape(hp.INPUT_SIZE, hp.INPUT_SIZE, hp.NUM_CHANNELS)
left, right, faster, slower = data["action"]
terminal, action, all_data, was_start = (
data["terminal"],
Action(left=left, right=right, faster=faster, slower=slower),
data["all_data"],
data["was_start"]
)
for i in range(len(all_data)):
data_dict=all_data[i]
speed_data = data_dict[u'speed']
position_data=data_dict[u'position']
result_action = agent.steps(image, 0.1, terminal, was_start, action, all_data)
if speed_data < 4000:
self.write(json.dumps(result_action.to_dict_faster()))
else:
self.write(json.dumps(result_action.to_dict_constant()))
def make_app():
return tornado.web.Application([
(r"/", MainHandler),
(r"/frame", FrameHandler),
(r"/static/(.*)", tornado.web.StaticFileHandler, {"path": static_path})
], debug=True)
if __name__ == "__main__":
app = make_app()
if "SERVER_PORT" in os.environ:
port = int(os.environ["SERVER_PORT"])
else:
port = 8880
print "LISTENING ON PORT: %d" % port
app.listen(port)
tornado.ioloop.IOLoop.current().run_sync(sampler)
tornado.ioloop.IOLoop.current().start()
You can move file writing to a different thread (using tornado's run_on_executor for example), so python interpreter will automatically switch from Sampler to main thread with FrameHandler on write. But you have to use thread-safe speed_data variable, I've used stdlib Queue.Queue as an example:
class Handler(tornado.web.RequestHandler):
#gen.coroutine
def get(self):
global speed_data
speed_data.put("REALLY BIG TEST DATA\n")
self.finish("OK")
class Sampler():
executor = concurrent.futures.ThreadPoolExecutor(max_workers=1)
def __init__(self, queue):
self._q = queue
#run_on_executor
def write_sample(self):
with open("foobar.txt", "w") as f:
while True:
data = self._q.get()
f.write(data)
if __name__ == '__main__':
application = Application(
[("/status", Handler)]
)
server = HTTPServer(application)
server.listen(8888)
speed_data = Queue.Queue()
smp = Sampler(speed_data)
IOLoop.current().add_callback(smp.write_sample)
IOLoop.current().start()

Python crashes running assemble_data.py for Flickr Style dataset

I'm trying to download the Flickr Style dataset using assemble_data.py provided in the examples folder. However, whenever I run this python crashes with error 'python quit unexpectedly'.
It seems to be related to multiprocessing and urllib. When I replace pool.map with a single threaded loop it works but is very slow. Also, if I run with multiprocessing but remove urlretrieve it seems to work too.
Answering my own question here... I resolved this by using urllib3 instead.
http = urllib3.PoolManager(10)
def download_image(args_tuple):
"For use with multiprocessing map. Returns filename on fail."
url, filename = args_tuple
try:
if not os.path.exists(filename):
print url + ' -> ' + filename
# Dont redirect.
response = http.request('GET', url, redirect=False)
with open(filename, 'wb') as f:
f.write(response.data)
with open(filename) as f:
assert hashlib.sha1(f.read()).hexdigest() != MISSING_IMAGE_SHA1
test_read_image = io.imread(filename)
return True
except KeyboardInterrupt:
raise Exception() # multiprocessing doesn't catch keyboard exceptions
except:
os.remove(filename)
return False
Gist here.

Preferred method for downloading a file generated on the fly in Flask

I have a page that displays a list of files in a directory. When the user clicks on the Download button, all of these files are zipped into a single file, which is then offered for download. I know how to send this file to the browser when the button is clicked, and I know how to reload the current page (or redirect to a different one), but is it possible to do both in the same step? Or would it make more sense to redirect to a different page with a download link?
My download is initiated with the Flask API's send_from_directory. Relevant test code:
#app.route('/download', methods=['GET','POST'])
def download():
error=None
# ...
if request.method == 'POST':
if download_list == None or len(download_list) < 1:
error = 'No files to download'
else:
timestamp = dt.now().strftime('%Y%m%d:%H%M%S')
zfname = 'reports-' + str(timestamp) + '.zip'
zf = zipfile.ZipFile(downloaddir + zfname, 'a')
for f in download_list:
zf.write(downloaddir + f, f)
zf.close()
# TODO: remove zipped files, move zip to archive
return send_from_directory(downloaddir, zfname, as_attachment=True)
return render_template('download.html', error=error, download_list=download_list)
Update: As a workaround, I am now loading a new page with the button click, which lets the user initiate the download (using send_from_directory) before returning to the updated listing.
Are you running the flask app behind a front end web server such as nginx or apache (which would be the best way to handle the downloading of files). If you're using nginx you can use the 'X-Accel-Redirect' header. For this example I'll use the directory /srv/static/reports as the directory you're creating the zipfiles in and wanting to serve them out of.
nginx.conf
in the server section
server {
# add this to your current server config
location /reports/ {
internal;
root /srv/static;
}
}
your flask method
send the header to nginx to server
from flask import make_response
#app.route('/download', methods=['GET','POST'])
def download():
error=None
# ..
if request.method == 'POST':
if download_list == None or len(download_list) < 1:
error = 'No files to download'
return render_template('download.html', error=error, download_list=download_list)
else:
timestamp = dt.now().strftime('%Y%m%d:%H%M%S')
zfname = 'reports-' + str(timestamp) + '.zip'
zf = zipfile.ZipFile(downloaddir + zfname, 'a')
for f in download_list:
zf.write(downloaddir + f, f)
zf.close()
# TODO: remove zipped files, move zip to archive
# tell nginx to server the file and where to find it
response = make_response()
response.headers['Cache-Control'] = 'no-cache'
response.headers['Content-Type'] = 'application/zip'
response.headers['X-Accel-Redirect'] = '/reports/' + zf.filename
return response
If you're using apache, you can use their sendfile directive http://httpd.apache.org/docs/2.0/mod/core.html#enablesendfile