I have a Dash app with a standard layout which I run with standard invocation:
#wsgi.py
from __init__ import app
from __init__ import server
if __name__ == "__main__":
app.run_server(port=5000, debug = True)
I would like to run a once-a-day updater script for some calculations with a general schematic like this:
#one of the application pages .py
def update_df_daily_loop():
while True:
if time since last update > 24h:
run update
last updated = now
else:
sleep for 15 minutes
#tried invoking the thread like this - but it blocks the server from starting
thread0 = threading.Thread(target=update_df_daily_loop)
thread0.start()
thread0.join()
Where should I put the invokation? If I just run it as a thread it blocks the server from ever starting.
Related
I have a function interpolate_to_particles written in c and wrapped with ctypes. I want to use dask.delayed to make a series of calls to this function.
The code runs successfully without dask
# Interpolate w/o dask
result = interpolate_to_particles(arg1, arg2, arg3)
and with the distributed schedular in single-threaded mode
# Interpolate w/ dask
from dask.distributed import Client
client = Client()
result = dask.delayed(interpolate_to_particles)(arg1, arg2, arg3)
result_c = result.compute(scheduler='single-threaded')
but if I instead call
result_c = result.compute()
I get the following KeyError:
> Traceback (most recent call last): File
> "/path/to/lib/python3.6/site-packages/distributed/worker.py",
> line 3287, in dumps_function
> result = cache_dumps[func] File "/path/to/lib/python3.6/site-packages/distributed/utils.py",
> line 1518, in __getitem__
> value = super().__getitem__(key) File "/path/to/lib/python3.6/collections/__init__.py",
> line 991, in __getitem__
> raise KeyError(key) KeyError: <function interpolate_to_particles at 0x1228ce510>
The worker logs accessed from the dask dashboard do not provide any information. Actually, I do not see any information that the workers have done anything besides starting up.
Any ideas on what could be occurring, or suggested tools that I can use to further debug? Thanks!
Given your comments it sounds like your function does not serialize well. To test this, you might try pickling the function in one process, and try unpickling it in another.
>>> import pickle
>>> print(pickle.dumps(interpolate_to_particles))
b'some bytes printed out here'
And then in another process
>>> import pickle
>>> interpolate_to_particles = pickle.loads(b'the same bytes you had before')
If this doesn't work then you'll know that that's your problem. I would encourage you to look up "how to make sure that ctypes functions are serializable" or something similar, or ask another question with that smaller scope here on Stack Overflow.
I am attempting to read a binary file in Cython. Previously this was working in Python, but I am looking to speed up the process. This code below was written as a familiarisation and logic check before writing the complete module. Once this section is complete the code will be expanded to read in multiple 400 Mb files and process.
A function was created that opens the file, reads in a number of data point and returns them to an array.
from libc.stdlib cimport malloc, free
from libc.stdio cimport fopen, fclose, FILE, fscanf, fread
def readin_binary(filename, int number_of_points):
"""
Test reading in a file and returning data
"""
header_bytes = <unsigned char*>malloc(number_of_points)
filename_byte_string = filename.encode("UTF-8")
cdef FILE *in_binary_file
in_binary_file = fopen(filename_byte_string, 'rb')
if in_binary_file is NULL:
print("file not found")
else:
print("Read file {}".format(filename))
fread(&header_bytes, 1, number_of_points, in_binary_file)
fclose(in_binary_file)
return header_bytes
print(hDVS.readin_binary(filename, 10))
The code compiles.
When the code is run the following error occurs:
Python has stopped working error
I've been playing with this for a few days now. I think there is a simple error but I can not see it. Any ideas?
My app is written in Python 2.7 and wxPython 2.8, and accesses a MySQL database. I have had issues with the program freezing when doing an add via:-
cursor.execute(sql ([idSession, TestDateTime, DataBLOb]))
Although this is in try: - except: construct it never executes the except portion. I have run this section from the command line and got:-
_mysql_exceptions.OperationalError: (2006, 'MySQL server has gone away')
Obviously I need to investigate the cause of the error but how can I get my software to execute the except: code rather than just freeze!
To solve this, you need to create a new Thread. Look the code:
from threading import Thread
class Spam(wx.Frame):
def doStuff(self, arg1,arg2,arg3)
# Do Something...
try:
cursor.execute(sql ([idSession, TestDateTime, DataBLOb]))
except Exception a e:
# Do the treatments or show a dialog
def ButtonClick(self, event): # Assuming a button like "do this operation"
t = Thread(target=self.doStuff,args=(arg1,arg2,arg3))
t.start()
Observation:
The function doStuff need to be free of errors, if some error raised and not is treated will create a deadlock, or a zombie thread.
I have a C code which takes a file as input, processes it and gives a number as output. I want to build a html webpage which takes the file path as input, gives it to the C code. C code processes it and output(integer) is displayed in browser. Can you please suggest me how to go about this? Are there are any prebuilt softwares to do this?
If C code is used to produce a command line utility then you could call it while generating a web page:
#!/usr/bin/env python
import subprocess
from bottle import request, route, run, template # http://bottlepy.org/
command = ['wc', '-c'] # <-- XXX put your command here
#route('/')
def index():
filename = request.query.filename or 'default' # query: /?filename=<filename>
output = subprocess.check_output(command + [filename]) # run the command
return template("""<dl>
<dt>Input</dt>
<dd>{{filename}}</dd>
<dt>Output</dt>
<dd>{{output}}</dd></dl>""", filename=filename, output=output)
run(host='localhost', port=8080)
Run this script or paste it into a Python console, then open your browser and pass a filename (a path on the server) as a query parameter:
$ python -mwebbrowser http://localhost:8080/?filename=/etc/passwd
wc -c prints number of bytes for each input file. It is executed on the server.
If C code is available as a library; you could use ctypes module to call a C function from Python e.g., to call printf() C function from libc library:
#!/usr/bin/env python
import ctypes
from ctypes.util import find_library
try:
libc = ctypes.cdll.msvcrt # Windows
except OSError:
libc = ctypes.cdll.LoadLibrary(find_library('c'))
n = libc.printf("abc ")
libc.printf("%d", n)
I have a view which queries a model to populate a form:
class AddServerForm(forms.Form):
…snip…
# Compile and present choices for HARDWARE CONFIG
hwChoices = HardwareConfig.objects.\
values_list('name','description').order_by('name')
hwChoices = [('', '----- SELECT ONE -----')] +\
[ (x,'{0} - {1}'.format(x,y)) for x,y in hwChoices ]
hwconfig = forms.ChoiceField(label='Hardware Config', choices=hwChoices)
…snip…
def addServers(request, template="manager/add_servers.html",
template_success="manager/add_servers_success.html"):
if request.method == 'POST':
# …snip… - process the form
else:
# Page was called via GET - use a default form
addForm = AddServerForm()
return render_to_response(template, dict(addForm=addForm),
context_instance=RequestContext(request))
Additions to the HardwareConfig model are done using the admin interface. Changes show up immediately in the admin interface as expected.
Running the query via the shell returns all results as expected:
michael#victory> python manage.py shell
Python 2.6 (r26:66714, Feb 21 2009, 02:16:04)
[GCC 4.3.2 [gcc-4_3-branch revision 141291]] on linux2
Type "help", "copyright", "credits" or "license" for more information.
(InteractiveConsole)
>>> from serverbuild.base.models import HardwareConfig
>>> hwChoices = HardwareConfig.objects.\
... values_list('name','description').order_by('name')
hwChoices now contains the complete set of results.
However, loading the addServers view (above) returns the old result set, missing the newly-added entries.
I have to restart the webserver in order for the changes to show up which makes it seem as though that query is being cached somewhere.
I'm not doing any explicit caching anywhere (grep -ri cache /project/root returns nothing)
It's not the browser caching the page - inspected via chrome tools, also tried using a different user & computer
What's going wrong and how do I fix it?
Versions:
MySQLdb: 1.2.2
django: 1.2.5
python: 2.6
hwChoices is evaluated when the form is defined - ie when the process starts.
Do the calculation in the form's __init__ method instead.