ESP32: Odd behavior with urequests - object with buffer protocol required - json

I am trying to wear out a battery to see how long it powers my ESP32. To do this, I wrote a simple program that writes "Still alive" to a website once a minute. However, the program crashes every time it has written the string successfully to the web file. I can't figure out why it works nine times, then fails. Here is the code:
from time import sleep_ms, ticks_ms
import network
import socket
import urequests
import machine
import json
SSID="my_ssid"
PASSWORD="my_password"
port=100
wlan=None
s=None
def connectWifi(ssid,passwd): #function to connect to the Web
global wlan #declare a WLAN object
wlan=network.WLAN(network.STA_IF) #create a wlan object
wlan.active(True) #Activate the network interface
wlan.disconnect() #Disconnect the last connected WiFi
wlan.connect(ssid,passwd) #connect wifi
while(wlan.ifconfig()[0]=='0.0.0.0'): #wait for connection
sleep_ms(1)
sleep_ms(1000) #hold on for 1 second
sendmessage("Connected to WLAN")
sleep_ms(1000) #hold on for 1 second
return True
def sendmessage(myMessage):
url = "http://www.sabulo.com/stillalive.php"
headers = {'content-type': 'application/json'}
data = {'message': myMessage}
jsonObj = json.dumps(data)
resp = urequests.post(url, data=jsonObj, headers=headers)
return True
def main():
connectWifi(SSID,PASSWORD)
hitcount = 0
while True:
sendmessage("Still alive " + str(hitcount))
print("Still alive " + str(hitcount))
hitcount = hitcount + 1
sleep_ms(1000)
main()
Then I get this:
>>>
>>> I (5906) phy: phy_version: 4007, 9c6b43b, Jan 11 2019, 16:45:07, 0, 0
Still alive 0
Still alive 1
Still alive 2
Still alive 3
Still alive 4
Still alive 5
Still alive 6
Still alive 7
Still alive 8
╝Traceback (most recent call last):
File "<stdin>", line 45, in <module>
File "<stdin>", line 40, in main
File "<stdin>", line 33, in sendmessage
File "urequests.py", line 111, in post
File "urequests.py", line 56, in request
OSError: 23
╝>
MicroPython v1.10-98-g4daee3170 on 2019-02-14; ESP32 module with ESP32
Type "help()" for more information.
I am using the same web upload code piece in various apps and it never gives me this error.
Any help gratefully acknowledged :)

Related

Sonoff 3.6.0 DIY v2.0

I bought a Sonoff Basic R3 to use use in DIY mode with a Raspberry Pi. The firmware I've confirmed is 3.6.0 so DIY v2.0. The DIY connection process works fine using 10.10.7.1 and it connects to the router and I have assigned it a static IP address. I can ping it no problem.
Ping statistics for 10.0.0.35:
Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
Minimum = 3ms, Maximum = 97ms, Average = 53ms
But when I run the Raspberry Pi (connected to the same router with the same subnet etc) http post json code
import requests
import json
#API details
url = "https://10.0.0.35:8081/zeroconf/switch"
body = {"deviceid": "", "data": {"switch": "on"} }
headers = {'Content-Type': 'application/json'}
#Making http post request
response = requests.post(url, headers=headers, data=body, verify=False)
print(response)
It just hangs up until I keyboard interrupt (Ctrl-C). I'm pretty sure it is some sort of connection/port issue as the outcome of the code is the same whether or not the device is powered up (I've also tried a number of variations of the json post code using info, switch: off etc all with the same outcome).
^CTraceback (most recent call last):
File "/home/pi/Programmes/sonoff3.py", line 10, in <module>
response = requests.post(url, headers=headers, data=body, verify=False)
File "/usr/lib/python3/dist-packages/requests/api.py", line 119, in post
return request('post', url, data=data, json=json, **kwargs)
File "/usr/lib/python3/dist-packages/requests/api.py", line 61, in request
return session.request(method=method, url=url, **kwargs)
File "/usr/lib/python3/dist-packages/requests/sessions.py", line 542, in request
resp = self.send(prep, **send_kwargs)
File "/usr/lib/python3/dist-packages/requests/sessions.py", line 655, in send
r = adapter.send(request, **kwargs)
File "/usr/lib/python3/dist-packages/requests/adapters.py", line 439, in send
resp = conn.urlopen(
File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 699, in urlopen
httplib_response = self._make_request(
File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 445, in _make_request
six.raise_from(e, None)
File "<string>", line 3, in raise_from
File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 440, in _make_request
httplib_response = conn.getresponse()
File "/usr/lib/python3.9/http/client.py", line 1347, in getresponse
response.begin()
File "/usr/lib/python3.9/http/client.py", line 307, in begin
version, status, reason = self._read_status()
File "/usr/lib/python3.9/http/client.py", line 268, in _read_status
line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
File "/usr/lib/python3.9/socket.py", line 704, in readinto
return self._sock.recv_into(b)
KeyboardInterrupt
So after 8 hours of trying yesterday and another few today I'm at a loss so any pointers as to what else to try would be great. Thanks.

'NoneType' object has no attribute 'read' when reading from JSON file

I am making a script for a school project that requires that I receive a JSON file that tells me if a license plate is visible in a picture. Right now the code sends a POST with an image to an API that then gives me a JSON in return, that JSON data is sent to the file "lastResponse.json."
The code that is giving out the error
with open('lastResponse.json', 'r+') as fp:
f = json.dump(r.json(), fp, sort_keys=True, indent=4) # Where the response data is sent to the JSON
data = json.load(f) # Line that triggers the error
print(data["results"]) # Debug code
print("------------------") # Debug code
print(data) # Debug code
# This statement just checks if a license plate is visible
if data["results"]["plate"] is None:
print("No car detected!")
else:
print("Car with plate number '" + data["results"]["plate"] + "' has been detected")
The Error
Traceback (most recent call last):
File "DetectionFinished.py", line 19, in <module>
data = json.load(f)
File "/usr/lib/python3.7/json/__init__.py", line 293, in load
return loads(fp.read(),
AttributeError: 'NoneType' object has no attribute 'read'
I am not very experienced in Python so I would appreciate explanations!
It turns out, after rereading the API's documentation and using their examples I was able to fix my issues
import requests
from pprint import pprint
regions = ['gb', 'it']
with open('/path/to/car.jpg', 'rb') as fp:
response = requests.post(
'https://api.platerecognizer.com/v1/plate-reader/',
data=dict(regions=regions), # Optional
files=dict(upload=fp),
headers={'Authorization': 'Token API_TOKEN'})
pprint(response.json())

Native-Messaging firefox extension example "ping_pong" not working because of TypeError

I'm trying to use native-messaging in a firefox extension. I tried to build the example from this https://developer.mozilla.org/en-US/docs/Mozilla/Add-ons/WebExtensions/Native_messaging
I copy/pasted all the code and set it up as described on the page exactly and it sends a "Ping" but doesn't receive a "Pong" back. The browser console says TypeError: a bytes-like object is required, not 'str'on line 17 in the python application. What can I do?
I use windows 7 and python 3.x
The web-extension is sending a json object to the application, which then tests the length and struct.unpacks the message. If the message is "ping" it tries to struct.pack and json.dumps a response "pong" which the web-extension receives as a response. Both messages and any errors get written into the console.log.
it says in the example:
Note that running python with the -u flag is required on Windows,
in order to ensure that stdin and stdout are opened in binary, rather
than text, mode.
And I did set up the .bat starter script to include the -u flag, but it appears that python still reads the stdin as a string which then gives the TypeError while trying to struct.unpack.
web-extension background.js for sending ping:
/*
On startup, connect to the "ping_pong" app.
*/
var port = browser.runtime.connectNative("ping_pong");
/*
Listen for messages from the app.
*/
port.onMessage.addListener((response) => {
console.log("Received: " + response);
});
/*
On a click on the browser action, send the app a message.
*/
browser.browserAction.onClicked.addListener(() => {
console.log("Sending: ping");
port.postMessage("ping");
});
python application to receive ping and send pong:
#!/usr/bin/python -u
# Note that running python with the `-u` flag is required on Windows,
# in order to ensure that stdin and stdout are opened in binary, rather
# than text, mode.
import json
import sys
import struct
# Read a message from stdin and decode it.
def get_message():
raw_length = sys.stdin.read(4)
if not raw_length:
sys.exit(0)
message_length = struct.unpack('=I', raw_length)[0]
message = sys.stdin.read(message_length)
return json.loads(message)
# Encode a message for transmission, given its content.
def encode_message(message_content):
encoded_content = json.dumps(message_content)
encoded_length = struct.pack('=I', len(encoded_content))
return {'length': encoded_length, 'content': encoded_content}
# Send an encoded message to stdout.
def send_message(encoded_message):
sys.stdout.write(encoded_message['length'])
sys.stdout.write(encoded_message['content'])
sys.stdout.flush()
while True:
message = get_message()
if message == "ping":
send_message(encode_message("pong"))
This is the line which gives the TypeError:
message_length = struct.unpack('=I', raw_length)[0]
The log should say:
Sending: ping
Received: pong
The log actually says:
Sending: ping
stderr output from native app ping_pong: Traceback (most recent call last):
stderr output from native app ping_pong: File "C:\\Users\\ping_pong\\ping_pong.py", line 37, in <module>
stderr output from native app ping_pong: message = get_message()
stderr output from native app ping_pong: File "C:\\Users\\ping_pong\\ping_pong.py", line 17, in get_message
stderr output from native app ping_pong: message_length = struct.unpack('=I', raw_length)[0]
stderr output from native app ping_pong: TypeError: a bytes-like object is required, not 'str'
Using Python 2 it works fine for me (on Linux).
#!/usr/bin/python2 -u
or
path\to\python2
in the windows batch file, respectively.
To make it work in Python 3 you need to pack the encoded string into a struct (I don't know why - but hey it works):
#!/usr/bin/python -u
# Note that running python with the `-u` flag is required on Windows,
# in order to ensure that stdin and stdout are opened in binary, rather
# than text, mode.
import json
import sys
import struct
# Read a message from stdin and decode it.
def get_message():
# use buffer to get bytes
raw_length = sys.stdin.buffer.read(4)
#raise ValueError(raw_length)
if not raw_length:
sys.exit(0)
message_length = struct.unpack('=I', raw_length)[0]
message = sys.stdin.buffer.read(message_length).decode("utf-8")
return json.loads(message)
# Encode a message for transmission, given its content.
def encode_message(message_content):
encoded_content = json.dumps(message_content).encode("utf-8")
encoded_length = struct.pack('=I', len(encoded_content))
return {'length': encoded_length, 'content': struct.pack(str(len(encoded_content))+"s",encoded_content)}
# Send an encoded message to stdout.
def send_message(encoded_message):
#raise ValueError(encoded_message)
sys.stdout.buffer.write(encoded_message['length'])
sys.stdout.buffer.write(encoded_message['content'])
sys.stdout.buffer.flush()
while True:
message = get_message()
if message == "ping":
send_message(encode_message("pong"))

CSV error opening file

I am getting an error opening a file that I can't resolve. I am able to open
this exact file with no issues using another small program I wrote.
First Program (doesn't work):
import csv
passwd = "f:\mark\python\etc_password.txt"
output = "f:\mark\python\output.txt"
with open(passwd, 'r') as passwd1, open(output, 'w') as output1:
ro = csv.reader(passwd1, delimiter=':')
wo = csv.writer(output1, delimiter='\t')
for record in ro:
# if not record[0].startswith('#'):
if len(record) > 1:
wo.writerow((record[0], record[2]))
Error:
Traceback (most recent call last):
File "C:/Users/Mark/PycharmProjects/main/main.py", line 11, in <module>
for record in ro:
ValueError: I/O operation on closed file.
Second Program (works):
etcfile = "f:\mark\python\etc_password.txt"
users = {}
with open(etcfile, "r") as datafile:
for line in datafile:
if not line.startswith("#"):
info = line.split(':')
users[info[0]] = info[2]
for username in sorted(users):
print("{}:{}".format(username, users[username]))
The first program has the issue that I can't figure out. The second program works just fine opening the same file.
The error ValueError: I/O operation on closed file. is telling you
you cannot read from a closed file. If you look at the indentation of your
first program, you are opening a csv reader to a file which is then closed
at the end of the with block. A simpler example of this behavior would be
In [1]: import csv
In [2]: file = open('test.csv')
In [3]: ro = csv.reader(file)
In [4]: file.close()
In [5]: for record in ro:
...: print(record)
...:
---------------------------------------------------------------------------
ValueError Traceback (most recent call last)
<ipython-input-5-1f7adaf76d31> in <module>()
----> 1 for record in ro:
2 print(record)
3
ValueError: I/O operation on closed file.

Selenium + Xvfb concurrency issues

test.py
#!/usr/bin/env python2
import os
from pyvirtualdisplay import Display
from selenium import webdriver
import sys
display = Display(visible=0, size=(800, 600))
display.start()
try:
capabilities = webdriver.common.desired_capabilities.DesiredCapabilities.CHROME
browser = webdriver.Chrome('/opt/chrome-driver/chromedriver', desired_capabilities=capabilities)
try:
browser.get('http://example.com')
browser.execute_script('return 1 + 1;')
finally:
browser.quit()
finally:
display.stop()
print 'Done'
And then run
seq 1 20 | parallel -j 5 ./test.py
I get a random number of errors like this:
Traceback (most recent call last):
File "./test.py", line 15, in <module>
browser.execute_script('return 1 + 1;')
File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/remote/webdriver.py", line 429, in execute_script
{'script': script, 'args':converted_args})['value']
File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/remote/webdriver.py", line 201, in execute
self.error_handler.check_response(response)
File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/remote/errorhandler.py", line 194, in check_response
raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.WebDriverException: Message: chrome not reachable
(Session info: chrome=50.0.2661.86)
(Driver info: chromedriver=2.21.371461 (633e689b520b25f3e264a2ede6b74ccc23cb636a),platform=Linux 3.19.0-59-generic x86_64)
If I remove the concurrency (parallel -j 1), it works.
If I remove Xvfb, it works.
What does this error mean, and how can I fix it (without giving up concurrency or virtual displays)?
Looks like starting xvfb has concurrency issues when starting. xvfb-run unreliable when multiple instances invoked in parallel
So I can fix this by requiring that xfvb sessions are started serially.
#!/usr/bin/env python2
import fcntl
import os
from pyvirtualdisplay import Display
from selenium import webdriver
import sys
import xvfbwrapper
display = None
browser = None
try:
with open('/tmp/xvfb.lock', 'w') as lock:
fcntl.lockf(lock, fcntl.LOCK_EX)
display = Display(visible=0, size=(800, 600))
display.start()
capabilities = webdriver.common.desired_capabilities.DesiredCapabilities.CHROME
browser = webdriver.Chrome('/opt/chrome-driver/chromedriver', desired_capabilities=capabilities)
browser.get('http://example.com')
browser.execute_script('return 1 + 1;')
finally:
if browser:
browser.quit()
if display:
display.stop()
print 'Done'
As long as everybody uses /tmp/xvfb.lock to control starting a Xvfb session starting, this works.
P.S. I don't think the selenium driver starting needed to be serialized too, but I did run into a transient problem that I think was fixed by this.