I am using PyCamera module of Raspberry Pi to capture an image and store as .jpg.
first encoding the image using base64.encodestring(). but while sending encoded string to PubNub server, I get error on my_publish_callback as
('ERROR: ', 'Expecting value: line 1 column 1 (char 0)')
('ERROR: ', JSONDecodeError('Expecting value: line 1 column 1 (char 0)',))
I have tried using base64.b64encode() but still get the same errors. I have tried the script in python 2 and 3;
def my_publish_callback(envelope, status):
if not status.is_error():
pass # Message successfully published to specified channel.
else:
#print("recv: ", envelope)
print("ERROR: ", status.error_data.information)
print("ERROR: ", status.error_data.exception)
def publish(channel, msg):
pubnub.publish().channel(channel).message(msg).async(my_publish_callback)
def captureAndSendImage():
camera.start_preview()
time.sleep(2)
camera.capture("/home/pi/Desktop/image.jpg")
camera.stop_preview()
with open("/home/pi/Desktop/image.jpg", "rb") as f:
encoded = base64.encodestring(f.read())
publish(myChannel, str(encoded))
I am not able to find or print full error traceback so that I can get some more clues about where the error is occurring. But it looks like PubNub is trying to parse the data in JSON, and its failing.
I realized the .jpg file size is 154KB, whereas PubNub max packet size is 32KB, so that should clearly say it all. PubNub recommends to send large messages by splitting them and re-arranging them in subscriber-end. Thanks #Craig for referring to that link, Its useful though support.pubnub.com/support/discussions/topics/14000006326
Related
I am very green when it comes to Python - so bare with me.
Using the out of the Box Python3 Sample Code provided by AWS to return the 'SecretString' from the AWS: Secrets Manager Service.
No issues there .. I get the returned object (note I have blanked out some details)
{"username":"postgres","password":"XXXXXXXXX","engine":"postgres","host":"srdataset.XXXXXXXXX.ap-southeast-2.rds.amazonaws.com","port":5432,"dbInstanceIdentifier":"srdataset"}
detail are all correct.
I am then using json.loads() to parse the above into my next function so I can extract the details like so
# request details
login_details = get_secret("pg_srdataset_login_details")
# load json
y = json.loads(login_details)
# extract result is a Python dictionary:
print(y["username"])
This again all works fine in my IDE (PyCharm). I can run the code, build in a Docker container .. and I then use the PyCharm AWS SAM CLI to deploy the code to the cloud .. no issues.
However when I test the function in AWS the code bugs out on the line y = json.loads(login_details) step.
the error being ..
{
"errorMessage": "Expecting value: line 1 column 1 (char 0)",
"errorType": "JSONDecodeError",
"stackTrace": [
" File \"/var/task/update_sp_changes.py\", line 229, in lambda_handler\n y = json.loads(login_details)\n",
" File \"/var/lang/lib/python3.8/json/__init__.py\", line 357, in loads\n return _default_decoder.decode(s)\n",
" File \"/var/lang/lib/python3.8/json/decoder.py\", line 337, in decode\n obj, end = self.raw_decode(s, idx=_w(s, 0).end())\n",
" File \"/var/lang/lib/python3.8/json/decoder.py\", line 355, in raw_decode\n raise JSONDecodeError(\"Expecting value\", s, err.value) from None\n"
]
}
To test this I also copied the JSON 'SecretString' returned from AWS, hard coded it as a variable, then passed this variable directly into the y = json.loads(login_details) step. Tested again and works a treat.
What am I doing wrong - how can I work around this issue.
For reference, I'd share Lambda print debug code.
Preparation:
from the Lambda-py38 initial state, SecretsManagerReadWrite policy is attached to Lamda
set SecretId and VersionId
Output:
json.loads() works for plaintext
print() output almost the same for plain_text:str and secret_dict:dict. (Difference is quotation:single/double. Which are treated as the same string when copy-pasted.)
import json
import boto3
def lambda_handler(event, context):
client = boto3.client('secretsmanager')
response = client.get_secret_value(
SecretId='arn:aws:secretsmanager:xxxxxxxxxxxxxxxxxxxxxxxxxxxxx',
VersionId='xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx',
VersionStage='AWSCURRENT'
)
print("type of response: ", type(response))
print("response: ", response)
plaintext = response['SecretString']
print("type of plaintext: ", type(plaintext))
print("plaintext: ", plaintext)
secret_dict = json.loads(plaintext)
print("type of secret_dict: ", type(secret_dict))
print("secret_dict: ", secret_dict)
return 200
I am sending a JSON file from the cloud to a device using Azure IoT cloud-to-device messages via the Python SDKs.
The file contains lots of new lines and tabs which I would like to preserve. The file received of course must be in the exact same format as the one being sent.
This is on the sending end (cloud) :
FILENAME = "my_file.json"
f = open (FILENAME, "r")
data = f.read()
registry_manager.send_c2d_message(DEVICE_ID, data)
And on the receiving end (device) :
message = client.receive_message()
received_file = open("output.json", "w")
received_file.write(str(message))
received_file.close()
However the file contains just one line with the special characters b' \n \t , and not the actual tabs and new lines etc. Here is just the beginning of it :
b'{\n "group1":\n [\n {\n
How should I get this to format properly and not print the special characters, but instead lines and tabs etc.? Thanks in advance.
I can't seem to catch an exception when using json.loads even though I specifically call it out. I largely see this when trying to stress my server with lots of client connection sending data very quickly. See my error below:
(I've replaced my IP address with X's in the error code)
EX: Unterminated string starting at: line 1 column 49 (char 48) Data:
'{"ap-hdop":0.55,"rtcmin":"38","ap-latdec":3.134,"a' error: uncaptured
python exception, closing channel
(:Unterminated string
starting at: line 1 column 49 (char 48)
[//faraday_server_handler.py|collect_incoming_data|34]
[/usr/lib/python2.7/json/init.py|loads|338]
[/usr/lib/python2.7/json/decoder.py|decode|366]
[/usr/lib/python2.7/json/decoder.py|raw_decode|382])
I understand this that the code fails because I simply miss a double quotes on the line:
'{"ap-hdop":0.55,"rtcmin":"38","ap-latdec":3.134,"a'
This line is usually a LOT longer so that "a.... was supposed to keep going and complete it's quotes.
Here's my relevant code:
def collect_incoming_data(self, data):
"""Read an incoming message from the client, place JSON message data into buffer"""
#self.logger.debug('collect_fing_data() -> (%d bytes)\n"""%s"""', len(data), data)
try:
loaded_data = json.loads(data)
except ValueError, ex:
self.handle_error()
type,value,traceback = sys.exc_info()
print type
#print "Collect Incoming Data: " . time.strftime("%Y-%m-%d %H:%M:%S",time.gmtime())
print "EX: ", ex
print "Data: ",repr(data)
Any ideas? I scoured the internet to see if I can find this issue, but I appear to be setting up to capture the exception which everyone else having this issue with loads seems to suggest to do.
EDIT 3/1/2016 - Evening
Commenting out my exception handle_error() let me see more of the error:
except ValueError, ex:
self.handle_error()
type,value,traceback = sys.exc_info()
print type
#print "Collect Incoming Data: " . time.strftime("%Y-%m-%d %H:%M:%S",time.gmtime())
print "EX: ", ex
print "Data: ",repr(data)
Below is my new error, I've commented out personal data. It's apparent that the issue I really have now is in-fact the unterminated string
EX: Unterminated string starting at:
line 1 column 49 (char 48) Data:
'{"ap-hdop":0.55,"rtcmin":"31","ap-latdec":XX.XXX,"a' EX: No JSON object could be decoded Data:
'p-latdeg":34,"adc6":2006,"adc7":2007,"adc4":2004,"adc5":2005,"adc2":2002,"adc3":2003,"adc0":2000,"adc1":2001,"gpio-0":30,"gpio-1":50,"gpio-2":70,"speed":5.0,"adc8":2008,"rtcday":"01","longdeg":118,"longdec":XX.XXX,"altitude":31.0,"ap-speed":0.0,"ap-pdop":0.77,"lat-dir":"N","long-dir":"W","hdop":0.01,"ap-rf":0,"alt-units":"M","rtcdow":"2","callsign":"XXXXX","ap-callsign":"XXXXX","id":1,"ap-id":1,"rtcyear":"2016","rtcmon":"03","ap-vdop":0.66,"ap-lat-dir":"N","vdop":0.02,"rtchour":"22","latdec":XX.XXX,"latdeg":34,"ap-longdeg":118,"ap-longdec":XX.XXX,"rtcsec":"15","ap-altitude":86.0,"ap-long-dir":"W","pdop":0.01,"ap-alt-units":"M","faraday-port":0}'
OK my original question was answered which was "Why am I not catching the ValueError exception even though I am providing code to do just that?"
#tadhg McDonald-jensen was correct with his comment to remove my call to handle_error().
I still have some other issues but they are a different question. Thanks!
I am receiving this error in Python 3.5.1.
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
Here is my code:
import json
import urllib.request
connection = urllib.request.urlopen('http://python-data.dr-chuck.net/comments_220996.json')
js = connection.read()
print(js)
info = json.loads(str(js))
If you look at the output you receive from print() and also in your Traceback, you'll see the value you get back is not a string, it's a bytes object (prefixed by b):
b'{\n "note":"This file .....
If you fetch the URL using a tool such as curl -v, you will see that the content type is
Content-Type: application/json; charset=utf-8
So it's JSON, encoded as UTF-8, and Python is considering it a byte stream, not a simple string. In order to parse this, you need to convert it into a string first.
Change the last line of code to this:
info = json.loads(js.decode("utf-8"))
in my case, some characters like " , :"'{}[] " maybe corrupt the JSON format, so use try json.loads(str) except to check your input
ok I am trying to create a definition which will read a list of IDS from an external Json file, Which it is doing. Its even putting the data into the database on load of the program, my issue is this. I cant seem to match the list IDs to a comparison. Here is my current code:
def check(account):
global ID_account
import json, httplib
if not hasattr(BigWorld, 'iddata'):
UID_DB = account['databaseID']
UID = ID_account
try:
conn = httplib.HTTPConnection('URL')
conn.request('GET', '/ids.json')
conn.sock.settimeout(2)
resp = conn.getresponse()
qresp = resp.read()
BigWorld.iddata = json.loads(qresp)
LOG_NOTE('[ABRO] Request of URL data successful.')
conn.close()
except:
LOG_NOTE('[ABRO] Http request to URL problem. Loading local data.')
if UID_DB is not None:
list = BigWorld.iddata["ids"]
#print (len(list) - 1)
for n in range(0, (len(list) - 1)):
#print UID_DB
#print list[n]
if UID_DB == list[n]:
#print '[ABRO] userid located:'
#print UID_DB
UID = UID_DB
else:
LOG_NOTE('[ABRO] userid not set.')
if 'databaseID' in account and account['databaseID'] != UID:
print '[ABRO] Account not active in database, game closing...... '
BigWorld.quit()
now my json file looks like this:
{
"ids":[
"1001583757",
"500687699",
"000000000"
]
}
now when I run this with all the commented out prints it seems to execute perfectly fine up till it tries to do the match inside the for loop. Even when the print shows UID_DB and list[n] being the same values, it does not set my variable, it doesn't post any errors, its just simply acting as if there was no match. am I possibly missing a loop break? here is the python log starting with the print of the length of the table print:
INFO: 2
INFO: 1001583757
INFO: 1001583757
INFO: 1001583757
INFO: 500687699
INFO: [ABRO] Account not active, game closing......
as you can see from the log, its never printing the User located print, so it is not matching them. its just continuing with the loop and using the default ID I defined above the definition. Anyone with an idea would definitely help me out as ive been poking and prodding this thing for 3 days now.
the answer to this was found by #VikasNehaOjha it was missing simply a conversion to match types before the match comparison I did this by adding in
list[n] = int(list[n])
that resolved my issue and it finally matched comparisons.