I keep getting this error in my lambda function:
{"errorMessage": "module initialization error"}
This happens when i try to turn the following string containing json data into a json dictionary object within python.
"{\n\"main\": {\n \"PART_NAME\": \"Genuine Cardboard Honda Wing\",\n \"BRAND\": \"Honda\",\n \"MJR_CAT\": \"Aero\",\n \"CAT\": \"Rear Wing\",\n \"SUB_CAT\": \"NA\",\n \"Power_Increase\": \"0\"\n},\n\"forza\":\n{\n \"power\": \"[0, True]\",\n \"Torque\": \"[0, True]\",\n \"Traction\": \"[50, True]\",\n \"Handling\": \"[100, True]\",\n \"Breaking\": \"[40, True]\"\n},\n\"custom\": {\n\"length\": 120,\n\"car max height[m]\": 2,\n\"RICER RANK\": -10\n\n}\n"
Here is my code to replicate this error:
client = boto3.client('s3')
result = client.get_object(Bucket=BUCKET, Key=FILE_TO_READ)
text = result['Body'].read().decode('utf-8')
text = json.load(text)
print(text)
without the print(text) it produces that string above.
Thanks :)
Here is the full lambda function (though not commented) if you are interested.
import json
import boto3
print('got this far')
BUCKET = '******'
FILE_TO_READ = 'example_honda_wing.json'
client = boto3.client('s3')
result = client.get_object(Bucket=BUCKET, Key=FILE_TO_READ)
text = result['Body'].read().decode('utf-8')
#text = str(text).replace("\n","")
#text = text.replace('\"',' ')
#text = json.load(text)
print(text) # Use your desired JSON Key for your value
def lambda_handler(event, context):
# TODO implement
return text
Related
Basic Information
I am creating a python script that can encrypt and decrypt a file with previous session data.
The Problem
I am able to decrypt my file and read it using a key. This returns a bytes string which I can in turn convert to a string. However, this string needs to be converted to a dictionary, which I cannot do. Using ast, json and eval I have run into errors.
Bytes string
decrypted = fernet.decrypt(encrypted)
String
string = decrypted.decode("UTF-8").replace("'", '"')
If I use eval() or ast.literal_eval() I get the following error:
Then I tried using json.loads() and I get the following error:
The information blocked out on both images is to protect my SSH connections. In the first image it is giving me a SyntaxError at the last digit of my ip address.
The Function
The function that is responsible for this when called looks like this:
def FileDecryption():
with open('enc_key.key', 'rb') as filekey:
key = filekey.read()
filekey.close()
fernet = Fernet(key)
with open('saved_data.txt', 'rb') as enc_file:
encrypted = enc_file.read()
enc_file.close()
decrypted = fernet.decrypt(encrypted)
print(decrypted)
string = decrypted.decode("UTF-8").replace("'", '"')
data = f'{string}'
print(data)
#data = eval(data)
data = json.loads(data)
print(type(data))
for key in data:
#command_string = ["load", data[key][1], data[key][2], data[key][3], data[key][4]]
#SSH.CreateSSH(command_string)
print(key)
Any help would be appreciated. Thanks!
Your data seems like it was written incorrectly in the first place, but without a complete example hard to say.
Here's a complete example that round-trips a JSON-able data object.
# requirement:
# pip install cryptography
from cryptography.fernet import Fernet
import json
def encrypt(data, data_filename, key_filename):
key = Fernet.generate_key()
with open(key_filename, 'wb') as file:
file.write(key)
fernet = Fernet(key)
encrypted = fernet.encrypt(json.dumps(data).encode())
with open(data_filename, 'wb') as file:
file.write(encrypted)
def decrypt(data_filename, key_filename):
with open(key_filename, 'rb') as file:
key = file.read()
fernet = Fernet(key)
with open(data_filename, 'rb') as file:
return json.loads(fernet.decrypt(file.read()))
data = {'key1': 'value1', 'key2': 'value2'}
encrypt(data, 'saved_data.txt', 'enc_key.key')
decrypted = decrypt('saved_data.txt', 'enc_key.key')
print(decrypted)
Output:
{'key1': 'value1', 'key2': 'value2'}
This line:
else:
#add to this
nutrients_totals_df = pd.read_json(total_nutrients_json, orient='split')
is throwing the error.
I write my json like:
nutrients_json = nutrients_df.to_json(date_format='iso', orient='split')
Then I stash it in a hidden div or dcc.Storage in one callback and get it in another callback. How do I fix this error?
When I read json files that i've written with Pandas, I use the function below and call inside of json.loads().
def read_json_file_from_local(fullpath):
"""read json file from local"""
with open(fullpath, 'rb') as f:
data = f.read().decode('utf-8')
return data
df = json.loads(read_json_file_from_local(fullpath))
I am using a python flask server API to call another python flask server API. Both these have been generated using swagger editor.
API2 returns an encoded string such as:
def Server2_API2():
payload = {
"value1": 'somedata',
"value2": 'somedata',
}
encoded_payload = jwt.encode(payload, SECRET, algorithm=ALGORITHM)
return encoded_payload <-- {str}'eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.'
API1 on server 1 calls API2 on server 2 as follows:
def Server1_API1():
resp = requests.post('http://localhost:1000/API2', data='value1=one&value2=two')
if resp.status_code == 200:
first_try = resp.content <-- b'"eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9"\n'
second_try = resp.content.decode() <-- {str} '"eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9"
' <- notice the new line here
return first_try
#return second_try
If I return first_try then I get the error: Object of type 'bytes' is not JSON serializable
If I return second_try then I get the following data returned:
"\"eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9\"\n"
My goal is to have just the data returned as follows without any error or any back slashes or new lines:
e.g. "eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9"
How can I achieve this?
SOLUTION:
third_try = resp.json() <-- This does the trick :)
This did the trick:
token = resp.json()
Using Python3, Django, and Django Rest Frameworks.
Previously I had a test case that was making a POST to an endpoint.
The payload was a dictionary:
mock_data = {some data here}
In order to send the data over the POST I was doing:
mock_data = base64.b64encode(json.dumps(mock_data).encode('utf-8'))
When doing the POST:
response = self.client.post(
'/some/path/here/', {
...,
'params': mock_data.decode('utf-8'),
},
)
And on the receiving end, in validate() method I was doing:
params = data['params']
try:
params_str = base64.b64decode(params).decode('utf-8')
app_data = json.loads(params_str)
except (UnicodeDecodeError, ValueError):
app_data = None
That was all fine, but now I need to use some hmac validation, and the json I am passing can no longer be a dict - its ordering would change each time, so hmac.new(secret, payload, algorithm) would be different.
I was trying to use a string:
payload = """{data in json format}"""
But when I am doing:
str_payload = payload.encode('utf-8')
b64_payload = base64.b64encode(str_payload)
I cannot POST it, and getting an error:
raise TypeError(repr(o) + " is not JSON serializable")
TypeError: b'ewog...Cn0=' is not JSON serializable
even if I do b64_payload.decode('utf-8') like before, still getting similar error:
raise TypeError(repr(o) + " is not JSON serializable")
TypeError: b'\xa2\x19...\xff' is not JSON serializable
Some python in the terminal:
>>> d = {'email': 'user#mail.com'} # regular dictionary
>>> d
{'email': 'user#mail.com'}
>>> dn = base64.b64encode(json.dumps(d).encode('utf-8'))
>>> dn
b'eyJlbWFpbCI6ICJ1c2VyQG1haWwuY29tIn0='
>>> s = """{"email": "user#mail.com"}""" # string
>>> s
'{"email": "user#mail.com"}'
>>> sn = base64.b64encode(json.dumps(s).encode('utf-8'))
>>> sn
b'IntcImVtYWlsXCI6IFwidXNlckBtYWlsLmNvbVwifSI=' # different!
>>> sn = base64.b64encode(s.encode('utf-8'))
>>> sn
b'eyJlbWFpbCI6ICJ1c2VyQG1haWwuY29tIn0=' # same
The issue is resolved.
The problem was NOT with the payload, but with a parameter I was passing:
signature': b'<signature'>
Solved by passing the signature field like this:
'signature': signature_b64.decode('utf-8')
Thank you!
I have the following code in my scrapy spider:
def parse(self, response):
jsonresponse = json.loads(response.body_as_unicode())
htmldata = jsonresponse["html"]
for sel in htmldata.xpath('//li/li'):
-- more xpath codes --
yield item
But i am having this error:
raise ValueError("No JSON object could be decoded")
exceptions.ValueError: No JSON object could be decoded
After checking the json reply, i found out about **<!--WPJM-->** and **<!--WPJM_END-->** which is causing this error.
<!--WPJM-->{"found_jobs":true,"html":"<html code>","max_num_pages":3}<!--WPJM_END-->
How do i parse my scrapy without looking at the !--WPJM-- and !--WPJM_END-- code?
EDIT: This is the error that i have:
File "/home/muhammad/Projects/project/project/spiders/crawler.py", line 150, in parse
for sel in htmldata.xpath('//li'):
exceptions.AttributeError: 'unicode' object has no attribute 'xpath'
def parse(self, response):
rawdata = response.body_as_unicode()
jsondata = rawdata.replace('<!--WPJM-->', '').replace('<!--WPJM_END-->', '')
# print jsondata # For debugging
# pass
data = json.loads(jsondata)
htmldata = data["html"]
# print htmldata # For debugging
# pass
for sel in htmldata.xpath('//li'):
item = ProjectjomkerjaItem()
item['title'] = sel.xpath('a/div[#class="position"]/div[#id="job-title-job-listing"]/strong/text()').extract()
item['company'] = sel.xpath('a/div[#class="position"]/div[#class="company"]/strong/text()').extract()
item['link'] = sel.xpath('a/#href').extract()
The easiest approach would be to get rid of the comments tags manually using replace():
data = response.body_as_unicode()
data = data.replace('<!--WPJM-->', '').replace('<!--WPJM_END-->', '')
jsonresponse = json.loads(data)
Though it is not quite pythonic and reliable.
Or, a better option would to be to get the text() by xpath:
$ scrapy shell index.html
>>> response.xpath('//text()').extract()[0]
u'{"found_jobs":true,"html":"<html code"}'