I have a Django app. In the view I call another function (in stats.py) which then makes a HTTP POST.
views.py
from stats import Stat
a = Stat(example="12345")
a.use(id='query')
stats.py
self.data = { example : "12345" }
req = urllib2.Request(api_url)
req.add_header('Content-Type', 'application/json')
response = urllib2.urlopen(req, json.dumps(self.data))
The problem that occurs is that I get the error,
<django.utils.functional.SimpleLazyObject object at 0x2b4d1fe47650> is not JSON serializable
Django Traceback
From looking at the Django Traceback I get the following,
/prod/tools/lx/views.py in update_input
a.use(id='query')
...
/prod/tools/main/stats.py in log_use
response = urllib2.urlopen(req, json.dumps(self.data))
...
/usr/local/lib/python2.7/json/__init__.py in dumps
return _default_encoder.encode(obj)
...
/usr/local/lib/python2.7/json/encoder.py in encode
chunks = self.iterencode(o, _one_shot=True)
...
/usr/local/lib/python2.7/json/encoder.py in iterencode
return _iterencode(o, 0)
...
/usr/local/lib/python2.7/json/encoder.py in default
raise TypeError(repr(o) + " is not JSON serializable")
...
Any ideas ?
Thanks,
Try this using urllib
import urllib
...
self.data = { example : "12345", 'Content-type':'application/json' }
self.data = urllib.urlencode(self.data)
req = urllib2.Request(api_url, self.data)
response = urllib2.urlopen(req)
Related
I'm trying to put python objects into a JSON file by getting the API from one of the sites but somehow when I run the code nothing has been put in the JSON file. API is working well, as well when I print out the code by json.load I get the output but I have no idea why does dump doesn't work.
here is my code:
from django.shortcuts import render
import requests
import json
import datetime
import re
def index(request):
now = datetime.datetime.now()
format = "{}-{}-{}".format(now.year, now.month, now.day)
source = []
author = []
title = []
date = []
url = "http://newsapi.org/v2/everything"
params = {
'q': 'bitcoin',
'from': format,
'sortBy': 'publishedAt',
'apiKey': '1186d3b0ccf24e6a91ab9816de603b90'
}
response = requests.request("GET", url, params=params)
for news in response.json()['articles']:
matching = re.match("\d+-\d+-\d+", news['publishedAt'])
if format == matching.group():
source.append(news['source'])
author.append(news['author'])
title.append(news['title'])
date.append(news['publishedAt'])
data = \
{
'source': source,
'author': author,
'title': title,
'date': date
}
with open('data.json', "a+") as fp:
x = json.dump(data, fp, indent=4)
return render(request, 'news/news.html', {'response': response})
Before sending the data I am using JSON.stringify to the data and it looks like this
{"data": [{"key1": value1, "key2": value2}, {"key1": value1, "key2": value2}]}
But once it passes through AWS API Gateway and Kinesis Firehose puts it to S3 it looks like this
{
"key1": value1,
"key2": value2
}{
"key1": value1,
"key2": value2
}
The seperator comma between the JSON objects are gone but I need it to process data properly.
Template in the API Gateway:
#set($root = $input.path('$'))
{
"DeliveryStreamName": "some-delivery-stream",
"Records": [
#foreach($r in $root.data)
#set($data = "{
""key1"": ""$r.value1"",
""key2"": ""$r.value2""
}")
{
"Data": "$util.base64Encode($data)"
}#if($foreach.hasNext),#end
#end
]
}
I had this same problem recently, and the only answers I was able to find were basically just to add line breaks ("\n") to the end of every JSON message whenever you posted them to the Kinesis stream, or to use a raw JSON decoder method of some sort that can process concatenated JSON objects without delimiters.
I posted a python code solution which can be found over here on a related Stack Overflow post:
https://stackoverflow.com/a/49417680/1546785
One approach you could consider is to configure data processing for your Kinesis Firehose delivery stream by adding a Lambda function as its data processor, which would be executed before finally delivering the data to the S3 bucket.
DeliveryStream:
...
Type: AWS::KinesisFirehose::DeliveryStream
Properties:
DeliveryStreamType: DirectPut
ExtendedS3DestinationConfiguration:
...
BucketARN: !GetAtt MyDeliveryBucket.Arn
ProcessingConfiguration:
Enabled: true
Processors:
- Parameters:
- ParameterName: LambdaArn
ParameterValue: !GetAtt MyTransformDataLambdaFunction.Arn
Type: Lambda
...
And in the Lambda function, ensure that '\n' is appended to the record's JSON string, see below the Lambda function myTransformData.ts in Node.js:
import {
FirehoseTransformationEvent,
FirehoseTransformationEventRecord,
FirehoseTransformationHandler,
FirehoseTransformationResult,
FirehoseTransformationResultRecord,
} from 'aws-lambda';
const createDroppedRecord = (
recordId: string
): FirehoseTransformationResultRecord => {
return {
recordId,
result: 'Dropped',
data: Buffer.from('').toString('base64'),
};
};
const processData = (
payloadStr: string,
record: FirehoseTransformationEventRecord
) => {
let jsonRecord;
// ...
// Process the orginal payload,
// And create the record in JSON
return jsonRecord;
};
const transformRecord = (
record: FirehoseTransformationEventRecord
): FirehoseTransformationResultRecord => {
try {
const payloadStr = Buffer.from(record.data, 'base64').toString();
const jsonRecord = processData(payloadStr, record);
if (!jsonRecord) {
console.error('Error creating json record');
return createDroppedRecord(record.recordId);
}
return {
recordId: record.recordId,
result: 'Ok',
// Ensure that '\n' is appended to the record's JSON string.
data: Buffer.from(JSON.stringify(jsonRecord) + '\n').toString('base64'),
};
} catch (error) {
console.error('Error processing record ${record.recordId}: ', error);
return createDroppedRecord(record.recordId);
}
};
const transformRecords = (
event: FirehoseTransformationEvent
): FirehoseTransformationResult => {
let records: FirehoseTransformationResultRecord[] = [];
for (const record of event.records) {
const transformed = transformRecord(record);
records.push(transformed);
}
return { records };
};
export const handler: FirehoseTransformationHandler = async (
event,
_context
) => {
const transformed = transformRecords(event);
return transformed;
};
Once the newline delimiter is in place, AWS services such as Athena will be able to work properly with the JSON record data in the S3 bucket, not just seeing the first JSON record only.
Once AWS Firehose dumps the JSON objects to s3, it's perfectly possible to read the individual JSON objects from the files.
Using Python you can use the raw_decode function from the json package
from json import JSONDecoder, JSONDecodeError
import re
import json
import boto3
NOT_WHITESPACE = re.compile(r'[^\s]')
def decode_stacked(document, pos=0, decoder=JSONDecoder()):
while True:
match = NOT_WHITESPACE.search(document, pos)
if not match:
return
pos = match.start()
try:
obj, pos = decoder.raw_decode(document, pos)
except JSONDecodeError:
# do something sensible if there's some error
raise
yield obj
s3 = boto3.resource('s3')
obj = s3.Object("my-bukcet", "my-firehose-json-key.json")
file_content = obj.get()['Body'].read()
for obj in decode_stacked(file_content):
print(json.dumps(obj))
# { "key1":value1,"key2":value2}
# { "key1":value1,"key2":value2}
source: https://stackoverflow.com/a/50384432/1771155
Using Glue / Pyspark you can use
import json
rdd = sc.textFile("s3a://my-bucket/my-firehose-file-containing-json-objects")
df = rdd.map(lambda x: json.loads(x)).toDF()
df.show()
source: https://stackoverflow.com/a/62984450/1771155
please use this code to solve your issue
__Author__ = "Soumil Nitin Shah"
import json
import boto3
import base64
class MyHasher(object):
def __init__(self, key):
self.key = key
def get(self):
keys = str(self.key).encode("UTF-8")
keys = base64.b64encode(keys)
keys = keys.decode("UTF-8")
return keys
def lambda_handler(event, context):
output = []
for record in event['records']:
payload = base64.b64decode(record['data'])
"""Get the payload from event bridge and just get data attr """""
serialize_payload = str(json.loads(payload)) + "\n"
hasherHelper = MyHasher(key=serialize_payload)
hash = hasherHelper.get()
output_record = {
'recordId': record['recordId'],
'result': 'Ok',
'data': hash
}
print("output_record", output_record)
output.append(output_record)
return {'records': output}
I use django-rest-framework first time. I use json format by default.
I need to send file for creating a new instance. This file has being saved in models.ImageField. However, I don't know which format this field requires for incoming file. I tried to send it in base64, but it isn't suitable.
TestCase for this problem:
class PersonTestCase(APITestCase):
def setUp(self):
self.c = APIClient()
def test_sign_up_with_valid_data(self):
with open('persons/test/bg.jpg', 'rb') as bg:
valid_registration_data = {
...,
'background': base64.b64encode(bg.read()).decode('utf-8'),
}
response = self.c.post('/persons/sign_up', valid_registration_data)
self.assertEqual(response.status_code, 201)
self.assertEqual(Person.objects.count(), 1)
self.assertEqual(Person.objects.get('username'), 'test_client74562984')
View:
class SignUpView(APIView):
"""
Create new user
"""
def post(self, request):
serializer = UserSerializer(data=request.data)
if serializer.is_valid():
serializer.save()
response = Response(serializer.data, status=status.HTTP_201_CREATED)
return response
response = Response(serializer.errors, status=status.HTTP_400_BAD_REQUEST)
return response
I am trying to build a REST service which accepts XML and convert it into JSON and call external Service which accepts JSON and put my JSON into it. I am able to put the json without pretty but I want to PUT the json in pretty format. Please suggest how to do, below is my code ...
package com.mypackge
import grails.converters.JSON
import grails.rest.RestfulController
import grails.plugins.rest.client.RestBuilder
class RestCustomerController extends RestfulController {
/*
static responseFormats = ['json', 'xml']
RestCustomerController() {
super(Customer)
}
*/
def index() {
convertXmlToJson()
}
def myJson = ''
def convertXmlToJson() {
def xml = ''' <Customer>
<customerid>9999999999999</customerid>
<ssn>8888</ssn>
<taxid>8888</taxid>
<address>
<addressline1>Yamber Ln</addressline1>
<addressline1>8664 SE</addressline1>
<city>CCCCC</city>
<state>CC</state>
<zipcode>97679</zipcode>
</address>
<firstname>Scott</firstname>
<middlename></middlename>
<lastname>David</lastname>
<account>
<accountno>576-294738943</accountno>
<accounttype>Lease</accounttype>
<accountsubtype></accountsubtype>
<accountstatus>complete</accountstatus>
<firstname>Scott</firstname>
<middlename></middlename>
<lastname>David</lastname>
<businessname></businessname>
<billingsystem>yoiuhn</billingsystem>
<brand></brand>
<plantype></plantype>
<billingaddress>
<addressline1>Yamber Ln</addressline1>
<addressline1>8664 SE </addressline1>
<city>CCCCC</city>
<state>CC</state>
<zipcode>97679</zipcode>
</billingaddress>
<job>
<jobid>8276437463728</jobid>
<jobstatus>SUCCESS</jobstatus>
</job>
</account>
</Customer>
'''.stripMargin()
// Parse it
def parsed = new XmlParser().parseText( xml )
def myId = parsed.customerid.text()
// Deal with each node:
def handle
handle = { node ->
if( node instanceof String ) {
node
}
else {
[ (node.name()): node.collect( handle ) ]
}
}
// Convert it to a Map containing a List of Maps
def jsonObject = [ (parsed.name()): parsed.collect { node ->
[ (node.name()): node.collect( handle ) ]
} ]
def json = new groovy.json.JsonBuilder(jsonObject) //.toPrettyString()
// Check it's what we expected
def mmyresp
try{
mmyresp = putRequest(myId,json)
}catch(Exception e) {
mmyresp = 'Please Validate JSON ....'
}
}
def putRequest(String id, JSON myJson) {
String url = "http://foo.com/customer/external/"+id
def rest = new RestBuilder()
def resp = rest.put(url){
contentType "application/json"
json{
myJson
}
}
return resp
}
}
The record is added in below format ...
{"Customer":[{"customerid":["9999999999999"]},{"ssn":["8888"]},
{"taxid":["8888"]},{"address":[{"addressline1":["Yamber Ln"]},
{"addressline1":["8664 SE"]},{"city":["CCCCC"]},{"state":["CC"]},{"zipcode":["97679"]}]},
{"firstname":["Scott"]},{"middlename":[]},{"lastname":["David"]},{"businessname":[]},
{"account":[{"accountno":["576-294738943"]},{"accounttype":["Lease"]},{"accountsubtype":[]},
{"accountstatus":["complete"]},{"firstname":["Scott"]},{"middlename":[]},{"lastname":["David"]},
{"businessname":[]},{"billingsystem":["yoiuhn"]},{"brand":[]},{"plantype":[]},
{"billingaddress":[{"addressline1":["Yamber Ln"]},{"addressline1":["8664 SE"]},
{"city":["CCCCC"]},{"state":["CC"]},{"zipcode":["97679"]}]},{"job":[{"jobid":["8276437463728"]},
,{"jobstatus":["SUCCESS"]}]}]}]}
But I want this to be inserted in pretty format. I tried .toPrettyString() but got casting exception when try to put as json. I am trying the REST services for the first time, not sure where I am doing wrong. Please suggest me on this.
You should set following field in you Config.groovy.
grails.converters.default.pretty.print = true
This will pretty print for both the xml and json.
you could optionally set it up for xml or json only like below:
For json:
grails.converters.json.pretty.print = true
For xml
grails.converters.xml.pretty.print = true
A sample of Config.groovy entry is:
environments {
development {
grails.converters.json.pretty.print = true
}
}
Hope it helps!!!
For Grails 4, try this:
def json = x as JSON
json.prettyPrint = true;
log.info(json.toString())
I am trying to Extract JSON data from URL with parameters to it and code is as follows:
import json
from flask import Flask, render_template, request, jsonify
import requests
app = Flask(__name__)
import urllib2
#app.route("/",methods=['GET','POST'])
def home():
if request.method == 'POST':
#user inputs
valueone= request.form.get('first')
valueTwo = request.form.get('second')
valueThree = request.form.get('third')
#api call
data = {"perfid" : {0}, "section" : {"hostname" : {1}, "iteration" : {2}, "sectionname" : "sysstat_M"}}
req = urllib2.Request('http://api-latx-dev.corp.netapp.com/ws/spm/spm-general', json.dumps(data))
response = urllib2.urlopen(req)
the_page = response.read()
url=response.getcode()
returnData = {}
if url.status_code == 200:
returnData["status"] = "SUCCESS"
returnData["result"] = the_page
return jsonify(returnData)
else:
returnData["status"] = "ERROR"
return jsonify(returnData)
#jsonify(response.json())
return render_template('index.html')
I need to fetch data But getcode() gives int which is not working with "if url.status_code == 200".Can anyone please suggest how to deal with it.