Problem printing json data from python script - json

I have a python script that should print json data.
This is what I have in my script:
finaldata = {
"date": datetime.datetime.utcnow().isoformat(),
"voltage_mv":emeter["voltage_mv"],
"current_ma":emeter["current_ma"],
"power_mw":emeter["power_mw"] ,
"energy_wh": emeter["total_wh"],
}
print(finaldata)
I am running the script from Node-RED because I need to send the data to a storage account (in json format of course). The Problem is that the data that is being sent looks like this:
{'power_mw': 0, 'date': '2019-04-16T07:12:19.858159', 'energy_wh': 2, 'voltage_mv': 225045, 'current_ma': 20}
when it should look like this in order to be correctly stored in my storage account:
{"power_mw": 0, "date": '2019-04-16T07:12:19.858159', "energy_wh": 2, "voltage_mv": 225045, "current_ma": 20}
(important for later use, since I already get errors in the storage account).
Does anyone know why this is happening and how I can fix it? Thanks in advance

You should use the python json module and dump your python dict into json:
import json
finaldata = {"power_mw": 0, "date": '2019-04-16T07:12:19.858159',
"energy_wh": 2, "voltage_mv": 225045, "current_ma": 20}
print(json.dumps(finaldata))
JSON Reference
For order check linked OrderedDict
or read the OrderedDict collection Reference

Related

How to take any CSV file and convert it to JSON?(with python as a script engine) [Novice user trying to learn NiFi]

1) There is a CSV file containing the following information (the first row is the header):
first,second,third,total
1,4,9,14
7,5,2,14
3,8,7,18
2) I would like to find the sum of individual rows and generate a final file with a modified header. The final file should look like this:
[
{
"first": 1,
"second": 4,
"third": 9,
"total": 14
},
{
"first": 7,
"second": 5,
"third": 2,
"total": 14
},
{
"first": 3,
"second": 8,
"third": 7,
"total": 18
}
]
But it does not work and I am not sure how to fix this. Can anyone provide me an understanding on how to approach this problem?
NiFi flow:
Although i'm not into Python, by just googling around i think this might do it:
import csv
with open("YOURFILE.csv") as f:
reader = csv.DictReader(f)
data = [r for r in reader]
import json
with open('result.json', 'w') as outfile:
json.dump(data, outfile)
You can use Query Record processor and add new property as
total
select first,second,third,first+second+third total from FLOWFILE
Configure the CsvReader controller service with matching avro schema with int as datatype for all the fields and Json Setwriter controller service,Include total field name so that the output from Query Record processor will be all the columns and the sum of the columns as total.
Connect total relationship from Query Record processor for further processing
Refer to these links regarding Query Record and Configure Record Reader/Writer

AWS X-Ray Python SDK get_service_graph

I am trying to get JSON using get_service_graph() provided by AWS X-Ray Python SDK in AWS Lambda function. reference link
import boto3
from datetime import datetime
def lambda_handler(event, context):
client = boto3.client('xray')
response1 = client.get_service_graph(
StartTime=datetime(2017, 5, 20, 12, 0),
EndTime=datetime(2017, 5, 20, 18, 0)
)
return response1
However, when I passed StartTime and EndTime parameters, stack trace reports datetime type is not JSON serializable. I even tried the following way.
response1 = client.get_service_graph(
StartTime="2017-05-20 00:00:00",
EndTime="2017-05-20 02:00:00"
)
What's weird is, if EndTime is set as "2017-05-20 01:00:00", there is no error generated. Other than that, the same error occurred.
{
"stackTrace": [
[
"/usr/lib64/python2.7/json/__init__.py",
251,
"dumps",
"sort_keys=sort_keys, **kw).encode(obj)"
],
[
"/usr/lib64/python2.7/json/encoder.py",
207,
"encode",
"chunks = self.iterencode(o, _one_shot=True)"
],
[
"/usr/lib64/python2.7/json/encoder.py",
270,
"iterencode",
"return _iterencode(o, 0)"
],
[
"/var/runtime/awslambda/bootstrap.py",
104,
"decimal_serializer",
"raise TypeError(repr(o) + \" is not JSON serializable\")"
]
],
"errorType": "TypeError",
"errorMessage": "datetime.datetime(2017, 5, 20, 1, 53, 13, tzinfo=tzlocal()) is not JSON serializable"
}
I did try only use date, like datetime(2017, 5, 20). However, if I use two consecutive days as StartTime and EndTime, the runtime complains the interval can't be more than 6 hours. If I use same date, it only returns empty JSON. I don't know how to get granularity of get_service_graph().
I think Python SDK for AWS X-Ray might be premature, but I'd still like to seek help from someone who had the same experience. Thanks!
the right way is using datetime(2017, 5, 20) not a string... but can you try using only date... without time? at least the AWS docs shows an example exactly like yours but only yyyy-mm-dd without time

Importing JSON file into Firebase error

I keep getting that there is an error uploading/importing my JSON file into Firebase. I initially had an excel spreadsheet that I saved as a CSV file, then I used a CSV to JSON converter.
I validated the JSON file (which have the .json extension) with a couple of online tools.
Though, I'm still getting an error.
Here is an example of my JSON:
{
"Rk": 1,
"Tm": "SEA",
"H/A": "H",
"DOW": "Sun",
"Opp": "CLE",
"QB": "Russell Wilson",
"Grade": "BLUE",
"Def mu pts": 4,
"Inj status": 0,
"Notes": "Got to wonder if not having a proven power RB under center will negatively impact Wilson's production.",
"TFS $50K": "$8,300",
"Init sal": "$8,300",
"Var": "$0",
"WC": 0
}
The issue is your key's..
Firebase keys must be:
UTF-8 encoded, cannot contain . $ # [ ] / or ASCII control characters
0-31 or 127
your $50k key and the H/A are the issues.

I m trying to use 'ffprobe' with Java or groovy

As per my understanding "ffprobe" will provide file related data in JSON format. So, I have installed the ffprobe in my Ubuntu machine but I don't know how to access the ffprobe JSON response using Java/Grails.
Expected response format:
{
"format": {
"filename": "/Users/karthick/Documents/videos/TestVideos/sample.ts",
"nb_streams": 2,
"nb_programs": 1,
"format_name": "mpegts",
"format_long_name": "MPEG-TS (MPEG-2 Transport Stream)",
"start_time": "1.430800",
"duration": "170.097489",
"size": "80425836",
"bit_rate": "3782576",
"probe_score": 100
}
}
This is my groovy code
def process = "ffprobe -v quiet -print_format json -show_format -show_streams HelloWorld.mpeg ".execute()
println "Found ${process.text}"
render process as JSON
I m able to get the process object and i m not able to get the json response
Should i want to convert the process object to json object?
OUTPUT:
Found java.lang.UNIXProcess#75566697
org.codehaus.groovy.grails.web.converters.exceptions.ConverterException: Error converting Bean with class java.lang.UNIXProcess
Grails has nothing to do with this. Groovy can execute arbitrary shell commands in a very simplistic way:
"mkdir foo".execute()
Or for more advanced features, you might look into using ProcessBuilder. At the end of the day, you need to execute ffprobe and then capture the output stream of JSON to use in your app.
Groovy provides a simple way to execute command line processes. Simply
write the command line as a string and call the execute() method.
The execute() method returns a java.lang.Process instance.
println "ffprobe <options>".execute().text
[Source]

Reading JSON data in a shell script

I have a JSON file containing data about some images:
{
"imageHeight": 1536,
"sessionID": "4340cc80cb532ecf106a7077fc2a166cb84e2c21",
"bottomHeight": 1536,
"imageID": 1,
"crops": 0,
"viewPortHeight": 1296,
"imageWidth": 2048,
"topHeight": 194,
"totalHeight": 4234
}
I wish to process these values in a simple manner in a shell script. I searched online but was not able to find any simple material to understand.
EDIT : What I wish to do with the values ?
I'm using convert (Imagemagick) to process the images. So, the whole workflow is something like. Read the an entry say crop from a line in the json file and then use the value in cropping the image :
convert -crop [image width from json]x[image height from json]+0+[crop value from json] [session_id from json]-[imageID from json].png [sessionID]-[ImageID]-cropped.png
I would recommend using jq. For example, to get the imageHeight, you can use:
jq ".imageHeight" data.json
Output:
1536
If you want to store the value in a shell variable use:
variable_name=$(jq ".imageHeight" data.json)
Python-solution
import json
from pprint import pprint
json_data=open('json_file')
data = json.load(json_data)
pprint(data)
data['bottomHeight']
output:
In [28]: pprint(data)
{u'bottomHeight': 1536,
u'crops': 0,
u'imageHeight': 1536,
u'imageID': 1,
u'imageWidth': 2048,
u'sessionID': u'4340cc80cb532ecf106a7077fc2a166cb84e2c21',
u'topHeight': 194,
u'totalHeight': 4234,
u'viewPortHeight': 1296}
In [29]: data['bottomHeight']
Out[29]: 1536