Serialize SQLAlchemy output to a JSON including column names - json

I've decided to migrate my Django project to work with SqlAlchemy instead of Django ORM and I'm trying to serialize my SqlAlchemy output to a JSON which contains column names.
In Django I had the following code:
logs = Log.objects.values('log_timestamp', 'message', 'exception', 'level__level', 'job_info__job_name', 'machine', 'user', 'job_report__id').filter(job_info__app_id = app_id).order_by('-time_added')[:1]
logs = json.dumps(list(logs), default=views_utils.default_json_serializer)
print(logs)
and here an output example (contains column names):
[{"user": "user", "level__level": "INFO", "message": "this is a message", "log_timestamp": null, "job_info__job_name": "MongoDB_Maintenance", "exception": "exception details", "machine": "machine", "job_report__id": 65}]
Here is my SqlAlchemy code:
res = session.query(func.DATETIME(LogObj.time_added), LogObj.message, LogObj.exception, LogLevelObj.level, LogObj.machine, LogObj.user).\
join(PeriodicJobInfoObj, LogLevelObj, LogObj,aliased=True).\
filter(PeriodicJobInfoObj.app_id == app_id).\
order_by(desc(LogObj.time_added))[:1]
res = json.dumps(res, default=views_utils.default_json_serializer)
print(res)
The output does not contain column names:
[["2015-09-28 15:36:33", "this is a message", "exception details", "CRITICAL", "machine", "user"]]
default_json_serializer code:
def default_json_serializer(obj):
"""Default JSON serializer."""
import calendar, datetime
if isinstance(obj, datetime.datetime):
if obj.utcoffset() is not None:
obj = obj - obj.utcoffset()
millis = int(
calendar.timegm(obj.timetuple()) * 1000 +
obj.microsecond / 1000
)
return millis
How can I achieve similar output like Django does?
(working with Python 3.4.2)

I have managed to do it this way:
def alchemyencoder(obj):
if isinstance(obj, datetime.date):
return obj.isoformat()
elif isinstance(obj, decimal.Decimal):
return float(obj)
I use the above encoder when retrieving records in following way:
row = conn.execute(Model.__table__.select())
d = json.dumps([dict(r) for r in row], default=alchemyencoder)
d = json.JSONDecoder().decode(d)
print(d)

Related

Receiving TypeError: 'NoneType' object is not subscriptable at random in Python3

I'm writing a simple program in Python that iterates through various API JSON links, and compares values from each of the data sources. As I cycle through one of the JSON files, I call a function that calls a different JSON file based off of one of the values of the initial JSON file.
Here is my Python code:
import requests
import json
import urllib.request as request
from urllib.request import Request, urlopen
import time
# THINGS
'''
international API: https://arsonwarehouse.com/api/v1/foreign-stock
domestic API: https://api.torn.com/market/ ?selections=itemmarket&key=KEY
personal API: https://api.torn.com/user/2519609?selections=money&key=KEY
'''
capacity = 21
moneyOverride = False
totalCost: int
totalIncome: int
#Takes in the cost per item, multiplies by capacity, returns true if it's affordable
def checkAfford(costPer):
global totalCost
totalCost = capacity * costPer
if totalCost < money:
return True
else:
return False
#Puts together the API link with the ID number and calls pullJSON()
def checkDomestic(id):
jsonPULL = pullJSON(f"https://api.torn.com/market/{id}?selections=itemmarket&key=YET")
if jsonPULL == None:
print("error")
else:
return jsonPULL
#Requests the JSON from the server
def pullJSON(url):
req = Request(url, headers={'User-Agent': 'Mozilla/5.0'})
webpage = urlopen(req).read()
with urlopen(req) as response:
if response.getcode() == 200:
source = response.read()
data = json.loads(source)
return data
else:
print('An error occurred while attempting to retrieve data from the API.')
internationalData = pullJSON("https://arsonwarehouse.com/api/v1/foreign-stock")
personalData = pullJSON("https://api.torn.com/user/2519609?selections=money&key=YET")
domesticData = checkDomestic(10)
#Money override
if moneyOverride == False:
money = personalData.get("money_onhand")
else:
money = 1000000000000000
print("onhand:", money)
i = 0
length = len(internationalData["items"])
print(length)
for x in internationalData["items"]:
time.sleep(0.5)
ident = x["item_id"]
domPrice = x["price"]
print("DOMESTIC: ID:", ident, " at price:", domPrice)
domestic = checkDomestic(ident).get("itemmarket")[1]
inPrice = domestic["cost"]
if inPrice == None:
print("error", domestic["ID"])
location = x["country"]
print("INTERNATIONAL: ID:", ident, " at price:", inPrice, "location: ", location)
print("end")
This code won't inherently work — I removed my API key from the links. However, I can show what the example JSON looks like.
This is what checkDomestic() pulls from:
{
"itemmarket": [
{
"ID": 101242786,
"cost": 17898,
"quantity": 1
},
{
"ID": 101242784,
"cost": 17898,
"quantity": 1
}
]
}
And this is what internationalData looks like:
{
"items": [
{
"item_id": 4,
"country": "South Africa",
"price": 750,
"stock": 247,
"reported_at": "2020-06-03T11:47:56Z"
},
{
"item_id": 8,
"country": "Mexico",
"price": 4200,
"stock": 59,
"reported_at": "2020-06-03T11:45:21Z"
}
]
}
I've searched around for an answer, but most questions have to do with the .sort() method. I should also mention that this happens at random—sometimes in the first 10 iterations, but never at the end, so it's not an out of range error. The JSON structures stay the same across.
A somewhat related question: I've completed this project before in Golang, and I noticed that in Go it finishes much, much, much faster than in Python. Is there some kind of redundancy that's making it take so long? I'm aware of the time.sleep() function I have in there-it was to make sure that it's not something with overcalling the API.
Any advice would be massively helpful!

How to read the text in excel cell and replace with someother value in json output using python?

My python code reads the excel sheet and converts it into a json file output. I have a column in the excel sheet, where the values are either "Planned" or "Unplanned".
1)In the json output, I want the Planned to be replaced with "1" and Unplanned to be replaced with "2" without changing anything in the excel file.
2)In the output I dont want "data" to appear.
3)In the excel, my Start time column value is like this "2018-11-16 08:00:00". I want the output to be "2018-11-16T08:00:00Z". Currently i am getting some garbage value.
Below is my code.
import xlrd, json, time, pytz, requests
from os import sys
from datetime import datetime, timedelta
from collections import OrderedDict
def json_from_excel():
excel_file = 'test.xlsx'
jsonfile = open('ExceltoJSON.json', 'w')
data = []
datestr = str(datetime.now().date())
loaddata = OrderedDict()
workbook = xlrd.open_workbook(excel_file)
worksheet = workbook.sheet_by_name('OMS-GX Data Extraction')
sheet = workbook.sheet_by_index(0)
for j in range(0, 6):
for i in range(1, 40):
temp = {}
temp["requestedStart"] = (sheet.cell_value(i,0)) #Start Time
temp["requestedComplete"] = (sheet.cell_value(i, 1)) #End Time
temp["location"] = (sheet.cell_value(i, 3)) #Station
temp["equipment"] = (sheet.cell_value(i, 4)) #Device Name
temp["switchOrderTypeID"] = (sheet.cell_value(i, 5)) #Outage Type
data.append(temp)
loaddata['data'] = data
json.dump(loaddata, jsonfile, indent=3, sort_keys=False)
jsonfile.write('\n')
return loaddata
if __name__ == '__main__':
data = json_from_excel()
Below is my sample output:
{
"data": [
{
"requestedStart": testtime,
"requestedComplete": testtime,
"location": "testlocation",
"equipment": "testequipment",
"switchOrderTypeID": "Planned"
},
{
"requestedStart": testtime,
"requestedComplete": testtime,
"location": "testlocation",
"equipment": "testequipment",
"switchOrderTypeID": "Unplanned"
}
]
}
Answer to the 1st question:
You may use conditional assignment.
temp["switchOrderTypeID"] = (1 if sheet.cell_value(i, 5) == "Planned" else 0)
Answer to the 2nd question:
Use loaddata = data which will be an array of the jsons without data as json key.
Answer to 3rd question:
from dateutil.parser import parse
t = "2018-11-16 08:00:00"
parse(t).strftime("%Y-%m-%dT%H:%M:%SZ")

add data to my Json in django

This is how my JSON data looks like
{"id": 50, "first_digit": "2", "second_digit": "1", "calculate": "Addition"}
I want to add extra data "result":"3" to my JSON
this is my view
def calc(request):
if request.method == 'POST':
first_number = request.POST['first number']
second_number = request.POST['second number']
operation = request.POST['operation']
result = do_calc(operation, first_number, second_number)
# how to pass the result to my tempelate
value = Calculation.objects.create(
first_digit=first_number,
second_digit=second_number,
calculate=operation
)
data = model_to_dict(value)
return JsonResponse(data)
Can anyone help me with it
Presumably the output of model_to_dict() is a dict. So you can simply add your value to that dict:
data = model_to_dict(value)
data['result'] = result
return JsonResponse(data)

Is there any tool to convert JSON object to SQLAlchemy model?

I'm looking for a way to convert list of JSON object into an approximate SQLAlchemy model. Is there any implementation?
This is what I did in one of my projects to convert json input from an http request to an sqlalchemy model dynamically:
if request.method == "POST" or request.method == "PUT":
data = request.get_json(force=True)
print("data:", data)
try:
TableClass = models.get_class_by_tablename(table_name)
if TableClass == None: raise Exception("Table not found: %s" % table_name)
if request.method == "POST": #insert data
object = TableClass(**data)
dbsession.add(object)
dbsession.commit()
else: #update data
object = dbsession.query(TableClass).filter_by(**{"id":id}).first()
if object == None: raise Exception("No data found.")
#object.update(**data)
for key in data.keys():
setattr(object, key, data[key])
#dbsession.add(object)
dbsession.commit()
return jsonify({
"status": "success",
"id": object.id,
})
except Exception as e:
return jsonify({
"status": "error",
"error": str(e),
})
data is the variable that gets converted from json input to the usual python dictionary. Then **data is passed as argument for the instance creation of sqlalchemy table class.

Parsing the response Json string using groovy

4 for checking the status of the process using httpbuilder and getting the below response.
{"result":[{"id":"167687","dapadmin":"false","status":"in progress","lastupdated":"2017-04-21 14:34:30.0","started":"2017-04-21 14:34:28.0","user":"sys","log":"Running a Stop action\n\nRunning command \n"}]}
Am unable to parse this response using jsonSlurper.parseText() giving error
When I use `
def json = JsonOutput.toJson(statusresponse)
println JsonOutput.prettyPrint(json)
I could see it is printed as json object
{
"result": [
{
"id": "167687",
"dapadmin": "false",
"status": "in progress",
"lastupdated": "2017-04-21 14:34:30.0",
"started": "2017-04-21 14:34:28.0",
"user": "dapsystem",
"log": "Running a Stop action\n\nRunning command \n"
}
]
}
When i check the getClass() , it prints as java.lang.String.
I need to get the lastupdated and status , id values from this response. Please help me to find a solution for this.
Many thanks
Hi Yanpei.
Thanks for the response.
I am using the below code as suggested by you.
def statusresponse = http.putText(baseurl,path,query,headerMap, Method.GET)
println("The status response value is" )
statusresponse.each{ k, v ->
println "${k}:${v}"
}
def json = JsonOutput.toJson(statusresponse)
println "JSON Object List : " + json
println "------------------"
println JsonOutput.prettyPrint(json)
println "The result class is "+json.getClass()
println "The result status is "+(json instanceof Map)
//def entry = slurper.parseText(json)
def slurper = new groovy.json.JsonSlurper()
def entry = slurper.parseText(statusresponse)
def lastupdated = entry.result.lastupdated
checkStatus = entry.result.status
def id = entry.result.id
Am getting the below error
The result class is class java.lang.String
Caught: groovy.lang.MissingMethodException: No signature of method: groovy.json.JsonSlurper.parseText() is applicable for argument types: (java.util.HashMap) values: [[result:[[id:170089, dapadmin:false, status:in progress, ...]]]]
Possible solutions: parseText(java.lang.String), parse(java.io.Reader)
groovy.lang.MissingMethodException: No signature of method: groovy.json.JsonSlurper.parseText() is applicable for argument types: (java.util.HashMap) values: [[result:[[id:170089, dapadmin:false, status:in progress, ...]]]]
Possible solutions: parseText(java.lang.String), parse(java.io.Reader)
at dap.Main.main(Main.groovy:171)
It works if i use the code as below
def json = JsonOutput.toJson(statusresponse)
def entry = slurper.parseText(json)
Am getting the results as below
The status of the action is :[in progress]
Last updated [2017-04-23 17:08:02.0]
the id is[170088]
First of all, am not sure why the code suggested is throwing this error
Secondly, why i am getting the results for the working solution, within the brackets?
def slurper = new groovy.json.JsonSlurper()
def entry = slurper.parseText('{"result":[{"id":"167687","dapadmin":"false","status":"in progress","lastupdated":"2017-04-21 14:34:30.0","started":"2017-04-21 14:34:28.0","user":"sys","log":"Running a Stop action\n\nRunning command \n"}]}')
def lastupdated = entry.result.lastupdated
def status = entry.result.status
def id = entry.result.id
Should work. Can't see your error so I can't give better info.