Create json sub objects using python collections module - json

I am attempting to build a json document to be passed to an api as a post body request.
I am pulling data from a mssql as an arrray list, then creating an ordered dictionary using collections model. Need help on how to create sub object
import json, collections
rowarray_list = []
for row in rows:
t = (row.NameLine1, row.NameLine2, row.Phone, row.Mobile,
row.Fax, row.Slogan, row.Address, row.City, row.State, row.Zip,
row.Email, row.WebSite, row.ApplyOnline,
row.Preflight, row.Facebook, row.LinkedIn, row.Username)
rowarray_list.append(t)
objects_list = []
for row in rows:
d = collections.OrderedDict()
d['Name'] = row.NameLine1
d['Phone'] = row.Phone
d['Mobile'] = row.Mobile
d['Fax'] = row.Fax
d['Slogan']=row.Slogan
d['Address']=row.Address
d['City'] = row.City
d['State'] = row.State
d['Zip'] = row.Zip
d['Email']=row.Email
d['Website']=row.WebSite
d['ApplyOnline']=row.ApplyOnline
d['Preflight']=row.Preflight
d['Facebook']=row.Facebook
d['LinkedIn']=row.LinkedIn
d['Username']=row.Username
objects_list.append(d)
json.dumps(objects_list)
I want to have the json objects to be built like this:
{"type": "task1",
"body": {"Name": row.NameLine
"Phone": row.Phone
... }}
I cant seem to figure out how to do this

I was able to get this resolved by using a different approach. Creating a dictionary object per each row and appending it to dictionary list. Then used json library to create the full json object
rowarray_list=[]
for row in rows:
subdt = dict(Name=row.NameLine1, Title=row.NameLine2, Phone=row.Phone, Mobile=row.Mobile,
Fax=row.Fax, Slogan=row.Slogan, Address=row.Address, City=row.City, State=row.State, Zip=row.Zip, Email=row.Email, WebSite=row.WebSite, ApplyOnline=row.ApplyOnline,
Preflight=row.Preflight, Facebook=row.Facebook, LinkedIn=row.LinkedIn, Username=row.Username)
dt=dict(action='fetchView', body=subdt)
rowarray_list.append(dt)
print(json.dumps(rowarray_list))
cursor.close()

Related

How to convert multi dimensional array in JSON as separate columns in pandas

I have a DB collection consisting of nested strings . I am trying to convert the contents under "status" column as separate columns against each order ID in order to track the time taken from "order confirmed" to "pick up confirmed". The string looks as follows:
I have tried the same using
xyz_db= db.logisticsOrders -------------------------(DB collection)
df =pd.DataFrame(list(xyz_db.find())) ------------(JSON to dataframe)
Using normalize :
parse1=pd.json_normalize(df['status'])
It works fine in case of non nested arrays. But status being a nested array the output is as follows:
Using for :
data = df[['orderid','status']]
data = list(data['status'])
dfy = pd.DataFrame(columns = ['statuscode','statusname','laststatusupdatedon'])
for i in range(0, len(data)):
result = data[i]
dfy.loc[i] = [data[i][0],data[i][0],data[i][0],data[i][0]]
It gives the result in form of appended rows which is not the format i am trying to achieve
The output I am trying to get is :
Please help out!!
i share you which i used json read, maybe help you:
you can use two and more list
def jsonify(z):
genr = []
if z==z and z is not None:
z = eval(z)
if type(z) in (dict, list, tuple):
for dic in z:
for key, val in dic.items():
if key == "name":
genr.append(val)
else:
return None
else:
return None
return genr
top_genr['genres_N']=top_genr['genres'].apply(jsonify)

Mulitlevel Pandas dataframe to nested json

I am trying to turn a data frame into some nested json and have been struggling a bit. Here is an example I created. The use case is to split a document for each guest at a hotel chain, with the guest at the top, the hotel details under the visit data, and the daily charges under the 'measurements' info.
The dataframe:
Here is an example of how I am trying to get the JSON to look
I have tried creating a multilevel index and using to_json:
Is there a way to do this using to_json() or will I need to build some nested loops to create nested dictionaries? This is the best I have been able to get:
I would recommend a programming approach. pandas.DataFrame.groupby can be useful.
def hotel_data_to_json(df):
return [
person_data_to_json(person_df)
for person_id, person_df
in df.groupby('person_id')
]
def person_data_to_json(df):
row = df.iloc[0]
return {
'person_id': row['person_id'],
'personal_name': row['personal_name'],
'family_name': row['family_name'],
'visits': [
visit_data_to_json(visit_df)
for visit_id, visit_df
in df.groupby('visit_id')
]
}
def visit_data_to_json(df):
row = df.iloc[0]
# and so on

Converting mongoengine objects to JSON

i tried to fetch data from mongodb using mongoengine with flask. query is work perfect the problem is when i convert query result into json its show only fields name.
here is my code
view.py
from model import Users
result = Users.objects()
print(dumps(result))
model.py
class Users(DynamicDocument):
meta = {'collection' : 'users'}
user_name = StringField()
phone = StringField()
output
[["id", "user_name", "phone"], ["id", "user_name", "phone"]]
why its show only fields name ?
Your query returns a queryset. Use the .to_json() method to convert it.
Depending on what you need from there, you may want to use something like json.loads() to get a python dictionary.
For example:
from model import Users
# This returns <class 'mongoengine.queryset.queryset.QuerySet'>
q_set = Users.objects()
json_data = q_set.to_json()
# You might also find it useful to create python dictionaries
import json
dicts = json.loads(json_data)

spark.RDD take(n) returns array of element n, n times

I'm using code from https://github.com/alexholmes/json-mapreduce to read a multi-line json file into an RDD.
var data = sc.newAPIHadoopFile(
filepath,
classOf[MultiLineJsonInputFormat],
classOf[LongWritable],
classOf[Text],
conf)
I printed out the first n elements to check if it was working correctly.
data.take(n).foreach { p =>
val (line, json) = p
println
println(new JSONObject(json.toString).toString(4))
}
However when I try to look at the data, the arrays returned from take don't seem to be correct.
Instead of returning an array of the form
[ data[0], data[1], ... data[n] ]
it is in the form
[ data[n], data[n], ... data[n] ]
Is this an issue with the RDD I've created, or an issue with how I'm trying to print it?
I figured out why take it was returning an array with duplicate values.
As the API mentions:
Note: Because Hadoop's RecordReader class re-uses the same Writable object
for each record, directly caching the returned RDD will create many
references to the same object. If you plan to directly cache Hadoop
writable objects, you should first copy them using a map function.
Therefore in my case it was reusing the same LongWritable and Text objects. For example if I did:
val foo = data.take(5)
foo.map( r => System.identityHashCode(r._1) )
The output was:
Array[Int] = Array(1805824193, 1805824193, 1805824193, 1805824193, 1805824193)
So in order to prevent it from doing this, I simply mapped the reused objects to their respective values:
val data = sc.newAPIHadoopFile(
filepath,
classOf[MultiLineJsonInputFormat],
classOf[LongWritable],
classOf[Text],
conf ).map(p => (p._1.get, p._2.toString))

How to fetch a JSON file to get a row position from a given value or argument

I'm using wget to fetch several dozen JSON files on a daily basis that go like this:
{
"results": [
{
"id": "ABC789",
"title": "Apple",
},
{
"id": "XYZ123",
"title": "Orange",
}]
}
My goal is to find row's position on each JSON file given a value or set of values (i.e. "In which row XYZ123 is located?"). In previous example ABC789 is in row 1, XYZ123 in row 2 and so on.
As for now I use Google Regine to "quickly" visualize (using the Text Filter option) where the XYZ123 is standing (row 2).
But since it takes a while to do this manually for each file I was wondering if there is a quick and efficient way in one go.
What can I do and how can I fetch and do the request? Thanks in advance! FoF0
In python:
import json
#assume json_string = your loaded data
data = json.loads(json_string)
mapped_vals = []
for ent in data:
mapped_vals.append(ent['id'])
The order of items in the list will be indexed according to the json data, since the list is a sequenced collection.
In PHP:
$data = json_decode($json_string);
$output = array();
foreach($data as $values){
$output[] = $values->id;
}
Again, the ordered nature of PHP arrays ensure that the output will be ordered as-is with regard to indexes.
Either example could be modified to use a mapped dictionary (python) or an associative array (php) if needs demand.
You could adapt these to functions that take the id value as an argument, track how far they are into the array, and when found, break out and return the current index.
Wow. I posted the original question 10 months ago when I knew nothing about Python nor computer programming whatsoever!
Answer
But I learned basic Python last December and came up with a solution for not only get the rank order but to insert the results into a MySQL database:
import urllib.request
import json
# Make connection and get the content
response = urllib.request.urlopen(http://whatever.com/search?=ids=1212,125,54,454)
content = response.read()
# Decode Json search results to type dict
json_search = json.loads(content.decode("utf8"))
# Get 'results' key-value pairs to a list
search_data_all = []
for i in json_search['results']:
search_data_all.append(i)
# Prepare MySQL list with ranking order for each id item
ranks_list_to_mysql = []
for i in range(len(search_data_all)):
d = {}
d['id'] = search_data_all[i]['id']
d['rank'] = i + 1
ranks_list_to_mysql.append(d)