I am trying to get a nested value from a json content I get from an API Rest (Swagger). I want to access to the first "url" value:
val getFeedVersionURL = url("https://api.transitfeeds.com/v1/getFeedVersions?").GET <<? List(
"key" -> someKey,
"feed" -> "consorcio-regional-de-transportes-de-madrid/743",
"page" -> "1",
"limit" -> "10000",
"err" -> "1 (default)",
"warn" ->"1 (default)")
val response : Future[String] = Http.configure(_ setFollowRedirects true)(getFeedVersionURL OK as.String)
val src: String = getFeedVersionURL.toString()
response onComplete {
case Success(content) => {
println("Successful response" + content)
//Parse the content
val jsonObject = Json.parse(content)
//Get url value
val fileDownloadURL = (jsonObject \ "results" \ "versions" \ "url").as[String]
//Save the file
new URL(fileDownloadURL) #> new File("C:/Users//Desktop//file.zip") !!
}
case Failure(t) => {
println("An error has occured: " + t.getMessage)
}
}
This is the json I receive:
{
"status": "OK",
"ts": 1513600944,
"results": {
"total": 4,
"limit": 100,
"page": 1,
"numPages": 1,
"versions": [
{
"id": "consorcio-regional-de-transportes-de-madrid/743/20171127",
"f": {
"id": "consorcio-regional-de-transportes-de-madrid/743",
"ty": "gtfs",
"t": "Metro Ligero de Madrid GTFS",
"l": {
"id": 167,
"pid": 166,
"t": "Madrid, Spain",
"n": "Madrid",
"lat": 40.416775,
"lng": -3.70379
},
"u": {
"i": "http://crtm.maps.arcgis.com/home/item.html?id=aaed26cc0ff64b0c947ac0bc3e033196"
}
},
"ts": 1511848526,
"size": 403388,
"url": "https://transitfeeds.com/p/consorcio-regional-de-transportes-de-madrid/743/20171127/download",
"d": {
"s": "20170106",
"f": "20180629"
},
"err": [],
"warn": []
},
{
"id": "consorcio-regional-de-transportes-de-madrid/743/20170927",
"f": {
"id": "consorcio-regional-de-transportes-de-madrid/743",
"ty": "gtfs",
"t": "Metro Ligero de Madrid GTFS",
"l": {
"id": 167,
"pid": 166,
"t": "Madrid, Spain",
"n": "Madrid",
"lat": 40.416775,
"lng": -3.70379
},
"u": {
"i": "http://crtm.maps.arcgis.com/home/item.html?id=aaed26cc0ff64b0c947ac0bc3e033196"
}
},
"ts": 1506554131,
"size": 403052,
"url": "https://transitfeeds.com/p/consorcio-regional-de-transportes-de-madrid/743/20170927/download",
"d": {
"s": "20170106",
"f": "20180925"
},
"err": [],
"warn": []
},
{
"id": "consorcio-regional-de-transportes-de-madrid/743/20170526",
"f": {
"id": "consorcio-regional-de-transportes-de-madrid/743",
"ty": "gtfs",
"t": "Metro Ligero de Madrid GTFS",
"l": {
"id": 167,
"pid": 166,
"t": "Madrid, Spain",
"n": "Madrid",
"lat": 40.416775,
"lng": -3.70379
},
"u": {
"i": "http://crtm.maps.arcgis.com/home/item.html?id=aaed26cc0ff64b0c947ac0bc3e033196"
}
},
"ts": 1495816290,
"size": 408985,
"url": "https://transitfeeds.com/p/consorcio-regional-de-transportes-de-madrid/743/20170526/download",
"d": {
"s": "20170106",
"f": "20180526"
},
"err": [],
"warn": []
},
{
"id": "consorcio-regional-de-transportes-de-madrid/743/20161012",
"f": {
"id": "consorcio-regional-de-transportes-de-madrid/743",
"ty": "gtfs",
"t": "Metro Ligero de Madrid GTFS",
"l": {
"id": 167,
"pid": 166,
"t": "Madrid, Spain",
"n": "Madrid",
"lat": 40.416775,
"lng": -3.70379
},
"u": {
"i": "http://crtm.maps.arcgis.com/home/item.html?id=aaed26cc0ff64b0c947ac0bc3e033196"
}
},
"ts": 1476308287,
"size": 420670,
"url": "https://transitfeeds.com/p/consorcio-regional-de-transportes-de-madrid/743/20161012/download",
"d": {
"s": "20160101",
"f": "20170621"
},
"err": [],
"warn": []
}
]
}
}
The problem is I receive this error:
play.api.libs.json.JsResultException: JsResultException(errors:List((,List(ValidationError(error.expected.jsstring,WrappedArray())))))
Any ideas?
cchantep in the comment is correct. You'll need to pull out the versions key and deal with it appropriately. Here's some Very rough code that does what you want:
import play.api.libs.json._
val jsonObject = Json.parse(s) // s is that string you gave in your q
val fileDownloadURL: String = (jsonObject \ "results" \ "versions") match {
case JsDefined(JsArray(versions)) => versions.headOption match {
case Some(jsValue) => (jsValue \ "url").as[String]
case _ => "NOPE hope you got some logic here"
}
case nope => throw new IllegalArgumentException("handle this better than this!")
}
And I get:
res0: String = https://transitfeeds.com/p/consorcio-regional-de-transportes-de-madrid/743/20171127/download
Note that I did this using play 2.4 in my console, and for 2.3 and other versions there may be differences. If you provide the version of play you're using I can probably give you some more rough ideas about how to accomplish what you're up to.
Note that the above with the matching is one way, another way would be to use as a bunch
val thisToo = (jsonObject \ "results" \ "versions").as[JsArray].value.headOption.map(first => (first \ "url").as[String]).getOrElse(throw new IllegalArgumentException("handle this better than this!"))
Related
i have a question:
Is it possible read a json file and convert to dataframe dynamically?
My example is the next code:
Having this json file, i need 3 table dataframes:
{
"date_time": "01-03-2022, 15:18:32",
"regions": {
"Home Region": "Madrid",
"Primary Region": "Barcelona",
"Secondary Region": "Rio"
},
"customers": [
{
"name": "campo santo",
"address": "rua trebal 1",
"phone": 987456321,
"parking": true
},
{
"name": "santo da silva",
"address": "rua sama 6",
"phone": 654321987,
"parking": false
},
{
"name": "roger campos",
"address": "av casal 10",
"phone": 684426654,
"parking": true
}
],
"office": [
{
"location": "madrid",
"co_working_spaces": 25,
"kitchen": false,
"food_track": 2,
"internal_staff": [
{
"id": 123,
"name": "pablo"
},
{
"id": 874,
"name": "saul"
},
{
"id": 741,
"name": "maria"
}
]
},
{
"location": "rio",
"co_working_spaces": 55,
"kitchen": true,
"food_track": 4,
"internal_staff": [
{
"id": 784,
"name": "raquel"
},
{
"id": 874,
"name": "pedro"
},
{
"id": 145,
"name": "maria"
},
{
"id": 365,
"name": "rocio"
}
]
},
{
"location": "barcelona",
"co_working_spaces": 5,
"kitchen": false,
"food_track": 1,
"internal_staff": [
]
},
{
"location": "la",
"co_working_spaces": 5,
"kitchen": true,
"food_track": 4,
"internal_staff": [
{
"id": 852,
"name": "maria"
},
{
"id": 748,
"name": "sara"
}
]
}
]
}
this is my python code:
import pandas as pd
# from pandas.io.json import json_normalize
import json
with open('offices.json') as f:
dt = json.load(f)
# df = pd.json_normalize(dt)
df1 = pd.json_normalize(dt, 'customers', 'date_time')[['name', 'address', 'phone', 'parking', 'date_time']]
print(df1)
df2 = pd.json_normalize(dt, 'office', 'date_time')[['location', 'co_working_spaces', 'kitchen', 'food_track']]
print(df2)
df3 = pd.json_normalize(dt['office'], 'internal_staff', 'location')
print(df3)
With this code, i got my 3 table dataframes. But i have to know the json structure to create the dataframes.
So is it possible to do it dynamically ?
Regards
Json body looks like this:
{
"access_key": "",
"erid": "",
"ch_sms": {
"messages": [
{
"urlsh": false,
"sp": "123",
"vp": 20,
"heid": 1,
"teid": ,
"peid": ,
"tmid": 4,
"msg": {
"txt": "hello"
},
"da": [
{
"number": "97278",
"cc": "IN",
"uid": "uid1",
"tags": [
"key",
"Value"
]
}
]
}
],
"metadata": {
"chver": "1.0",
"cburl": "",
"heid": 1,
"teid": ,
"peid": ,
"tmid": 4,
"Oa": "",
"flash": false,
"tags": [
"key",
"tag1"
]
}
}
}
We need to add multiple "da" sections based on user input. For example if I give input from a CSV file as 3 then "da" section will repeat 3 times and it will look like as follow:
{
"access_key": "",
"erid": "",
"ch_sms": {
"messages": [
{
"urlsh": false,
"sp": "123",
"vp": 20,
"heid": 1,
"teid": ,
"peid": ,
"tmid": 4,
"msg": {
"txt": "hello"
},
"da": [
{
"number": "97278",
"cc": "IN",
"uid": "uid1",
"tags": [
"key",
"Value"
]
},
{
"number": "97278",
"cc": "IN",
"uid": "uid1",
"tags": [
"key",
"Value"
]
},
{
"number": "97278",
"cc": "IN",
"uid": "uid1",
"tags": [
"key",
"Value"
]
}
]
}
],
"metadata": {
"chver": "1.0",
"cburl": "",
"heid": 1,
"teid": ,
"peid": ,
"tmid": 4,
"Oa": "",
"flash": false,
"tags": [
"key",
"tag1"
]
}
}
}
You can try this,
import org.json.simple.JSONObject
JSONObject data = new JSONObject()
data.put("number", "97278")
data.put("cc", "IN")
data.put("uid", "uid1")
data.put("tags", ["key","Value"])
int da = Integer.parseInt(vars.get("da"))
ArrayList<Object> listData = new ArrayList<Object>()
for(int i=0; i < da; i++)
{
listData.add(data.toString())
}
log.info("da: " + listData )
Result log:
da: [{"cc":"IN","number":"97278","uid":"uid1","tags":["key","Value"]}, {"cc":"IN","number":"97278","uid":"uid1","tags":["key","Value"]}, {"cc":"IN","number":"97278","uid":"uid1","tags":["key","Value"]}]
I'm totally new to python. I want to merge two JSON files who have the same objects but different keys.
Here is a basic example of the result I would love to get :
JSON1 :
{
"json1" : {
"1" : {
"id": 1,
"name": "first_artist",
"imageUrl": "https://1.jpg",
"genre": "Rap "
},
"2" : {
"id": 2,
"name": "second_artist",
"imageUrl": "https://2.jpg",
"genre": "Hip-Hop"
}
}
}
JSON2:
{
"json2" : {
"1" : {
"date": 17/07/19,
"venue": "venue1"
},
"2" : {
"date": 19/07/19,
"venue": "venue2"
}
}
}
Expected JSON:
{
"expected_json" : {
"1" : {
"id": 1,
"name": "first_artist",
"imageUrl": "https://1.jpg",
"genre": "Rap "
"date": 17/07/19,
"venue": "venue1"
},
"2" : {
"id": 2,
"name": "second_artist",
"imageUrl": "https://2.jpg",
"genre": "Hip-Hop"
"date": 19/07/19,
"venue": "venue2"
}
}
}
Can someone give tips and direction to make this possible ? Thanks
You can simplify your input to:
A.json:
{
"1" : {
"id": 1,
"name": "first_artist",
"imageUrl": "https://1.jpg",
"genre": "Rap "
},
"2" : {
"id": 2,
"name": "second_artist",
"imageUrl": "https://2.jpg",
"genre": "Hip-Hop"
}
}
B.json:
{
"1" : {
"date": "17/07/19",
"venue": "venue1"
},
"2" : {
"date": "19/07/19",
"venue": "venue2"
}
}
and you have to change 19/07/19 to "19/07/19" for it to be valid json.
Now you can use the json module:
import json
#from pprint import pprint
# load json from files
with open('A.json') as A_file:
A = json.load(A_file) # returns a dict()
#print('A:')
#pprint(A)
with open('B.json') as B_file:
B = json.load(B_file)
#print('\nB:')
#pprint(A)
# get a list of unique keys -> {'1', '2'}
keys = set()
keys.update(A.keys())
keys.update(B.keys())
#print(f'\nkeys: {keys}')
# for each key merge values from dicts A and B
result = {}
for key in keys:
#print(f'\n{key}:')
merge = {}
if key in A:
merge.update(A[key])
if key in B:
merge.update(B[key])
#pprint(merge)
result[key] = merge
#print('\nresult:')
#pprint(result)
# write the result to expected.json
with open('expected.json', 'w+') as expected_file:
expected_file.write(json.dumps(result, sort_keys=True, indent='\t'))
This writes:
expected.json:
{
"1": {
"date": "17/07/19",
"genre": "Rap ",
"id": 1,
"imageUrl": "https://1.jpg",
"name": "first_artist",
"venue": "venue1"
},
"2": {
"date": "19/07/19",
"genre": "Hip-Hop",
"id": 2,
"imageUrl": "https://2.jpg",
"name": "second_artist",
"venue": "venue2"
}
}
I am trying to create multiple files by reading my JSON log file and separate file has to be created per each element in my JSON array. Any help would be appreciated.
Approach : Able to read the json log file using json.load() and read the number of elements inside the json array and then copy first 5 elements in to file1 as an json array , and rest 5 into second file as an json array and so on till the length of my json log file array.
update :
I got it working till splitting the list/array in to 10 individual lists/array till end of the file. now have to write those list to separate json.gz file and upload to s3.
import json
json_data = json.dumps([
{
"a": "1",
"b": "2"
},
{
"d": "3"
},
{
"d": "4"
},
{
"e": "5"
},
{
"e": "6"
},
{
"e": "7"
},
{
"e": "8"
},
{
"e": "9"
},
{
"e": "5"
},
{
"e": "10"
},
{
"e": "11"
},
{
"e": "12"
},
{
"s": "y"
},
{
"s": "y"
},
{
"s": "y"
},
{
"s": "y"
},
{
"s": "y"
},
{
"s": "y"
},
{
"s": "y"
},
{
"s": "y"
},
{
"s": "y"
},
{
"s": "y"
},
{
"s": "y"
},
{
"s": "y"
},
{
"s": "y"
},
{
"s": "y"
},
{
"s": "y"
},
{
"abc": [
{
"te": "tre"
}
]
}
])
item_dict = json.loads(json_data)
lens = len(item_dict)
first = 0
last = 10
b = item_dict[first:last]
#print (item_dict[0:2])
print (lens)
for i in range(lens):
if last < lens:
n = 0
res = item_dict[first:last]
print(range(lens))
print(res)
first = last
last = first + 10
i = i + 1
print(first)
print(last)
print(i)
if last > lens:
diff = last-lens
last = last-diff
print("second" + str(first))
print("second" + str(last))
ins = item_dict[first:last]
print(ins)```
my json log file:
```[
{
"id": 1000,
"type": "BigBox",
"name": "Mall of America",
"address": "340 W Market",
"address2": "",
"city": "Bloomington",
"state": "MN",
"zip": "55425",
"location": {
"lat": 44.85466,
"lon": -93.24565
},
"hours": "Mon: 10-9:30; Tue: 10-9:30; Wed: 10-9:30; Thurs: 10-9:30; Fri: 10-9:30; Sat: 10-9:30; Sun: 11-7",
"services": [
"Geek Squad Services",
"Best Buy Mobile",
"Best Buy For Business"
]
},
{
"id": 1002,
"type": "BigBox",
"name": "Tempe Marketplace",
"address": "1900 E Rio Salado Pkwy",
"address2": "",
"city": "Tempe",
"state": "AZ",
"zip": "85281",
"location": {
"lat": 33.430729,
"lon": -111.89966
},
"hours": "Mon: 10-9; Tue: 10-9; Wed: 10-9; Thurs: 10-9; Fri: 10-10; Sat: 10-10; Sun: 10-8",
"services": [
"Windows Store",
"Geek Squad Services",
"Best Buy Mobile",
"Best Buy For Business"
]}
]```
I'm want to make my json to csv so that i can upload it on google sheets and make it as json api. Whenever i have change data i will just change it on google sheets. But I'm having problems on converting my json file to csv because it changes the variables whenever i convert it. I'm using https://toolslick.com/csv-to-json-converter to convert my json file to csv.
What is the best way to convert json nested to csv ?
JSON
{
"options": [
{
"id": "1",
"value": "Jumbo",
"shortcut": "J",
"textColor": "#FFFFFF",
"backgroundColor": "#00000"
},
{
"id": "2",
"value": "Hot",
"shortcut": "D",
"textColor": "#FFFFFF",
"backgroundColor": "#FFFFFF"
}
],
"categories": [
{
"id": "1",
"order": 1,
"name": "First Category",
"active": true
},
{
"id": "2",
"order": 2,
"name": "Second Category",
"shortcut": "MT",
"active": true
}
],
"products": [
{
"id": "03c6787c-fc2a-4aa8-93a3-5e0f0f98cfb2",
"categoryId": "1",
"name": "First Product",
"shortcut": "First",
"options": [
{
"optionId": "1",
"price": 23
},
{
"optionId": "2",
"price": 45
}
],
"active": true
},
{
"id": "e8669cea-4c9c-431c-84ba-0b014f0f9bc2",
"categoryId": "2",
"name": "Second Product",
"shortcut": "Second",
"options": [
{
"optionId": "1",
"price": 11
},
{
"optionId": "2",
"price": 20
}
],
"active": true
}
],
"discounts": [
{
"id": "1",
"name": "S",
"type": 1,
"amount": 20,
"active": true
},
{
"id": "2",
"name": "P",
"type": 1,
"amount": 20,
"active": true
},
{
"id": "3",
"name": "G",
"type": 2,
"amount": 5,
"active": true
}
]
}
Using python, this can be easily done or almost done. Maybe this code will help you in some way to understand that.
import json,csv
data = []
with open('your_json_file_here.json') as file:
for line in file:
data.append(json.loads(line))
length = len(data)
with open('create_new_file.csv','w') as f:
writer = csv.writer(f)
writers = csv.DictWriter(f, fieldnames=['header1','header2'])
writers.writeheader()
for iter in range(length):
writer.writerow((data[iter]['specific_col_name1'],data[iter]['specific_col_name2']))
f.close()