I got a JSON text which I should parse, but for some reason I can't parse it because it has another array inside. My JSON looks like that:
{
"statementId": "1",
"movements": [
{
"id": 65,
"date": "2019-02-05",
"number": 32,
"balance": -4.62,
"purpose": "1"
},
{
"id": 1,
"date": "2019-02-05",
"number": 22,
"balance": -3,
"purpose": "23"
},
{
"id": 32,
"date": "2019-02-05",
"number": 12,
"balance": -11,
"purpose": "2"
}
],
"startPointer": "1122",
"endPointer": "3333"
}
I am using JsonSlurper. I want to know if it is possible to catch all the data inside "movements", I have tried to use this script:
JsonSlurper slurper = new JsonSlurper()
Map parsedJson = slurper.parseText(bodyContent)
String parsed_movements = parsedJson["movements"]
I have no problem with parsing single strings, like statementId or startPointer, but when I try to parse movements with my script it gives me result as null. I have also tried parsedJson["movements"][0] to catch first movement but it also gives me an error.
I have found a lot of things about json parsers on internet and also on stackoverflow but nothing what I seek. I really don't think that it is a duplicate question.
EDIT: I tried for statement also to put each object in array like that:
def movements_array = []
for(def i = 0; i < parsedJson.movements.size(); i++) {
movements_array << parsedJson.movements[i].id
println(movements_array)
}
But it gives me an error: Cannot invoke method size() on null object, because parsedJson.movements is null.
When you do:
String parsed_movements = parsedJson["movements"]
You're sticking a map into a String, which isn't what you want.
Given the json in your question, you can just do
def movementIds = new JsonSlurper().parseText(bodyContents).movements.id
To get a list of [65, 1, 32]
If you're getting NPEs I assume the json isn't what you show in the question
I have file that is list of JSON objects. It looks like this :
[
{
"id": 748,
"location": {
"slug": "istanbul",
"parent": {
"id": 442,
"slug": "turkey"
}
},
"rank": 110
},
{
"id": 769,
"location": {
"slug": "dubai",
"parent": {
"id": 473,
"slug": "uae"
}
},
"rank": 24
}
]
I want to create a list of hotel parent names, so i write this code to do this, I read the JSON file and assigned it to a variable, that part is correct. But look at this code :
with open('hotels.json', 'r', encoding="utf8") as hotels_data:
hotels = json.load(hotels_data)
parents_list = []
for item in hotels:
if item["location"]["parent"]["slug"] not in parents_list:
parents_list.append(item["location"]["parent"])
when i run this code, i give this error :
if item["location"]["parent"]["slug"] not in parents_list:
TypeError: 'NoneType' object is not subscriptable
This code does not work, so I tried to print the JSON objects so I wrote this in the loop:
print(item["location"]["parent"]["slug"])
This code prints the values I want, but also give me the exact same error.
thank you for any help.
I tried running the code and it seems to be working fine with your dataset.
However, instead of opening the file to read the data, I just assigned hotels with your dataset, hotels = [...].
The result I got was this:
[{'id': 442, 'slug': 'turkey'}, {'id': 473, 'slug': 'uae'}]
What is your result if you print hotels, is it the same as you shown here?
If you actually have a lot more data in your dataset, then I can presume that some of the dictionaries don't contain item["location"]["parent"]["slug"]. If that is the case, you should skip those by checking if that element exists in each item first before reading off from the parents_list.
For example:
try:
item["location"]["parent"]["slug"]
except (KeyError, TypeError) as e:
pass
else:
if item["location"]["parent"]["slug"] not in parents_list:
parents_list.append(item["location"]["parent"])
I cannot replicate the same error as you. The only thing that I can think of is that the last item in each object in the JSON shouldn't have a comma after it. See if that fixes your error
I am getting some data from a JSON file "new.json", and I want to filter some data and store it into a new JSON file. Here is my code:
import json
with open('new.json') as infile:
data = json.load(infile)
for item in data:
iden = item.get["id"]
a = item.get["a"]
b = item.get["b"]
c = item.get["c"]
if c == 'XYZ' or "XYZ" in data["text"]:
filename = 'abc.json'
try:
outfile = open(filename,'ab')
except:
outfile = open(filename,'wb')
obj_json={}
obj_json["ID"] = iden
obj_json["VAL_A"] = a
obj_json["VAL_B"] = b
And I am getting an error, the traceback is:
File "rtfav.py", line 3, in <module>
data = json.load(infile)
File "/usr/lib64/python2.7/json/__init__.py", line 278, in load
**kw)
File "/usr/lib64/python2.7/json/__init__.py", line 326, in loads
return _default_decoder.decode(s)
File "/usr/lib64/python2.7/json/decoder.py", line 369, in decode
raise ValueError(errmsg("Extra data", s, end, len(s)))
ValueError: Extra data: line 88 column 2 - line 50607 column 2 (char 3077 - 1868399)
Here is a sample of the data in new.json, there are about 1500 more such dictionaries in the file
{
"contributors": null,
"truncated": false,
"text": "#HomeShop18 #DreamJob to professional rafter",
"in_reply_to_status_id": null,
"id": 421584490452893696,
"favorite_count": 0,
"source": "Mobile Web (M2)",
"retweeted": false,
"coordinates": null,
"entities": {
"symbols": [],
"user_mentions": [
{
"id": 183093247,
"indices": [
0,
11
],
"id_str": "183093247",
"screen_name": "HomeShop18",
"name": "HomeShop18"
}
],
"hashtags": [
{
"indices": [
12,
21
],
"text": "DreamJob"
}
],
"urls": []
},
"in_reply_to_screen_name": "HomeShop18",
"id_str": "421584490452893696",
"retweet_count": 0,
"in_reply_to_user_id": 183093247,
"favorited": false,
"user": {
"follow_request_sent": null,
"profile_use_background_image": true,
"default_profile_image": false,
"id": 2254546045,
"verified": false,
"profile_image_url_https": "https://pbs.twimg.com/profile_images/413952088880594944/rcdr59OY_normal.jpeg",
"profile_sidebar_fill_color": "171106",
"profile_text_color": "8A7302",
"followers_count": 87,
"profile_sidebar_border_color": "BCB302",
"id_str": "2254546045",
"profile_background_color": "0F0A02",
"listed_count": 1,
"profile_background_image_url_https": "https://abs.twimg.com/images/themes/theme1/bg.png",
"utc_offset": null,
"statuses_count": 9793,
"description": "Rafter. Rafting is what I do. Me aur mera Tablet. Technocrat of Future",
"friends_count": 231,
"location": "",
"profile_link_color": "473623",
"profile_image_url": "http://pbs.twimg.com/profile_images/413952088880594944/rcdr59OY_normal.jpeg",
"following": null,
"geo_enabled": false,
"profile_banner_url": "https://pbs.twimg.com/profile_banners/2254546045/1388065343",
"profile_background_image_url": "http://abs.twimg.com/images/themes/theme1/bg.png",
"name": "Jayy",
"lang": "en",
"profile_background_tile": false,
"favourites_count": 41,
"screen_name": "JzayyPsingh",
"notifications": null,
"url": null,
"created_at": "Fri Dec 20 05:46:00 +0000 2013",
"contributors_enabled": false,
"time_zone": null,
"protected": false,
"default_profile": false,
"is_translator": false
},
"geo": null,
"in_reply_to_user_id_str": "183093247",
"lang": "en",
"created_at": "Fri Jan 10 10:09:09 +0000 2014",
"filter_level": "medium",
"in_reply_to_status_id_str": null,
"place": null
}
As you can see in the following example, json.loads (and json.load) does not decode multiple json object.
>>> json.loads('{}')
{}
>>> json.loads('{}{}') # == json.loads(json.dumps({}) + json.dumps({}))
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "C:\Python27\lib\json\__init__.py", line 338, in loads
return _default_decoder.decode(s)
File "C:\Python27\lib\json\decoder.py", line 368, in decode
raise ValueError(errmsg("Extra data", s, end, len(s)))
ValueError: Extra data: line 1 column 3 - line 1 column 5 (char 2 - 4)
If you want to dump multiple dictionaries, wrap them in a list, dump the list (instead of dumping dictionaries multiple times)
>>> dict1 = {}
>>> dict2 = {}
>>> json.dumps([dict1, dict2])
'[{}, {}]'
>>> json.loads(json.dumps([dict1, dict2]))
[{}, {}]
Iterate over the file, loading each line as JSON in the loop:
tweets = []
for line in open('tweets.json', 'r'):
tweets.append(json.loads(line))
This avoids storing intermediate python objects. As long as you write one full tweet per append() call, this should work.
I came across this because I was trying to load a JSON file dumped from MongoDB. It was giving me an error
JSONDecodeError: Extra data: line 2 column 1
The MongoDB JSON dump has one object per line, so what worked for me is:
import json
data = [json.loads(line) for line in open('data.json', 'r')]
This may also happen if your JSON file is not just 1 JSON record.
A JSON record looks like this:
[{"some data": value, "next key": "another value"}]
It opens and closes with a bracket [ ], within the brackets are the braces { }. There can be many pairs of braces, but it all ends with a close bracket ].
If your json file contains more than one of those:
[{"some data": value, "next key": "another value"}]
[{"2nd record data": value, "2nd record key": "another value"}]
then loads() will fail.
I verified this with my own file that was failing.
import json
guestFile = open("1_guests.json",'r')
guestData = guestFile.read()
guestFile.close()
gdfJson = json.loads(guestData)
This works because 1_guests.json has one record []. The original file I was using all_guests.json had 6 records separated by newline. I deleted 5 records, (which I already checked to be bookended by brackets) and saved the file under a new name. Then the loads statement worked.
Error was
raise ValueError(errmsg("Extra data", s, end, len(s)))
ValueError: Extra data: line 2 column 1 - line 10 column 1 (char 261900 - 6964758)
PS. I use the word record, but that's not the official name. Also, if your file has newline characters like mine, you can loop through it to loads() one record at a time into a json variable.
I just got the same error while my json file is like this
{"id":"1101010","city_id":"1101","name":"TEUPAH SELATAN"}
{"id":"1101020","city_id":"1101","name":"SIMEULUE TIMUR"}
And I found it malformed, so I changed it to:
{
"datas":[
{"id":"1101010","city_id":"1101","name":"TEUPAH SELATAN"},
{"id":"1101020","city_id":"1101","name":"SIMEULUE TIMUR"}
]
}
One-liner for your problem:
data = [json.loads(line) for line in open('tweets.json', 'r')]
If you want to solve it in a two-liner you can do it like this:
with open('data.json') as f:
data = [json.loads(line) for line in f]
The error is due to the \nsymbol if you use the read()method of the file descriptor... so don't bypass the problem by using readlines()& co but just remove such character!
import json
path = # contains for example {"c": 4} also on multy-lines
new_d = {'new': 5}
with open(path, 'r') as fd:
d_old_str = fd.read().replace('\n', '') # remove all \n
old_d = json.loads(d_old_str)
# update new_d (python3.9 otherwise new_d.update(old_d))
new_d |= old_d
with open(path2, 'w') as fd:
fd.write(json.dumps(new_d)) # save the dictionary to file (in case needed)
... and if you really really want to use readlines() here an alternative solution
new_d = {'new': 5}
with open('some_path', 'r') as fd:
d_old_str = ''.join(fd.readlines()) # concatenate the lines
d_old = json.loads(d_old_str)
# then as above
I think saving dicts in a list is not an ideal solution here proposed by #falsetru.
Better way is, iterating through dicts and saving them to .json by adding a new line.
Our 2 dictionaries are
d1 = {'a':1}
d2 = {'b':2}
you can write them to .json
import json
with open('sample.json','a') as sample:
for dict in [d1,d2]:
sample.write('{}\n'.format(json.dumps(dict)))
And you can read json file without any issues
with open('sample.json','r') as sample:
for line in sample:
line = json.loads(line.strip())
Simple and efficient
My json file was formatted exactly as the one in the question but none of the solutions here worked out. Finally I found a workaround on another Stackoverflow thread. Since this post is the first link in Google search, I put the that answer here so that other people come to this post in the future will find it more easily.
As it's been said there the valid json file needs "[" in the beginning and "]" in the end of file. Moreover, after each json item instead of "}" there must be a "},". All brackets without quotations! This piece of code just modifies the malformed json file into its correct format.
https://stackoverflow.com/a/51919788/2772087
If your data is from a source outside your control, use this
def load_multi_json(line: str) -> [dict]:
"""
Fix some files with multiple objects on one line
"""
try:
return [json.loads(line)]
except JSONDecodeError as err:
if err.msg == 'Extra data':
head = [json.loads(line[0:err.pos])]
tail = FrontFile.load_multi_json(line[err.pos:])
return head + tail
else:
raise err
I have a web-service call (HTTP Get) that my Python script makes in which returns a JSON response. The response looks to be a list of Dictionaries. The script's purpose is to iterate through the each dictionary, extract each piece of metadata (i.e. "ClosePrice": "57.74",) and write each dictionary to its own row in Mssql.
The issue is, I don't think Python is recognizing the JSON output from the API call as a list of dictionaries, and when I try a for loop, I'm getting the error must be int not str. I have tried converting the output to a list, dictionary, tuple. I've also tried to make it work with List Comprehension, with no luck. Further, if I copy/paste the data from the API call and assign it to a variable, it recognizes that its a list of dictionaries without issue. Any help would be appreciated. I'm using Python 2.7.
Here is the actual http call being made: http://test.kingegi.com/Api/QuerySystem/GetvalidatedForecasts?user=kingegi&market=us&startdate=08/19/13&enddate=09/12/13
Here is an abbreviated JSON output from the API call:
[
{
"Id": "521d992cb031e30afcb45c6c",
"User": "kingegi",
"Symbol": "psx",
"Company": "phillips 66",
"MarketCap": "34.89B",
"MCapCategory": "large",
"Sector": "basic materials",
"Movement": "up",
"TimeOfDay": "close",
"PredictionDate": "2013-08-29T00:00:00Z",
"Percentage": ".2-.9%",
"Latency": 37.48089483333333,
"PickPosition": 2,
"CurrentPrice": "57.10",
"ClosePrice": "57.74",
"HighPrice": null,
"LowPrice": null,
"Correct": "FALSE",
"GainedPercentage": 0,
"TimeStamp": "2013-08-28T02:31:08 778",
"ResponseMsg": "",
"Exchange": "NYSE "
},
{
"Id": "521d992db031e30afcb45c71",
"User": "kingegi",
"Symbol": "psx",
"Company": "phillips 66",
"MarketCap": "34.89B",
"MCapCategory": "large",
"Sector": "basic materials",
"Movement": "down",
"TimeOfDay": "close",
"PredictionDate": "2013-08-29T00:00:00Z",
"Percentage": "16-30%",
"Latency": 37.4807215,
"PickPosition": 1,
"CurrentPrice": "57.10",
"ClosePrice": "57.74",
"HighPrice": null,
"LowPrice": null,
"Correct": "FALSE",
"GainedPercentage": 0,
"TimeStamp": "2013-08-28T02:31:09 402",
"ResponseMsg": "",
"Exchange": "NYSE "
}
]
Small Part of code being used:
import os,sys
import subprocess
import glob
from os import path
import urllib2
import json
import time
try:
data = urllib2.urlopen('http://api.kingegi.com/Api/QuerySystem/GetvalidatedForecasts?user=kingegi&market=us&startdate=08/10/13&enddate=09/12/13').read()
except urllib2.HTTPError, e:
print "HTTP error: %d" % e.code
except urllib2.URLError, e:
print "Network error: %s" % e.reason.args[1]
list_id=[x['Id'] for x in data] #test to see if it extracts the ID from each Dict
print(data) #Json output
print(len(data)) #should retrieve the number of dict in list
UPDATE
Answered my own question, here is the method below:
`url = 'some url that is a list of dictionaries' #GetCall
u = urllib.urlopen(url) # u is a file-like object
data = u.read()
newdata = json.loads(data)
print(type(newdata)) # printed data type will show as a list
print(len(newdata)) #the length of the list
newdict = newdata[1] # each element in the list is a dict
print(type(newdict)) # this element is a dict
length = len(newdata) # how many elements in the list
for a in range(1,length): #a is a variable that increments itself from 1 until a number
var = (newdata[a])
print(var['Correct'], var['User'])`
var data = xhr.responseText;
When I output this console.log(xhr.responseText). Below is my output
["{id:1,name\":\"JOHN\",\"city\":\"null\"}"
,"{\"id\":2,\"name\":\"MICHEAL\,\"city\":\"null\"}"]
How do I get id, name. I tried like this data.id but I get this error
jquery JSON.parse: unexpected end of data.
Update
I am using code igniter with data mapper so my data mapper is giving that json response. Do you know, how I can resolve it.
You've already been told what the problem is in the comments: the JSON generated by the server is invalid. You are probably not using a library to encode your JSON, don't ever encode it by hand.
Your JSON should probably look like the following (when pretty printed) http://jsfiddle.net/7FKWr/
[
{"id": 1, "name": "JOHN", "city": null},
{"id": 2, "name": "MICHEAL", "city": null}
]