How to skip text at the beginning of a json file in python - json

I have a json object file and at the beginning of this file a header of text, date, time, IP address.
I've tried f.readlines()[5:] to no avail.
I've tried next(f)
I wish to skip 5 or 6 lines of text and go directly into the json data.
Here is an example.
import jsonlines
import json
data_file = input("Enter a file to parse: ")
with jsonlines.open(data_file) as file:
for obj in file:
try:
jsonparse = json.loads(obj)
except Exception as e:
pass
print(obj)
Error:
jsonlines.jsonlines.InvalidLineError: line contains invalid json: Expecting value: line 1 column 1 (char 0) (line 1)
Top of json file:
Start: 07/02/2019 14:59:40.686
Connected To:
192.168.11.203
Here is the full long error:
Enter a file to parse: Play.raw
Traceback (most recent call last):
File "C:\Users\sdickey\AppData\Local\Programs\Python\Python37\lib\site-packages\jsonlines\jsonlines.py", line 159, in read
value = self._loads(line)
File "C:\Users\sdickey\AppData\Local\Programs\Python\Python37\lib\json\__init__.py", line 348, in loads
return _default_decoder.decode(s)
File "C:\Users\sdickey\AppData\Local\Programs\Python\Python37\lib\json\decoder.py", line 337, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "C:\Users\sdickey\AppData\Local\Programs\Python\Python37\lib\json\decoder.py", line 355, in raw_decode
raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "C:/Users/sdickey/PycharmProjects/Python Testing/Actall Data/testing.py", line 6, in <module>
for obj in file:
File "C:\Users\sdickey\AppData\Local\Programs\Python\Python37\lib\site-packages\jsonlines\jsonlines.py", line 204, in iter
skip_empty=skip_empty)
File "C:\Users\sdickey\AppData\Local\Programs\Python\Python37\lib\site-packages\jsonlines\jsonlines.py", line 164, in read
six.raise_from(exc, orig_exc)
File "<string>", line 3, in raise_from
jsonlines.jsonlines.InvalidLineError: line contains invalid json: Expecting value: line 1 column 1 (char 0) (line 1)
Process finished with exit code 1

It is always preferable not to read the whole file in memory but one line at a time.
Assuming your input file contains:
first line
second line
third line
fourth line
fifth line
{ "k1": "val1", "k2": "val2" }
{ "k3": "val3", "k4": "val4" }
if you just want to skip 5 lines, you could do it brutally as:
import json
with open("test.txt") as f:
for _ in range(5):
next(f)
for line in f:
obj = json.loads(line)
print(obj)
or using enumerate:
import json
with open("test.txt") as f:
for i, line in enumerate(f):
if i<5:
continue
obj = json.loads(line)
print(obj)
or use itertools' dropwhile:
import itertools as it
import json
with open("test.txt") as f:
for i, line in it.dropwhile(lambda i: i[0]<5, enumerate(f)):
obj = json.loads(line)
print(obj)

I think you are trying to convert to json line by line. You need to join all lines by \n igoring the first 5 lines and then load it:
import json
with open("test.txt") as f:
json_obj = "\n".join(f.readlines()[5:])
jsonparse = json.loads(json_obj)
print(jsonparse)

Related

How can I fix this error: json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

I am trying to create an REST API and a part of the project requires working with JSON files.
I am trying to open a JSON file, to check whether certain content exists in the file. If it doesn't, it will be added. For that reason, I am trying to load the JSON file in order to use the dictionary methods. When I open the file and try to use the json.loads() function, I get an error. I am attaching the part of the code that doesn't work.
file = open(self.storage_file_name, 'r')
if file_is_empty:
content_to_write = json.dumps(self.states_population_json, indent=4)
file.write(content_to_write)
else:
current_date = str(date.today())
print(file.read())
json_content = json.loads(file.read()) # THIS IS WHERE THE ISSUE APPEARS <<<
if current_date in self.states_population_json["features"].keys():
pass
else:
json_content["features"][current_date] = self.states_population_json["features"][current_date]
Here's the error as well:
Traceback (most recent call last):
File "C:\Users\denis\AppData\Local\Programs\Python\Python39-32\lib\json\decoder.py", line 355, in raw_decode
raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
I tried using json.load() instead of json.loads(), but then I get the following error:
Traceback (most recent call last):
File "C:\Users\denis\Desktop\Programming\ESRI_Internship\test_file.py", line 89, in <module>
data.save_data_json()
File "C:\Users\denis\Desktop\Programming\ESRI_Internship\test_file.py", line 79, in save_data_json
json_content = json.load(file)
File "C:\Users\denis\AppData\Local\Programs\Python\Python39-32\lib\json\__init__.py", line 293, in load
return loads(fp.read(),
File "C:\Users\denis\AppData\Local\Programs\Python\Python39-32\lib\json\__init__.py", line 346, in loads
return _default_decoder.decode(s)
File "C:\Users\denis\AppData\Local\Programs\Python\Python39-32\lib\json\decoder.py", line 337, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "C:\Users\denis\AppData\Local\Programs\Python\Python39-32\lib\json\decoder.py", line 355, in raw_decode
raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
Process finished with exit code 1
I made sure that the file isn't empty.
Thanks in advance!
EDIT:
Here is the content of the file (shortened a bit):
{
"features": {
"2022-01-06": [
{
"STATE_NAME": "Alabama",
"POPULATION": 5028316
},
{
"STATE_NAME": "North Carolina",
"POPULATION": 10736879
},
{
"STATE_NAME": "North Dakota",
"POPULATION": 321419
}
]
}
}
The line print(file.read()) moves the file cursor for reading to the end of the file (the command reads the whole file). A subsequent file.read() thus will not be able to read anything. Just remove the print statement, or if needed, move the file cursor back to the beginning of the file by executing file.seek(0) before the json.loads() statement.

Safely parsing nested json double quotes

The issue:
>>> import json
>>> json_str='{"message": "John said "come here!""}'
>>> json.loads(json_str)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/Users/ram/.pyenv/versions/3.9.6/lib/python3.9/json/__init__.py", line 346, in loads
return _default_decoder.decode(s)
File "/Users/ram/.pyenv/versions/3.9.6/lib/python3.9/json/decoder.py", line 337, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/Users/ram/.pyenv/versions/3.9.6/lib/python3.9/json/decoder.py", line 353, in raw_decode
obj, end = self.scan_once(s, idx)
json.decoder.JSONDecodeError: Expecting ',' delimiter: line 1 column 25 (char 24)
Then I attempted to escape the double-quotes. Like so:
>>> json_str='{"message": "John said \"come here!\""}'
>>> json.loads(json_str)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/Users/ram/.pyenv/versions/3.9.6/lib/python3.9/json/__init__.py", line 346, in loads
return _default_decoder.decode(s)
File "/Users/ram/.pyenv/versions/3.9.6/lib/python3.9/json/decoder.py", line 337, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/Users/ram/.pyenv/versions/3.9.6/lib/python3.9/json/decoder.py", line 353, in raw_decode
obj, end = self.scan_once(s, idx)
json.decoder.JSONDecodeError: Expecting ',' delimiter: line 1 column 25 (char 24)
The following works though:
>>> json_str='{"message": "John said \'come here!\'"}'
>>> json.loads(json_str)
{'message': "John said 'come here!'"}
What am I doing wrong? Is it possible to flag the json library to be more forgiving (helps me leave json parsing to the json library)?
PS: Following the answer here, I attempted the following:
>>> data=json.dumps('{"message":"John said "hello!""}')
>>> print(data)
"{\"message\":\"John said \"hello!\"\"}"
>>> dict = json.loads(data)
>>> print(dict)
{"message":"John said "hello!""}
>>> print(dict['message'])
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: string indices must be integers
>>> print(dict[0])
{
The last json.loads is returning a string instead of dict. But the correctly escaped quotes in data did sound promising.
You can simply use either a raw string or double backslashes so that you both escape the backslashes in python as well as in the json string so that json.loads can see the backslashes.
>>> json.loads(r'{"message": "John said \"come here!\""}')
{'message': 'John said "come here!"'}
>>> json.loads('{"message": "John said \\"come here!\\""}')
{'message': 'John said "come here!"'}

Write the correct format of JSON to file using json.dumps() in Python 3

Let say I have a simple dictionary as below:
a = {'animal' : 'fish', 'fruit' : 'apple','vehicle' : 'car'}
I wish to convert a into JSON and write to a file. If I used json.dumps() as below:
b = jason.dumps(a)
with open('output.json', 'w') as f:
f.write(pprint.pformat(b))
The content of the JSON file output.json became:
'{"animal": "fish", "fruit": "apple", "vehicle": "car"}'
What I wish to actually have (in the file) is without quote as below:
{"animal": "fish", "fruit": "apple", "vehicle": "car"}
What is the right way to code it so that I can get the desired output?
Thanks in advance.
[EDIT]: The reason why I need the desired output in file is because I received the error below when I loaded the file.
>>> with open("output.json") as f:
... data = json.load(f)
...
Traceback (most recent call last):
File "<stdin>", line 2, in <module>
File "/Users/louis.kok/Analyses/devel/anaconda3/lib/python3.7/json/__init__.py", line 296, in load
parse_constant=parse_constant, object_pairs_hook=object_pairs_hook, **kw)
File "/Users/louis.kok/Analyses/devel/anaconda3/lib/python3.7/json/__init__.py", line 348, in loads
return _default_decoder.decode(s)
File "/Users/louis.kok/Analyses/devel/anaconda3/lib/python3.7/json/decoder.py", line 337, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/Users/louis.kok/Analyses/devel/anaconda3/lib/python3.7/json/decoder.py", line 355, in raw_decode
raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
Import json library
import json
Save your file properly using json.dump
with open('output.json', 'w') as f:
json.dump(a, f)
Read it using json.load
with open("output.json") as f:
data = json.load(f)
Try this:
with open('task.json', 'w') as f:
s = json.dumps(mydict, indent=2)
f.write('%s\n' % (s))

Error json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

I want to create a temporary JSON file to store the google calendar credentials, that were stored in a "job" object.
I am using the ServiceAccountCredentials to then get the credentials from the file.
Client = {
"clientID": job.getClientID(),
"clientSecret": job.getClientSecret()
}
temp = tempfile.NamedTemporaryFile(mode="w+b", suffix=".json")
complex_data = open(temp.name, "w", encoding="UTF-8")
complex_data.write(json.dumps(Client))
# data = complex_data.write(json.dumps(Client))
# z = json.loads(data)
credentials = ServiceAccountCredentials.from_json(
temp.name
)
```
I get the following error:
Traceback (most recent call last):
File "/var/www/library-offers-google-calendar/main.py", line 209, in <module>
temp.name
File "/var/www/library-offers-google-calendar/venv/lib/python3.7/site-packages/oauth2client/service_account.py", line 436, in from_json
json_data = json.loads(_helpers._from_bytes(json_data))
File "/usr/lib/python3.7/json/__init__.py", line 348, in loads
return _default_decoder.decode(s)
File "/usr/lib/python3.7/json/decoder.py", line 337, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/usr/lib/python3.7/json/decoder.py", line 355, in raw_decode
raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
From the docs, the method ServiceAccountCredentials.from_json accepts a json data rather than a json file name.
If you want to use a json file name, you need to use the method ServiceAccountCredentials.from_json_keyfile_name
Or else, you can directly use Client dictionary as a parameter to ServiceAccountCredentials.from_json without creating and reading from temporary file.

json decod error when using POST in chalice

When I try to use app.current_request.json_body in chalice I get a decode error:
Traceback (most recent call last): File "/var/task/chalice/app.py",
line 659, in _get_view_function_response
response = view_function(**function_args) File "/var/task/app.py", line 34, in post_item
data = app.current_request.json_body File "/var/task/chalice/app.py", line 303, in json_body
self._json_body = json.loads(self.raw_body) File "/var/lang/lib/python3.6/json/init.py", line 354, in loads
return _default_decoder.decode(s) File "/var/lang/lib/python3.6/json/decoder.py", line 339, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end()) File "/var/lang/lib/python3.6/json/decoder.py", line 357, in raw_decode
raise JSONDecodeError("Expecting value", s, err.value) from None json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char
0)
It doesn't matter how simple the data is. Example: {"Company":"ABC"} or {}.
As can be seen in the following code in the API Gateway all I try to do is return the data that has been sent so I don't think this is the problem:
#app.route('/test', methods=['POST'], content_types=['application/json'], cors=cors_config)
def post_item(data):
data = app.current_request.json_body
return data
Does anyone know what I might have done wrong?
You must remove the data from the parameters of the function.
That is used for pass url parameters.
#app.route('/test', methods=['POST'], content_types=['application/json'], cors=cors_config)
def post_item():
data = app.current_request.json_body
return data