I am trying to parse a .json file into a .kml file to be used by a plotting program. I am going to give a set a data samples to simplify the issue:
I have a LocationHistory.json file that has the following structure:
{
"data" : {
"items" : [ {
"kind" : "latitude#location",
"timestampMs" : "1374870896803",
"latitude" : 34.9482949,
"longitude" : -85.3245474,
"accuracy" : 2149
}, {
"kind" : "latitude#location",
"timestampMs" : "1374870711762",
"latitude" : 34.9857898,
"longitude" : -85.3526902,
"accuracy" : 2016"
}]
}
}
I define a function that parses the json data and then I want to feed it into a "placemark" string to write (output) into a "location.kml" file.
def parse_jason_data_to_kml_file():
kml_file = open('location.kml', "r+")
#Here I parse the info inside the LocationHistory.json file
json_file = open('LocationHistory.json')
json_string = json_file.read()
json_data = json.loads(json_string)
locations = json_data["data"]["items"]
# Here I get the placemark string structure
placemark = ["<Placemark>",
"<TimeStamp><when>the “timestampMS” value from the JSON data item</when></TimeStamp>",
"<ExtendedData>",
"<Data name=”accuracy”>",
"<value> the “accuracy” value from the JSON data item </value>]",
"</Data>",
"</ExtendedData><Point><coordinates>”longitude,latitude”</coordinates></Point>",
"</Placemark>"]
placemark = "\n".join(placemark)
# Now i try to find certain substrings in the "placemark" string that I would like to replace:
find_timestamp = placemark.find("the “timestampMS” value from the JSON data item")
find_accuracy = placemark.find("the “accuracy” value from the JSON data item")
find_longitude = placemark.find("”longitude")
find_latitude = placemark.find("latitude”")
# Next, I want to loop through the LocationHistory.json file (the data list above)
# and replace the strings with the parsed LocationHistory.json data saved as:
# location["timestampMs"], location["accuracy"], location["longitude"], location["latitude"]:
for location in locations:
if find_timestamp != -1:
placemark.replace('the “timestampMS” value from the JSON data item', location["timestampMs"])
if find_accuracy != -1:
placemark.replace('the “accuracy” value from the JSON data item', location["accuracy"])
if find_longitude != -1:
placemark.replace('”longitude', location["longitude"])
if find_latitude != -1:
placemark.replace('latitude”', location["latitude"])
kml_file.write(placemark)
kml_file.write("\n")
kml_file.close()
The aim of this code is to write the placemark string that contains the .json data, into the location.ml file, from the placemark string that looks like this:
<Placemark>
<TimeStamp><when>the “timestampMS” value from the JSON data item</when></TimeStamp>
<ExtendedData>
<Data name=”accuracy”>
<value> the “accuracy” value from the JSON data item </value>
</Data>
</ExtendedData><Point><coordinates>”longitude,latitude”</coordinates></Point>
</Placemark>
To an output, which should look like this:
<Placemark>
<TimeStamp><when>2013-07-26T22:34:56Z</when></TimeStamp>
<ExtendedData>
<Data name=”accuracy”>
<value>2149</value>
</Data>
</ExtendedData><Point><coordinates>-85.3245474,34.9482949</coordinates></Point>
</Placemark>
<Placemark>
<TimeStamp><when>2013-07-26T22:31:51Z</when></TimeStamp>
<ExtendedData>
<Data name=”accuracy”>
<value>2016</value>
</Data>
</ExtendedData><Point><coordinates>-85.3526902,34.9857898</coordinates></Point>
</Placemark>
If I try to run this code, I get the following error:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/Users/Elysium/anaconda/lib/python2.7/site-packages/spyderlib/widgets/externalshell/sitecustomize.py", line 685, in runfile
execfile(filename, namespace)
File "/Users/Elysium/anaconda/lib/python2.7/site-packages/spyderlib/widgets/externalshell/sitecustomize.py", line 78, in execfile
builtins.execfile(filename, *where)
File "/Users/Elysium/Dropbox/LeeV/Task 5 - Location Plotting/ParseToKML.py", line 77, in <module>
parse_jason_data_to_kml_file()
File "/Users/Elysium/Dropbox/LeeV/Task 5 - Location Plotting/ParseToKML.py", line 51, in parse_jason_data_to_kml_file
placemark.replace('the “timestampMS” value from the JSON data item', location["timestampMs"])
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 33: ordinal not in range(128)
Is this a problem with my code or a software problem?
From various sources I can find that it can be related to PIP or ascii or something, but no answer to help me... I'm sure I have pip installed...
Any help would be appreciated. Any suggestions to improve my code is also welcome :)
Thank You
You can try this as follows-
placemark = ["<Placemark>",
"<TimeStamp><when>%(timestampMs)r</when></TimeStamp>",
"<ExtendedData>",
"<Data name=\"accuracy\">",
"<value>%(accuracy)r</value>]",
"</Data>",
"</ExtendedData><Point><coordinates>%(longitude)r, %(latitude)r</coordinates></Point>",
"</Placemark>"]
placemark = "\n".join(placemark)
for location in locations:
temp = placemark % location
kml_file.write(temp)
kml_file.write("\n")
Here, in %(any_dict_key)r, r converts any Python object (here, the value of the any_dict_key) and then inserts it into the string. So, for your timestamps you have to convert it into a datetime object.
You can read this part of the documentation to check out the details - string-formatting-operations
Related
I'm a bit stumped on how to convert a text file of JSON to JSON array in Python?
So I have a text file that is constantly being written to with JSON entries, however the text file does not have bracket to call it as a JSON Array.
{"apple": "2", "orange": "3"}, {"apple": "5", "orange": "6"}, {"apple": "9", "orange": "10"} ...
How can I put that into a JSON array? My end goal is read the latest entry every time there is a new entry added. The JSON file is created by some other program that I have no control over, and its being written to constantly, so I can't just slap brackets to the start and end of the file.
Thanks
After you read in the file, you can treat it as a string, to which you add brackets to. Then, pass that to the json library to decode it as JSON for you.
import json
with open('data.txt') as f:
raw_data = f.read().splitlines()[-1]
list_data = f'[{raw_data}]'
json_data = json.loads(list_data)
print(json_data)
# [{'apple': '2', 'orange': '3'}, {'apple': '5', 'orange': '6'}, {'apple': '9', 'orange': '10'}]
This assumes that each line of the file should become a new array. The code extracts the last line of the file and converts it.
I am creating a JSON file in an s3 bucket using the following code -
def myconverter(o):
if isinstance(o, datetime.datetime):
return o.__str__()
s3.put_object(
Bucket='sample-bucket',
Key="sample.json",
Body = json.dumps(whole_file, default=myconverter)
)
Here, the whole_file variable is a list.
Sample of the "whole_file" variable -
[{"sample_column1": "abcd","sample_column2": "efgh"},{"sample_column1": "ijkl","sample_column2": "mnop"}]
The output "sample.json" file that I get should be in the following format -
{"sample_column1": "abcd","sample_column2": "efgh"}
{"sample_column1": "ijkl","sample_column2": "mnop"}
The output "sample.json" that I am getting is -
[{"sample_column1": "abcd","sample_column2": "efgh"},{"sample_column1": "ijkl","sample_column2": "mnop"}]
What changes should be made to get each JSON object in a single line?
You can write each entry to the file, then upload the file object to s3
import json
whole_file = [{"sample_column1": "abcd","sample_column2": "efgh"},
{"sample_column1": "ijkl","sample_column2": "mnop"}
]
with open("temp.json", "w") as temp:
for record in whole_file:
temp.write(json.dumps(record, default=str))
temp.write("\n")
The lookput should look like this
~ cat temp.json
{"sample_column1": "abcd", "sample_column2": "efgh"}
{"sample_column1": "ijkl", "sample_column2": "mnop"}
upload the file
import boto3
s3 = boto3.client("s3")
s3.upload_file("temp.json", bucket, object_name="whole_file.json")
I am currently trying to write 2 pieces of user inputted code to a .json file without clearing the existing data. I believe it is a problem with the logins.append as it says there is no such thing. What would i have to use?
I have been searching around trying to find different suffixes to the logins.
def i():
path_to_json = "./logins.json"
with open("logins.json", "r") as content:
logins = json.load(content)
with open('logins.json', 'a') as outfile:
username = str(input('New Username: '))
password = str(input('New Password: '))
logins.append({username:password})
I am getting the error:
Traceback (most recent call last):
File "<pyshell#5>", line 1, in <module>
i()
File "N:\NEA Computing\NEA code.py", line 188, in i
logins.append({username: password})
AttributeError: 'dict' object has no attribute 'append'
I am expecting it to add data to the .json file without deleting the other data however i am getting an error and nothing is being written to the .json file.
Try to open the file with 'w' option
def i():
path_to_json = "./logins.json"
with open("logins.json", "w") as content:
logins = json.load(content)
with open('logins.json', 'a') as outfile:
username = str(input('New Username: '))
password = str(input('New Password: '))
logins.append({username:password})
Maybe this is a misunderstanding of the question, but what is happening, as far as I can see, is that you use json.load, which seems to load an object literal like {key1: value1, key2: value2,...} and not an array literal like [value1, value2, ...]. Python converts the first one into a python dict, and a dictionary has no append function, only an array / a list has. You get your error and execution halts. When you have the content of logins.json as object {...}, it doesn't work for me but as array/list [...] it works
So I have the following dictionaries that I get by parsing a text file
keys = ["scientific name", "common names", "colors]
values = ["somename1", ["name11", "name12"], ["color11", "color12"]]
keys = ["scientific name", "common names", "colors]
values = ["somename2", ["name21", "name22"], ["color21", "color22"]]
and so on. I am dumping the key value pairs using a dictionary to a json file using a for loop where I go through each key value pair one by one
for loop starts
d = dict(zip(keys, values))
with open("file.json", 'a') as j:
json.dump(d, j)
If I open the saved json file I see the contents as
{"scientific name": "somename1", "common names": ["name11", "name12"], "colors": ["color11", "color12"]}{"scientific name": "somename2", "common names": ["name21", "name22"], "colors": ["color21", "color22"]}
Is this the right way to do it?
The purpose is to query the common name or colors for a given scientific name. So then I do
with open("file.json", "r") as j:
data = json.load(j)
I get the error, json.decoder.JSONDecodeError: Extra data:
I think this is because I am not dumping the dictionaries in json in the for loop correctly. I have to insert some square brackets programatically. Just doing json.dump(d, j) won't suffice.
JSON may only have one root element. This root element can be [], {} or most other datatypes.
In your file, however, you get multiple root elements:
{...}{...}
This isn't valid JSON, and the error Extra data refers to the second {}, where valid JSON would end instead.
You can write multiple dicts to a JSON string, but you need to wrap them in an array:
[{...},{...}]
But now off to how I would fix your code. First, I rewrote what you posted, because your code was rather pseudo-code and didn't run directly.
import json
inputs = [(["scientific name", "common names", "colors"],
["somename1", ["name11", "name12"], ["color11", "color12"]]),
(["scientific name", "common names", "colors"],
["somename2", ["name21", "name22"], ["color21", "color22"]])]
for keys, values in inputs:
d = dict(zip(keys, values))
with open("file.json", 'a') as j:
json.dump(d, j)
with open("file.json", 'r') as j:
print(json.load(j))
As you correctly realized, this code failes with
json.decoder.JSONDecodeError: Extra data: line 1 column 105 (char 104)
The way I would write it, is:
import json
inputs = [(["scientific name", "common names", "colors"],
["somename1", ["name11", "name12"], ["color11", "color12"]]),
(["scientific name", "common names", "colors"],
["somename2", ["name21", "name22"], ["color21", "color22"]])]
jsonData = list()
for keys, values in inputs:
d = dict(zip(keys, values))
jsonData.append(d)
with open("file.json", 'w') as j:
json.dump(jsonData, j)
with open("file.json", 'r') as j:
print(json.load(j))
Also, for python's json library, it is important that you write the entire json file in one go, meaning with 'w' instead of 'a'.
How to write the returned dict into a json file.
until now I am able to returned the correct data and print it, but when I tried to write it into a JSON file it just write the last record.
I will appreciate any help to fix this
example :
printed data:
[{"file Name": "test1.txt", "searched Word": "th", "number of occurence": 1}]
[{"file Name": "test2.txt", "searched Word": "th", "number of occurence": 1}]
json file
[{"file Name": "test2.txt", "searched Word": "th", "number of occurence": 1}]
code:
import re
import json
import os.path
import datetime
for counter, myLine in enumerate(textList):
thematch=re.sub(searchedSTR,RepX,myLine)
matches = re.findall(searchedSTR, myLine, re.MULTILINE | re.IGNORECASE)
if len(matches) > 0:
# add one record for the match (add one because line numbers start with 1)
d[matches[0]].append(counter + 1)
self.textEdit_PDFpreview.insertHtml(str(thematch))
'''
loop over the selected file and extract 3 values:
==> name of file
==> searched expression
==> number of occurence
'''
for match, positions in d.items():
listMetaData = [{"file Name":fileName,"searched Word":match,"number of occurence":len(positions)}]
jsondata = json.dumps(listMetaData)
print("in the for loop ==>jsondata: \n{0}".format(jsondata))
'''
create a folder called 'search_result' that includes all the result of the searching as JSON file
where the system check if the folder exist will continue if not the system create the folder
insode the folder the file will be created as ==> today_searchResult.js
'''
if not(os.path.exists("./search_result")):
try:
#print(os.mkdir("./search_result"))
#print(searchResultFoder)
today = datetime.date.today()
fileName = "{}_searchResult.json".format(today)
#fpJ = os.path.join(os.mkdir("./search_result"),fileName)
#print(fpJ)
with open(fileName,"w") as jsf:
jsf.write(jsondata)
print("finish writing")
except Exception as e:
print(e)