How remove from my json dump backslashes?
my python code is :
#sio.on('donation')
def on_message(data):
y = json.loads(data)
with open('donate.json', 'w') as outfile:
json.dump(data, outfile)
if i make print all fine and no backslashes!But if i open my json file he look like this :
"{\"id\":107864345,\"alert_type\":\"1\",\"is_shown\":\"0\",\"additional_data\":\"{\\\"randomness\\\":811}\",\"billing_system\":\"fake\",\"billing_system_type\":null,\"username\":\"test24\",\"amount\":\"1.00\",\"amount_formatted\":\"1\",\"amount_main\":1,\"currency\":\"USD\",\"message\":\"aaaaaa aaaa\",\"header\":\"\",\"date_created\":\"2022-12-17 21:57:10\",\"emotes\":null,\"ap_id\":null,\"_is_test_alert\":true,\"message_type\":\"text\",\"preset_id\":0}"
i try all what i know
Your code, annotated:
def on_message(data):
When this function is called, it is provided with the argument data, which is a string containing the JSON encoding for a complex object.
y = json.loads(data)
Now data is still the same string, and y is the complex object which was represented by data.
with open('donate.json', 'w') as outfile:
json.dump(data, outfile)
json.dump takes a data object and turns it into a string. With two arguments, as here, it writes the string to a file. But despite its name, data is not the dara object. It's a string. The data object is y.
json.dump will convert any Python object with a JSON representation to a string representing that object, and a string can be represented in JSON. So in this case, the string in data is encoded as a JSON representation. That means that the string must be enclosed in double quotes and any special characters escaped.
But that's not what you wanted. You wanted to dump the data object, which you have named y. Changing that line to
json.dump(y, outfile)
Will probably do what you want.
But if you just wanted to write out the string, there wasn't much point converting it to JSON and back to a string. You could just write it out:
outfile.write(data)
Then you can get rid of y (unless you need it somewhere else).
Related
I have the below json file which i want to read into a dataframe but i wm getting error as the json file has double quotes within the string.for example:
data:{
"Field1":"val"ue 1",
"Field2":"value2",
"Field3":"va"lu"e3"
}
Required output"
Field1,Field2,Field3
Value1,value2,value2
Your json is not valid (because of the nested double quotes), this is why you have an error when you read the file with Spark data source API or with any other Json parser.
What you can do is to read your file as a dataset of String, then clean each String using a Regex to remove the useless double quotes, and finally use "from_json" function in order to parse each string and convert your dataset from a Dataset[String] to a Dataset[< your case class >].
I have uploaded a *.mat file that contains a 'struct' to my jupyter lab using:
from pymatreader import read_mat
data = read_mat(mat_file)
Now I have a multi-dimensional dictionary, for example:
data['Forces']['Ss1']['flap'].keys()
Gives the output:
dict_keys(['lf', 'rf', 'lh', 'rh'])
I want to convert this into a JSON file, exactly by the keys that already exist, without manually do so because I want to perform it to many *.mat files with various key numbers.
EDIT:
Unfortunately, I no longer have access to MATLAB.
An example for desired output would look something like this:
json_format = {
"Forces": {
"Ss1": {
"flap": {
"lf": [1,2,3,4],
"rf": [4,5,6,7],
"lh": [23 ,5,6,654,4],
"rh": [4 ,34 ,35, 56, 66]
}
}
}
}
ANOTHER EDIT:
So after making lists of the subkeys (I won't elaborate on it), I did this:
FORCES = []
for ind in individuals:
for force in forces:
for wing in wings:
FORCES.append({
ind: {
force: {
wing: data['Forces'][ind][force][wing].tolist()
}
}
})
Then, to save:
with open(f'{ROOT_PATH}/Forces.json', 'w') as f:
json.dump(FORCES, f)
That worked but only because I looked manually for all of the keys... Also, for some reason, I have squared brackets at the beginning and at the end of this json file.
The json package will output dictionaries to JSON:
import json
with open('filename.json', 'w') as f:
json.dump(data, f)
If you are using MATLAB-R2016b or later, and want to go straight from MATLAB to JSON check out JSONENCODE and JSONDECODE. For your purposes JSONENCODE
encodes data and returns a character vector in JSON format.
MathWorks Docs
Here is a quick example that assumes your data is in the MATLAB variable test_data and writes it to a file specified in the variable json_file
json_data = jsonencode(test_data);
writematrix(json_data,json_file);
Note: Some MATLAB data formats cannot be translate into JSON data due to limitations in the JSON specification. However, it sounds like your data fits well with the JSON specification.
From a separate system I get a String parameter "messageJson" whose content is in the form:
{"agent1":"smith","agent2":"brown","agent3":{"agent3_1":"jones","agent3_2":"johnson"}}
To use it in my program I parse it with JsonSlurper.
def myJson = new JsonSlurper().parseText(messageJson)
But the resulting Json has the form:
[agent1:smith, agent2:brown, agent3:[agent3_1:jones, agent3_2:johnson]]
Note the square brackets and the lack of double quotes. How can I parse messageJson so that the original structure is kept?
Ok, thanks to the hint by cfrick, I was able to find a solution. In case anyone else has a similar problem, all I needed to do was using JsonOutput in the end to convert the map back to a Json
I.E. :
def myJson = new JsonSlurper().parseText(messageJson)
myJson << [agent4:"jane"]
def backToJson = JsonOutput.toJson(myJson)
I have a JSON file with some lines like:
"updatedAt" : ISODate("2018-11-20T09:32:16.732+0000"),
I tried json.loads but it has an error json.decoder.JSONDecodeError: Expecting value: line 2 column 13 (char 15).
I believe that the problem is at ISODate () but how could I handle that with Python?
Many thanks
This is not valid JSON, to begin with. I guess the ISODATE("...") is generated from MongoDB, maybe dumping the ISODate() helper directly instead of its string representation into the JSON?
In any case, you could use a regex on the whole JSON-string to get rid of the ISODate("..."), retrieve the date as a string and then use python-dateutil to parse the value to a datetime.datetime.
Something to the tune of
import json
import dateutil.parse
import re
json_str = ....
clean_json = re.compile('ISODate\(("[^"]+")\)').sub('\\1', json_str)
json_obj = json.loads(clean_json)
# use dateutil.parser.parse(s) to parse each date into a datetime.datetime
I created a config file in JSON format, and I want to use KDB to read it in as a dictionary.
In Python, it's so easy:
with open('data.json') as f:
data = json.load(f)
Is there a similar function in KDB?
To read your JSON file into kdb+, you should use read0. This returns the lines of the file as a list of strings.
q)read0`:sample.json
,"{"
"\"name\":\"John\","
"\"age\":30,"
"\"cars\":[ \"Ford\", \"BMW\", \"Fiat\" ]"
,"}"
kdb+ allows for the de-serialisation (and serialisation) of JSON objects to dictionaries using the .j namespace. The inbuilt .j.k expects a single string of characters containing json and converts this into a dictionary. A raze should be used to flatten our list of strings:
q)raze read0`:sample.json
"{\"name\":\"John\",\"age\":30,\"cars\":[ \"Ford\", \"BMW\", \"Fiat\" ]}"
Finally, using .j.k on this string yields the dictionary
q).j.k raze read0`:sample.json
name| "John"
age | 30f
cars| ("Ford";"BMW";"Fiat")
For a particularly large JSON file, it may be more efficient to use read1 rather than raze read0 on your file, e.g.
q).j.k read1`:sample.json
name| "John"
age | 30f
cars| ("Ford";"BMW";"Fiat")
If you're interested in the reverse operation, you can use .j.j to convert a dictionary into a list of strings and use 0: to save.
Further information on the .j namespace can be found here.
You can also see more examples on the Kx wiki of read0, read1 and 0:.
Working with JSON is handled by the .j namespace where .j.j serialises and .j.k deserialises the messages. Note the you will need to use raze to convert the JSON into a single string first.
There is more information available on the Kx wiki, where the following example is presented:
q).j.k "{\"a\":[0,1],\"b\":[\"hello\",\"world\"]}"
a| 0 1
b| "hello" "world"
When using .j.j both symbols and strings in kdb will be encoded into a JSON string while kdb will decode JSON strings to kdb strings except keys where they will be symbols.
To encode a kdb table in JSON an array of objects with identical keys should be sent. kdb will also encode tables as arrays of objects in JSON.
q).j.k "[{\"a\":1,\"b\":2},{\"a\":3,\"b\":4}]"
a b
---
1 2
3 4
When encoding q will use the value of \P to choose the precision, which is by default 7 which could lead to unwanted rounding.
This can be changed with 0 meaning maximum precision although the final digits are unreliable as shown below. See here for more info https://code.kx.com/q/ref/cmdline/#-p-display-precision.
q).j.j 1.000001 1.0000001f
"[1.000001,1]"
q)\P 0
q).j.j 1.000001 1.0000001f
"[1.0000009999999999,1.0000001000000001]"