I have uploaded a *.mat file that contains a 'struct' to my jupyter lab using:
from pymatreader import read_mat
data = read_mat(mat_file)
Now I have a multi-dimensional dictionary, for example:
data['Forces']['Ss1']['flap'].keys()
Gives the output:
dict_keys(['lf', 'rf', 'lh', 'rh'])
I want to convert this into a JSON file, exactly by the keys that already exist, without manually do so because I want to perform it to many *.mat files with various key numbers.
EDIT:
Unfortunately, I no longer have access to MATLAB.
An example for desired output would look something like this:
json_format = {
"Forces": {
"Ss1": {
"flap": {
"lf": [1,2,3,4],
"rf": [4,5,6,7],
"lh": [23 ,5,6,654,4],
"rh": [4 ,34 ,35, 56, 66]
}
}
}
}
ANOTHER EDIT:
So after making lists of the subkeys (I won't elaborate on it), I did this:
FORCES = []
for ind in individuals:
for force in forces:
for wing in wings:
FORCES.append({
ind: {
force: {
wing: data['Forces'][ind][force][wing].tolist()
}
}
})
Then, to save:
with open(f'{ROOT_PATH}/Forces.json', 'w') as f:
json.dump(FORCES, f)
That worked but only because I looked manually for all of the keys... Also, for some reason, I have squared brackets at the beginning and at the end of this json file.
The json package will output dictionaries to JSON:
import json
with open('filename.json', 'w') as f:
json.dump(data, f)
If you are using MATLAB-R2016b or later, and want to go straight from MATLAB to JSON check out JSONENCODE and JSONDECODE. For your purposes JSONENCODE
encodes data and returns a character vector in JSON format.
MathWorks Docs
Here is a quick example that assumes your data is in the MATLAB variable test_data and writes it to a file specified in the variable json_file
json_data = jsonencode(test_data);
writematrix(json_data,json_file);
Note: Some MATLAB data formats cannot be translate into JSON data due to limitations in the JSON specification. However, it sounds like your data fits well with the JSON specification.
I upload from my backend the attached JSON to Google bucket,
Now I'm trying to connect this JSON to the Bigquery table but getting the following error, what are the changes I need to make?
Error while reading table: XXXXX, error message: Failed to parse JSON: No object found when new array is started.; BeginArray returned false; Parser terminated before end of string
[["video_screen","click_on_screen","false","202011231958","1","43","0"],["buy","error","2","202011231807","1","6","0"],["sign_in","enter","user_details","202011231220","2","4","0"],["video_screen","click_on_screen","false","202011230213","1","4","0"],["video_screen","click_on_screen","false","202011230633","1","4","0"],["video_screen","click_on_screen","false","202011230709","1","4","0"],["video_screen","click_on_screen","false","202011230712","1","4","0"],["video_screen","click_on_screen","false","202011230723","1","4","0"],["video_screen","click_on_screen","false","202011230725","1","4","0"],["video_screen","click_on_screen","false","202011231739","1","4","0"],["category","select","MTV","202011232228","1","3","0"],["sign_in","enter","user_details","202011230108","2","3","0"],["sign_in","enter","user_details","202011230442","2","3","0"],["video","select","youtube","202011230108","1","3","0"],["video","select","youtube","202011230633","1","3","0"],["video_screen","click_on_screen","false","202011230458","1","3","0"],["video_screen","click_on_screen","false","202011230552","1","3","0"],["video_screen","click_on_screen","false","202011230612","1","3","0"],["video_screen","click_on_screen","false","202011231740","1","3","0"],["category","select","Disney Karaoke","202011232228","1","2","0"],["category","select","Duet","202011232228","1","2","0"],["category","select","Free","202011230726","1","2","0"],["category","select","Free","202011231830","2","2","0"],["category","select","Free","202011232228","1","2","0"],["category","select","Love","202011232228","1","2","0"],["category","select","New","202011232228","1","2","0"],["category","select","Pitch Perfect 2","202011232228","1","2","0"],["developer","click","hithub","202011230749","1","2","0"],["sign_in","enter","user_details","202011230134","1","2","0"],["sign_in","enter","user_details","202011230211","1","2","0"],["sign_in","enter","user_details","202011230219","1","2","0"]]
Bigquery reads JSONL files. The example is not in that format.
JSONL uses \n as the delimeter between records. The example is all on one line with commas delimiting.
Every JSONL line is a json object, so starts with { and ends with }. The example has JSON arrays which are not supported.
JSONL is based on JSON. Every data element needs to be named. So the first record might appear as { "field1_name": "video_screen", "field2_name": "click_on_screen", "field3_name": false, "field4_name": 202011231958, "Field5_name": 1, "field6_name": 43, "field7_name": 0}
JSONL does not have the outer pair of brackets []. The first line starts { not [{ and the last line ends } not }].
I have the file log.txt with following data:
{"__TIMESTAMP":"2020-07-09T19:05:20.858013","__LABEL":"web_channel","__LEVEL":4,"__DIAGNOSE_SLOT":"","msg":"Port web_channel/diagnose_client not connected!"}
{"__TIMESTAMP":"2020-07-09T19:05:21.229737","__LABEL":"context_logging_addon","__LEVEL":4,"__DIAGNOSE_SLOT":"","msg":"startup component"}
{"__TIMESTAMP":"2020-07-09T19:05:21.229761","__LABEL":"context_logging_addon","__LEVEL":4,"__DIAGNOSE_SLOT":"","msg":"activate component"}
{"__TIMESTAMP":"2020-07-09T19:05:21.229793","__LABEL":"context_monitoring_addon","__LEVEL":4,"__DIAGNOSE_SLOT":"","msg":"startup component"}
{"__TIMESTAMP":"2020-07-09T19:05:21.229805","__LABEL":"context_monitoring_addon","__LEVEL":4,"__DIAGNOSE_SLOT":"","msg":"activate component"}
If I define a single row, I can convert in real JSON type:
import json
import datetime
from json import JSONEncoder
log = {
"__TIMESTAMP":"2020-07-09T19:05:21.229737",
"__LABEL":"context_logging_addon",
"__LEVEL":4,
"__DIAGNOSE_SLOT":"",
"msg":"Port web_channel/diagnose_client not connected!"}
class DateTimeEncoder(JSONEncoder):
#Override the default method
def default(self,obj):
if isinstance(obj,(datetime.date,datetime.datetime)):
return obj.isoformat()
print("Printing to check how it will look like")
print(DateTimeEncoder().encode(log))
I have the following output, which format is perfect JSON.
Printing to check how it will look like
{"__TIMESTAMP": "2020-07-09T19:05:21.229737", "__LABEL": "context_logging_addon", "__LEVEL": 4, "__DIAGNOSE_SLOT": "", "msg": "Port web_channel/diagnose_client not connected!"}
But I don't know how should I open the log.txt file, read the data to convert into JSON without any failure.
Could you help me please? Thanks in advance.
Let us say your log.txt file is in the same directory than your .py file.
Just open it with with open(... and then parse your file according to your syntax to create a list of dictionaries (each item corresponding to a row, then parse each dictionary as you're currently doing).
Here is how you could open and parse your file:
with open("log.txt","r") as file:
all_text = file.readlines()
parsed_line = list()
for text in all_text:
parsed_line.append(dict([item.split('":"') for item in text[2:-2].split('","')]))
If you have any question about the parsing let me know. This one is pretty straightforward.
Hope this helped you.
Try it this way:
logs = """[your log file above]"
for log in logs.splitlines():
print(DateTimeEncoder().encode(log))
Output:
"{\"__TIMESTAMP\":\"2020-07-09T19:05:20.858013\",\"__LABEL\":\"web_channel\",\"__LEVEL\":4,\"__DIAGNOSE_SLOT\":\"\",\"msg\":\"Port web_channel/diagnose_client not connected!\"}"
"{\"__TIMESTAMP\":\"2020-07-09T19:05:21.229737\",\"__LABEL\":\"context_logging_addon\",\"__LEVEL\":4,\"__DIAGNOSE_SLOT\":\"\",\"msg\":\"startup component\"}"
"{\"__TIMESTAMP\":\"2020-07-09T19:05:21.229761\",\"__LABEL\":\"context_logging_addon\",\"__LEVEL\":4,\"__DIAGNOSE_SLOT\":\"\",\"msg\":\"activate component\"}"
"{\"__TIMESTAMP\":\"2020-07-09T19:05:21.229793\",\"__LABEL\":\"context_monitoring_addon\",\"__LEVEL\":4,\"__DIAGNOSE_SLOT\":\"\",\"msg\":\"startup component\"}"
"{\"__TIMESTAMP\":\"2020-07-09T19:05:21.229805\",\"__LABEL\":\"context_monitoring_addon\",\"__LEVEL\":4,\"__DIAGNOSE_SLOT\":\"\",\"msg\":\"activate component\"}"