I have a JSON as follows but need a way to remove nulls before putting it into elasticsearch. Looking for a simple jq command to remove nulls that i can incorporate into my bash script unless there's a way to do this in elasticsearch
{
"master_no": {
"master_no": 100000000,
"barcode": "E00000000",
"external_key": null,
"umid": null
},
"cust_id": {
"other_cust_id": null,
"cust_reference": null,
"external_key": null,
"list_id": null,
"cust_id": null
},
"customer_name": null,
"master_desc": "test Custom Patch - test",
"barcode": "E00000000",
"container_master_no": null,
"master_status": "I",
"length": "0:00",
"format_no": {
"format_desc": null,
"external_key": null,
"format_no": null
},
"lib_master_audio": [
{
"master_no": 10000000,
"audio_channel_no": {
"audio_channel_no": 10,
"audio_channel": "1",
"external_key": null
}
},
{
"master_no": 100000000,
"audio_channel_no": {
"audio_channel_no": 10,
"audio_channel": "2",
"external_key": null
}
}
]
}
Thanks
This GitHub issue on remove null key and values from JSON can help you, in short, some handful of command might help you like, mentioned in this link :
del(.[][] | nulls)
Please note there are several methods of doing this, please check which one works for you.
As pointed out in comments by #oguz, Please use https://github.com/stedolan/jq/issues/104#issuecomment-289637207 which work with the latest version.
Related
Hey guys I have a json file connections.json with the following airflow connections
{
"google_cloud_default": {
"conn_type": "google_cloud_platform",
"host": null,
"login": null,
"password": null,
"schema": null,
"port": null,
"extra": "{\"extra__google_cloud_platform__key_path\": \"/usr/local/sample_path/credentials/sample.json\", \"extra__google_cloud_platform__num_retries\": 5, \"extra__google_cloud_platform__project\": \"sample-project\", \"extra__google_cloud_platform__scope\": \"https://www.googleapis.com/auth/spreadsheets\"}"
}
}
Is it possible to extend the "extra" parameter so it spans multiple lines for readability?
Like so:
{
"google_cloud_default": {
"conn_type": "google_cloud_platform",
"host": null,
"login": null,
"password": null,
"schema": null,
"port": null,
"extra": "{\"extra__google_cloud_platform__key_path\": \"/usr/local/sample_path/credentials/sample.json\",
\"extra__google_cloud_platform__num_retries\": 5,
\"extra__google_cloud_platform__project\": \"sample-project\",
\"extra__google_cloud_platform__scope\": \"https://www.googleapis.com/auth/spreadsheets\"}"
}
}
I tried wrapping the entire extra value in three double quotes i.e. """ but that still seems to throw an error. Thoughts?
I want to achieve following JSON transformation using Jolt processor in NIFI
By focusing on values field, in the first input in json (id 900551), values are populated as the following
input JSON
{
"id": 900551,
"internal_name": [],
"values": [
{
"id": 1430156,
"form_field_id": 900551,
"pos": 0,
"weight": null,
"category": null,
"created_at": "2020-10-15 12:55:02",
"updated_at": "2020-11-27 10:45:09",
"deleted_at": null,
"settings": {
"image": "myimage.png"
"fix": false,
"bold": false,
"exclusive": false
},
"internal_value": "494699DV7271000,6343060SX0W1000,619740BWR0W1000",
"css_class": null,
"value": "DIFFERENCE",
"settings_lang": {},
"value_html": ""
}
]
}
On the second input Json file to parse, values is null.
{
"id": 900552,
"internal_name": [],
"values": []
}
I would like to convert null values to an empty array in my conversion
Is there a way to do this using existing Jolt operations ?
Thanks.
The default operation is what you are looking for:
Defaultr walks the spec and asks "Does this exist in the data? If not, add it."
In our case:
if the value for "values" key is null, put the empty array instead
Here is the spec:
[
{
"operation": "default",
"spec": {
"values": []
}
}
]
tested with https://jolt-demo.appspot.com/
edit: answering the question from the comment:
Maybe this workaround will work for you
I had an issue today with Filemaker on how to get the first element out of a json result without knowing the key.
Example $json result from an API call
{
"26298070": {
"task_id": "26298070",
"parent_id": "0",
"name": "DEPOT-0045 Research ODBC Model Extraction via Django To cut down on development time from Filemaker to Postgres",
"external_task_id": "32c8fd51-2066-42b9-b88b-8a2275fafc3f",
"external_parent_id": "64e7c829-d88e-48ae-9ba4-bb7a3871a7ce",
"level": "1",
"add_date": "2018-06-04 21:45:16",
"archived": "0",
"color": "#34C644",
"tags": "DEPOT-0045",
"budgeted": "1",
"checked_date": null,
"root_group_id": "91456",
"assigned_to": null,
"assigned_by": null,
"due_date": null,
"note": "",
"context": null,
"folder": null,
"repeat": null,
"billable": "0",
"budget_unit": "hours",
"public_hash": null,
"modify_time": null
}
}
I tried JSONGetElement( $json, "") and got the original json.
I tried JSONGetElement( $json, ".") and got the original json.
I tried JSONGetElement( $json, 1 ) and got nothing.
How do you get the first element out of a JSON String without knowing the name of the element in FileMaker 16 or 17?
Try this for the root element:
JSONListKeys ( $json ; "" )
result: 26298070
Once you get the root, you can get the child keys.
I remembered that FileMaker has a function to extract words from text so I thought I'd see what happened if I extracted the first word as a key.
I tried
JSONGetElement ( $json ; MiddleWords ( $json,1,1 ) )
and got the result I was looking for.
{
"add_date": "2018-06-04 21:45:16",
"archived": "0",
"assigned_by": null,
"assigned_to": null,
"billable": "0",
"budget_unit": "hours",
"budgeted": "1",
"checked_date": null,
"color": "#34C644",
"context": null,
"due_date": null,
"external_parent_id": "64e7c829-d88e-48ae-9ba4-bb7a3871a7ce",
"external_task_id": "32c8fd51-2066-42b9-b88b-8a2275fafc3f",
"folder": null,
"level": "1",
"modify_time": null,
"name": "DEPOT-0045 Research ODBC Model Extraction via Django To cut down on development time from Filemaker to Postgres",
"note": "",
"parent_id": "0",
"public_hash": null,
"repeat": null,
"root_group_id": "91456",
"tags": "DEPOT-0045",
"task_id": "26298070"
}
which makes it easy to parse simple JSON schema's that use attributes for keys.
I am trying to persist a nested object in sessionAttributes. However, it seems Amazon Lex does not recognize if I send a structure like this in response:
{
"sessionAttributes": {
"previousIntent": "TCAIntent",
"prevSlots": {
"test": 1
}
},
"dialogAction": {
"type": "Delegate",
"slots": {
"firstName": null,
"lastName": null,
"rate": null,
"rentMortgage": null,
"downPayment": null,
"purchasePrice": null,
"email": null
}
}
}
When I replace {"test": 1} with a value, it works fine. Is there a way around this or possible mistake?
Error:
I have figured out that Lex does not allow storing json object or array on sessionAttributes. As a work around, I converted the json object to string and convert it back when I need to use it.
I am currently a happy user of ngx_postgres. However I recently discover I need to do something very weird. Basically I need to produce the following json output:
{
"PatientName": {
"Tag": "00100010",
"VR": "PN",
"PersonName": [
{
"SingleByte": "Wang^XiaoDong",
"Ideographic": "王^小東"
}
]
},
},
{
"PatientName": {
"Tag": "00100010",
"VR": "PN",
"PersonName": [
{
"SingleByte": "John^Doe",
}
]
},
}
With a little reading of the DICOM standard it is easy to create (simplified) a table of equivalent for Keyword, Tag and VR:
CREATE TABLE equiv (
"Keyword" varchar(64) PRIMARY KEY,
"Tag" char(8) NOT NULL,
"VR" char(2) NOT NULL,
);
Well now the tricky part is this indirection with PatientName that I do not understand, I tried:
CREATE TABLE patientname (
"SingleByte" varchar(64) primary key,
"Ideographic" varchar(64),
);
CREATE TABLE patientlevel_impl_detail (
"PatientName" varchar(64) references patientname("SingleByte"),
"PatientID" character varying(64) NOT NULL
);
CREATE view patientlist as select
patientname."SingleByte",
patientname."Ideographic",
patientname."Phonetic",
patientlevel_impl_detail."PatientID"
FROM patientlevel_impl_detail,patientname where patientlevel_impl_detail."PatientName" = patientname."SingleByte";
However in any case a TABLE and/or a VIEW is always flatten, and instead I am getting something like:
$ curl http://localhost:8080/patients
[
{
"Tag": "00100010",
"VR": "PN",
"SingleByte": "John^Doe",
"Ideographic": null,
},
]
So I do not see how I can make PersonName an array of nested string (nested json tree).
Note: I am not using 9.3, I need to use 9.1 for now
On 9.1, do yourself a favor and get the json extension for 9.1. It will save you a lot of work.
The second thing you need to do is to create a nested data structure as a view which matches your json structure. You will use array_agg() for this:
CREATE view patientlist as select
arrayagg(patientname) as "PatientName"
patientlevel_impl_detail."PatientID"
FROM patientlevel_impl_detail,patientname
where patientlevel_impl_detail."PatientName" = patientname."SingleByte";
Then you should be able to:
SELECT row_to_json(patientlist) FROM patientlist;