How to transform a json structure with jq - json

How can I transform (using jq) the following json structure from this:
{
"_internal_messages": {
"error": [
{
"date": "16:12:30 - 07/02/2023",
"id": 1,
"origin": "A",
"text": "This is an error message"
},
{
"date": "16:12:31 - 07/02/2023",
"id": 5,
"origin": "A",
"text": "This is a second error message"
}
],
"info": [
{
"date": "16:12:29 - 07/02/2023",
"id": 0,
"origin": "A",
"text": "This is an info message"
},
{
"date": "16:12:30 - 07/02/2023",
"id": 4,
"origin": "C",
"text": "This is a second info message"
}
],
"success": [
{
"date": "16:12:30 - 07/02/2023",
"id": 2,
"origin": "B",
"text": "This is a success message"
},
{
"date": "16:12:30 - 07/02/2023",
"id": 3,
"origin": "B",
"text": "This is a second success message"
},
{
"date": "16:12:31 - 07/02/2023",
"id": 6,
"origin": "C",
"text": "This is a third success message"
}
]
}
}
To this:
{"_internal_messages":[
{
"type":"info",
"date": "16:12:29 - 07/02/2023",
"id": 0,
"origin": "A",
"text": "This is an info message"
},{
"type":"error",
"date": "16:12:30 - 07/02/2023",
"id": 1,
"origin": "A",
"text": "This is an error message"
},{
"type":"success",
"date": "16:12:30 - 07/02/2023",
"id": 2,
"origin": "B",
"text": "This is a success message"
},{
"type":"success",
"date": "16:12:30 - 07/02/2023",
"id": 3,
"origin": "B",
"text": "This is a second success message"
},{
"type":"info",
"date": "16:12:30 - 07/02/2023",
"id": 4,
"origin": "C",
"text": "This is a second info message"
},{
"type":"error",
"date": "16:12:31 - 07/02/2023",
"id": 5,
"origin": "A",
"text": "This is a second error message"
},{
"type":"success",
"date": "16:12:31 - 07/02/2023",
"id": 6,
"origin": "C",
"text": "This is a third success message"
}
]}
I checked the jq Manual and some other previous answered questions at SO, but I was not able to crack this one out... I´m thinking in combining jq with bash to do it, but I´m sure there must be a better way using just jq but my skills are not there yet. Can someone help me out, please? Thanks!
In the original structure the messages are stored in 3 arrays: error, info and success. Each time a message is added to those arrays, it receives a consecutive/incremental id. What I want is to take all those messages out from the three arrays, put them in one array while adding an extra attribute to know where each message came from (error, info or success) and finally order them in asc order by their id. The tricky part is knowing which one is an error, info or a success once you put them all together.

I don't understand why one of the items is missing in your desired output, but apart from that it's flattening the object with mapping the key to another field:
jq '._internal_messages |= (to_entries | map({type: .key} + .value[]))'
{
"_internal_messages": [
{
"type": "error",
"date": "16:12:30 - 07/02/2023",
"id": 1,
"origin": "A",
"text": "This is an error message"
},
{
"type": "error",
"date": "16:12:31 - 07/02/2023",
"id": 5,
"origin": "A",
"text": "This is a second error message"
},
{
"type": "info",
"date": "16:12:29 - 07/02/2023",
"id": 0,
"origin": "A",
"text": "This is an info message"
},
{
"type": "info",
"date": "16:12:30 - 07/02/2023",
"id": 4,
"origin": "C",
"text": "This is a second info message"
},
{
"type": "success",
"date": "16:12:30 - 07/02/2023",
"id": 2,
"origin": "B",
"text": "This is a success message"
},
{
"type": "success",
"date": "16:12:30 - 07/02/2023",
"id": 3,
"origin": "B",
"text": "This is a second success message"
},
{
"type": "success",
"date": "16:12:31 - 07/02/2023",
"id": 6,
"origin": "C",
"text": "This is a third success message"
}
]
}
If you want the items to be sorted by .id, append a sort_by:
jq '._internal_messages |= (to_entries | map({type: .key} + .value[]) | sort_by(.id))'
{
"_internal_messages": [
{
"type": "info",
"date": "16:12:29 - 07/02/2023",
"id": 0,
"origin": "A",
"text": "This is an info message"
},
{
"type": "error",
"date": "16:12:30 - 07/02/2023",
"id": 1,
"origin": "A",
"text": "This is an error message"
},
{
"type": "success",
"date": "16:12:30 - 07/02/2023",
"id": 2,
"origin": "B",
"text": "This is a success message"
},
{
"type": "success",
"date": "16:12:30 - 07/02/2023",
"id": 3,
"origin": "B",
"text": "This is a second success message"
},
{
"type": "info",
"date": "16:12:30 - 07/02/2023",
"id": 4,
"origin": "C",
"text": "This is a second info message"
},
{
"type": "error",
"date": "16:12:31 - 07/02/2023",
"id": 5,
"origin": "A",
"text": "This is a second error message"
},
{
"type": "success",
"date": "16:12:31 - 07/02/2023",
"id": 6,
"origin": "C",
"text": "This is a third success message"
}
]
}

Related

How to solve JQ processing as multiple dictionaries if document has an array and JQ filter uses group_by?

I have the following JSON document
[
{
"id": 6,
"description": "Component 1",
"due": "20211122T183000Z",
"entry": "20211119T181735Z",
"modified": "20211119T181735Z",
"project": "product1",
"status": "pending",
"uuid": "55bf0497-208c-492a-8f76-bb692d48afaa",
"tags": [
"abc",
"123"
],
"urgency": 13.9699
},
{
"id": 10,
"description": "Component 2",
"due": "20211129T183000Z",
"entry": "20211130T045620Z",
"modified": "20211130T045620Z",
"project": "product2",
"status": "pending",
"uuid": "d57eb8f7-e5ec-497c-ac47-f1cf34b005db",
"tags": [
"foo",
"bar"
],
"urgency": 14.0151
},
{
"id": 11,
"description": "Component 3",
"due": "20211202T183000Z",
"entry": "20211130T121529Z",
"completed": "20211130T123915Z",
"project": "product3",
"status": "pending",
"uuid": "9f15e6a4-5cef-4b0f-915b-fc916ab152c7",
"tags": [
"xyz",
"676"
],
"urgency": 14.0096
},
{
"id": 12,
"description": "Component 4",
"due": "20211202T183000Z",
"entry": "20211130T122537Z",
"pending": "20211130T122537Z",
"project": "product1",
"status": "pending",
"uuid": "91c9ec76-42a7-4ebc-9649-b3a12027feb1",
"tags": [
"def"
],
"urgency": 13.9096
}
]
I have written below JQ filter to parse the JSON, the expected output is not to generate multiple dictionaries.
group_by(.project,.status)
| .[]
| { project: .[0].project , status: .[0].status ,
description: [{"\(.[].description)" : (.[].tags | join(";"))}] }
After applying the filter, i get the below output with multiple dictionaries because of the tags array
{
"project": "product1",
"status": "pending",
"description": [
{
"Component 1": "abc;123"
},
{
"Component 1": "def"
},
{
"Component 4": "abc;123"
},
{
"Component 4": "def"
}
]
}
{
"project": "product2",
"status": "completed",
"description": [
{
"Component 2": "foo;bar"
}
]
}
{
"project": "product3",
"status": "completed",
"description": [
{
"Component 3": "xyz;676"
}
]
}
The output I am expecting is without multiple dictionaries as below
{
"project": "product1",
"status": "pending",
"description": [
{
"Component 1": "abc;123"
},
{
"Component 4": "def"
}
]
}
{
"project": "product2",
"status": "completed",
"description": [
{
"Component 2": "foo;bar"
}
]
}
{
"project": "product3",
"status": "completed",
"description": [
{
"Component 3": "xyz;676"
}
]
}
How can I generate the above-expected output using JQ?
One similar option to yours would be
jq 'group_by(.project)[]
| { project: .[0].project, status:.[0].status, "description": [.[]
| { (.description) : .tags|join(";") } ] }'
Demo
To just bring together .description and .tags use
jq '.[] | del(.description, .tags) + ({(.description): .tags | join(";")})'
Demo
To also group by .project and just consider .project, .status and an array with the .description and .tags from above, go
jq '
group_by(.project)[]
| (first | {project, status})
+ {description: map({(.description): .tags | join(";")})}
'
Demo

Json parsing losgstash

i can't parse this json with logstash... someone could help me?
seems like the way it is parsed can't be readed by logstash.
there is a ruby code to parse this?
I cannot extract the fields nested in the square brackets
[
{
"capacity": 0,
"created_at": "2021-04-06T16:18:34+02:00",
"decisions": [
{
"duration": "22h16m4.141220361s",
"id": 842,
"origin": "CAPI",
"scenario": "crowdsecurity/http-bad-user-agent",
"scope": "ip",
"simulated": false,
"type": "ban",
"value": "3.214.184.223/32"
},
.
.
.
{
"duration": "22h16m4.195897491s",
"id": 904,
"origin": "CAPI",
"scenario": "crowdsecurity/http-backdoors-attempts",
"scope": "ip",
"simulated": false,
"type": "ban",
"value": "51.68.11.195/32"
}
],
"events": null,
"events_count": 0,
"id": 12,
"labels": null,
"leakspeed": "",
"machine_id": "N/A",
"message": "",
"scenario": "update : +63/-0 IPs",
"scenario_hash": "",
"scenario_version": "",
"simulated": false,
"source": {
"scope": "Community blocklist",
"value": ""
},
"start_at": "2021-04-06 16:18:34.750588276 +0200 +0200",
"stop_at": "2021-04-06 16:18:34.750588717 +0200 +0200"
}
]
Require JSON
JSON.parse(yourString)
Would likely be what you're looking for.
The module is described here

How to confirm Webhook POST is being received in Coldfusion

To be clear upfront, I'm not a professional - just trying to get a task done and I have minimal expertise. What I thought would be easy is apparently not so. I've dealt with APIs and XML but it's always been me initiating. This is apparently different.
Goal:
Receive HTTP POST Webhook data (JSON) from my Wordpress/Woocommerce website.
Problem:
The Webhook is working and being received perfectly by RequestBin.The JSON string is being sent over HTTP. But I'm not even sure the request is being heard by my CF Template. When I try to write the received data into a session variable so I can at least see it, nothing is there?!?! How do I find out if my code being triggered?
First line of code in my CF Template:
<cfset HTTPRequestData = deserializeJSON(ToString(getHTTPRequestData().content))>
Example from RequestBin of data being sent:
{
"id": 259609,
"parent_id": 0,
"number": "CW-104-270219",
"order_key": "xxxxxxxxxxxx",
"created_via": "checkout",
"version": "3.5.5",
"status": "completed",
"currency": "USD",
"date_created": "2019-02-27T15:11:58",
"date_created_gmt": "2019-02-27T20:11:58",
"date_modified": "2019-02-27T15:11:59",
"date_modified_gmt": "2019-02-27T20:11:59",
"discount_total": "0.00",
"discount_tax": "0.00",
"shipping_total": "0.00",
"shipping_tax": "0.00",
"cart_tax": "0.00",
"total": "9.68",
"total_tax": "0.00",
"prices_include_tax": false,
"customer_id": 2,
"customer_ip_address": "xxxxxxxxxxxx",
"customer_user_agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.119 Safari/537.36",
"customer_note": "",
"billing": {
"first_name": "Valerie",
"last_name": "Criswell",
"company": "",
"address_1": "",
"address_2": "",
"city": "",
"state": "",
"postcode": "",
"country": "",
"email": "xxxxxxxxxx",
"phone": ""
},
"shipping": {
"first_name": "",
"last_name": "",
"company": "",
"address_1": "",
"address_2": "",
"city": "",
"state": "",
"postcode": "",
"country": ""
},
"payment_method": "authorize_net_aim",
"payment_method_title": "Credit Card",
"transaction_id": "xxxxxxxxxxxxx",
"date_paid": "2019-02-27T15:11:59",
"date_paid_gmt": "2019-02-27T20:11:59",
"date_completed": "2019-02-27T15:11:59",
"date_completed_gmt": "2019-02-27T20:11:59",
"cart_hash": "xxxxxxxxxxxxx",
"meta_data": [
{
"id": 8785641,
"key": "_wcson_order_number",
"value": "CWD-259609-022719-108"
},
{
"id": 8785642,
"key": "_order_number",
"value": "104"
},
{
"id": 8785643,
"key": "_order_number_formatted",
"value": "CW-104-270219"
},
{
"id": 8785644,
"key": "_order_number_meta",
"value": {
"prefix": "CW-",
"suffix": "-{DD}{MM}{YY}",
"length": "2"
}
},
{
"id": 8785645,
"key": "_wc_authorize_net_aim_retry_count",
"value": "0"
},
{
"id": 8785646,
"key": "_wc_authorize_net_aim_trans_id",
"value": "xxxxxxxx"
},
{
"id": 8785647,
"key": "_wc_authorize_net_aim_trans_date",
"value": "2019-02-27 15:11:59"
},
{
"id": 8785648,
"key": "_wc_authorize_net_aim_environment",
"value": "production"
},
{
"id": 8785649,
"key": "_wc_authorize_net_aim_account_four",
"value": "xxxxxxxx"
},
{
"id": 8785650,
"key": "_wc_authorize_net_aim_authorization_amount",
"value": "xxxxxxx"
},
{
"id": 8785651,
"key": "_wc_authorize_net_aim_authorization_code",
"value": "xxxxxxx"
},
{
"id": 8785652,
"key": "_wc_authorize_net_aim_charge_captured",
"value": "yes"
},
{
"id": 8785653,
"key": "_wc_authorize_net_aim_card_expiry_date",
"value": "21-08"
},
{
"id": 8785654,
"key": "_wc_authorize_net_aim_card_type",
"value": "mastercard"
},
{
"id": 8785659,
"key": "_pip_invoice_number",
"value": "CW-104-270219"
},
{
"id": 8785660,
"key": "_wc_pip_invoice_email_count",
"value": "1"
},
{
"id": 8785661,
"key": "_wc_pip_packing_list_email_count",
"value": "1"
}
],
"line_items": [
{
"id": 17,
"name": "I Give Myself Away - MP3 Download - 30 Day Access",
"product_id": 192127,
"variation_id": 192128,
"quantity": 1,
"tax_class": "",
"subtotal": "9.68",
"subtotal_tax": "0.00",
"total": "9.68",
"total_tax": "0.00",
"taxes": [],
"meta_data": [
{
"id": 182,
"key": "pa_format",
"value": "download-30-day"
}
],
"sku": "131599-DL30DAY",
"price": 9.68
}
],
"tax_lines": [],
"shipping_lines": [],
"fee_lines": [],
"coupon_lines": [],
"refunds": []
}
Let's start with the stated goal:
Receive HTTP POST Webhook data from my Wordpress/Woocommerce website.
Can your Wordpress site hit the intended URL on your ColdFusion site? If so, can you record / dump the HTTP POST request to a file?
<cfdump var="#form#" format="html" output="/full/path/to/file">
This will tell you if you're even receiving the request correctly.

Convert nested json to csv to sheets json api

I'm want to make my json to csv so that i can upload it on google sheets and make it as json api. Whenever i have change data i will just change it on google sheets. But I'm having problems on converting my json file to csv because it changes the variables whenever i convert it. I'm using https://toolslick.com/csv-to-json-converter to convert my json file to csv.
What is the best way to convert json nested to csv ?
JSON
{
"options": [
{
"id": "1",
"value": "Jumbo",
"shortcut": "J",
"textColor": "#FFFFFF",
"backgroundColor": "#00000"
},
{
"id": "2",
"value": "Hot",
"shortcut": "D",
"textColor": "#FFFFFF",
"backgroundColor": "#FFFFFF"
}
],
"categories": [
{
"id": "1",
"order": 1,
"name": "First Category",
"active": true
},
{
"id": "2",
"order": 2,
"name": "Second Category",
"shortcut": "MT",
"active": true
}
],
"products": [
{
"id": "03c6787c-fc2a-4aa8-93a3-5e0f0f98cfb2",
"categoryId": "1",
"name": "First Product",
"shortcut": "First",
"options": [
{
"optionId": "1",
"price": 23
},
{
"optionId": "2",
"price": 45
}
],
"active": true
},
{
"id": "e8669cea-4c9c-431c-84ba-0b014f0f9bc2",
"categoryId": "2",
"name": "Second Product",
"shortcut": "Second",
"options": [
{
"optionId": "1",
"price": 11
},
{
"optionId": "2",
"price": 20
}
],
"active": true
}
],
"discounts": [
{
"id": "1",
"name": "S",
"type": 1,
"amount": 20,
"active": true
},
{
"id": "2",
"name": "P",
"type": 1,
"amount": 20,
"active": true
},
{
"id": "3",
"name": "G",
"type": 2,
"amount": 5,
"active": true
}
]
}
Using python, this can be easily done or almost done. Maybe this code will help you in some way to understand that.
import json,csv
data = []
with open('your_json_file_here.json') as file:
for line in file:
data.append(json.loads(line))
length = len(data)
with open('create_new_file.csv','w') as f:
writer = csv.writer(f)
writers = csv.DictWriter(f, fieldnames=['header1','header2'])
writers.writeheader()
for iter in range(length):
writer.writerow((data[iter]['specific_col_name1'],data[iter]['specific_col_name2']))
f.close()

How to COMBINE 4 JSON objects/files to create ONE JSON file

OK, this is probably a REAL simple request.
But, I have 4 JSON files, well, I get JSON when doing a GET to a
//some IP address/some directory/some sub directory/name of folder
So I get back let's just say, NETWORKS, SITES, RESOURCES, COMPONENTS
Each one below NETWORKS is the CHILD...
Ok, next: I need to COMBINE these JSON files to populate JSTree (from jstree.com)
I have a basic JSON file like this and it WORKS beautifully:
(NOTE: the ID's below are irrelevant and DO NOT match in the REAL examples farther down here.)
The intent here is to JOIN all four JSON objects which I'm getting through a RESTful environment to JAVA API's that GET the data from the database.
[
{
"data": "Network 1",
"metadata": {"id" : "n1"},
"children": [ {
"data": "Site 1",
"metadata": {"id" : "s1"},
"children": [ {
"data": "Resource 1",
"metadata": {"id" : "r1"},
"children": [
{
"data": "Component 1",
"metadata": {"id" : "c1"}
},
{
"data": "Component 2",
"metadata": {"id" : "c2"}
},
{
"data": "Component 3",
"metadata": {"id" : "c3"}
} ] } ] },
"Site 2",
"Site 3",
"Site 4"]
}
]
Here's my dilemma:
I need to combine the following JSON files: SITES, RESOURCES, and COMPONENTS
Simple right? Not so much.
Here's a sample of each of the lower level JSON Objects:
NETWORKS:
[
{
"id": "23ef0d23-0d8d-4466-98da-81ef30791773",
"notes": "This is a network for network 1",
"name": "n1"
},
{
"id": "b4b46748-511a-49bf-9d22-8da014c76cc2",
"notes": "This is a network for network 2",
"name": "n2"
},
{
"id": "678b4a01-a6a6-449f-966d-c50c74964729",
"notes": "This is a network for network 3",
"name": "n3"
},
{
"id": "8e2822b1-49a8-498e-979b-2849cfa82148",
"notes": "This is a network for network 4",
"name": "n4"
}
]
SITES:
[
{
"id": "05683e7b-e471-4417-bead-317cfcbfaf30",
"name": "s1",
"networkId": "23ef0d23-0d8d-4466-98da-81ef30791773",
"notes": "This is site 1"
},
{
"id": "de8d654c-f9c4-4a4e-8742-32794b218b54",
"name": "s2",
"networkId": "23ef0d23-0d8d-4466-98da-81ef30791773",
"notes": "This is site 2"
},
{
"id": "16b2b1cf-2991-4717-ae65-2158700fa95d",
"name": "s3",
"networkId": "8e2822b1-49a8-498e-979b-2849cfa82148",
"notes": "This is site 3"
}
]
RESOURCES:
[
{
"id": "26db6a18-5099-4117-9adb-b8c808a3c478",
"networkId": "23ef0d23-0d8d-4466-98da-81ef30791773",
"siteId": "05683e7b-e471-4417-bead-317cfcbfaf30",
"name": "r1",
"notes": "This is Resource 1"
},
{
"id": "26ad2b53-f4b2-41c1-a618-d9e710452b7f",
"networkId": "23ef0d23-0d8d-4466-98da-81ef30791773",
"siteId": "05683e7b-e471-4417-bead-317cfcbfaf30",
"name": "r2",
"notes": "This is Resource 2"
}
]
And Finally, COMPONENTS:
[
{
"id": "6e8d13ad-9eb6-4213-bf84-a6e91d2e1460",
"resourceId": "26ad2b53-f4b2-41c1-a618-d9e710452b7f",
"siteId": "05683e7b-e471-4417-bead-317cfcbfaf30",
"networkId": "23ef0d23-0d8d-4466-98da-81ef30791773",
"name": "Component 1",
"notes": "This is my very first component - Yay!"
},
{
"id": "8f18cca3-378e-4f9b-8a39-eb2285fa61fd",
"resourceId": "26ad2b53-f4b2-41c1-a618-d9e710452b7f",
"siteId": "05683e7b-e471-4417-bead-317cfcbfaf30",
"networkId": "23ef0d23-0d8d-4466-98da-81ef30791773",
"name": "Component 2",
"notes": "This is my Second component - Yay!"
},
{
"id": "539370a6-577f-477d-a6ea-d45efd7e65aa",
"resourceId": "26ad2b53-f4b2-41c1-a618-d9e710452b7f",
"siteId": "05683e7b-e471-4417-bead-317cfcbfaf30",
"networkId": "23ef0d23-0d8d-4466-98da-81ef30791773",
"name": "Component 3",
"notes": "This is my Third component - Yay!"
}
]
Ultimately, when combined, it's GOT to look like the very first JSON example which WORKS for JSTree.
Thank you all for helping.
Regards