Shell variables as additional fields in json - json

I'm able to retrieve each user's data into m4-$u.json file with the below shell script
#!/bin/bash
USERID=ricardo.sanchez
PASSWORD=password
PORT=2728
for u in `cat user-list.txt`;
do echo $u;
curl --user $USERID:$PASSWORD http://198.98.99.12:46567/$PORT/protects/$u | jq '.' > m4-$u.json
done
One for the user's output of m4-daniel.json file few lines as follows.
[
{
"depotFile": "//ABND/JJEB/...",
"host": "*",
"isgroup": "",
"line": "16",
"perm": "open",
"user": "5G_USER_GROUP"
},
{
"depotFile": "//LIB/...",
"host": "*",
"isgroup": "",
"line": "19",
"perm": "write",
"user": "6G_USER_GROUP"
},
{
"depotFile": "//AND/RIO/...",
"host": "*",
"isgroup": "",
"line": "20",
"perm": "write",
"user": "AND_USER_GROUP"
},
During shell script run time additionally $PORT & $u variable values need to be added in output of each json file.
Expected json output:-
{
"depotFile": "//ABND/JJEB/...",
"host": "*",
"isgroup": "",
"line": "16",
"perm": "open",
"user": "5G_USER_GROUP",
"port": "2728",
"userid": "daniel"
},
To achieve this any help will be appreciated.

By using --arg with jq you can pass paremeter and use as a variables
( more info https://stedolan.github.io/jq/manual/v1.5/#Invokingjq )
by using map and + you can iterate over the array and add a new property for each associative array
in you case :
curl --user $USERID:$PASSWORD http://198.98.99.12:46567/$PORT/protects/$u \
| jq --arg a_port $PORT --arg $u $USER 'map(.+{"userid":$a_userid}+{"port":$a_port|tonumber})' > m4-$u.json
see addition and map in
https://stedolan.github.io/jq/manual/v1.5/#Builtinoperatorsandfunctions

Related

Why is my JSON request body showing as invalid when earlier, a similar request is working?

I have the following code snippet where I'm trying to send notification data to a discord webhook, but it is returning {"code": 50109, "message": "The request body contains invalid JSON."}%.
Here is the non-working code (bolded the JSON request part):
if [ "$DISCORD_WEBHOOK_URL" != "" ]; then
rclone_sani_command="$(echo $rclone_command | sed 's/\x1b\[[0-9;]*[a-zA-Z]//g')" # Remove all escape sequences
# Notifications assume following rclone ouput:
# Transferred: 0 / 0 Bytes, -, 0 Bytes/s, ETA - Errors: 0 Checks: 0 / 0, - Transferred: 0 / 0, - Elapsed time: 0.0s
transferred_amount=${rclone_sani_command#*Transferred: }
transferred_amount=${transferred_amount%% /*}
** send_notification() {
output_transferred_main=${rclone_sani_command#*Transferred: }
output_transferred_main=${output_transferred_main% Errors*}
output_errors=${rclone_sani_command#*Errors: }
output_errors=${output_errors% Checks*}
output_checks=${rclone_sani_command#*Checks: }
output_checks=${output_checks% Transferred*}
output_transferred=${rclone_sani_command##*Transferred: }
output_transferred=${output_transferred% Elapsed*}
output_elapsed=${rclone_sani_command##*Elapsed time: }
notification_data='{
"username": "'"$DISCORD_NAME_OVERRIDE"'",
"avatar_url": "'"$DISCORD_ICON_OVERRIDE"'",
"content": null,
"embeds": [
{
"title": "Rclone Local Backup: Finished!",
"color": 65408,
"fields": [
{
"name": "Directories synced",
"value": "'"$SOURCE_DIR"' to '"$DESTINATION_DIR"'"
},
{
"name": "Transferred",
"value": "'"$output_transferred_main"'"
},
{
"name": "Errors",
"value": "'"$output_errors"'"
},
{
"name": "Checks",
"value": "'"$output_checks"'"
},
{
"name": "Transferred",
"value": "'"$output_transferred"'"
},
{
"name": "Elapsed time",
"value": "'"$output_elapsed"'"
},
{
"name": "Finished",
"value": "Finished backup on '"$END_DATE"' at '"$END_TIME"'"
},
],
"thumbnail": {
"url": null
}
}
]
}'
curl -H "Content-Type: application/json" -d "$notification_data" $DISCORD_WEBHOOK_URL
}**
if [ "$transferred_amount" != "0" ]; then
send_notification
fi
fi
rm -f "$LOCK_FILE"
trap - SIGINT SIGTERM
exit
fi
Earlier in the same script, the following code snippet does correctly send a notification to discord:
touch "$LOCK_FILE"
echo "Starting to backup $SOURCE_DIR to $DESTINATION_DIR on $START_DATE at $START_TIME" | tee -a $LOG_FILE
**send_notification_start() {
notification_data='{
"username": "'"$DISCORD_NAME_OVERRIDE"'",
"avatar_url": "'"$DISCORD_ICON_OVERRIDE"'",
"content": null,
"embeds": [
{
"title": "Rclone Local Backup: Started!",
"color": 4094126,
"fields": [
{
"name": "Started",
"value": "Started backup on '$START_DATE' at '$START_TIME'"
},
{
"name": "Directories being synced",
"value": "'$SOURCE_DIR' to '$DESTINATION_DIR'"
}
],
"thumbnail": {
"url": null
}
}
]
}'
curl -H "Content-Type: application/json" -d "$notification_data" $DISCORD_WEBHOOK_URL
}**
I've tried messing with the quotes and other characters to see if that is the issue but I am unable to figure out why this JSON request body is incorrect.
Running either of those two JSON snippets themselves (bolded) through a JSON validator I just get that it is incorrectly parsing right the beginning (even though the latter is working fine).
I am pretty new to using JSON but I wanted to just get a notification sent to discord when my script finishes backing up with some details on what exactly occurred.

Extracting values from nested arrays

I'm trying to extract values from nested arrays in JSON below and output as CSV.
Fields to extract:
templates.name
items.name
triggers.name
Output as:
templates.name; items.name; triggers.name
Anticipated output something like:
"Template App Agent"; "Host name of zabbix_agentd running"; "Host name of zabbix_agentd was changed on {HOST.NAME}"
"Template App Agent"; "Agent ping"; "Zabbix agent on {HOST.NAME} is unreachable for 5 minutes"
"Template App Agent"; "Version of zabbix_agent(d) running"; ""
Note:
Not every item has a trigger.
Several triggers may exist for an item.
I'm new to JQ. So far only success is extracting the template name.
jq '.[] | {templates: [.templates[].name]}'
Data:
{
"zabbix_export": {
"version": "5.4",
"date": "2022-05-17T06:25:59Z",
"groups": [
{
"uuid": "7df96b18c230490a9a0a9e2307226338",
"name": "Templates"
}
],
"templates": [
{
"uuid": "e60e6598cf19448089a5f5a6c5d796a2",
"template": "Template App Agent",
"name": "Template App Agent",
"groups": [
{
"name": "Templates"
}
],
"items": [
{
"uuid": "24c03ed734d54dc8868a282a83a02200",
"name": "Host name of zabbix_agentd running",
"key": "agent.hostname",
"delay": "1h",
"history": "1w",
"trends": "0",
"value_type": "CHAR",
"request_method": "POST",
"tags": [
{
"tag": "Application",
"value": "Zabbix agent"
}
],
"triggers": [
{
"uuid": "d2d12d9e7dfe4fedb252f19b85e5e6aa",
"expression": "(last(/Template App Agent/agent.hostname,#1)<>last(/Template App Agent/agent.hostname,#2))>0",
"name": "Host name of zabbix_agentd was changed on {HOST.NAME}",
"priority": "INFO"
}
]
},
{
"uuid": "abacad4ca5eb46d29864d8a4998f1cbb",
"name": "Agent ping",
"key": "agent.ping",
"history": "1w",
"description": "The agent always returns 1 for this item. It could be used in combination with nodata() for availability check.",
"valuemap": {
"name": "Zabbix agent ping status"
},
"request_method": "POST",
"tags": [
{
"tag": "Application",
"value": "Zabbix agent"
}
],
"triggers": [
{
"uuid": "6d2a73199f3b4288bf36331a142c1725",
"expression": "nodata(/Template App Agent/agent.ping,5m)=1",
"name": "Zabbix agent on {HOST.NAME} is unreachable for 5 minutes",
"priority": "AVERAGE"
}
]
},
{
"uuid": "2cc337555efd43d181c28c792f8cbbdb",
"name": "Version of zabbix_agent(d) running",
"key": "agent.version",
"delay": "1h",
"history": "1w",
"trends": "0",
"value_type": "CHAR",
"request_method": "POST",
"tags": [
{
"tag": "Application",
"value": "Zabbix agent"
}
]
}
],
"valuemaps": [
{
"uuid": "3d66c59a28c04b0ca8227c87902ddb4d",
"name": "Zabbix agent ping status",
"mappings": [
{
"value": "1",
"newvalue": "Up"
}
]
}
]
}
]
}
}
.zabbix_export.templates[] | .name as $tn | .items[] | [ $tn, .name, .triggers[]?.name? ] | join("; ")
Loop over the templates
.zabbix_export.templates[]
Save the template name in a var
.name as $tn
Loop over the items
.items[]
Create an array with fields you like (including the name from step 1
[ $tn, .name, .triggers[]?.name? ]
Join the array to a string
join("; ")
Will output:
"Template App Agent; Host name of zabbix_agentd running; Host name of zabbix_agentd was changed on {HOST.NAME}"
"Template App Agent; Agent ping; Zabbix agent on {HOST.NAME} is unreachable for 5 minutes"
"Template App Agent; Version of zabbix_agent(d) running"
Online demo
This is a nested structure, you need to iterate level by level and add up the items you want to be in one output line. Store values from previous levels in variables.
To account for an inexistent .triggers array, you may use the Error Suppression Operator ? in combination with Alternative Operator //.
Finally, wrap the items in quotes (here using map), join them using join, and output them as raw text using the -r option
jq -r '
.[].templates[] | .name as $t
| .items[] | .name as $i
| [$t, $i, (.triggers[].name)? // ""]
| map("\"\(.)\"") | join("; ")
'
"Template App Agent"; "Host name of zabbix_agentd running"; "Host name of zabbix_agentd was changed on {HOST.NAME}"
"Template App Agent"; "Agent ping"; "Zabbix agent on {HOST.NAME} is unreachable for 5 minutes"
"Template App Agent"; "Version of zabbix_agent(d) running"; ""
Demo
Also consider using the #csv builtin, which gives you valid CSV right away (properly encoded (not just quoted) items, but separated with commas, not semicolons):
jq -r '
.[].templates[] | .name as $t
| .items[] | .name as $i
| [$t, $i, (.triggers[].name)? // ""]
| #csv
'
"Template App Agent","Host name of zabbix_agentd running","Host name of zabbix_agentd was changed on {HOST.NAME}"
"Template App Agent","Agent ping","Zabbix agent on {HOST.NAME} is unreachable for 5 minutes"
"Template App Agent","Version of zabbix_agent(d) running",""
Demo

Converting json into csv with jq while only capturing specific keys

This is my first post so apologies if I make mistakes. Consider I have the following json output
{
"records": [
{
"title": "root logon",
"login": "61819009",
"uid": "ajsd879asdjksasda123asd1asd1",
"password": "OizfD19jC$ySaV$MKpSF",
"login_url": "http://192.168.0.1/",
"notes": ""
},
{
"title": "important admin account",
"login": "admin",
"uid": "asdjhkasdh89eoajdiuas98ue9aoi",
"password": "0z5gDUC#Rb354TlLq$KJ",
"login_url": "",
"notes": "",
"folders": [
{
"shared_folder": "Department",
"folder": "Important",
"option1": false,
"option2": false
}
]
}
{
"title": "another important admin",
"login": "admin#domain.com",
"uid": "asjhe98asiajsijeouiaueiaiu",
"password": "3pUs#uXEqsxCv7PRkDlJ",
"login_url": "http://192.168.0.2/",
"notes": ""
},
{
"title": "switch admin",
"login": "admin",
"uid": "asjhe89ausiodjakljskea90ik",
"password": "hMB!eMsAE8q4aDQuM4LY",
"login_url": "",
"notes": "",
"folders": [
{
"shared_folder": "Department2",
"folder": "network\\switches",
"option1": false,
"option2": false
}
]
}
]
}
from the list above, I would like to only export title, login, uid, shared folder, and folder fields to csv file. When I run the following command
jq --raw-output '.records[] | [.title, .login, .uid, .shared_folder, .folder] | # tsv > file
what I am expecting as a CSV output is
title
login
uid
shared_folder
folder
root logon
61819009
ajsd879asdjksasda123asd1asd1
important admin account
admin
asdjhkasdh89eoajdiuas98ue9aoi
Department
Important
another important admin
admin#domain.com
asjhe98asiajsijeouiaueiaiu
switch admin
admin
asjhe89ausiodjakljskea90ik
Department2
network\switches
instead I'm getting the following CSV output:
title
login
uid
root logon
61819009
ajsd879asdjksasda123asd1asd1
important admin account
admin
asdjhkasdh89eoajdiuas98ue9aoi
another important admin
admin#domain.com
asjhe98asiajsijeouiaueiaiu
switch admin
admin
asjhe89ausiodjakljskea90ik
my goal is to eliminate capturing following fields
password
login_url
notes
option1
option2
You an error in your JSON, you must add a ,.
Once done, the command is this
jq --raw-output '.records[] | [.title, .login, .uid, .folders[0].shared_folder,.folders[0].folder] |#tsv'
Here's one way:
.records[]
| [.title, .login, .uid] + ((.folders[]? // null) | [.shared_folder, .folder])
| #tsv

Format json output with jq

I'm writing a script which will obtain certain info from my Kubernetes cluster.
The following command
kubectl get --context <my-context> svc --selector='<my-selectors>' -o json |
jq -r ' .items[]| {Name:.metadata.name, Port:.spec.ports[0].port} + {Group: "test", Group: "group name", SSLMode: "prefer", MaintenanceDB: "postgres"} '>file.json
will output something like this
{
"Name": "db-name",
"Port": 3000,
"Group": "group name",
"SSLMode": "prefer",
"MaintenanceDB": "postgres"
}
{
"Name": "db-name",
"Port": 5432,
"Group": "group name",
"SSLMode": "prefer",
"MaintenanceDB": "postgres"
}
I've been trying to get the above into the following format
{
"Servers":{
"1": {
"Name": "db-name",
"Port": 3000,
"Group": "Server Group 1",
"SSLMode": "prefer",
"MaintenanceDB": "postgres"
},
"2": {
"Name": "db-name",
"Port": 5432,
"Group": "Server Group 1",
"SSLMode": "prefer",
"MaintenanceDB": "postgres"
}
}
}
Only just discovered jq so the few things I've tried have been unsuccessful. Would be grateful for any pointers.
Given the stream of JSON objects shown in the question, the following jq filter will produce the desired output assuming the stream is somehow "slurped":
. as $in
| reduce range(0;length) as $i ({};.[$i+1|tostring] = $in[$i])
| {Servers: .}
To avoid having to call jq twice, you could wrap your filter in square brackets, and pipe that into the above; better yet, you can streamline everything, e.g. along the following lines:
.items
| [to_entries[]
| {(.key+1|tostring): .value}
| map_values(
{Name:.metadata.name,
Port:.spec.ports[0].port,
Group: "group name",
SSLMode: "prefer",
MaintenanceDB: "postgres"}) ]
| {Servers: add}

How to select words after /slash from a json file in a powershell script

I've a json file from a az cli command, I want to deploy with a powershell script from a tool like jenkins or rundeck, I need to take the field "name" from json but not entirely only the word after forward slash
[
{
"Group": "KC-EMEA-RSGP-BPTRAINING-01",
"id": "/subscriptions/KC-EMEA-RSGP-BPTRAINING-01/providers/Microsoft.Sql/servers/",
"name": "kc-emea-sqsrv-bptraining-dev-01/master"
},
{
"Group": "KC-EMEA-RSGP-NAVISIONKM-DEV-01",
"id": "/subscriptions/KC-EMEA-RSGP-NAVISIONKM-DEV-01/providers/Microsoft.Sql/servers/",
"name": "km-emea-sqsrv-navision-tst-01/km-emea-sqdb-navision-tst-01"
},
{
"Group": "KC-EMEA-RSGP-NAVISIONKM-DEV-01",
"id": "/subscriptions/KC-EMEA-RSGP-NAVISIONKM-DEV-01/providers/Microsoft.Sql/servers/",
"name": "km-emea-sqsrv-navision-tst-01/master"
},
{
"Group": "KC-EMEA-RSGP-PROJECTS-DEV-01",
"id": "/subscriptions/KC-EMEA-RSGP-PROJECTS-DEV-01/providers/Microsoft.Sql/servers/",
"name": "kc-emea-sqsrv-projects-dev-01/KC-EMEA-SQDB-BPTRAINING-TRAINEE-01"
}
]
$file = "pathtojsonfile.json"
$jsonContent = Get-Content $file | ConvertFrom-Json;
$namedb = $jsonContent.name;
$value= $namedb.Substring($namedb.IndexOf('/')+1) --> this doesn't work.
Try
$namedb[0].Substring($namedb[0].IndexOf('/')+1)
And for all of them:
foreach($name in $namedb) {$name.Substring($name.IndexOf('/')+1)}