I run a command on one of my systems and it spits out JSON "pretty" format like so:
[
{
"server": "servename1",
"i.p": 127.0.0.1,
"domain": "generic",
"OS": "RHEL",
"Version": 7.0
},
{
"server": "servename2",
"i.p": 127.0.0.1,
"domain": "generic",
"OS": "RHEL",
"Version": 7.0
},
{
"server": "servename3",
"i.p": 127.0.0.1,
"domain": "generic",
"OS": "RHEL",
"Version": 7.0
}
]
I need to parse these paragraphs in one liners like so:
[{"server":"servename1","i.p":127.0.0.1,"domain":"generic","OS":"RHEL","Version":7.0},
{"server":"servename2","i.p":127.0.0.1,"domain":"generic","OS":"RHEL","Version":7.0},
{"server":"servename3","i.p":127.0.0.1,"domain":"generic","OS":"RHEL","Version":7.0},]
What is the easiest way to do this? I am tried using SED and JQ but couldn't get it.
You can try this in jq if you're OK with newline-delimited JSON:
$ jq -c ".[]" test.json
{"server":"servename1","i.p":"127.0.0.1","domain":"generic","OS":"RHEL","Version":7}
{"server":"servename2","i.p":"127.0.0.1","domain":"generic","OS":"RHEL","Version":7}
{"server":"servename3","i.p":"127.0.0.1","domain":"generic","OS":"RHEL","Version":7}
Note that I had to quote the IP addresses from your sample, since the JSON you posted is not valid JSON.
If your input is always that regular:
$ awk '{ORS=(/},|\]/?RS:""); gsub(/[[:blank:]]+/,""); sub(/}$/,"},")}1' file
[{"server":"servename1","i.p":127.0.0.1,"domain":"generic","OS":"RHEL","Version":7.0},
{"server":"servename2","i.p":127.0.0.1,"domain":"generic","OS":"RHEL","Version":7.0},
{"server":"servename3","i.p":127.0.0.1,"domain":"generic","OS":"RHEL","Version":7.0},]
If that doesn't work for your real input then edit your question to include more truly representative sample input.
To skip the [ and ] lines:
$ awk '!/^[][]/{ORS=(/},/?RS:""); gsub(/[[:blank:]]+/,""); sub(/}$/,"},\n"); print}' file
{"server":"servename1","i.p":127.0.0.1,"domain":"generic","OS":"RHEL","Version":7.0},
{"server":"servename2","i.p":127.0.0.1,"domain":"generic","OS":"RHEL","Version":7.0},
{"server":"servename3","i.p":127.0.0.1,"domain":"generic","OS":"RHEL","Version":7.0},
Related
I am describing a AWS security group and passing the output by jq in order to get all the CIDRs of the inbound rules.
I have reached so far:
▶ aws ec2 describe-security-groups --group-ids sg-123456789 | jq '.SecurityGroups[0].IpPermissions[0].IpRanges'
[
{
"CidrIp": "11.22.33.44/32",
"Description": "Something"
},
{
"CidrIp": "22.33.44.12/32",
"Description": "Something else"
},
{
"CidrIp": "22.11.33.55/32",
"Description": "Something different"
},
]
I know I can grep but is there a way to get just the CidrIp from each json element of this array?
Sure, change your pipeline to
jq -r '.SecurityGroups[0].IpPermissions[0].IpRanges[].CidrIp'
Demo
Note that I also added the -r flag which makes the output raw text instead of JSON.
jq '.SecurityGroups[0].IpPermissions[0].IpRanges | values[].CidrIp'
this seemed to work as well.
I know how to retrieve an entire JSON attribute with jq, but I want to extract only a specific substring. Consider the following sample input:
[
{
"name": "test",
"output": "",
"error": "",
"state": "unknown",
"startTime": 1571292623936,
"endTime": 0,
"extra": {},
"warning": "************************* test Warnings *************************\n*\n* \n*****************************************************************",
"hasWarning": false
},
{
"name": "npm run test",
"output": "\n> DISPLAY was set to: \":99\"\n\nCypress will attempt to fix the problem and rerun.\n\n\n Running: consumer/oct.js... (1 of 1) \nPROCESSING JS RESOURCE FILE FROM:/PMT1469/workspace/E2EI/cypress/e2e/consumer/kindle.js\n{\"dataFile\":\"scripts/regression/transfers/card/kindle.csv\"}\nSENDING JS RESOURCE FILE FROM: /PMT-1469/workspace/E2E-UI { startedTestsAt: '2019-10-17T06:10:59.339Z',\n endedTestsAt: '2019-10-17T06:11:53.542Z',\n totalDuration: 54203,\n totalSuites: 4,\n totalTests: 2,\n totalFailed: 2,\n totalPassed: 0,\n totalPending: 0,\n totalSkipped: 0,\n\n browserPath: '',\n browserName: 'electron',\n reporter: 'mochawesome',\n taskTimeout: 60000,\n video: true,\n known: true }\n",
"error": null,
"state": "success",
"startTime": 1571292631223,
"endTime": 1571292718780,
"extra": {},
"warning": "************************* npm run test Warnings *************************\n*\n* \n*************************************************************************",
"hasWarning": false
}
]
I just want to pick the following values in the above JSON payload which is in "output" attribute.
Expected output:
totalDuration: 54203
totalSuites: 4
totalFailed: 2
totalPassed: 0
totalSkipped: 0
We can easily fetch the attribute values using jq -r '.[].output', but I'm trying to only capture substrings of the form total<something>: <number>.
The inefficient-but-easy answer is to do the bulk of the work in a separate pipeline stage. Assuming GNU tools:
jq -r '.[].output' <in.json \
| grep -Eo '^[[:space:]]+(total[[:alpha:]]+: [[:digit:]]+)' \
| sed -re 's/^[[:space:]]+//'
However, with modern jq, one can do much better:
jq -r '.[].output | scan("total[[:alpha:]]+: [[:digit:]]+")' <in.json
Here is a json file named test.json for testing
{
"name": "Google",
"location": {
"street": "1600 Amphitheatre Parkway",
"city": "Mountain View",
"state": "California",
"country": "US"
},
"employees": [
{
"name": "Michael",
"division": "Engineering"
},
{
"name": "Laura",
"division": "HR"
},
{
"name": "Elise",
"division": "Marketing * test"
}
]
}
if I use the jq code to parser it like below:
cat test.json | jq -r '.employees[2].division'
it will work well and give a correct result:
Marketing * test
but I use $(), the bad thing will happen!
echo $(cat test.json | jq -r '.employees[2].division')
the result will list all file names under current folder! like:
my1.json my2.json test.json test ...
I guess it $() run asterisk * as a shell script, but a string.
so how to make asterisk (*) in json file just as a string when I am using jq ?. I am using Google cloud platform and Ubuntu 17.10
Always use double-quotes around command-substitution to avoid * to be treated literally. The * is a special character in shell that is a wildcard entry that expands to all the files available in the current working directory. You need to quote it to deprive of its special meaning (Refer GNU bash man page under Parameters section).
Also jq can process the file directly, you can avoid useless cat usage.
result="$(jq -r '.employees[2].division' < test.json)"
echo "$result"
should produce the result as expected.
I have a command that I run and it gives an output like below:
{
"endpointApplications": {
"App_Name": {
"connectionState": "Disconnected",
"connectionTime": "No connection was established",
"linkAttributes": {
"ackSettings": {
"dataAckEnabled": "true",
"dataAckTimeout": "5000",
"dataNakRetryLimit": "0",
"retransmitDelay": "500"
},
"keepAliveSettings": {
"keepAliveAckTimeout": "5000",
"keepAliveInterval": "30000"
},
"logTraffic": "false",
"port": "9999",
"role": "server"
},
"protocol": "snmp"
}
},
"queueStats": {}
}
I would need the output to be in one line like below:
{"endpointApplications": {"app_name": {"connectionState": "Disconnected","connectionTime": "No connection was established","linkAttributes": {"ackSettings":{"dataAckEnabled": "true","dataAckTimeout": "5000","dataNakRetryLimit": "0","retransmitDelay": "500"},"keepAliveSettings":{"keepAliveAckTimeout": "5000","keepAliveInterval": "30000"},"logTraffic": "false","port": "9999","role": "server"},"protocol": "snmp"}},"queueStats":{}}
I tried using awk and sed combining different parameters but I can't get to work without losing the JSON format.
You should use jq for stuff like that:
jq -c . input.txt
An alternative quick a dirty solution would be to use sed & tr:
sed -e 's/^ *//' < input.txt | tr -d '\n'
although I would recommend using jq which is designed for manipulating JSON. jq is like sed for JSON. Manipulating JSON textually with sed/awk/etc is not guaranteed to produce semantically equivalent JSON.
jq or any other json aware tool is best suited for json file manipulation.However here is awk based solution.
awk -v RS= '{$1=$1}1' input.json
{ "endpointApplications": { "App_Name": { "connectionState": "Disconnected", "connectionTime": "No connection was established", "linkAttributes": { "ackSettings": { "dataAckEnabled": "true", "dataAckTimeout": "5000", "dataNakRetryLimit": "0", "retransmitDelay": "500" }, "keepAliveSettings": { "keepAliveAckTimeout": "5000", "keepAliveInterval": "30000" }, "logTraffic": "false", "port": "9999", "role": "server" }, "protocol": "snmp" } }, "queueStats": {} }
Note: This solution is mainly for the legacy systems not having tools like jq and have no chance to get them installed due to some reasons.
My json looks like this :
{
"20160522201409-jobsv1-1": {
"vmStateDisplayName": "Ready",
"servers": {
"20160522201409 jobs_v1 1": {
"serverStateDisplayName": "Ready",
"creationDate": "2016-05-22T20:14:22.000+0000",
"state": "READY",
"provisionStatus": "PENDING",
"serverRole": "ROLE",
"serverType": "SERVER",
"serverName": "20160522201409 jobs_v1 1",
"serverId": 2902
}
},
"isAdminNode": true,
"creationDate": "2016-05-22T20:14:23.000+0000",
"totalStorage": 15360,
"shapeId": "ot1",
"state": "READY",
"vmId": 4353,
"hostName": "20160522201409-jobsv1-1",
"label": "20160522201409 jobs_v1 ADMIN_SERVER 1",
"ipAddress": "10.252.159.39",
"publicIpAddress": "10.252.159.39",
"usageType": "ADMIN_SERVER",
"role": "ADMIN_SERVER",
"componentType": "jobs_v1"
}
}
My key keeps changing from time to time. So for example 20160522201409-jobsv1-1 may be something else tomorrow. Also I may more than one such entry in the json payload.
I want to echo $KEYS and I am trying to do it using jq.
Things I have tried :
| jq .KEYS is the command i use frequently.
Is there a jq command to display all the primary keys in the json?
I only care about the hostname field. And I would like to extract that out. I know how to do it using grep but it is NOT a clean approach.
You can simply use: keys:
% jq 'keys' my.json
[
"20160522201409-jobsv1-1"
]
And to get the first:
% jq -r 'keys[0]' my.json
20160522201409-jobsv1-1
-r is for raw output:
--raw-output / -r: With this option, if the filter’s result is a string then it will be written directly to standard output rather than being formatted as a JSON string with quotes. This can be useful for making jq filters talk to non-JSON-based systems.
Source
If you want a known value below an unknown property, eg xxx.hostName:
% jq -r '.[].hostName' my.json
20160522201409-jobsv1-1