I have a command that I run and it gives an output like below:
{
"endpointApplications": {
"App_Name": {
"connectionState": "Disconnected",
"connectionTime": "No connection was established",
"linkAttributes": {
"ackSettings": {
"dataAckEnabled": "true",
"dataAckTimeout": "5000",
"dataNakRetryLimit": "0",
"retransmitDelay": "500"
},
"keepAliveSettings": {
"keepAliveAckTimeout": "5000",
"keepAliveInterval": "30000"
},
"logTraffic": "false",
"port": "9999",
"role": "server"
},
"protocol": "snmp"
}
},
"queueStats": {}
}
I would need the output to be in one line like below:
{"endpointApplications": {"app_name": {"connectionState": "Disconnected","connectionTime": "No connection was established","linkAttributes": {"ackSettings":{"dataAckEnabled": "true","dataAckTimeout": "5000","dataNakRetryLimit": "0","retransmitDelay": "500"},"keepAliveSettings":{"keepAliveAckTimeout": "5000","keepAliveInterval": "30000"},"logTraffic": "false","port": "9999","role": "server"},"protocol": "snmp"}},"queueStats":{}}
I tried using awk and sed combining different parameters but I can't get to work without losing the JSON format.
You should use jq for stuff like that:
jq -c . input.txt
An alternative quick a dirty solution would be to use sed & tr:
sed -e 's/^ *//' < input.txt | tr -d '\n'
although I would recommend using jq which is designed for manipulating JSON. jq is like sed for JSON. Manipulating JSON textually with sed/awk/etc is not guaranteed to produce semantically equivalent JSON.
jq or any other json aware tool is best suited for json file manipulation.However here is awk based solution.
awk -v RS= '{$1=$1}1' input.json
{ "endpointApplications": { "App_Name": { "connectionState": "Disconnected", "connectionTime": "No connection was established", "linkAttributes": { "ackSettings": { "dataAckEnabled": "true", "dataAckTimeout": "5000", "dataNakRetryLimit": "0", "retransmitDelay": "500" }, "keepAliveSettings": { "keepAliveAckTimeout": "5000", "keepAliveInterval": "30000" }, "logTraffic": "false", "port": "9999", "role": "server" }, "protocol": "snmp" } }, "queueStats": {} }
Note: This solution is mainly for the legacy systems not having tools like jq and have no chance to get them installed due to some reasons.
Related
Been struggling with this for a while and I'm no closer to a solution. I'm not very experienced using jq.
I'd like to take the values from one json file and add them to another file when other values in the dict match. The example files below demonstrate what I'd like more clearly than an explanation.
hosts.json:
{
"hosts": [
{
"host": "hosta.example.com",
"hostid": "101",
"proxy_hostid": "1"
},
{
"host": "hostb.example.com",
"hostid": "102",
"proxy_hostid": "1"
},
{
"host": "hostc.example.com",
"hostid": "103",
"proxy_hostid": "2"
}
]
}
proxies.json:
{
"proxies": [
{
"host": "proxy1.example.com",
"proxyid": "1"
},
{
"host": "proxy2.example.com",
"proxyid": "2"
}
]
}
I also have the above file available with proxyid as the key, if this makes it easier:
{
"proxies": {
"1": {
"host": "proxy1.example.com",
"proxyid": "1"
},
"2": {
"host": "proxy2.example.com",
"proxyid": "2"
}
}
}
Using these json files above (from the Zabbix API), I'd like to add the value of .proxies[].host (from proxies.json) as .hosts[].proxy_host (to hosts.json).
This would only be when .hosts[].proxy_hostid equals .proxies[].proxyid
Desired output:
{
"hosts": [
{
"host": "hosta.example.com",
"hostid": "101",
"proxy_hostid": "1",
"proxy_host": "proxy1.example.com"
},
{
"host": "hostb.example.com",
"hostid": "102",
"proxy_hostid": "1",
"proxy_host": "proxy1.example.com"
},
{
"host": "hostc.example.com",
"hostid": "103",
"proxy_hostid": "2",
"proxy_host": "proxy2.example.com"
}
]
}
I've tried many different ways of doing this, and think I need to use jq -s or jq --slurpfile, but I've reached a lot of dead-ends and can't find a solution.
jq 'input as $p | map(.[].proxy_host = $p.proxies[].proxyid)' hosts.json proxies.json
I think I would need something like this as well, but not sure how to use it.
if .hosts[].proxy_hostid == .proxies[].proxyid then .hosts[].proxy_host = .proxies[].host else empty end'
I've found these questions but they haven't helped :(
How do I use a value as a key reference in jq? <- I think this one is the closest
Lookup values from one JSON file and replace in another
Using jq find key/value pair based on another key/value pair
This indeed is easier with the alternative version of your proxies.json. All you need is to store proxies in a variable as reference, and retrieve proxy hosts from it while updating hosts.
jq 'input as { $proxies } | .hosts[] |= . + { proxy_host: $proxies[.proxy_hostid].host }' hosts.json proxies.json
Online demo
I have a file:
{
"test_data": [
{
"id": "1",
"pm": "30",
"mp": "40"
}
],
"test": [
"id",
"pm",
"mp"
]
}
I want to extract test_data. Output:
"test_data": [
{
"id": "1",
"pm": "30",
"mp": "40"
}
],
I try this command: cat myFile | sed -n '/^"test_data": \[$/,/^\],$/p'
But it's don't work. An idea ?
Thanks you !
jq seems the right tool for the job :
$ jq '.|{test_data:.test_data}' filename
{
"test_data": [
{
"id": "1",
"pm": "30",
"mp": "40"
}
]
}
Solution 1st: With sed
sed -n '/"test_data"/,/],/p' Input_file
OR: as per OP, OP needs to append a string/data after a line matches:
sed -n '/"test_data"/,/],/p;/],/s/$/"test"/p' Input_file
OR2: If one wants to add an another file's content to a match then following may help in same:
sed -n '/"test_data"/,/],/p;/],/r another_file' Input_file
Solution 2nd: Following simple awk may help you in same.
awk '/test_data/, /],/' Input_file
Output will be as follows.
"test_data": [
{
"id": "1",
"pm": "30",
"mp": "40"
}
],
Logic for above solutions:
For sed: -n option in sed will turn OFF the printing of any line till it is explicitly mentioned to print it, then by doing /"test_data"/,/],/ I am letting sed know that I need to get the data from test_data to till ,/] and mentioning p after that will make sure those lines which are coming in this range are getting printed here/
For awk: Simply mentioning the range from /"test_data"/,/],/ and not mentioning any action so when any line comes into this range condition becomes true and since no action mentioned so by default print of that line happens then.
You can try that with gnu
csplit -s -z infile %test_data%-1 /],/1;rm xx01;echo "Add Text here" >> xx00;cat xx00
The right way is jq tool:
jq 'del(.test)' inputfile
The output:
{
"test_data": [
{
"id": "1",
"pm": "30",
"mp": "40"
}
]
}
I have two .json-files.
The first is 1.json
{
"id": "107709375",
"type": "page",
"title": "SomeTitle",
"space": {
"key": "BUSINT"
},
"version": {
"number": 62
}
}
And the second one logg.json:
{
"id": "228204270",
"type": "page",
"status": "current",
"title": "test-test",
"version": {
"when": "2016-11-23T16:54:18.313+07:00",
"number": 17,
"minorEdit": false
},
"extensions": {
"position": "none"
}
}
Can I paste version.number from logg.json into version.number 1.json using jq? I need something like that (it's absolutely wrong):
jq-win64 ".version.number 1.json" = ".version.number +1" logg.json
Read logg.json as an argument file. You could then access its values to make changes to the other.
$ jq --argfile logg logg.json '.version.number = $logg.version.number + 1' 1.json
Of course you'll need to use double quotes to work in the Windows Command prompt.
> jq --argfile logg logg.json ".version.number = $logg.version.number + 1" 1.json
Although the documentation says to use --slurpfile instead, we only have a single object in the file so it would be totally appropriate to use --argfile instead.
My json looks like this :
{
"20160522201409-jobsv1-1": {
"vmStateDisplayName": "Ready",
"servers": {
"20160522201409 jobs_v1 1": {
"serverStateDisplayName": "Ready",
"creationDate": "2016-05-22T20:14:22.000+0000",
"state": "READY",
"provisionStatus": "PENDING",
"serverRole": "ROLE",
"serverType": "SERVER",
"serverName": "20160522201409 jobs_v1 1",
"serverId": 2902
}
},
"isAdminNode": true,
"creationDate": "2016-05-22T20:14:23.000+0000",
"totalStorage": 15360,
"shapeId": "ot1",
"state": "READY",
"vmId": 4353,
"hostName": "20160522201409-jobsv1-1",
"label": "20160522201409 jobs_v1 ADMIN_SERVER 1",
"ipAddress": "10.252.159.39",
"publicIpAddress": "10.252.159.39",
"usageType": "ADMIN_SERVER",
"role": "ADMIN_SERVER",
"componentType": "jobs_v1"
}
}
My key keeps changing from time to time. So for example 20160522201409-jobsv1-1 may be something else tomorrow. Also I may more than one such entry in the json payload.
I want to echo $KEYS and I am trying to do it using jq.
Things I have tried :
| jq .KEYS is the command i use frequently.
Is there a jq command to display all the primary keys in the json?
I only care about the hostname field. And I would like to extract that out. I know how to do it using grep but it is NOT a clean approach.
You can simply use: keys:
% jq 'keys' my.json
[
"20160522201409-jobsv1-1"
]
And to get the first:
% jq -r 'keys[0]' my.json
20160522201409-jobsv1-1
-r is for raw output:
--raw-output / -r: With this option, if the filter’s result is a string then it will be written directly to standard output rather than being formatted as a JSON string with quotes. This can be useful for making jq filters talk to non-JSON-based systems.
Source
If you want a known value below an unknown property, eg xxx.hostName:
% jq -r '.[].hostName' my.json
20160522201409-jobsv1-1
Could somebody help me to deal with jq command line utility to update JSON object's inner value?
I want to alter object interpreterSettings.2B263G4Z1.properties by adding several key-values, like "spark.executor.instances": "16".
So far I only managed to fully replace this object, not add new properties with command:
cat test.json | jq ".interpreterSettings.\"2B188AQ5T\".properties |= { \"spark.executor.instances\": \"16\" }"
This is input JSON:
{
"interpreterSettings": {
"2B263G4Z1": {
"id": "2B263G4Z1",
"name": "sh",
"group": "sh",
"properties": {}
},
"2B188AQ5T": {
"id": "2B188AQ5T",
"name": "spark",
"group": "spark",
"properties": {
"spark.cores.max": "",
"spark.yarn.jar": "",
"master": "yarn-client",
"zeppelin.spark.maxResult": "1000",
"zeppelin.dep.localrepo": "local-repo",
"spark.app.name": "Zeppelin",
"spark.executor.memory": "2560M",
"zeppelin.spark.useHiveContext": "true",
"spark.home": "/usr/lib/spark",
"zeppelin.spark.concurrentSQL": "false",
"args": "",
"zeppelin.pyspark.python": "python"
}
}
},
"interpreterBindings": {
"2AXUMXYK4": [
"2B188AQ5T",
"2AY8SDMRU"
]
}
}
I also tried the following but this only prints contents of interpreterSettings.2B263G4Z1.properties, not full object.
cat test.json | jq ".interpreterSettings.\"2B188AQ5T\".properties + { \"spark.executor.instances\": \"16\" }"
The following works using jq 1.4 or jq 1.5 with a Mac/Linux shell:
jq '.interpreterSettings."2B188AQ5T".properties."spark.executor.instances" = "16" ' test.json
If you have trouble adapting the above for Windows, I'd suggest putting the jq program in a file, say my.jq, and invoking it like so:
jq -f my.jq test.json
Notice that there is no need to use "cat" in this case.
p.s. You were on the right track - try replacing |= with +=