I have a JSON file:
{
"header": {
"uuid": "c578592a-a751-4993-9060-53f488597e59",
"timestamp": 1522938800,
"productionDateTime": "2018-04-05T14:33:20.000+00:00",
"producers": {
"operator": null,
"application": {
"com.example": {
"id": {
"string": "1"
},
"name": {
"string": "Test"
}
}
},
"station": null,
"equipment": null,
"partner": null,
"flow": null
},
"correlationContext": {
"array": [{
"correlationId": "98440498-3104-479e-9c99-f4449ba8f4b6",
"correlationDateTime": {
"string": "2018-04-05T14:30:39.000+00:00"
}
}]
}
},
}
I would like to create n files that copy the JSON file but with a random correlationId and correlationDateTime.
Do you have any tips or suggestions? Many thanks in advance!
Using jq and leaving the generation of appropriate random values as an exercise for the reader:
$ jq --arg cid "foo" --arg cdt "bar" '
.header.correlationContext.array[0]={
correlationId: $cid,
correlationDateTime: {string: $cdt}
}' tmp.json
Here you go:-
#!/bin/bash
alias uid="python -c 'import sys,uuid; sys.stdout.write(uuid.uuid4().hex)' | pbcopy && pbpaste && echo"
dt=$(date "+%Y%m%d-%H%M%S")
# For above two values I am leaving with you. If you have best option please go for it
i=0
while [ $i -le 20 ]
do
sed "/correlationId/s/98440498-3104-479e-9c99-f4449ba8f4b6/"$uid"/;s/2018-04-05T14:30:39.000+00:00/"$dt"/" yourJsonFile.json > testcase$i.json
i=$((i+1))
done
Related
I have the following string as below:
{
"name": {
"value": "Demo"
},
"activity": {
"value": "CLOSED",
"timestamp": "2020-11-19T10:58:17.534+0000"
},
"state": {
"value": "OK",
"timestamp": "2020-11-19T10:58:17.570+0000"
},
"lastErrorCode": {
"value": "NO_MESSAGE",
"timestamp": "2020-11-19T10:58:17.570+0000"
}
}
How can I convert all timestamps in that string to another format (Timezone)? Example like (on cmd line):
echo '2020-11-19T10:58:17.534+0000' | xargs date +'%Y-%m-%d %H:%M:%S' -d
results in:
2020-11-19 11:58:17
That string of yours is actually a JSON(-string). Please use a tool that supports JSON and can do dateTime-conversions, like xidel.
First of all, the value of the timestamp-keys is not a valid dateTime. You'd have to change the timezone property 0000 to 00:00 to make it a valid one:
$ xidel -s input.json -e '
for $x in $json//timestamp return
dateTime(replace($x,"0000","00:00"))
'
2020-11-19T10:58:17.534Z
2020-11-19T10:58:17.57Z
2020-11-19T10:58:17.57Z
Then to change the timezone to +01:00 use adjust-dateTime-to-timezone():
$ xidel -s input.json -e '
for $x in $json//timestamp return
adjust-dateTime-to-timezone(
dateTime(replace($x,"0000","00:00")),
duration("PT1H")
)
'
2020-11-19T11:58:17.534+01:00
2020-11-19T11:58:17.57+01:00
2020-11-19T11:58:17.57+01:00
(You can remove duration("PT1H") if your timezone already is +01:00)
Finally to customize your output use format-dateTime():
$ xidel -s input.json -e '
for $x in $json//timestamp return
format-dateTime(
adjust-dateTime-to-timezone(
dateTime(replace($x,"0000","00:00")),
duration("PT1H")
),
"[Y]-[M01]-[D01] [H01]:[m01]:[s01]"
)
'
2020-11-19 11:58:17
2020-11-19 11:58:17
2020-11-19 11:58:17
If instead you want to update the JSON with these customized dateTimes... that can be done, but requires a more advanced recursive function:
$ xidel -s input.json --xquery '
declare function local:change-timestamp($a){
if (exists($a)) then
if ($a instance of map(*)) then
map:merge(
map:keys($a) ! map{
.:if (.="timestamp") then
format-dateTime(
adjust-dateTime-to-timezone(
dateTime(replace($a(.),"0000","00:00")),
duration("PT1H")
),
"[Y]-[M01]-[D01] [H01]:[m01]:[s01]"
)
else
local:change-timestamp($a(.))
}
)
else
$a
else
()
};
local:change-timestamp($json)
'
{
"name": {
"value": "Demo"
},
"activity": {
"value": "CLOSED",
"timestamp": "2020-11-19 11:58:17"
},
"state": {
"value": "OK",
"timestamp": "2020-11-19 11:58:17"
},
"lastErrorCode": {
"value": "NO_MESSAGE",
"timestamp": "2020-11-19 11:58:17"
}
}
Also check the xidel playground.
Should be possible using regex. Something like this:
str='{
"name": {
"value": "Demo"
},
"activity": {
"value": "CLOSED",
"timestamp": "2020-11-19T10:58:17.534+0000"
},
"state": {
"value": "OK",
"timestamp": "2020-11-19T10:58:17.570+0000"
},
"lastErrorCode": {
"value": "NO_MESSAGE",
"timestamp": "2020-11-19T10:58:17.570+0000"
}
}'
re='(.*?-[0-9]+-[0-9]+)T([0-9:]+)(\.[0-9+]+)(.*)'
while [[ $str =~ $re ]]; do
str="${BASH_REMATCH[1]} ${BASH_REMATCH[2]}${BASH_REMATCH[4]}"
done
echo "$str"
Returns:
{
"name": {
"value": "Demo"
},
"activity": {
"value": "CLOSED",
"timestamp": "2020-11-19 10:58:17"
},
"state": {
"value": "OK",
"timestamp": "2020-11-19 10:58:17"
},
"lastErrorCode": {
"value": "NO_MESSAGE",
"timestamp": "2020-11-19 10:58:17"
}
}
My json file has the below content:
{
"Fruits": {
"counter": 1,
"protocols": [
{
"id": "100",
"name": "lemon",
"category": "citrus"
},
{
"id": "350",
"name": "Orange",
"category": "citrus"
},
{
"id": "150",
"name": "lime",
"category": "citrus"
}
]
}
}
I am expecting an output as below
Fruits:lemon:citrus
Fruits:Orange:citrus
Fruits:lime:citrus
Easy to do with jq:
$ jq -r '.Fruits.protocols[] | "Fruits:\(.name):\(.category)"' input.json
Fruits:lemon:citrus
Fruits:Orange:citrus
Fruits:lime:citrus
The jq answer is better. Still posting a Ruby solution (if you cannot use jq), but it is less elegant:
ruby -e '
require "json";
l="";
ARGF.each { |x| l+=x };
obj=JSON.parse(l);
obj["Fruits"]["protocols"].each { |x| puts "Fruits:#{x["name"]}:#{x["category"]}" }
'
Here is the full example:
echo '{"Fruits":{"counter":1,"protocols":[{"id":"100","name":"lemon","category":"citrus"},{"id":"350","name":"Orange","category":"citrus" },{"id":"150","name":"lime","category":"citrus"}]}}' \
| ruby -e 'require "json";l="";ARGF.each { |x| l+=x } ; obj=JSON.parse(l) ; obj["Fruits"]["protocols"].each { |x| puts "Fruits:#{x["name"]}:#{x["category"]}" }'
Output:
Fruits:lemon:citrus
Fruits:Orange:citrus
Fruits:lime:citrus
I have two JSON files a.json and b.json. The contents in a.json file is a JSON object and inside b.json its an array.I want to add/update status field in each mappings in a.json by retrieving the value from b.json file.
a.json:
{
"title": 25886,
"data": {
"request": {
"c": 46369,
"t1": 1562050127.376641
},
},
"rs": {
"mappings": {
"12345": {
"id": "12345",
"name": "test",
"customer_id": "11228",
},
"45678": {
"id": "45678",
"name": "abc",
"customer_id": "11206",
}
}
}}
b.json:
[
{
"status": "pending",
"extra": {
"name": "test"
},
"enabled": true,
"id": "12345"
},
{
"status": "not_started",
"extra": {
"name": "abc"
},
"enabled": true,
"id": "45678"
}
]
Below is my expected output:
{
"title": 25886,
"data": {
"request": {
"c": 46369,
"t1": 1562050127.376641
},
},
"rs": {
"mappings": {
"12345": {
"id": "12345",
"name": "test",
"customer_id": "11228",
"status":"pending"
},
"45678": {
"id": "45678",
"name": "abc",
"customer_id": "11206",
"status":"not_started"
}
}
}}
In this expected JSON file we have status field whose value is retrieved from b.json file based on a matching id value. How to do this using jq ?
For the purposes of this problem, b.json essentially defines a dictionary, so for simplicity, efficiency and perhaps elegance,
it make sense to start by using the builtin function INDEX to create the relevant dictionary:
INDEX( $b[] | {id, status}; .id )
This assumes an invocation of jq along the lines of:
jq --argfile b b.json -f update.jq a.json
(Yes, I know --argfile has been deprecated. Feel free to choose another way to set $b to the contents of b.json.)
Now, to perform the update, it will be simplest to use the "update" operator, |=, in conjunction with map_values. (Feel free to check the jq manual :-)
Putting everything together:
INDEX( $b[] | {id, status}; .id ) as $dict
| .rs.mappings |= map_values( .status = $dict[.id].status )
I'd like to know a quick way to insert a json to json.
$ cat source.json
{
"AWSEBDockerrunVersion": 2,
"containerDefinitions": [
{
"environment": [
{
"name": "SERVICE_MANIFEST",
"value": ""
},
{
"name": "SERVICE_PORT",
"value": "4321"
}
]
}
]
}
The SERVICE_MANIFEST is content of another json file
$ cat service_manifest.json
{
"connections": {
"port": "1234"
},
"name": "foo"
}
I try to make it with jq command
cat service_manifest.json |jq --arg SERVICE_MANIFEST - < source.json
But seems it doesn't work
Any ideas? The final result still should be a valid json file
{
"AWSEBDockerrunVersion": 2,
"containerDefinitions": [
{
"environment": [
{
"name": "SERVICE_MANIFEST",
"value": {
"connections": {
"port": "1234"
},
"name": "foo"
}
},
...
]
}
],
...
}
Updates.
Thanks, here is the command I run from your sample.
$ jq --slurpfile sm service_manifest.json '.containerDefinitions[].environment[] |= (select(.name=="SERVICE_MANIFEST").value=$sm)' source.json
But the result is an array, not list.
{
"AWSEBDockerrunVersion": 2,
"containerDefinitions": [
{
"environment": [
{
"name": "SERVICE_MANIFEST",
"value": [
{
"connections": {
"port": "1234"
},
"name": "foo"
}
]
},
{
"name": "SERVICE_PORT",
"value": "4321"
}
]
}
]
}
You can try this jq command:
jq --slurpfile sm SERVICE_MANIFEST '.containerDefinitions[].environment[] |= (select(.name=="SERVICE_MANIFEST").value=$sm[])' file
--slurpfile assigns the content of the file to the variable sm
The filter replaces the array .containerDefinitions[].environment[] with the content of the file only on the element having SERVICE_MANIFEST as name.
A simple solution would use --argfile and avoid select:
< source.json jq --argfile sm service_manifest.json '
.containerDefinitions[0].environment[0].value = $sm '
Or if you want only to update the object(s) with .name == "SERVICE_MANIFEST" you could use the filter:
.containerDefinitions[].environment
|= map(if .name == "SERVICE_MANIFEST"
then .value = $sm
else . end)
Variations
There is no need for any "--arg"-style parameter at all, as illustrated by the following:
jq -s '.[1] as $sm
| .[0] | .containerDefinitions[0].environment[0].value = $sm
' source.json service_manifest.json
I have a json of this structure:
{
"nodes": {
"60e327ee58a0": {
"nodeinfo": {
"network": {
"mesh": {
"bat0": {
"interfaces": {
"wireless": [
"<mac-address-removed>"
],
"tunnel": [
"<mac-address-removed>"
]
}
}
},
"mac": "<mac removed>",
"addresses": [
"<ipv6 removed>",
"<ipv6 removed>"
]
},
"hardware": {
"model": "TP-Link TL-WR841N/ND v10",
"nproc": 1
},
"software": {
"batman-adv": {
"compat": 15,
"version": "2015.1"
},
"autoupdater": {
"branch": "stable",
"enabled": true
},
"firmware": {
"release": "v2016.1+1.0.1",
"base": "gluon-v2016.1"
},
"status-page": {
"api": 1
},
"fastd": {
"enabled": true,
"version": "v17"
}
},
"hostname": "Antoniusweg12",
"system": {
"site_code": "ffmsd03"
},
"node_id": "60e327ee58a0"
},
"lastseen": "2016-04-14T12:39:04",
"flags": {
"gateway": false,
"online": true
},
"firstseen": "2016-03-16T15:14:04",
"statistics": {
"clients": 1,
"gateway": "de:ad:be:ef:43:02",
"rootfs_usage": 0.6041666666666667,
"loadavg": 0.09,
"uptime": 1822037.41,
"memory_usage": 0.8124737210932025,
"traffic": {
"rx": {
"packets": 50393821,
"bytes": 5061895206
},
"forward": {
"packets": 173,
"bytes": 17417
},
"mgmt_rx": {
"packets": 47453745,
"bytes": 6623785282
},
"tx": {
"packets": 1205695,
"bytes": 173509528,
"dropped": 5683
},
"mgmt_tx": {
"packets": 37906725,
"bytes": 11475209742
}
}
}
},
"30b5c2b042f4": {
<next block...>
And I want to query it with jq for the hostname, the mac or the IPv6.
cat nodes.json |jq -c '.nodes[] | select(.nodes[]| contains("Antoniusweg12"))'
Most examples do not fit this kind of json structure as the objects have an index
Thanks for help in advance.
If you're going to filter, you need to drill down to the property that you want to check for and see if it matches your criteria. You can't expect to just give a name and you'll magically be presented with the results you want.
Searching by hostname, it is found on the .nodeinfo.hostname property of each node:
$ jq -c --arg hostname "Antoniusweg12" \
'.nodes[] | select(.nodeinfo.hostname == $hostname)' nodes.json
Similarly for the mac address, it's found on the .nodeinfo.network.mac property:
$ jq -c --arg mac "aa:bb:cc:dd:ee:ff" \
'.nodes[] | select(.nodeinfo.network.mac == $mac)' nodes.json
For the ip addresses, there's an array of them but it's not that much different in the query. They're found on the .nodeinfo.network.addresses property:
$ jq -c --arg ip "aaaa:bbbb:cccc:dddd::1" \
'.nodes[] | select(.nodeinfo.network.addresses[] == $ip)' nodes.json
Here's another take on the question. Suppose you want to find all occurrences of the key "hostname" for which the value is "Antoniusweg12",
no matter where the key/value combination occurs.
The following will reveal the path to the key/value combination of interest:
paths as $p
| select ( $p[-1] == "hostname" and getpath($p) == "Antoniusweg12" )
| $p
The result for the given input JSON:
[
"nodes",
"60e327ee58a0",
"nodeinfo",
"hostname"
]
If you wanted the path to the containing object, then replace the final $p with $p[0:-1]; and if you want the containing object itself: getpath($p[0:-1])
Here is a solution which searches for nodes where the specified $needle is present in any of the addresses, mac or hostname fields.
"<ipv6 removed>" as $needle # set to whatever you like
| foreach (.nodes|keys[]) as $k (
.
; .
; ( .nodes[$k].nodeinfo.network.addresses?
+ [ .nodes[$k].nodeinfo.network.mac?
, .nodes[$k].nodeinfo.hostname?
]
) as $haystack
| if $haystack | index($needle)
then {($k): .nodes[$k]}
else empty
end
)
EDIT: I now realize a filter of the form foreach E as $X (.; .; R) can almost always be rewritten as E as $X | R so the above is really just
"<ipv6 removed>" as $needle
| (.nodes|keys[]) as $k
| ( .nodes[$k].nodeinfo.network.addresses?
+ [ .nodes[$k].nodeinfo.network.mac?
, .nodes[$k].nodeinfo.hostname?
]
) as $haystack
| if $haystack | index($needle)
then {($k): .nodes[$k]}
else empty
end