I have a json which looks like this:-
{
"name": "abc",
"version": "20.02.01",
"tag": "24",
"abc_version": "20.02",
"registry": "registry.abc.com",
"vendor": "greenplanet",
"apps": [
{
"name": "abc-app",
"version": "20.02.01-16",
"volumes": [
"/dev/log:/dev/log"
],
"max_instances": "1"
},
{
"name": "xyz-app",
"version": "2.0.0-2",
"volumes": [
"/dev/log:/dev/log"
],
"max_instances": "1"
}
]
}
based on a condition I need to increment the abc-app's or xyz-app's version. At present its at "20.02.01-16" for abc-app and I need to change it to "20.02.01-17". Also I need to increment the tag of the parent app which is "abc" to "25"
I am able to increment the version with sed but that is not working for me:-
./jq -r ".apps" version.json | ./jq -r ".[] | .version" | grep -v '1.0.2' |sed -r 's/([0-9]+.[0-9]+.[0-9]+)(.*)([0-9]+)/echo "\1\2$((\3+1))"/e'
I need to increment all the above conditions in-place in json or maybe into a temporary file which I can move to original.
Thank you in advance.
First, let's define a helper function to perform the incrementation:
# return a string
def inc:
(capture("(?<pre>.*)-(?<post>[0-9]+$)") | "\(.pre)-\( (.post|tonumber) + 1 )")
// "\(tonumber+1)" ;
The remainder of the solution can now be written in two lines, one for each value to be incremented:
.tag |= inc
| .apps |= map(if .name == "abc-app" then .version |= inc else . end)
In-place editing
You could use sponge, or a temporary file, or ... (see e.g.
"How can "in-place" editing of a JSON file be accomplished?"
on jq's FAQ).
I'm having trouble parsing a JSON file from which I would like to extract elements that does not contains the string "To Do" as label.
Here is the file I'm trying to parse
[
{
"id": 1,
"title": "Task A",
"labels": [
"Low",
"To Do"
]
},
{
"id": 2,
"title": "Task B",
"labels": [
"Medium",
"Done"
]
},
{
"id": 3,
"title": "Task C",
"labels": [
"High",
"To Do"
]
}
]
I can successfully extract elements which contains "To Do" with the following command :
$ cat tmp.json | jq 'keys[] as $k | select(.[$k].labels[] | contains("To Do")) | "\(.[$k].id) -- \(.[$k].title) -- \(.[$k].labels)"'
"1 -- Task A -- [\"Low\",\"To Do\"]"
"3 -- Task C -- [\"High\",\"To Do\"]"
However when I negate the condition jq has the following behaviour :
$ cat tmp.json | jq 'keys[] as $k | select(.[$k].labels[] | contains("To Do") | not) | "\(.[$k].id) -- \(.[$k].title) -- \(.[$k].labels)"'
"1 -- Task A -- [\"Low\",\"To Do\"]"
"2 -- Task B -- [\"Medium\",\"Done\"]"
"2 -- Task B -- [\"Medium\",\"Done\"]"
"3 -- Task C -- [\"High\",\"To Do\"]"
The only difference here is the condition's negation, but all three elements are printed and the one I want to filter is printed twice.
Thanks.
Your query seems overcomplicated for such a trivial task (I didn't debug but you're probably misusing contains there; I suspect it's run for each array element, not just once for the array itself).
Anyway, here is a clean and straightforward approach:
.[] | select(.labels | index("To Do") | not) | "\(.id) -- \(.title) -- \(.labels)"
I'd go with all(_; . != "To Do"):
.[]
| select( all(.labels[]; . != "To Do"))
| "\(.id) -- \(.title) -- \(.labels)"
I'm trying to convert 7z file content list to json and can't fix missing separator between output converted blocks.
I'm little bit newbie in json conversion, but found that jq could do the job.
I read the jq documentation and found examples inside here and there also elsewhere without solution.
Please find the use case:
The command line:
jq -f pf_7z.jq -R
The input file demo.lst:
Date Time Attr Size Compressed Name
------------------- ----- ------------ ------------ ------------------------
2018-06-23 14:02:16 D.... 0 0 Installer
2018-06-23 14:02:16 ..... 3381 1157 Installer\Readme
2018-06-23 14:02:16 ..... 4646 1157 Installer\License.txt
2018-06-23 14:02:16 ..... 138892 136152 Installer\Setup.exe
The filter file pf7z.jq:
def parse:
def parse_line:
. | map(match("(\\d+-\\d+-\\d+) (\\d+:\\d+:\\d+) (D|.).* +(\\d+) +(\\d+) +(.*\\\\)([^\\\\]*)\\.(.*)")) | .[] |
({
"date" :(.captures[0].string),
"time" :(.captures[1].string),
"attr" :(.captures[2].string),
"size" :(.captures[3].string),
"path" :(.captures[5].string),
"name" :(.captures[6].string),
"extn" :(.captures[7].string)
});
split("\n") | ( {} + (parse_line));
parse
The expected result should be:
{
"date": "2018-06-23",
"time": "14:02:16",
"attr": ".",
"size": "4646",
"path": "Installer\",
"name": "License",
"extn": "txt"
},
{
"date": "2018-06-23",
"time": "14:02:16",
"attr": ".",
"size": "138892",
"path": "Installer\",
"name": "Setup",
"extn": "exe"
}
And I only got :
{
"date": "2018-06-23",
"time": "14:02:16",
"attr": ".",
"size": "4646",
"path": "Installer\",
"name": "License",
"extn": "txt"
}
{
"date": "2018-06-23",
"time": "14:02:16",
"attr": ".",
"size": "138892",
"path": "Installer\",
"name": "Setup",
"extn": "exe"
}
without the comma separator between blocks.
Thanks ;-)
Your def for parse_line produces a stream of JSON entities, whereas you evidently want a JSON array. Using your regex, you could write:
def parse:
def parse_line:
match("(\\d+-\\d+-\\d+) (\\d+:\\d+:\\d+) (D|.).* +(\\d+) +(\\d+) +(.*\\\\)([^\\\\]*)\\.(.*)")
| .captures
| map(.string)
| { "date" :.[0],
"time" :.[1],
"attr" :.[2],
"size" :.[3],
"path" :.[5],
"name" :.[6],
"extn" :.[7] } ;
[inputs | parse_line];
parse
Invocation
jq -nR -f 7z.jq 7z.txt
Alternative regex
The regex fragment (D|.).* does not make much sense.
You should consider replacing it by (.)[^ ]* or some such.
A simpler solution
def parse_line:
capture("(?<date>\\d+-\\d+-\\d+) "
+ "(?<time>\\d+:\\d+:\\d+) "
+ "(?<attr>.)[^ ]* +"
+ "(?<size>\\d+) +\\d+ +"
+ "(?<path>.*\\\\)"
+ "(?<name>[^\\\\]*)\\."
+ "(?<extn>.*)");
[inputs | parse_line]
An alternative approach
From the comment about JSONEdit, it seems likely to me that your overall approach might be suboptimal. Have you considered using jq rather than jq with JSONEdit?
Using jq, I'd like to cherry-pick key/value pairs from the following json:
{
"project": "Project X",
"description": "This is a description of Project X",
"nodes": [
{
"name": "server001",
"detail001": "foo",
"detail002": "bar",
"networks": [
{
"net_tier": "network_tier_001",
"ip_address": "10.1.1.10",
"gateway": "10.1.1.1",
"subnet_mask": "255.255.255.0",
"mac_address": "00:11:22:aa:bb:cc"
}
],
"hardware": {
"vcpu": 1,
"mem": 1024,
"disks": [
{
"disk001": 40,
"detail001": "foo"
},
{
"disk002": 20,
"detail001": "bar"
}
]
},
"os": "debian8",
"geo": {
"region": "001",
"country": "Sweden",
"datacentre": "Malmo"
},
"detail003": "baz"
}
],
"detail001": "foo"
}
For the sake of an example, I'd like to parse the following keys and their values: "Project", "name", "net_tier", "vcpu", "mem", "disk001", "disk002".
I'm able to parse individual elements without much issue, but due to the hierarchical nature of the full parse, I've not had much luck parsing down different branches (i.e. both networks and hardware > disks).
Any help appreciated.
Edit:
For clarity, the output I'm going for is a comma-separated CSV. In terms of parsing all combinations, covering the sample data in the example will do for now. I will hopefully be able to expand on any suggestions.
Here is a different filter which computes the unique set of network tier and disk names and then generates a result with columns appropriate to the data.
{
tiers: [ .nodes[].networks[].net_tier ] | unique
, disks: [ .nodes[].hardware.disks[] | keys[] | select(startswith("disk")) ] | unique
} as $n
| def column_names($n): [ "project", "name" ] + $n.tiers + ["vcpu", "mem"] + $n.disks ;
def tiers($n): [ $n.tiers[] as $t | .networks[] | if .net_tier==$t then $t else null end ] ;
def disks($n): [ $n.disks[] as $d | map(select(.[$d]!=null)|.[$d])[0] ] ;
def rows($n):
.project as $project
| .nodes[]
| .name as $name
| tiers($n) as $tier_values
| .hardware
| .vcpu as $vcpu
| .mem as $mem
| .disks
| disks($n) as $disk_values
| [$project, $name] + $tier_values + [$vcpu, $mem] + $disk_values
;
column_names($n), rows($n)
| #csv
The benfit of this approach becomes apparent if we add another node to the sample data:
{
"name": "server002",
"networks": [
{
"net_tier": "network_tier_002"
}
],
"hardware": {
"vcpu": 1,
"mem": 1024,
"disks": [
{
"disk002": 40,
"detail001": "foo"
}
]
}
}
Sample Run (assuming filter in filter.jq and amended data in data.json)
$ jq -Mr -f filter.jq data.json
"project","name","network_tier_001","network_tier_002","vcpu","mem","disk001","disk002"
"Project X","server001","network_tier_001","",1,1024,40,20
"Project X","server002",,"network_tier_002",1,1024,,40
Try it online!
Here's one way you could achieve the desired output.
program.jq:
["project","name","net_tier","vcpu","mem","disk001","disk002"],
[.project]
+ (.nodes[] | .networks[] as $n |
[
.name,
$n.net_tier,
(.hardware |
.vcpu,
.mem,
(.disks | add["disk001","disk002"])
)
]
)
| #csv
$ jq -r -f program.jq input.json
"project","name","net_tier","vcpu","mem","disk001","disk002"
"Project X","server001","network_tier_001",1,1024,40,20
Basically, you'll want to project the fields that you want into arrays so you may convert those arrays to csv rows. Your input makes it seem like there could potentially be multiple networks for a given node. So if you wanted to output all combinations, that would have to be flattened out.
Here's another approach, that is short enough to speak for itself:
def s(f): first(.. | f? // empty) // null;
[s(.project), s(.name), s(.net_tier), s(.vcpu), s(.mem), s(.disk001), s(.disk002)]
| #csv
Invocation:
$ jq -r -f value-pairs.jq input.json
Result:
"Project X","server001","network_tier_001",1,1024,40,20
With headers
Using the same s/1 as above:
. as $d
| ["project", "name", "net_tier", "vcpu", "mem", "disk001","disk002"]
| (., map( . as $v | $d | s(.[$v])))
| #csv
With multiple nodes
Again with s/1 as above:
.project as $p
| ["project", "name", "net_tier", "vcpu", "mem", "disk001","disk002"] as $h
| ($h,
(.nodes[] as $d
| $h
| map( . as $v | $d | s(.[$v]) )
| .[0] = $p)
) | #csv
Output with the illustrative multi-node data:
"project","name","net_tier","vcpu","mem","disk001","disk002"
"Project X","server001","network_tier_001",1,1024,40,20
"Project X","server002","network_tier_002",1,1024,,40
I've got a jq question. Given a file file.json containing:
[
{
"type": "A",
"name": "name 1",
"url": "http://domain.com/path/to/filenameA.zip"
},
{
"type": "B",
"name": "name 2",
"url": "http://domain.com/otherpath/to/filenameB.zip"
},
{
"type": "C",
"name": "name 3",
"url": "http://otherdomain.com/otherpath/to/filenameB.zip"
}
]
I'm looking to create another file using jq with url modified only if the url's value matches some pattern. For example, I'd want to update any url matching the pattern:
http://otherdomain.com.*filenameB.*
to some fixed string such as:
http://yetanotherdomain.com/new/path/to/filenameC.tar.gz
with the resulting json:
[
{
"type": "A",
"name": "name 1",
"url": "http://domain.com/path/to/filenameA.zip"
},
{
"type": "B",
"name": "name 2",
"url": "http://domain.com/otherpath/to/filenameB.zip"
},
{
"type": "C",
"name": "name 3",
"url": "http://yetanotherdomain.com/new/path/to/filenameB.tar.gz"
}
]
I haven't gotten far even on being able to find the url, let alone update it. This is as far as I've gotten (wrong results and doesn't help me with the update issue):
% cat file.json | jq -r '.[] | select(.url | index("filenameB")).url'
http://domain.com/otherpath/to/filenameB.zip
http://otherdomain.com/otherpath/to/filenameB.zip
%
Any ideas on how to get the path of the key that has a value matching a regex? And after that, how to update the key with some new string value? If there are multiple matches, all should be updated with the same new value.
The good news is that there's a simple solution to the problem:
map( if .url | test("http://otherdomain.com.*filenameB.*")
then .url |= sub( "http://otherdomain.com.*filenameB.*";
"http://yetanotherdomain.com/new/path/to/filenameC.tar.gz")
else .
end)
The not-so-good news is that it's not so easy to explain unless you understand the key cleverness here - the "|=" filter. There is plenty of jq documentation about it, so I'll just point out that it is similar to the += family of operators in the C family of programming languages.
Specifically, .url |= sub(A;B) is like .url = (.url|sub(A;B)). That is how the update is done "in-place".
Here is a solution which identifies paths to url members with tostream and select and then updates the values using reduce and setpath
"http://otherdomain.com.*filenameB.*" as $from
| "http://yetanotherdomain.com/new/path/to/filenameC.tar.gz" as $to
| reduce (tostream | select(length == 2 and .[0][-1] == "url")) as $p (
.
; setpath($p[0]; $p[1] | sub($from; $to))
)