I have different language files like these:
file1
{
"Pack": [
{
"id": "item1",
"lang": {
"en": {
}
}
},
{
"id": "item2",
"lang": {
"en": {
}
}
}
]
}
file2
{
"Pack": [
{
"id": "item1",
"lang": {
"sp": {
}
}
}
]
}
and I need to merge the same ids by lang field. Final file should looks like:
{
"Pack": [
{
"id": "item1",
"lang": {
"en": {
},
"sp": {
}
}
},
{
"id": "item2",
"lang": {
"en": {
}
}
}
]
}
I think I should use something like more complex command but my starting point is:
jq -s '{ attributes: map(.attributes[0]) }' file*.json
First you'll want to read in all files as input, then combine all Pack items and aggregating them into groups by id, then take those groups and arrange them to the result you need.
$ jq -n '
{Pack: ([inputs.Pack[]] | group_by(.id) | map({id: .[0].id, lang: (map(.lang) | add)}))}
' file*.json
This results in:
{
"Pack": [
{
"id": "item1",
"lang": {
"en": {},
"sp": {}
}
},
{
"id": "item2",
"lang": {
"en": {}
}
}
]
}
Related
HERE is sample json file.
sample.json
{
"apps": [
{
"name": "app1"
},
{
"name": "app2"
},
{
"name": "app3"
}
],
"test": [
{
"name": "test1"
},
{
"name": "test2"
}
]
}
I want to divide the above JSON file into the following two files.
I want to manage the entire configuration file with one JSON, divide the file when necessary and give it to the tool.
apps.json
{
"apps": [
{
"name": "app1"
},
{
"name": "app2"
},
{
"name": "app3"
}
}
test.json
{
"test": [
{
"name": "test1"
},
{
"name": "test1"
}
]
}
jq .apps sample.json outputs only value.
[
// Not contain the key
{
"name": "app1"
},
{
"name": "app2"
},
{
"name": "app3"
}
]
Can you have any idea?
Construct a new object using {x} which is a shorthand for {x: .x}.
jq '{apps}' sample.json
{
"apps": [
{
"name": "app1"
},
{
"name": "app2"
},
{
"name": "app3"
}
]
}
Demo
And likewise with {test}.
You can do
{apps}, {test}
Demo
https://jqplay.org/s/P_9cc2uANV
I have two JSON objects in two files (result_0.json and result_1.json) which look like this
{
"data": {
"pools": [
{
"id": "1"
},
{
"id": "2"
}
]
}
}
and like this:
{
"data": {
"pools": [
{
"id": "3"
},
{
"id": "4"
}
]
}
}
What I would like to get looks like this:
{
"data": {
"pools": [
{
"id": "1"
},
{
"id": "2"
},
{
"id": "3"
},
{
"id": "4"
}
]
}
}
How can it be done? I tried
jq -s add result_0.json result_1.json
but it just overwrites the values in result_0.json with the values of result_1.json.
If .data and .pool are the only keys in the json files, you can use
jq -n '{ data: { pools: [inputs.data.pools] | add } }' result0 result1
This will create the desired output:
{
"data": {
"pools": [
{
"id": "1"
},
{
"id": "2"
},
{
"id": "3"
},
{
"id": "4"
}
]
}
}
Regarding the inputs keyword, consider reading JQ's docs on this part.
I have this json i got from aws, this is just a test i created not my actual rules
[
{
"Name": "Fortinet-all_rules",
"Priority": 0,
"Statement": {
"ManagedRuleGroupStatement": {
"VendorName": "Fortinet",
"Name": "all_rules",
"ExcludedRules": [
{
"Name": "Database-Vulnerability-Exploit-01"
},
{
"Name": "Database-Vulnerability-Exploit-02"
},
{
"Name": "Database-Vulnerability-Exploit-03"
},
{
"Name": "Malicious-Robot"
},
{
"Name": "OS-Command-Injection-01"
},
{
"Name": "OS-Command-Injection-02"
},
{
"Name": "SQL-Injection-01"
},
{
"Name": "SQL-Injection-02"
},
{
"Name": "SQL-Injection-03"
},
{
"Name": "Source-Code-Disclosure"
},
{
"Name": "Web-Application-Injection-01"
},
{
"Name": "Web-Application-Injection-02"
},
{
"Name": "Web-Application-Vulnerability-Exploit-01"
},
{
"Name": "Web-Application-Vulnerability-Exploit-02"
},
{
"Name": "Web-Application-Vulnerability-Exploit-03"
},
{
"Name": "Web-Application-Vulnerability-Exploit-04"
},
{
"Name": "Web-Application-Vulnerability-Exploit-05"
},
{
"Name": "Web-Application-Vulnerability-Exploit-06"
},
{
"Name": "Web-Application-Vulnerability-Exploit-07"
},
{
"Name": "Web-Scanner-01"
},
{
"Name": "Web-Scanner-02"
},
{
"Name": "Web-Scanner-03"
},
{
"Name": "Web-Server-Vulnerability-Exploit-01"
},
{
"Name": "Web-Server-Vulnerability-Exploit-02"
},
{
"Name": "Web-Server-Vulnerability-Exploit-03"
},
{
"Name": "Web-Server-Vulnerability-Exploit-04"
}
],
"ScopeDownStatement": {
"RegexPatternSetReferenceStatement": {
"ARN": "",
"FieldToMatch": {
"UriPath": {}
},
"TextTransformations": [
{
"Priority": 0,
"Type": "NONE"
}
]
}
}
}
},
"OverrideAction": {
"None": {}
},
"VisibilityConfig": {
"SampledRequestsEnabled": true,
"CloudWatchMetricsEnabled": true,
"MetricName": "Fortinet-all_rules"
}
},
{
"Name": "DDOS_rate_rule",
"Priority": 1,
"Statement": {
"RateBasedStatement": {
"Limit": 350,
"AggregateKeyType": "FORWARDED_IP",
"ScopeDownStatement": {
"NotStatement": {
"Statement": {
"IPSetReferenceStatement": {
"ARN": "",
"IPSetForwardedIPConfig": {
"HeaderName": "X-Forwarded-For",
"FallbackBehavior": "MATCH",
"Position": "FIRST"
}
}
}
}
},
"ForwardedIPConfig": {
"HeaderName": "X-Forwarded-For",
"FallbackBehavior": "MATCH"
}
}
},
"Action": {
"Block": {}
},
"VisibilityConfig": {
"SampledRequestsEnabled": true,
"CloudWatchMetricsEnabled": true,
"MetricName": "DDOS_rate_rule"
}
}
]
So what i want for example is to delete the element { "Name": "OS-Command-Injection-01" }
I need to do it conditionally
So i tried using select jq '. | select([].Statement.ManagedRuleGroupStatement.ExcludedRules[].Name == "Malicious-Robot")'
problem is it errors jq: error (at :150): Cannot iterate over null (null)
also if i try to chain selects it doesn't work
I will also need to delete several object at once, but if i can delete one i can run the query several times so that's not an issue
|= is useful for modifying elements of a data structure.
The left-hand side should return the things to modify. (Use parens if it contains |.)
The right-hand side is evaluated as if | was used instead of |=.
The right-hand side should return the new value. (Use parens if it contains |.)
The whole returns . with the modifications made.
jq '
( .[].Statement.ManagedRuleGroupStatement.ExcludedRules | arrays ) |=
map(select(.Name != "OS-Command-Injection-01"))
'
jqplay
To delete objects from arrays, you could use the template:
walk(if type == "array”
then map(select(
( type=="object" and
(.Name|IN( ... ) ) ) | not ))
else . end)
You can try this :
jq 'walk(if type=="object" and
(.Name|IN("OS-Command-Injection-01","SQL-Injection-03"))
then empty
else . end)' input-file
I have a collection of a class that looks something like that:
[
{
"_id": 1,
"title": "dummy title",
"assignments": [
{
"_id": 1,
"name": "a1",
"members": [
{
"_id": 11,
"full_name": "john doe",
"aga": 18
},
{
"_id": 12,
"full_name": "john doe2",
"aga": 18
}
]
}
],
"settings": [
{
"type": "light",
"status": "enabled"
},
{
"type": "flare",
"status": "disabled"
},
{
"type": "toolbar",
"status": "enabled"
}
]
}
]
I have 2 nested documents here "assignments" which have a nested "members"
and "settings". the result i want should look something like that:
{
"_id": 1,
"title": "dummy title",
"assignments": [
{
"_id": 1,
"name": "a1",
"member_ids": [11, 18]
}
],
"active_settings": ["light", "toolbar"]
}
Meaning in each "assignment" I should only return the ids of the members and not the whole member data. and in settings I should only return the settings that are set to "active"
is it possible?
Playground here:
https://mongoplayground.net/p/Le4BdTm_gOv
You can try with $map to go one by one. $mergeObjects helps to merge the output value with same object
[
{
$project: {
title: 1,
assignments: {
$map: {
input: "$assignments",
in: {
$mergeObjects: [
"$$this",
{
members: {
$map: {
input: "$$this.members",
in: "$$this._id"
}
}
}
]
}
}
},
active_settings: {
$reduce: {
input: "$settings",
initialValue: [],
in: {
$cond: [
{
$eq: [
"$$this.status",
"enabled"
]
},
{
$setUnion: [
"$$value",
[
"$$this.type"
]
]
},
"$$value"
]
}
}
}
}
}
]
Working Mongo playground
You can try,
get assignments member ids using $map and $reduce
get active_settings using $reduce
db.collection.aggregate([
{
$project: {
_id: 1,
title: 1,
assignments: {
$map: {
input: "$assignments",
in: {
_id: "$$this._id",
name: "$$this.name",
memberIds: {
$reduce: {
input: "$$this.members",
initialValue: [],
in: { $concatArrays: ["$$value", ["$$this._id"]] }
}
}
}
}
},
active_settings: {
$reduce: {
input: "$settings",
initialValue: [],
in: {
$cond: [
{ $eq: ["$$this.status", "enabled"] },
{ $concatArrays: ["$$value", ["$$this.type"]] },
"$$value"
]
}
}
}
}
}
])
Playground
I have an Artifactory AQL Spec file in JSON format. The spec file is as follows:
{
"files": [
{
"aql": {
"items.find": {
"repo": "release-repo",
"modified": { "$before": "30d" },
"type": { "$eq": "folder" },
"depth": "2"
}
}
}
]
}
let's say I run a gitlab api query to acquire a list of SHAs that I want to iterate through and add to this json spec file.. The list of SHAs are assigned to a variable..
"a991fef6bb9e9759d513fd4b277fe3674b44e4f4"
"5a562d34bb1d4ab4264acc2c61327651218524ad"
"d4e296c35644743e58aed35d1afb87e34d6c8823"
I would like to iterate through all these commit IDs in and add them one by one to the json so that they are in this format:
{
"files": [
{
"aql": {
"items.find": {
"repo": "release-repo",
"modified": { "$before": "30d" },
"type": { "$eq": "folder" },
"$or": [
{
"$and": [
{
"name": {
"$nmatch": "*a991fef6bb9e9759d513fd4b277fe3674b44e4f4*"
}
}
]
},
{
"$and": [
{
"name": {
"$nmatch": "*5a562d34bb1d4ab4264acc2c61327651218524ad*"
}
}
]
},
{
"$and": [
{
"name": {
"$nmatch": "*d4e296c35644743e58aed35d1afb87e34d6c8823*"
}
}
]
}
],
"depth": "2"
}
}
}
]
}
The list of SHAs returned from the gitlab api query will be different everything and that's why I'd like this to be a dynamic entry or update every time. The number of returned SHAs will also be different... Could return 10 one day or it could return 50 on another day.
#!/usr/bin/env bash
template='{
"files": [
{
"aql": {
"items.find": {
"repo": "release-repo",
"modified": { "$before": "30d" },
"type": { "$eq": "folder" },
"$or": [],
"depth": "2"
}
}
}
]
}'
shas=(
"a991fef6bb9e9759d513fd4b277fe3674b44e4f4"
"5a562d34bb1d4ab4264acc2c61327651218524ad"
"d4e296c35644743e58aed35d1afb87e34d6c8823"
)
jq -n \
--argjson template "$template" \
--arg shas_str "${shas[*]}" \
'
reduce ($shas_str | split(" ") | .[]) as $sha ($template;
.files[0].aql["items.find"]["$or"] += [{
"$and": [{"name": {"$nmatch": ("*" + $sha + "*")}}]
}]
)
'
...emits as output:
{
"files": [
{
"aql": {
"items.find": {
"repo": "release-repo",
"modified": {
"$before": "30d"
},
"type": {
"$eq": "folder"
},
"$or": [
{
"$and": [
{
"name": {
"$nmatch": "*a991fef6bb9e9759d513fd4b277fe3674b44e4f4*"
}
}
]
},
{
"$and": [
{
"name": {
"$nmatch": "*5a562d34bb1d4ab4264acc2c61327651218524ad*"
}
}
]
},
{
"$and": [
{
"name": {
"$nmatch": "*d4e296c35644743e58aed35d1afb87e34d6c8823*"
}
}
]
}
],
"depth": "2"
}
}
}
]
}
Here is a reduce-free solution. It makes some inessential assumptions -
that the sha strings are presented as a stream of strings on STDIN, and that the Artifactory spec is in a file named spec.json. Here is the jq program:
map( {"$and": [ {name: { "$nmatch": "*\(.)*" }}]} ) as $x
| $spec[0] | (.files[0].aql."items.find"."$or" = $x)
The jq invocation might look like this:
jq -s --slurpfile spec spec.json -f program.jq <<< "${shas[*]}"