I have a json that looks like this:
{
"HostedZones": [
{
"ResourceRecordSetCount": 2,
"CallerReference": "test20150527-2",
"Config": {
"Comment": "test2",
"PrivateZone": true
},
"Id": "/hostedzone/Z119WBBTVP5WFX",
"Name": "dev.devx.company.services."
},
{
"ResourceRecordSetCount": 2,
"CallerReference": "test20150527-1",
"Config": {
"Comment": "test",
"PrivateZone": true
},
"Id": "/hostedzone/Z3P5QSUBK4POTI",
"Name": "test.devx.company.services."
}
],
"IsTruncated": false,
"MaxItems": "100"
}
And my goal is to fetch a specific Name (in my case it's the test.devx.company.services), however the Name field contains an extra "." at the end that I'd like to remove from the output.
This is what I have so far:
jq --raw-output '.HostedZones[] | select(.Name | test("test")?) | (.Name[:-1] | sub("."; ""))'
The problem with that it is removing the first character from the output also.
So the output currently is: est.devx.company.services (JQ play snippet)
Not sure what I'm doing wrong :/
To always remove the last character, if it contains "test":
jq '(.HostedZones[].Name | select(contains("test"))) |= .[:-1]'
To remove it only if it is a dot:
jq '(.HostedZones[].Name | select(contains("test"))) |= sub("[.]$"; "")'
Related
I am trying to use jq to filter the latest Docker Image version from a curl output. So far I could come up to here:
Command
curl https://docker.hub.example.net/api/v1.0/projects/myapp/repositories/artifacts | jq -r '(.[] | {digest, tags})'
Output:
Note: Some sub-keys have been removed and real values have been replaced with some example values in the output.
{
"digest": "sha256:.......",
"tags": [
{
"artifact_id": 123456,
"name": "latest",
},
{
"artifact_id": 123456,
"name": "1.0.1234567890.ab12cd3",
}
]
}
{
"digest": "sha256:.......",
"tags": [
{
"artifact_id": 234567,
"name": "1.0.1234567890.bc23de4",
}
]
}
{
"digest": "sha256:.......",
"tags": [
{
"artifact_id": 345678,
"name": "1.0.1234567890.cd34ef5",
}
]
}
As you can see in the above output, only one digest has two tags with the same contents except the name sub-key values are different. One is "name": "latest" and the other is the image version (e.g. "name": "1.0.1234567890.ab12cd3"). Other digests have only one tag.
I need to get the image version from the digest that has the other tag with "name": "latest". I prefer to avoid scripted loop, if possible, and just use the jq options.
How can I achieve this?
Use select in combination with any:
curl ... | jq -r '
.[] | select(.tags | any(.name == "latest"))
| first(.tags[] | select(.name != "latest")).name
'
1.0.1234567890.ab12cd3
Demo
Preface: If the following is not possible with jq, then I completely accept that as an answer and will try to force this with bash.
I have two files that contain some IDs that, with some massaging, should be able to be combined into a single file. I have some content that I'll add to that as well (as seen in output). Essentially "mitre_test" should get compared to "sys_id". When compared, the "mitreid" from in2.json becomes technique_ID in the output (and is generally the unifying field of each output object).
Caveats:
There are some junk "desc" values placed in the in1.json that are there to make sure this is as programmatic as possible, and there are actually numerous junk inputs on the true input file I am using.
some of the mitre_test values have pairs and are not in a real array. I can split on those and break them out, but find myself losing the other information from in1.json.
Notice in the "metadata" for the output that is contains the "number" values from in1.json, and stored in a weird way (but the way that the receiving tool requires).
in1.json
[
{
"test": "Execution",
"mitreid": "T1204.001",
"mitre_test": "90b"
},
{
"test": "Defense Evasion",
"mitreid": "T1070.001",
"mitre_test": "afa"
},
{
"test": "Credential Access",
"mitreid": "T1556.004",
"mitre_test": "14b"
},
{
"test": "Initial Access",
"mitreid": "T1200",
"mitre_test": "f22"
},
{
"test": "Impact",
"mitreid": "T1489",
"mitre_test": "fa2"
}
]
in2.json
[
{
"number": "REL0001346",
"desc": "apple",
"mitre_test": "afa"
},
{
"number": "REL0001343",
"desc": "pear",
"mitre_test": "90b"
},
{
"number": "REL0001366",
"desc": "orange",
"mitre_test": "14b,f22"
},
{
"number": "REL0001378",
"desc": "pineapple",
"mitre_test": "90b"
}
]
The output:
[{
"techniqueID": "T1070.001",
"tactic": "defense-evasion",
"score": 1,
"color": "",
"comment": "",
"enabled": true,
"metadata": [{
"name": "DET_ID",
"value": "REL0001346"
}],
"showSubtechniques": true
},
{
"techniqueID": "T1204.001",
"tactic": "execution",
"score": 1,
"color": "",
"comment": "",
"enabled": true,
"metadata": [{
"name": "DET_ID",
"value": "REL0001343"
},
{
"name": "DET_ID",
"value": "REL0001378"
}],
"showSubtechniques": true
},
{
"techniqueID": "T1556.004",
"tactic": "credential-access",
"score": 1,
"color": "",
"comment": "",
"enabled": true,
"metadata": [{
"name": "DET_ID",
"value": "REL0001366"
}],
"showSubtechniques": true
},
{
"techniqueID": "T1200",
"tactic": "initial-access",
"score": 1,
"color": "",
"comment": "",
"enabled": true,
"metadata": [{
"name": "DET_ID",
"value": "REL0001366"
}],
"showSubtechniques": true
}
]
I'm assuming I have some splitting to do on mitre_test with something like .mitre_test |= split(",")), and there are some joins I'm assuming, but doing so causes data loss or mixing up of the data. You'll notice the static data in the output exists as well, but is likely easy to place in and as such isn't as much of an issue.
Edit: reduced some of the match IDs so that it is easier to look at while analyzing the in1 and in2 files. Also simplified the two inputs to have a similar structure so that the answer is easier to understand later.
The requirements are somewhat opaque but it's fairly clear that if the task can be done by computer, it can be done using jq.
From the description, it would appear that one of the unusual aspects of the problem is that the "dictionary" defined by in1.json must be derived by splitting the key names that are CSV (comma-separated values). Here therefore is a jq def that will do that:
# Input: a JSON dictionary for which some keys are CSV,
# Output: a JSON dictionary with the CSV keys split on the commas
def refine:
. as $in
| reduce keys_unsorted[] as $k ({};
if ($k|index(","))
then ($k/",") as $keys
| . + ($keys | map( {(.): $in[$k]}) | add)
else .[$k] = $in[$k]
end );
You can see how this works by running:
INDEX($mitre.records[]; .mitre_test) | refine
using an invocation of jq such as:
jq --argfile mitre in1.json -f program.jq in2.json
For the joining part of the problem, there are many relevant Q&As on SO, e.g.
How to join JSON objects on particular fields using jq?
There is probably a much more elegant way to do this, but I ended up manually walking around things and piping to new output.
Explanation:
Read in both files, pull the fields I need.
Break out the mitre_test values that were previously just a comma separated set of values with map and try.
Store the none-changing fields as a variable and then manipulate mitre_test to become an appropriately split array, removing nulls.
Group by mitre_test values, since they are the common thing that the output is based on.
Cleanup more nulls.
Sort output to look like I want it.
jq . in1.json in2.json | \
jq '.[] |{number: .number, test: .test, mitreid: .mitreid, mitre_test: .mitre_test}' |\
jq -s '[. |map(try(.mitre_test |= split(",")) // .)|\
.[] | [.number,.test,.mitreid] as $h | .mitre_test[] |$h + [.] | \
{DET_ID: .[0], tactic: .[1], techniqueID: .[2], mitre_test: .[3]}] |\
del(.[][] | nulls)' |jq '[group_by(.mitre_test)[]|{mitre_test: .[0].mitre_test, techniqueID: [.[].techniqueID],tactic: [.[].tactic], DET_ID: [.[].DET_ID]}]|\
del(.[].techniqueID[] | nulls) | del(.[].tactic[] | nulls) | del(.[].DET_ID[] | nulls)' | \
jq '.[]| [{techniqueID: .techniqueID[0],tactic: .tactic[0], metadata: [{name: "DET_ID",value: .DET_ID[]}]}] | .[] | \
select((.metadata|length)>0)'
It was a long line, so I split it among some of the basic ideas.
I have the following json file for exemple:
{
"FOO": {
"name": "Donald",
"location": "Stockholm"
},
"BAR": {
"name": "Walt",
"location": "Stockholm"
},
"BAZ": {
"name": "Jack",
"location": "Whereever"
}
}
and i have this jq command :
cat json | jq .[] | {newname : select(.location=="Stockholm") | .name , contains_w : select(.location=="Stockholm") | .name | startswith("W")}
so i get the result :
{
"newname": "Donald",
"contains_w": false
}
{
"newname": "Walt",
"contains_w": true
}
my question is : is there any way to DRY my command ?
i mean how can i get the same result without duplicate the part :
select(.location=="Stockholm") | .name
how can i reuse the result of newname feild ?
i have a really big file to work with so i don't want to waste time and resources.
You are filtering multiple times during object construction. You could filter first and then do the construction on the filtered list eg.
map(select(.location=="Stockholm"))
| map({newname: .name, contains_w: (.name | startswith("W"))})
https://jqplay.org/s/aXjlgOEDnb
I have a json file that looks like so:
[
{
"code": "1234",
"files": [
{
"fileType": "pdf",
"url": "http://.../a.pdf"
},
{
"fileType": "video",
"url": "http://.../b.mp4"
}
]
},
{
"code": "4321",
"files": [
{
"fileType": "pdf",
"url": "http://.../c.pdf"
},
{
"fileType": "video",
"url": "http://.../d.mp4"
}
]
},
{
"code": "9999",
"files": [
{
"fileType": "pdf",
"url": "http://.../e.pdf"
}
]
}
]
I would like to print out only the files that are of fileType == video in the files array such that I end up with output that looks like so:
1234, "http://.../b.mp4"
4321, "http://.../d.mp4"
So far I am only able to output something that looks like this:
1234, "http://.../a.pdf", "http://.../b.mp4",
4321, "http://.../c.pdf", "http://.../d.mp4"
Using the following:
jq -r '.[] | select(.files[]?.fileType == "video") | [.code, .files[].url] | #csv'
I was wondering how I can filter the .files[] based on the fileType as I am outputting them?
The following pipeline makes the solution fairly self-explanatory, assuming one understands the basic syntax and the -r command-line option:
< input.json jq -r '
.[]
| .code as $code
| .files[]
| select(.fileType == "video")
| "\($code), \"\(.url)\""
'
So I am trying to pull out some JSON with jq.
Here is an example:
{
"limit":10,
"offset":20,
"values": [
{
"id": "abcd"
"type": "file"
"users": {
"total": 2,
"profiles": [
{
"first_name": "John",
"last_name": "Smith"
},
{
"first_name": "Sue",
"last_name": "Johnson"
}
]
}
},
{
"id": "efgh"
"type": "folder"
"users": {
"total": 1,
"profiles": [
{
"first_name": "Steve",
"last_name": "Gold"
}
]
}
},
]
}
I would love to get the following in my result
limit:10
offset:20
id:abcd, type:file, [users.total:2, users.profiles.first_name: [John, Sue], users.profiles.last_name: [Smith, Johnson]],
id:efgh, type:folder, [users.total:1, users.profiles.first_name: [Steve], users.profiles.last_name: [Gold]],
I know I can pipe this to jq, but I don't know how. I can get stuff in an array easily, but I can't seem to figure out how to do it and add in the top level elements. So it's 'limit' and 'offset' that are throwing me fits.
I can get something out
jq ".values[] | .users.total, .users[].first_name, .users[].last_name"
But I cannot seem to figure out how to add in limit and offset.
You can use commas at the top-level to get those fields. Use parentheses to group the expression that gets everything from .values.
jq '.limit, .offset, (.values[] | .id, .type, .users.total, .users.profiles[].first_name, .users.profiles[].last_name)'
DEMO
So it's 'limit' and 'offset' that are throwing me fits.
It looks like to_entries might be what you're looking for. Consider, for example:
to_entries[]
| select(.value | type != "array")
| [.key, .value]
| join(": ")
With your JSON, this produces the lines you are having difficulty with, without any explicit mention of the key names:
limit: 10
offset: 20
You could then (for example) use the comma operator (,) to produce the other lines you want.