I'm trying to change # to %23 in every context value, but I'm having problem with null values.
The shortened JSON is:
{
"stats": {
"suites": 1
},
"results": [
{
"uuid": "676-a46b-47a1-a49f-4da4e46c1120",
"title": "",
"suites": [
{
"uuid": "gghjh-56a9-4713-b139-0d5b36bc7fbc",
"title": "Login process",
"tests": [
{
"pass": false,
"fail": true,
"pending": false,
"context": "\"screenshots/login.spec.js/Login process -- should login #11 (failed).png\""
},
{
"pass": false,
"fail": false,
"pending": true,
"context": null
}
]
}
]
}
]
}
And the JQ command I think it's closest to correct is:
jq '.results[].suites[].tests[].context | strings | sub("#";"%23")'
But the problem is that I need to get in return full edited file. How could I achieve that?
You were close. To retain the original structure, you need to use the update operator (|=) instead of pipe. Enclosing the entire expression to the left of it in parentheses is also necessary, otherwise the original input will be invisible to |=.
(.results[].suites[].tests[].context | strings) |= sub("#"; "%23")
Online demo
change # to %23 in every context value
You might wish to consider:
walk( if type=="object" and (.context|type)=="string"
then .context |= gsub("#"; "%23")
else . end )
Related
When I run the jq command to parse a json document from the amazon cli I have the following problem.
I’m parsing through the IP address and a tag called "Enviroment". The enviroment tag in the instance does not exist therefore it does not throw me any result.
Here's an example of the relevant output returned by the AWS CLI
{
"Reservations": [
{
"Instances": [
{
"PrivateIpAddress": "10.0.0.1",
"Tags": [
{
"Key": "Name",
"Value": "Balance-OTA-SS_a"
},
{
"Key": "Environment",
"Value": "alpha"
}
]
}
]
},
{
"Instances": [
{
"PrivateIpAddress": "10.0.0.2",
"Tags": [
{
"Key": "Name",
"Value": "Balance-OTA-SS_a"
}
]
}
]
}
]
}
I’m running the following command
aws ec2 describe-instances --filters "Name=tag:Name,Values=Balance-OTA-SS_a" | jq -c '.Reservations[].Instances[] | ({IP: .PrivateIpAddress, Ambiente: (.Tags[]|select(.Key=="Environment")|.Value)})'
## output
empty
How do I show the IP address in the output of the command even if the enviroment tag does not exist?
Regards,
Let's assume this input:
{
"Reservations": [
{
"Instances": [
{
"PrivateIpAddress": "10.0.0.1",
"Tags": [
{
"Key": "Name",
"Value": "Balance-OTA-SS_a"
},
{
"Key": "Environment",
"Value": "alpha"
}
]
}
]
},
{
"Instances": [
{
"PrivateIpAddress": "10.0.0.2",
"Tags": [
{
"Key": "Name",
"Value": "Balance-OTA-SS_a"
}
]
}
]
}
]
}
This is the format returned by describe-instances, but with all the irrelevant fields removed.
Note that tags is always a list of objects, each of which has a Key and a Value. This format is perfect for from_entries, which can transform this list of tags into a convenient mapping object. Try this:
.Reservations[].Instances[] |
{
IP: .PrivateIpAddress,
Ambiente: (.Tags|from_entries.Environment)
}
{"IP":"10.0.0.1","Ambiente":"alpha"}
{"IP":"10.0.0.2","Ambiente":null}
That answers how to do it. But you probably want to understand why your approach didn't work.
.Reservations[].Instances[] |
{
IP: .PrivateIpAddress,
Ambiente: (.Tags[]|select(.Key=="Environment")|.Value)
}
The .[] filter you're using on the tags can return zero or multiple results. Similarly, the select filter can eliminate some or all items. When you apply this inside an object constructor (the expression from { to }), you're causing that whole object to be created a variable number of times. You need to be very careful where you use these filters, because often that's not what you want at all. Often you instead want to do one of the following:
Wrap the expression that returns multiple results in an array constructor [ ... ]. That way instead of outputting the parent object potentially zero or multiple times, you output it once containing an array that potentially has zero or multiple items. E.g.
[.Tags[]|select(.Key=="Environment")]
Apply map to the array to keep it an array but process its contents, e.g.
.Tags|map(select(.Key=="Environment"))
Apply first(expr) to capture only the first value emitted by the expression. If the expression might emit zero items, you can use the comma operator to provide a default, e.g.
first((.Tags[]|select(.Key=="Environment")),null)
Apply some other array-level function, such as from_entries.
.Tags|from_entries.Environment
You can either use an if ... then ... else ... end construct, or //. For example:
.Reservations[].Instances[]
| {IP: .PrivateIpAddress} +
({Ambiente: (.Tags[]|select(.Key=="Environment")|.Value)}
// null)
I have two json files here, both have the same content only in a different order, and these are to be checked for equality with a diff.
I already sort the keys with jq -S, but now I have to make sure that the strings are sorted equally within the arrays.
Unfortunately, I fail at the moment, I am not quite clear how I get to the right level and how I can sort the content.
Here is an example structure of the jsons, the array 'allowed-test-mapper-data' should be sorted in descending order
{
"accessCodeLife": 60,
"accessCodeLifespan": 1800,
"accessCodeType": 300,
"components": {
"test.data.app": [
{
"config": {
"allow-default-test-scopes": [
"true"
]
},
"name": "Allowed Test Client",
"id": "allowed-testdata",
"subComponents": {},
"subType": "testdata"
},
{
"config": {
"allowed-test-mapper-data": [
"alfred",
"usa",
"canada",
"somedata",
"alcohol",
"brother"
]
}
}
]
}
}
Can someone help me here ?
Would be great :)
Use the update assignment |= operator to change a part of the structure:
jq '.components."test.data.app"[].config."allowed-test-mapper-data"
|= if . then sort else empty end' file.json
Here is a simplified json file of a terraform state file (let's call it dev.ftstate)
{
"version": 4,
"terraform_version": "0.12.9",
"serial": 2,
"lineage": "ba56cc3e-71fd-1488-e6fb-3136f4630e70",
"outputs": {},
"resources": [
{
"module": "module.rds.module.reports_cpu_warning",
"mode": "managed",
"type": "datadog_monitor",
"name": "alert",
"each": "list",
"provider": "module.rds.provider.datadog",
"instances": []
},
{
"module": "module.rds.module.reports_lag_warning",
"mode": "managed",
"type": "datadog_monitor",
"name": "alert",
"each": "list",
"provider": "module.rds.provider.datadog",
"instances": []
},
{
"module": "module.rds.module.cross_region_replica_lag_alert",
"mode": "managed",
"type": "datadog_monitor",
"name": "alert",
"each": "list",
"provider": "module.rds.provider.datadog",
"instances": []
},
{
"module": "module.rds",
"mode": "managed",
"type": "aws_db_instance",
"name": "master",
"provider": "provider.aws",
"instances": [
{
"schema_version": 0,
"attributes": {
"address": "dev-database.123456.us-east-8.rds.amazonaws.com",
"allocated_storage": 10,
"password": "",
"performance_insights_enabled": false,
"tags": {
"env": "development"
},
"timeouts": {
"create": "6h",
"delete": "6h",
"update": "6h"
},
"timezone": "",
"username": "admin",
"vpc_security_group_ids": [
"sg-1234"
]
},
"private": ""
}
]
}
]
}
There are many modules at the same level of module.rds inside the instances. I took out many of them to create the simplified version of the raw data. The key takeway: do not assume the array index will be constant in all cases.
I wanted to extract the password field in the above example.
My first attempt is to use equality check to extract the relevant modules
` jq '.resources[].module == "module.rds"' dev.tfstate`
but it actually just produced a list of boolean values. I don't see any mention of builtin functions like filter in jq's manual
then I tried to just access the field:
> jq '.resources[].module[].attributes[].password?' dev.tfstate
then it throws the following error
jq: error (at dev.tfstate:1116): Cannot iterate over string ("module.rds")
So what is the best way to extract the value? Hopefully it can only focus on the password attribute in module.rds module only.
Edit:
My purpose is to detect if a password is left inside a state file. I want to ensure the passwords are exclusively stored in AWS secret manager.
You can extract the module you want like this.
jq '.resources[] | select(.module == "module.rds")'
I'm not confident that I understand the requirements for the rest of the solution. So this might not only not be the best way of doing what you want; it might not do what you want at all!
If you know where password will be, you can do this.
jq '.resources[] | select(.module == "module.rds") | .instances[].attributes.password'
If you don't know exactly where password will be, this is a way of finding it.
jq '.resources[] | select(.module == "module.rds") | .. | .password? | values'
According to the manual under the heading "Recursive Descent," ..|.a? will "find all the values of object keys “a” in any object found “below” ."
values filters out the null results.
You could also get the password value out of the state file without jq by using Terraform outputs. Your module should define an output with the value you want to output and you should also output this at the root module.
Without seeing your Terraform code you'd want something like this:
modules/rds/main.tf
resource "aws_db_instance" "master" {
# ...
}
output "password" {
value = aws_db_instance.master.password
sensitive = true
}
example/main.tf
module "rds" {
source = "../modules/rds"
# ...
}
output "rds_password" {
value = module.rds.password
sensitive = true
}
The sensitive = true parameter means that Terraform won't print the output to stdout when running terraform apply but it's still held in plain text in the state file.
To then access this value without jq you can use the terraform output command which will retrieve the output from the state file and print it to stdout. From there you can use it however you want.
I'm trying to modify the deluge web.conf file with jq and I'm having some issues. The deluge web config seems to be invalid json
{
"file": 1,
"format": 1
}
{
"sidebar_show_zero": false,
"show_session_speed": false,
"pwd_sha1": "CHANGEME",
"show_sidebar": true,
"sessions": {},
"enabled_plugins": [],
"base": "/",
"first_login": true,
"theme": "gray",
"pkey": "ssl/daemon.pkey",
"default_daemon": "",
"cert": "test",
"session_timeout": 3600,
"https": false,
"interface": "0.0.0.0",
"sidebar_multiple_filters": true,
"pwd_salt": "salt",
"port": 8112
}
It has multiple top level elements which aren't separated by a comma so if I try to edit the file with jq using something like this jq '.pwd_sha1 = "NEW HASH"' web.conf I get the following
{
"file": 1,
"format": 1,
"pwd_sha1": "NEW HASH"
}
{
"sidebar_show_zero": false,
"show_session_speed": false,
"pwd_sha1": "NEW HASH",
"show_sidebar": true,
"sessions": {},
"enabled_plugins": [],
"base": "/",
"first_login": true,
"theme": "gray",
"pkey": "ssl/daemon.pkey",
"default_daemon": "",
"cert": "test",
"session_timeout": 3600,
"https": false,
"interface": "0.0.0.0",
"sidebar_multiple_filters": true,
"pwd_salt": "salt",
"port": 8112
}
jq is adding a new element to the first top level object and changing the second top level element's value. How can I get this to only change the existing item in the second top level element?
The web.conf you show is a stream of JSON entities. Fortunately for you, jq is stream-oriented, and it appears from your example that you could simply write:
jq 'if .pwd_sha1 then .pwd_sha1 = "NEW HASH" else . end' web.conf
In general, though, it might be more appropriate to write something with a more stringent test, e.g.
jq 'if type == "object" and has("pwd_sha1")
then .pwd_sha1 = "NEW HASH" else . end' web.conf
"changing the second top level element's value"
To edit the second top-level item only, you could use foreach inputs with the -n command-line option:
foreach inputs as $in (0; .+1;
if . == 2 then $in | .pwd_sha1 = "NEW_HASH"
else $in end)
I have an input JSON document with roughly the following form (actual data has additional keys, which should be passed through unmodified; the whitespace is adjusted for human readability, and there's no expectation that it be maintained):
{
"Rules": [
{"Filter": { "Prefix": "to_me/" }, "Status": "Enabled" },
{"Filter": { "Prefix": "from_me/" }, "Status": "Enabled" },
{"Filter": { "Prefix": "__bg/" }, "Status": "Enabled" }
]
}
I need to match .Rules[].Filter.Prefix=="to_me/" and then change the associated "Status": "Enabled" to "Disabled". Since only the first rule above has a prefix of to_me/, status of that rule would be changed to Disabled, making correct output look like the following:
{
"Rules": [
{"Filter": { "Prefix": "to_me/" }, "Status": "Disabled" },
{"Filter": { "Prefix": "from_me/" }, "Status": "Enabled" },
{"Filter": { "Prefix": "__bg/" }, "Status": "Enabled" }
]
}
I've tried several different combinations but can't seem to get it right.
Anyone have ideas?
I prefer the idiom ARRAY |= map(...) over ARRAY[] |= ..., mainly because the former can be used reliably whether or not any of the substitutions evaluate to empty:
jq '.Rules |= map(if .Filter.Prefix == "to_me/"
then .Status="Disabled" else . end)'
To overwrite the input file, you might like to consider sponge from moremutils.
Doing in-place updates can be done with |=, and deciding whether to modify content in-place can be done with if/then/else. Thus:
jq '.Rules[] |= (if .Filter.Prefix == "to_me/" then .Status="Disabled" else . end)'