I am doing some work with Azure Devops (ADO) Variable Groups and I would like to query an existing Variable Group to get all its variables to build a list of parameters to send to ADO CLI method.
Here is the JSON representation of an existing Variable Group:
{
"authorized": true,
"description": "test",
"name": "TESTGROUP",
"providerData": null,
"type": "Vsts",
"variables": {
"app_container_environment": {
"value": "dev"
},
"aws_region": {
"value": "us-west-2"
}
}
}
Problem:
What I'd like to do is use jq to read each variable definition and extract the variable name and value. Then, I'd build a string prefixed by "--variables" followed by a list of all the key/value pairs, space-delimited as follows:
--variables app_container_environment="dev" aws_region="us-west-2"
Note that the list begins with "--variables" followed by a key/value pair delimited by a space between each key/value pair.
I have tried to use join which is kind of close. But the main problem I'm having is due to the way the JSON is structured. I'm not sure how to refer to the variable name. The value element is easy to get, but I can't seem to get it's parent(e.g. the variable name). For example, "us-west-2" value's variable name is "aws_region".
How would I do this?
With your sample, the invocation:
jq -r '
.variables
| [to_entries[]
| "\(.key)=\"\(.value.value|tostring)\""]
| "--variables " + join(" ")
' sample.json
produces:
--variables app_container_environment="dev" aws_region="us-west-2"
Related
I have an object that looks like this
{
"my_list": [
{
"name": "an_image",
"image": "an_image.com"
},
{
"name": "another_image",
"image": "another_image.com"
},
...<more objects with image property>
],
"foo": {
"image": "foobar.io"
},
"bar": {
"image": "bar_image.io"
},
...<more objects with image property>
}
I'd like to get all of the image properties from each of the objects, and from each object in my_list and other lists that have objects that include an image property. So in this example I'd like to get
"an_image.com"
"another_image.com"
"foobar.io"
"bar_image.io"
We don't know the keys of any of these objects at runtime, so we can't reference my_list, foo, or bar in this example.
Previously we didn't have my_list in the object and jq '.[].image' worked, but now that results in jq: error (at bar.json:18): Cannot index array with string "image".
The problem is that we don't know the name of the objects that contain image properties so we can't reference them explicitly, and now that we've added another element that's a list we're running into type errors, that I'm not sure how to solve.
I've tried various combinations of .[].image, but they all seem to run into issues with typing.
If you don't mind the terseness, you could perhaps go with:
jq '..|select(.image?).image'
You could select by objects and items of arrays:
jq '.[] | ., arrays[] | objects.image'
"an_image.com"
"another_image.com"
"foobar.io"
"bar_image.io"
Demo
Using recursive descent .. is more elegant:
jq '.. | .image? // empty'
If the input is large, you might want to consider streaming the data in:
$ jq --stream -r 'select(.[0][-1] == "image")[1] // empty' input.json
an_image.com
another_image.com
foobar.io
bar_image.io
When streamed, your input will be processed as path/value pairs for the most part. Filter the paths you want, then return the value.
I have the following code:
json_string = '{
"EmployeeName": {
"FirstName": "Lex",
"LastName": "Sharma"
}
}'
final_data = JSON.parse(json_string)
And I have a variable that is read in with the value of the given field, Ie:
location = ['EmployeeName', 'FirstName']
So I would like to retrieve the value somehow in one line preferably, something like
value = final_data[location]
Unfortunately even Ruby isn't this flexible, is there something wrong with my syntax? Can anyone recommend how I can get the value Lex efficiently in a single line?
Use Hash#dig to Specify Keys at Runtime
An easy way to drill down into a dynamic list of key objects is to use Ruby's Hash#dig method. For example:
require 'json'
json = <<~'JSON'
{
"EmployeeName": {
"FirstName": "Lex",
"LastName": "Sharma"
}
}
JSON
# the path/query elements
keys = ['EmployeeName', 'FirstName']
# splat the array to turn array elements
# into positional arguments for Hash#dig
JSON.parse(json).dig *keys
#=> "Lex"
There are certainly other ways to navigate a Hash, but the #dig method allows you to chain keys as a list of arguments, rather than hard-coding the number of keys as you would need to do with a series of element reference calls like json[key1][key2].
I'm setting up a devops pipeline so that certain data profiles stored in JSON format can be shifted across different servers. While downloading it from the current server I need to clean up all the protected keys and unique identifiers. I'm looking for the cleanest way to do the following in JQ
Input:
{
"TopKey1":{
"some_key":"some_value"
},
"TopKey2":{
"some_key2":"some_value2"
},
"KeytoSearch":[
{
"_id":"sdf",
"non_relevant_key1":"val"
},
{
"_id":"sdfdsdf",
"non_relevant_key2":"val"
},
{
"_id":"sgf",
"non_relevant_key3":"val"
}
]
}
Output:
{
"TopKey1":{
"some_key":"some_value"
},
"TopKey2":{
"some_key2":"some_value2"
},
"KeytoSearch":[
{
"non_relevant_key1":"val"
},
{
"non_relevant_key2":"val"
},
{
"non_relevant_key3":"val"
}
]
}
In python terms if this were a dictionary
for json_object in dictionary["KeytoSearch"]:
json_object.pop("_id")
I've tried combinations of map and del but can't seem to figure out the nested indexing with this. The error messages I get are along the lines of jq: error (at <stdin>:277): Cannot index string with string "_id" which sort of tells me I haven't fundamentally understood how jq works or is to be used, but this is the route I need to go because using a Python script to clean up JSON objects is something I'd rather avoid
Going with your input JSON and assuming there are other properties in your KeytoSearch object along with the _id fields, you could just do below.
jq 'del(.KeytoSearch[]._id)'
See this jqplay.org snippet for a demo. The quotes around the property key containing _ are not needed as confirmed in one of the comments below. Some meta-characters (e.g. . in the property key values needs be accessed with quotes as ".id") needs to be quoted properly, but _ is clearly not one of them.
I've tried combinations of map and del
Good! You were probably just missing the '|=' magic ingredient:
.Keytosearch |= map( del(._id) )
alternatively, you could use a walk-path unix tool for JSON: jtc and apply changes right into the sourse json file (-f):
bash $ jtc -fpw'[KeytoSearch]<_id>l:' file.json
bash $
bash $
bash $ jtc file.json
{
"KeytoSearch": [
{
"non_relevant_key1": "val"
},
{
"non_relevant_key2": "val"
},
{
"non_relevant_key3": "val"
}
],
"TopKey1": {
"some_key": "some_value"
},
"TopKey2": {
"some_key2": "some_value2"
}
}
bash $
if given json snippet is a part of a larger JSON (and [KeytoSearch] is not addressable from the root), then replace it with the search lexeme: <KeytoSearch>l.
PS> Disclosure: I'm the creator of the jtc tool
How can one write a jq query that will extract a property from an element that may appear as singleton or list?
For example, extract the URL property from the creator in both example JSON strings below.
Example #1:
{
"#type": "example1",
"creator":{
"#type":"Organization",
"url": "https://www.ncei.noaa.gov/"
}
}
Example #2:
{
"#type": "example2",
"creator": [{
"#type":"Person",
"url": "https://www.example.com/homepage"
},
{
"url": "https://www.example.com/another"
}]
}
I have tried using .creator for the first one and .creator[] for the second one, but these two are not compatible. Is there a way to write so that it works for both examples?
One possibility that is very straightforward is simply to test whether .creator is an array or not:
if .creator|type == "array" then .creator[] else .creator end
| .url
Streaming parser
At the other end of the spectrum of straightforwardness, here is another possibility that would be relevant if (a) the input JSON document was ginormous, and (b) the goal is to list all .url values that occur as immediate children of .creator, no matter where the keys are located in the input JSON document. The invocation would use the --stream command-line option, e.g.
jq --stream -f filter.jq input.json
The jq filter in this case is:
select(length==2 and
(.[0][-1] == "url") and
last( .[0][:-1][]| strings ) == "creator")
| .[1]
I have json that looks like this:
{
"Workflow": [
{
"Parameters": {
"Project": "/Path/To/File",
"OtherParam": "True"
}
}
],
"Overrides": [
{
"Special": {
"Parameters": {
"NewParam": "NewStuffGoesHere",
"OtherParam": "False"
}
}
}
]
}
... where I want to use the Overrides.Special section to add or update fields in the workflow object. In other words, given the json above, I want to do something like this:
$config = Get-Content workflow.json | out-string | ConvertFrom-Json
$configWithOverrides = Merge-Object $config.Workflow $config.Overrides.Special
And end up with something like this:
$configWithOverrides
Parameters
----------
#{Project=/Path/To/File; NewParam=NewStuffGoesHere; OtherParam=False}
I can certainly write the Merge-Object function above to add or update values as needed based on what's in the override section, but it seems there should (could?) be a built-in or one-liner way to handle this.
I tried this:
$test = $config.Workflow + $config.Overrides.Special
...but that doesn't quite work.
$test
Parameters
----------
#{Project=/Path/To/File; OtherParam=True}
#{NewParam=NewStuffGoesHere; OtherParam=False}
This enables adding parameters:
>$test.Parameters.NewParam
NewStuffGoesHere
...but it's not so great for updating them
>$test.Parameters.OtherParam
True
False
Note - in this example, I'm choosing to handle the merge after converting the json to a psobject, but that's not a requirement.
I have a one-liner to do what you're asking for. Notice that, as far as I know, PowerShell does not deal directly with json strings. But, once converted to PowerShell objects, it's like any other object.
So, firstly, define your json file, and read it as a single string:
# Requires -Version 4
$jsonFile='c:\temp\jsonfile.json'
$jsonObj=#(gc $jsonFile -raw)|ConvertFrom-Json
Define the property upon which you want to merge the json's objects, and the 1st and 2nd objects:
$property='Parameters'
$1=$jsonObj.Workflow.$property
$2=$jsonObj.Overrides.Special.$property
Now, see the one-liner (which I've splitted in 3, for the sake of clarity):
$MergedJson=[pscustomobject]#{
$property=$2.psobject.properties|%{$11=$1}{$11|add-member $_.name $_.value -ea Ignore}{$11}
}|ConvertTo-Json
You see? $MergedJson holds the following string (using your json string):
{
"Parameters": {
"Project": "/Path/To/File",
"OtherParam": "True",
"NewParam": "NewStuffGoesHere"
}
}
Is that what you're looking for?
P.S.: if you swap the roles of $1 and $2, the common parameters' (like OtherParam) values that prevail, change.