Merging json objects in powershell - json

I have json that looks like this:
{
"Workflow": [
{
"Parameters": {
"Project": "/Path/To/File",
"OtherParam": "True"
}
}
],
"Overrides": [
{
"Special": {
"Parameters": {
"NewParam": "NewStuffGoesHere",
"OtherParam": "False"
}
}
}
]
}
... where I want to use the Overrides.Special section to add or update fields in the workflow object. In other words, given the json above, I want to do something like this:
$config = Get-Content workflow.json | out-string | ConvertFrom-Json
$configWithOverrides = Merge-Object $config.Workflow $config.Overrides.Special
And end up with something like this:
$configWithOverrides
Parameters
----------
#{Project=/Path/To/File; NewParam=NewStuffGoesHere; OtherParam=False}
I can certainly write the Merge-Object function above to add or update values as needed based on what's in the override section, but it seems there should (could?) be a built-in or one-liner way to handle this.
I tried this:
$test = $config.Workflow + $config.Overrides.Special
...but that doesn't quite work.
$test
Parameters
----------
#{Project=/Path/To/File; OtherParam=True}
#{NewParam=NewStuffGoesHere; OtherParam=False}
This enables adding parameters:
>$test.Parameters.NewParam
NewStuffGoesHere
...but it's not so great for updating them
>$test.Parameters.OtherParam
True
False
Note - in this example, I'm choosing to handle the merge after converting the json to a psobject, but that's not a requirement.

I have a one-liner to do what you're asking for. Notice that, as far as I know, PowerShell does not deal directly with json strings. But, once converted to PowerShell objects, it's like any other object.
So, firstly, define your json file, and read it as a single string:
# Requires -Version 4
$jsonFile='c:\temp\jsonfile.json'
$jsonObj=#(gc $jsonFile -raw)|ConvertFrom-Json
Define the property upon which you want to merge the json's objects, and the 1st and 2nd objects:
$property='Parameters'
$1=$jsonObj.Workflow.$property
$2=$jsonObj.Overrides.Special.$property
Now, see the one-liner (which I've splitted in 3, for the sake of clarity):
$MergedJson=[pscustomobject]#{
$property=$2.psobject.properties|%{$11=$1}{$11|add-member $_.name $_.value -ea Ignore}{$11}
}|ConvertTo-Json
You see? $MergedJson holds the following string (using your json string):
{
"Parameters": {
"Project": "/Path/To/File",
"OtherParam": "True",
"NewParam": "NewStuffGoesHere"
}
}
Is that what you're looking for?
P.S.: if you swap the roles of $1 and $2, the common parameters' (like OtherParam) values that prevail, change.

Related

jq get values from complex object

I have an object that looks like this
{
"my_list": [
{
"name": "an_image",
"image": "an_image.com"
},
{
"name": "another_image",
"image": "another_image.com"
},
...<more objects with image property>
],
"foo": {
"image": "foobar.io"
},
"bar": {
"image": "bar_image.io"
},
...<more objects with image property>
}
I'd like to get all of the image properties from each of the objects, and from each object in my_list and other lists that have objects that include an image property. So in this example I'd like to get
"an_image.com"
"another_image.com"
"foobar.io"
"bar_image.io"
We don't know the keys of any of these objects at runtime, so we can't reference my_list, foo, or bar in this example.
Previously we didn't have my_list in the object and jq '.[].image' worked, but now that results in jq: error (at bar.json:18): Cannot index array with string "image".
The problem is that we don't know the name of the objects that contain image properties so we can't reference them explicitly, and now that we've added another element that's a list we're running into type errors, that I'm not sure how to solve.
I've tried various combinations of .[].image, but they all seem to run into issues with typing.
If you don't mind the terseness, you could perhaps go with:
jq '..|select(.image?).image'
You could select by objects and items of arrays:
jq '.[] | ., arrays[] | objects.image'
"an_image.com"
"another_image.com"
"foobar.io"
"bar_image.io"
Demo
Using recursive descent .. is more elegant:
jq '.. | .image? // empty'
If the input is large, you might want to consider streaming the data in:
$ jq --stream -r 'select(.[0][-1] == "image")[1] // empty' input.json
an_image.com
another_image.com
foobar.io
bar_image.io
When streamed, your input will be processed as path/value pairs for the most part. Filter the paths you want, then return the value.

Creating valid JSON for container_definitions in Terraform

I’m creating an aws_ecs_task_definition resource. Within that resource, I need a container_definitions, which needs to be a JSON string. I’d like to add multiple secrets to that definition from a list of strings; [“var1”, “var2”].
The output I need looks like:
“secrets”: [
{
“name” = “var1”,
“valueFrom” = “arn:somestuffvar1”
},
{
“name” = “var2”,
“valueFrom” = “arn:somestuffvar2”
}
],
I have tried string interpolation and templatefile, this is the section from my .tftpl
  "secrets": [
   %{ for myvar in myvars ~}
    {
      "name": "${myvar}",
     "valueFrom”: “arn:somestuff${myvar}"
    }
    %{ endfor }
  ],
The problem is the commas. the above gives me
[
{
“name” = “var1”,
“valueFrom” = “arn:somestuffvar1”
}
{
“name” = “var2”,
“valueFrom” = “arn:somestuffvar2”
}
],
with no commas between the braces, if I add a comma, then i get a trailing comma
[
{
},
{
},
],
I’ve tried a zillion syntax variations, I’ve tried jsonencode on the interpolated string, I’ve tried stripping the trailing comma. Nothing gives me valid JSON. What am I missing?
The templatefile function documentation has a section specifically about Generating JSON or YAML from a template, which explicitly discourages using string templating to try to build valid JSON from string fragments like this.
Instead, you should use the jsonencode function as the entire definition of your template, and thus let Terraform be the one to worry about generating valid JSON syntax. You then only need to worry about writing an expression that describes the data structure that remote system expects.
In your case, a template generating a JSON object with just this "secrets" property would look like this:
${jsonencode({
secrets = [
for myvar in myvars : {
name = myvar
valueFrom = "arn:somestuff${myvar}"
}
],
})}
Notice that the entire template consists of a single interpolation sequence ${ ... } and the expression inside it is one big call to the jsonencode function, with the argument describing the data structure to serialize. Therefore inside that argument we're using normal Terraform expression syntax (like you'd write in a resource argument in a .tf file) rather than the special template interpolation/repetition syntaxes. In particular, the value of "secrets" is defined using a for expression.
With your { myvars = ["var1", "var2"] } template variables, this will produce a minified version of the following JSON structure, which I'm showing with manually-added indentation and newlines just so you can read it:
{
"secrets": [
{
"name": "var1",
"valueFrom": "arn:somestuffvar1"
},
{
"name": "var2",
"valueFrom": "arn:somestuffvar2"
}
]
}
I understand that you're only showing a fragment of the template here and so the above won't include all of the other properties included in your template, but hopefully you can see how to use Terraform expression syntax to describe those properties as Terraform object attributes too, so that the overall result of this template will be a valid JSON serialization of the total data structure.

How Can I Create a Space-Delimited String from JSON?

I am doing some work with Azure Devops (ADO) Variable Groups and I would like to query an existing Variable Group to get all its variables to build a list of parameters to send to ADO CLI method.
Here is the JSON representation of an existing Variable Group:
{
"authorized": true,
"description": "test",
"name": "TESTGROUP",
"providerData": null,
"type": "Vsts",
"variables": {
"app_container_environment": {
"value": "dev"
},
"aws_region": {
"value": "us-west-2"
}
}
}
Problem:
What I'd like to do is use jq to read each variable definition and extract the variable name and value. Then, I'd build a string prefixed by "--variables" followed by a list of all the key/value pairs, space-delimited as follows:
--variables app_container_environment="dev" aws_region="us-west-2"
Note that the list begins with "--variables" followed by a key/value pair delimited by a space between each key/value pair.
I have tried to use join which is kind of close. But the main problem I'm having is due to the way the JSON is structured. I'm not sure how to refer to the variable name. The value element is easy to get, but I can't seem to get it's parent(e.g. the variable name). For example, "us-west-2" value's variable name is "aws_region".
How would I do this?
With your sample, the invocation:
jq -r '
.variables
| [to_entries[]
| "\(.key)=\"\(.value.value|tostring)\""]
| "--variables " + join(" ")
' sample.json
produces:
--variables app_container_environment="dev" aws_region="us-west-2"

Is there a way to delete the same key from a list of objects within a nested field?

I'm setting up a devops pipeline so that certain data profiles stored in JSON format can be shifted across different servers. While downloading it from the current server I need to clean up all the protected keys and unique identifiers. I'm looking for the cleanest way to do the following in JQ
Input:
{
"TopKey1":{
"some_key":"some_value"
},
"TopKey2":{
"some_key2":"some_value2"
},
"KeytoSearch":[
{
"_id":"sdf",
"non_relevant_key1":"val"
},
{
"_id":"sdfdsdf",
"non_relevant_key2":"val"
},
{
"_id":"sgf",
"non_relevant_key3":"val"
}
]
}
Output:
{
"TopKey1":{
"some_key":"some_value"
},
"TopKey2":{
"some_key2":"some_value2"
},
"KeytoSearch":[
{
"non_relevant_key1":"val"
},
{
"non_relevant_key2":"val"
},
{
"non_relevant_key3":"val"
}
]
}
In python terms if this were a dictionary
for json_object in dictionary["KeytoSearch"]:
json_object.pop("_id")
I've tried combinations of map and del but can't seem to figure out the nested indexing with this. The error messages I get are along the lines of jq: error (at <stdin>:277): Cannot index string with string "_id" which sort of tells me I haven't fundamentally understood how jq works or is to be used, but this is the route I need to go because using a Python script to clean up JSON objects is something I'd rather avoid
Going with your input JSON and assuming there are other properties in your KeytoSearch object along with the _id fields, you could just do below.
jq 'del(.KeytoSearch[]._id)'
See this jqplay.org snippet for a demo. The quotes around the property key containing _ are not needed as confirmed in one of the comments below. Some meta-characters (e.g. . in the property key values needs be accessed with quotes as ".id") needs to be quoted properly, but _ is clearly not one of them.
I've tried combinations of map and del
Good! You were probably just missing the '|=' magic ingredient:
.Keytosearch |= map( del(._id) )
alternatively, you could use a walk-path unix tool for JSON: jtc and apply changes right into the sourse json file (-f):
bash $ jtc -fpw'[KeytoSearch]<_id>l:' file.json
bash $
bash $
bash $ jtc file.json
{
"KeytoSearch": [
{
"non_relevant_key1": "val"
},
{
"non_relevant_key2": "val"
},
{
"non_relevant_key3": "val"
}
],
"TopKey1": {
"some_key": "some_value"
},
"TopKey2": {
"some_key2": "some_value2"
}
}
bash $
if given json snippet is a part of a larger JSON (and [KeytoSearch] is not addressable from the root), then replace it with the search lexeme: <KeytoSearch>l.
PS> Disclosure: I'm the creator of the jtc tool

How to write jq script to extract elements that might appear as singleton or list?

How can one write a jq query that will extract a property from an element that may appear as singleton or list?
For example, extract the URL property from the creator in both example JSON strings below.
Example #1:
{
"#type": "example1",
"creator":{
"#type":"Organization",
"url": "https://www.ncei.noaa.gov/"
}
}
Example #2:
{
"#type": "example2",
"creator": [{
"#type":"Person",
"url": "https://www.example.com/homepage"
},
{
"url": "https://www.example.com/another"
}]
}
I have tried using .creator for the first one and .creator[] for the second one, but these two are not compatible. Is there a way to write so that it works for both examples?
One possibility that is very straightforward is simply to test whether .creator is an array or not:
if .creator|type == "array" then .creator[] else .creator end
| .url
Streaming parser
At the other end of the spectrum of straightforwardness, here is another possibility that would be relevant if (a) the input JSON document was ginormous, and (b) the goal is to list all .url values that occur as immediate children of .creator, no matter where the keys are located in the input JSON document. The invocation would use the --stream command-line option, e.g.
jq --stream -f filter.jq input.json
The jq filter in this case is:
select(length==2 and
(.[0][-1] == "url") and
last( .[0][:-1][]| strings ) == "creator")
| .[1]