I have two JSON objects and I am trying to build one object out of it. I am using an example here, but the properties/names are dynamic.
I have looked into JOIN function, MERGE, Combine-Object none of them seems to do what I want. As they either replace the values with the one or give me random results.
This is my example,
$aVar = '{ "oldEmployees" : [ { "firstName": "Jane", "lastName": "Doe" }, { "firstName": "John", "lastName": "Doe" } ] }'
$bVar = '{ "newEmployees" : [ { "firstName": "Joe", "lastName": "Doe" } ] }'
I would like to have one JSON as,
{
"oldEmployees": [
{
"firstName": "Jane",
"lastName": "Doe"
},
{
"firstName": "John",
"lastName": "Doe"
}
],
"newEmployees": [
{
"firstName": "Joe",
"lastName": "Doe"
}
]
}
BACKGROUND: I am using a JSON object to deploy multiple resources in Azure DevOps pipeline using an ARM template. I have hit a point where I am limited to use Powershell script to combine multiple variables due to the size limit on the Azure Pipeline Variables to a max of 4K characters.
This will give you the exact JSON that you seek:
$combinedObject = New-Object -TypeName PSObject -Property #{
oldEmployees = #($aVar | ConvertFrom-Json | Select-Object -ExpandProperty oldEmployees)
newEmployees = #($bVar | ConvertFrom-Json | Select-Object -ExpandProperty newEmployees)
} | ConvertTo-Json
It's the same approach as Panomosh's answer, except that instead of using an array, I used a property map. This prevents the output from having an array.
Simply pipe your definition to |Convertfrom-Json
$aVar = '{ "oldEmployees" : [ { "firstName": "Jane", "lastName": "Doe" }, { "firstName": "John", "lastName": "Doe" } ] }' |ConvertFrom-Json
$bVar = '{ "newEmployees" : [ { "firstName": "Joe", "lastName": "Doe" } ] }'|ConvertFrom-Json
$aVar,$bVar|ConvertTo-Json
The result is ugly formatted with PSv5 it get's better with core.
[
{
"oldEmployees": [
"#{firstName=Jane; lastName=Doe}",
"#{firstName=John; lastName=Doe}"
]
},
{
"newEmployees": [
"#{firstName=Joe; lastName=Doe}"
]
}
]
you are going to have to turn both of the JSON into powershell objects, combine them, and then turn them back into JSON. In the example above, powershell will just see them as strings.
You can do this like so:
$container = #()
$container += ConvertFrom-Json '{ "oldEmployees" : [ { "firstName": "Jane", "lastName": "Doe" }, { "firstName": "John", "lastName": "Doe" } ] }'
$container += ConvertFrom-Json '{ "newEmployees" : [ { "firstName": "Joe", "lastName": "Doe" } ] }'
$combinedObject = $container | ConvertTo-Json
now if you view $combinedObject you will see:
[
{
"newEmployees": [
"#{firstName=Joe; lastName=Doe}"
]
},
{
"oldEmployees": [
"#{firstName=Jane; lastName=Doe}",
"#{firstName=John; lastName=Doe}"
]
}
]
Related
I am trying to pass a parameter through a function in powershell but it is not working
Code
function test($test1, $test2)
{
$details = #"
{ "updateDetails": [
{
"customer": "John",
"rank": $test1
},
{
"school": "western",
"address": $test2
}
]
}
"#
return $details
}
test 0 florida
Current issue
{ "updateDetails": [
{
"customer": "John",
"rank":
},
{
"school": "western",
"address": florida
}
]
}
I tried running test but the value 0 is not filled in the details json, florida is filled in correctly. How can I replace the two values. Also how can florida be in string
0 fills in for me, but florida doesn't have quotes, which is invalid JSON.
To make life a little easier, instead building a here-string, consider building an object and converting it to JSON with the built-in cmdlet ConvertTo-Json.
In this example I'll show you how to do it using a hashtable
function test($test1, $test2)
{
$details = #{
"updateDetails"=
#(
#{
"customer" = "John"
"rank" = $test1
},
#{
"school" = "western"
"address" = $test2
}
)
}
return $details | ConvertTo-Json
}
test 0 florida
output
{
"updateDetails": [
{
"customer": "John",
"rank": 0
},
{
"school": "western",
"address": "florida"
}
]
}
Your code is perfectly fine. I ran your example and it worked as expected. Maybe you missed updating the function. Close the shell and then try again.
To include "florida" as string you could simply add quotes around the variable "$test2", or even safer: Use ConvertTo-Json to output a properly quoted and escaped JSON string:
function test {
param ([int]$rank, [string]$address)
return #"
{ "updateDetails": [
{
"customer": "John",
"rank": $rank
},
{
"school": "western",
"address": $(ConvertTo-Json $address)
}
]
}
"#
}
test 0 florida
Is there a way to create a new element in an existing json object using jq? Example below:
Let's say I have this json object and would like to add a new element to foo:
json='{
"id": "<id>>",
"name": "<name>",
"properties": {
"State": "<state>",
"requests": [],
"foo": [
{
"id": "<id1>",
"bar1": [
{
"baz1": "*"
}
]
},
{
"id": "<id2>",
"bar2": [
{
"baz2": "*"
}
]
}
]
}
}'
This command works to do that:
json2=$($json1 | jq '.properties.foo += [ { "id": "<id3>", "bar3": [ { "baz3": "*"} ] } ]')
However, running that same command without a preexisting foo element fails (example array below):
json3='{
"id": "<id>>",
"name": "<name>",
"properties": {
"State": "<state>",
"requests": []
}
}'
Is there a way in jq to create that element in the json object if one already does not exist?
Thanks!
There is nothing wrong with your jq program, which can be seen by running:
jq '.properties.foo += [ { "id": "<id3>", "bar3": [ { "baz3": "*"} ] } ]' <<< "$json3"
It looks like the problem is with your invocation but since it's not clear what $json1 is, I'll just guess that the above is sufficient for you to resolve the issue.
I have below JSON where in I need to add key and values at a specific place from env variable. Moreover would try to avoid any hardcoding in JSON if possible
Sample JSON:
{
"abc": {
"admin": {
"username": "admin",
"password": "pa55word",
"tenant_name": "t1"
},
"users": [
{
"username": "n1",
"password": "password",
"user_domain_name": "Default",
"project_name": "p1",
"project_domain_name": "Default"
},
{
"username": "n2",
"password": "password",
"user_domain_name": "Default",
"project_name": "p2",
"project_domain_name": "Default"
}
]
}
}
Output json needs to be like below where in $AUTH and $REG are env variables.
{
"abc": {
"auth_url": "$AUTH",
"region_name": "$REG",
"endpoint_type": "PRIVATE",
"admin": {
"username": "admin",
"password": "pa55word",
"tenant_name": "t1"
},
"users": [
{
"username": "n1",
"password": "password",
"user_domain_name": "Default",
"project_name": "p1",
"project_domain_name": "Default"
},
{
"username": "n2",
"password": "password",
"user_domain_name": "Default",
"project_name": "p2",
"project_domain_name": "Default"
}
]
}
}
Looked into jq but it seemed confusing. How this can be achieved using jq?
Grab AUTH and REG environment variables from $ENV preset variable and create fields using them, then add the resulting object to the first member of the input. E.g:
first(.[]) |= ( $ENV | {
auth_url: .AUTH,
region_name: .REG,
endpoint_type: "PRIVATE"
}
) + .
The simplest approach would be to take advantage of the fact that + on objects respects the ordering of keys:
.abc |= {auth_url: env.AUTH, region_name: env.REG, endpoint_type: "PRIVATE"} + .
This assumes that AUTH and REG are environment variables. If they are not, then you could use the --arg and/or --argjson command-line options, e.g.
jq --arg AUTH "$AUTH" --arg REG "$REG" '
.abc |= {auth_url: $AUTH, region_name: $REG, endpoint_type: "PRIVATE"} + .
' sample.json
I have an existing json which have a from like this:
{
"arg1": "Admin",
"arg2": 0,
"data": [
{
"arg3": "11",
"user": "user1",
"age": 51,
"arg4": "11"
},
{
"arg3": "22",
"user": "user2",
"age": 52,
"arg4": "22"
},
{
"arg3": "33",
"user": "user3",
"age": 53,
"arg4": "33"
},
{
"arg3": "44",
"user": "user4",
"age": 54,
"arg4": "44"
}
]
}
Then I try this command:
$ cat tmp.json|jq '.data|.[]|{user,age}'
{
"user": "user1",
"age": 51,
}
{
"user": "user2",
"age": 52,
}
{
"user": "user3",
"age": 53,
}
{
"user": "user4",
"age": 54,
}
What I expect to output is:
{
"department": "Admin",
"user_age": {
"user1": "51",
"user2": "52",
"user3": "53",
"user4": "54"
},
"year": 2016
}
In jq's manual, my request is close to example 23.
So I tried to use from_entries function
cat tmp.json|jq '.data|.[]|{user,age}|from_entries'
but get this error:
jq: error (at :30): Cannot index string with string "key"
I know this is because of its format not equal an entry.
So, what should I do to convert to the expected output?
from_entries expects an array of objects with key and value properties. You are however generating separate objects with user and age properties which just would not work. First of all, you would need to make sure that the properties are in an array, then have the separate key/value pairs for each of the values you want in the array.
With that said, you don't really need to use entries to build out the result, it's not the right tool for the job. You just want to pull some properties from each of the data objects and add to another object mapping users to ages. There's many ways this can be achieved, I would use reduce to accomplish this.
reduce .data[] as $d ({}; .[$d.user] = $d.age)
To get your final result, just combine the parts you need into the object you're requiring. I'm not sure where you got that 2016 from, I'm assuming it's just hardcoded.
{department: .arg1, user_age: (reduce .data[] as $d ({}; .[$d.user] = $d.age)), year: 2016}
you can also try:
cat tmp.json|jq '{department: .arg1, user_age: (.data|map({(.user): .age})|add), year: 2016}'
or
cat tmp.json|jq '{department: .arg1, user_age: .data|map({(.user): .age})|add, year: 2016}'
As Jeff explained, from_entries expects input of the form
[ {key:xxx, value:yyy} ]
so Hao's filter
.data | .[] | {user,age} | from_entries
requires two small modifications to generate the desired user_age object:
the values must be {key,value} objects
the values need to be collected into an array
e.g.
.data | [.[]|{key:user, value:age|tostring}] | from_entries
(we include tostring because the values in the example output are strings)
But since [ .[]| ... ] is the same as map(...) this could be written
.data | map({key:user, value:age|tostring}) | from_entries
Here is a complete solution:
{
"department": .arg1,
"user_age": .data | map({key:.user, value:.age|tostring}) | from_entries,
"year": 2016
}
Sample output
{
"department": "Admin",
"user_age": {
"user1": "51",
"user2": "52",
"user3": "53",
"user4": "54"
},
"year": 2016
}
I have below sample nested json response and I need to convert this response with specific values into CSV file. Below is the sample nested Json response:
{
"transaction": {
"id": "TestTransID",
"testCode": "NEW",
"TestStatus": "SUCCESS",
"client": {
"TestNumber": "112112111"
},
"subject": {
"individual": {
"additionalAttributes": {
"extraid": "787877878"
},
"addressList": [
{
"city": "New York",
"country": {
"name": "United States"
},
"postalCode": "123456789",
"stateOrProvince": {
"codeValue": "NY"
}
}
],
"gender": "F",
"identificationDocumentList": [
{
"number": "1214558520",
"type": "TestId"
}
],
"name": {
"firstName": "Qusay TestFull",
"lastName": "TestLast",
"middleName": "Middle 3"
}
}
},
"PROCESSConfiguration": {
"id": 1
},
"testProductList": [
{
"product": {
"id": 00,
"name": "Test PROCESS",
"productCode": "EFG",
"disclaimer": "TestDisclaimer"
},
"testSourceResponseList": [
{
"testSource": {
"id": 1,
"name": "TEST"
},
"testSourceRecordList": [
{
"type": "TestRecord",
"alertReasonCode": "TESTS",
"alertReasonDescription": "ACTION LIST HIT - TEST",
"testSource": "TEST",
"varListNameFull": "TEST FULL NAME",
"varListNameShort": "TEST SHORT",
"varProgList": [
"SHORT"
],
"varListId": "3421",
"subject": {
"individual": {
"TestScore": {
"TestScore": 100,
"triggeredRule": "TestRule"
},
"aNameList": [
{
"fullName": " TestNameA",
"lastName": "TestNameA"
},
{
"firstName": "TestFirst",
"fullName": "TestFirst HUSAYN",
"lastName": "TestLast"
},
{
"firstName": "TestFirst",
"fullName": "TestFull",
"lastName": "TestLast"
},
{
"firstName": "TestFirst",
"fullName": "TestFull",
"lastName": "TestLast"
}
],
"birthList": [
{
"dateOfBirth": "12 Apr 1910",
"dateOfBirthVerified": "true"
}
],
"name": {
"firstName": "TestFirst",
"fullName": "TestFull",
"lastName": "TestLast"
},
"varNationality": [
{
"verified": "true"
}
],
"remarks": "remark1"
}
}
},
{
"testSource": "TEST",
"varListNameFull": "TEST FULL",
"varListNameShort": "TEST SHORT",
"varProgList": [
"XYZ"
],
"varListId": "1234",
"subject": {
"individual": {
"overallScore": {
"TestScore": 100,
"triggeredRule": "Testing"
},
"birthList": [
{
"dateOfBirth": "1965",
},
{
"dateOfBirth": "1966",
}
],
"name": {
"firstName": "TestFirst",
"fullName": "TestFull",
"lastName": "TestLast",
},
"varNationality": [
{
"verified": "true"
}
],
"remarks": "REMARK2"
}
}
}
]
}
]
}
],
}
}
I need to take response from ""PROCESSConfiguration": {
"id": 1"
from row # 40. If u'll take above code in notepad ++.
Also I need the response with respect to var value like first name, last name full name, DOB etc.
I am still unsure what is being asked for. Let me assume that you want a fully qualified path for each of the elements in the JSON file.
I started by making some small adjustments to the JSON based on http://jsonlint.com/.
Based on that, I did a proof of concept that works for ONE OBJECT like the JSON you posted. It works for THIS CASE. I wrapped the logic to deal with multiple objects making assumptions about when one object started and the next began. Also, logic would should be added to deal with nested series/array containing multiple properties (i.e. Threads in Get-Process) to handle a generic case. A single element with an array of values (like .transaction.varProgList in this case) is handled by putting a ‘;’ between them.
CSV files assume that all the objects are symmetric (have the same properties). There is no check to see that the properties for each object align with the properties of the other objects. Note the handling of nested series is related to this. You may see an example of this by uncommenting the [System.Collections.ICollection] section and trying something like (Get-Process r*) | Select-Object Name,Threads | Expand-NestedProperty | Out-File .\t.csv -Width 100000
The repro goes as follows where $a is the adjusted JSON content and the function is saved as Expand-NestedProperty.ps1.
# Load the function
. .\Expand-NestedProperty.ps1
# Create PowerShell object(s) based on the JSON
$b = $a | ConvertFrom-Json
# Create a file with the CSV contents.
$b | Expand-NestedProperty | Out-File -FilePath .\my.csv -Width 100000
Save this as Expand-NestedProperty.ps1
function Expand-NestedProperty {
[CmdletBinding()]
param (
[Parameter( Position=0,
Mandatory=$true,
ValueFromPipeline=$true,
ValueFromPipelineByPropertyName=$true,
ValueFromRemainingArguments=$false,
HelpMessage='Object required...' )]
$InputObject
)
begin {
function ExpandNestedProperty {
[CmdletBinding()]
param (
[Parameter( Position=0,
Mandatory=$true,
ValueFromPipeline=$true,
ValueFromPipelineByPropertyName=$true,
ValueFromRemainingArguments=$false,
HelpMessage='Object required...' )]
$InputObject,
[Parameter( Position=1,
Mandatory=$false,
ValueFromPipeline=$false,
ValueFromPipelineByPropertyName=$true,
ValueFromRemainingArguments=$false,
HelpMessage='String required...' )]
[string]
$FullyQualifiedName = ""
)
begin {
$localResults =#()
$FQN = $FullyQualifiedName
$nestedProperties = $null
}
process {
foreach ($obj in $InputObject.psobject.Properties) {
if ($(try {$obj.Value[0] -is [PSCustomObject]} catch {$false})) { # Catch 'Cannot index into a null array' for null values
# Nested properties
$FQN = "$($FullyQualifiedName).$($obj.Name)"
$nestedProperties = $obj.value | ExpandNestedProperty -FullyQualifiedName $FQN
}
elseif ($obj.Value -is [array]) {
# Array property
$FQN = "$($FullyQualifiedName).$($obj.Name)"
[psobject]$nestedProperties = #{
$FQN = ($obj.Value -join ';')
}
}
# Example of how to deal with generic case.
# This needed for the Get-Process values ([System.Collections.ReadOnlyCollectionBase] and [System.Diagnostics.FileVersionInfo]) that are not [array] collection type.
<#
elseif ($obj.Value -is [System.Collections.ICollection]) {
# Nested properties
$FQN = "$($FullyQualifiedName).$($obj.Name)"
$nestedProperties = $obj.value | ExpandNestedProperty -FullyQualifiedName $FQN
}
#>
else { # ($obj -is [PSNoteProperty]) for this case, but could be any type
$FQN = "$($FullyQualifiedName).$($obj.Name)"
[psobject]$nestedProperties = #{
$FQN = $obj.Value
}
}
$localResults += $nestedProperties
} #foreach $obj
}
end {
[pscustomobject]$localResults
}
} # function ExpandNestedProperty
$objectNumber = 0
$firstObject = #()
$otherObjects = #()
}
process {
if ($objectNumber -eq 0) {
$objectNumber++
$firstObject = $InputObject[0] | ExpandNestedProperty
}
else {
if ($InputObject -is [array]) {
foreach ($nextInputObject in $InputObject[1..-1]) {
$objectNumber++
$otherObjects += ,($nextInputObject | ExpandNestedProperty)
}
}
else {
$objectNumber++
$otherObjects += ,($InputObject | ExpandNestedProperty)
}
}
}
end {
# Output CSV header and a line for each object which was the specific requirement here.
# Could create an array of objects using $firstObject + $otherObjects that is then piped to Export-CSV if we want a generic case.
Write-Output "`"$($firstObject.Keys -join '","')`""
Write-Output "`"$($firstObject.Values -join '","')`""
foreach ($otherObject in $otherObjects) {
Write-Output "`"$($otherObject.Values -join '","')`""
}
}
} # function Expand-NestedProperty