I have a question about convert from/to json with powershell. I need a json with a specific format (for a REST Call) but if I do an convertfrom-json and convertto-json the format is wrong and I don't know how I can change it.
This is the original format:
{
"property-content": [
{
"statKey": "Test|Test1",
"timestamps": [1522678260000],
"values": ["compliant"],
"others": "",
"otherAttributes": ""
}
]
}
and if I do an converfrom-json ... then I change some values and then I do an convertto-json the format is without the brackets:
{
"property-content": [
{
"statKey": "Test|Test1",
"timestamps": "1522678260000",
"values": "compliant",
"others": "",
"otherAttributes": ""
}
]
}
Thanks!
The arrays are flattened by the ConvertTo-Json cmdlet because of the -Depth parameter which is set to 2 by default. If you higher the -Depth it will resolve your issue:
PS C:\> $a = ConvertFrom-JSON #'
>> {
>> "property-content": [
>> {
>> "statKey": "Test|Test1",
>> "timestamps": [1522678260000],
>> "values": ["compliant"],
>> "others": "",
>> "otherAttributes": ""
>> }
>> ]
>> }
>> '#
PS C:\> $a | convertTo-JSON -Depth 5
{
"property-content": [
{
"statKey": "Test|Test1",
"timestamps": [
1522678260000
],
"values": [
"compliant"
],
"others": "",
"otherAttributes": ""
}
]
}
Related
I have a text file with below data which is Dynamically generated, so the contents of the repo.txt keep changing
repo.txt -> Content as below
repo1
repo2
repo_3
And a JSON file
template.json -> Content as below
{
"name": "new-report-test-1",
"resources": {
"repositories": [
{
"name": "repoa"
},
{
"name": "repo_b"
}
]
},
"filters": {
"severities": [
"High"
]
}
}
now i want to read the repo list from the repo.txt file and update the template.json to below
{
"name": "new-report-test-1",
"resources": {
"repositories": [
{
"name": "repoa"
},
{
"name": "repo_b"
},
{
"name": "repo1"
},
{
"name": "repo2"
},
{
"name": "repo_3"
}
]
},
"filters": {
"severities": [
"High",
"Medium",
"Critical"
]
}
}
I have this bash script that does the job thiugh I'm unsure how to translate it to PoweShell. Any inputs would be highly appreciated
cat ./tmpgen_reponames.txt | while read repoEntry do echo " Repository: $repoEntry" cat <<< $(jq --arg REPOID "$repoEntry" '[ .[] , {"name":$REPOID} ]' tmpgen_repoarray.json) > tmpgen_repoarray.json done
You can accomplish this in PowerShell using ConvertFrom-Json to convert your Json into objects then following an object-oriented approach adding new custom objects to the .resources.repositories property and adding the new values Medium and Critical to the .filters.severities property of your Json, lastly, converting it back to a Json string with ConvertTo-Json:
$json = Get-Content path\to\template.json -Raw | ConvertFrom-Json
$json.resources.repositories = #(
$json.resources.repositories
(Get-Content path\to\repo.txt).Trim() | ForEach-Object {
[pscustomobject]#{ Name = $_ }
}
)
$json.filters.severities = #(
$json.filters.severities
'Medium'
'Critical'
)
$json | ConvertTo-Json -Depth 99 | Set-Content path\to\newJson.Json
I have a big target json file (parameters_general.json) where all common settings for a deployment are set.
For each tier I have another json file (ex: parameters_dev.json, parameters_test.json, ....) Settings set in one needs tobe added to the general.json, or overwrite it when already in general.
ex: parameters_general.json
{
"general": {
"db_Policy": {
"type": "Periodic",
"databaseAccountOfferType": "Standard",
"periodicModeProperties": {
"backupIntervalInMinutes": 240,
"backupRetentionIntervalInHours": 8,
"backupStorageRedundancy": "Local"
}
},
"databases": [
{
"name": "CtrlWps",
"Containers": [
{
"name": "ControllerAuthentication",
"partitionKey": "id"
}
],
"ContainersTTL": []
},
{
"name": "CpoOcpi",
"Containers": [
{
"name": "Cpos",
"partitionKey": "cpoId"
},
{
"name": "OcpiCdrLastRecoveries",
"partitionKey": "id"
},
{
"name": "Routes",
"partitionKey": "ocpiCpoId"
}
],
"ContainersTTL": [
{
"name": "OcpiCdrs",
"partitionKey": "pk",
"ttl": 172800
},
{
"name": "OcppTransactionIds",
"partitionKey": "pk",
"ttl": 172800
},
{
"name": "Sessions",
"partitionKey": "pk",
"ttl": 172800
}
]
}
],
"system_engineers": [
],
}
}
If I want to updatethis with.
ex: parameters_test.json
{
"general": {
"system_engineers": [
{
"name": "hans",
"AppPrincipalId": "<id>",
"permissions": [
"get",
"list"
]
},
{
"name": "John do",
"AppPrincipalId": "<pid>",
"permissions": [
"all"
]
}
]
}
}
This works, the users are added to the empty "sytem_engineers" node in the parameters_general.json.
However, If I just want to change a setting on a lower node example:
ex: parameters_dev.json
{
"general": {
"databases": [
{
"name": "CtrlWps",
"Containers": [
{
"name": "ControllerAuthentication",
"partitionKey": "pk"
}
]
}
]
}
}
for replacing the partitionKey in one of the databases it replaces the whole "databases" part so I lose all other database configurations in the target.
The code I use is the following.
function ExtendJSON($base, $ext)
{
$propNames = $($ext | Get-Member -MemberType *Property).Name
foreach ($propName in $propNames) {
if ($base.PSObject.Properties.Match($propName).Count) {
if ($base.$propName.GetType().Name -eq "PSCustomObject")
{
$base.$propName = ExtendJSON $base.$propName $ext.$propName
}
else
{
$base.$propName = $ext.$propName
}
}
else
{
$base | Add-Member -MemberType NoteProperty -Name $propName -Value $ext.$propName
}
}
return $base
}
$tier = 'dev'
$parametersJsonGeneral = Get-Content -Path "./parameters/parameters_general.json" | ConvertFrom-Json
#Write-Output "#####################"
$parametersJsonTier = Get-Content -Path "./parameters/parameters_$($tier).json" | ConvertFrom-Json # overwrites existing values in $parametersJsonGeneral
ExtendJSON $parametersJsonGeneral $parametersJsonTier
Is there a way to loop over the settings from the lowest level up to the higher, and replace only these?
The proposed answer does only work one 1st level THIS WORKS
function merger ($target, $source) {
$source.psobject.Properties | ForEach-Object {
if ($_.TypeNameOfValue -eq 'System.Management.Automation.PSCustomObject' -and $target."$($_.Name)" ) {
merger $target."$($_.Name)" $_.Value
}
else {
$target | Add-Member -MemberType $_.MemberType -Name $_.Name -Value $_.Value -Force
}
}
}
$Json1 ='{
"a": {
"b":"asda"
},
"c": "asdasd"
}
' | ConvertFrom-Json
$Json2 = '{
"a": {
"b":"d"
}
}
' | ConvertFrom-Json
merger $Json1 $Json2
However with this I loose data in $Json1
$Json1 ='{
"a": {
"b": [
{
"name": "admin",
"appconfig": {
"test1": true,
"test2": false
}
}
]
},
"c": "asdasd"
}
' | ConvertFrom-Json
$Json2 = '{
"a": {
"b": [
{
"name": "admin",
"appconfig": {
"test1": false
}
}
]
}
}
' | ConvertFrom-Json
merger $Json1 $Json2
{
"a": {
"b": [
{
"name": "admin",
"appconfig": {
"test1": false
}
}
]
},
"c": "asdasd"
}
test2 is gone!
I don't have enough rep to post general comments yet, so I'll have to post it as an answer:
I tested the code in this answer to another question, and it did what you asked. The advantage of this answer is that it doesn't depend on importing a separate module.
There's another answer in the same question that provides link to a more general module that provides for different types of merges (Left Join, Inner Join, etc.) here, which links to here. However, one commenter said that it didn't work with powershell 7.0.
Below function is giving me required result except the numbers (0,1,2,0) populating before JSON object beginning. Can someone help me to get rid of those numbers.
$V1='11' $V2='1881' $V3='111','222','333'
$body=CreateJsonBody $V1 $V2 $V3
function CreateJsonBody{
param($objectId,$AttributeId,$AttributeValues)
$jsonBase=#{"objectId"= $objectId;}
$vlist = New-Object System.Collections.ArrayList
$alist = New-Object System.Collections.ArrayList
foreach ($Val in $AttributeValues) {
$vlist.Add(#{"value"= $Val;})
}
$alist.Add(#{"AttributeId"= $AttributeId;"AttributeValues"=$vlist;})
$jsonBase.Add("attributes",$alist)
return $jsonBase | ConvertTo-Json -Depth 10
}
Output:
PS C:\WINDOWS\system32> $body
0
1
2
0
{
"objectId": "105",
"attributes": [
{
"AttributeId": "1887",
"AttributeValues": [
{
"value": "111"
},
{
"value": "222"
},
{
"value": "333"
}
]
}
]
}
I'm trying to convert a text file that has an array of json objects to an NDJSON formatted file for another team to consume.
I've almost got it, except for one problem. I have an array of objects nested inside the JSON (which then has nested arrays and objects inside of it, the structure gets pretty complex, I'll include a sample below) and for whatever reason, when I use ConvertFrom-JSON it drops this nested array and in my output, I end up with a blank string for that key, instead of the nested array object. I tried using the -Depth flag but when I do that my output file ends up blank, which doesn't make a ton of sense to me. I don't have a whole lot of experience with powershell, so I'm not really sure where I'm going wrong here.
Code:
$JSONSourceFile = Get-Content -Path "input/sample.json" | ConvertFrom-JSON
$NDJSONTargetFile = "output/sample.json"
New-Item $NDJSONTargetFile -ItemType file
for ( $i = 0 ; $i -lt $JSONSourceFile.Length ; $i++) {
$item = $JSONSourceFile.item($i)
$row = ($item | ConvertTo-JSON -Compress)
Add-Content $NDJSONTargetFile $row
}
Input File:
[
{
"id": "1",
"TransactionDttm": "2021-02-22T15:45:00:00.000-05:00",
"Array1": [
{
"UserID": "ak2354",
"Preferences": [
{
"CagegoryID": "01",
"CategoryName": "Reminder",
"Channels": [
{
"ChannelID": "1",
"ChannelName": "Email",
"Preference": "Y"
},
{
"ChannelID": "2",
"ChannelName": "Text",
"Preference": "N"
}
]
}
]
}
]
},
{
"id": "2",
"TransactionDttm": "2021-02-22T15:45:00:00.000-05:00",
"Array1": [
{
"UserID": "ak1234",
"Preferences": [
{
"CagegoryID": "01",
"CategoryName": "Reminder",
"Channels": [
{
"ChannelID": "1",
"ChannelName": "Email",
"Preference": "Y"
},
{
"ChannelID": "2",
"ChannelName": "Text",
"Preference": "Y"
}
]
}
]
}
]
},
{
"id": "3",
"TransactionDttm": "2021-02-22T15:45:00:00.000-05:00",
"Array1": [
{
"UserID": "ak5678",
"Preferences": [
{
"CagegoryID": "01",
"CategoryName": "Reminder",
"Channels": [
{
"ChannelID": "1",
"ChannelName": "Email",
"Preference": "N"
},
{
"ChannelID": "2",
"ChannelName": "Text",
"Preference": "N"
}
]
}
]
}
]
}
]
And then when I convert it to the output, this is what I get:
{"id":"1","TransactionDttm":"2021-02-22T15:45:00:00.000-05:00","Array1":[{"UserID":"ak2354","Preferences":""}]}
{"id":"2","TransactionDttm":"2021-02-22T15:45:00:00.000-05:00","Array1":[{"UserID":"ak1234","Preferences":""}]}
{"id":"3","TransactionDttm":"2021-02-22T15:45:00:00.000-05:00","Array1":[{"UserID":"ak5678","Preferences":""}]}
Thanks to the comment from Doug Maurer I figured it out, I was adding -Depth to my ConvertFrom-Json command when it should have been in my ConvertTo-Json command. This is the final script and what it gives me:
$JSONSourceFile = Get-Content -Path "input/sample.json" | ConvertFrom-JSON
$NDJSONTargetFile = "output/sample.json"
New-Item $NDJSONTargetFile -ItemType file
for ( $i = 0 ; $i -lt $JSONSourceFile.Length ; $i++) {
$item = $JSONSourceFile.item($i)
$row = ($item | ConvertTo-JSON -Compress -Depth 20)
Add-Content $NDJSONTargetFile $row
}
and the output:
{"id":"1","TransactionDttm":"2021-02-22T15:45:00:00.000-05:00","Array1":[{"UserID":"ak2354","Preferences":[{"CagegoryID":"01","CategoryName":"Reminder","Channels":[{"ChannelID":"1","ChannelName":"Email","Preference":"Y"},{"ChannelID":"2","ChannelName":"Text","Preference":"N"}]}]}]}
{"id":"2","TransactionDttm":"2021-02-22T15:45:00:00.000-05:00","Array1":[{"UserID":"ak1234","Preferences":[{"CagegoryID":"01","CategoryName":"Reminder","Channels":[{"ChannelID":"1","ChannelName":"Email","Preference":"Y"},{"ChannelID":"2","ChannelName":"Text","Preference":"Y"}]}]}]}
{"id":"3","TransactionDttm":"2021-02-22T15:45:00:00.000-05:00","Array1":[{"UserID":"ak5678","Preferences":[{"CagegoryID":"01","CategoryName":"Reminder","Channels":[{"ChannelID":"1","ChannelName":"Email","Preference":"N"},{"ChannelID":"2","ChannelName":"Text","Preference":"N"}]}]}]}
I have two file json. I want to append two array of SomeFile2.json to SomeFile1.json as below.
SomeFile1.json
[
{
"DNSName": "CLB-test-112a877451.ap-northeast-1.elb.amazonaws.com",
"Instances": [
{
"InstanceId": "i-0886ed703de64028a"
}
]
},
{
"DNSName": "CLB-test1-156925981.ap-northeast-1.elb.amazonaws.com",
"Instances": [
{
"InstanceId": "i-0561634c4g3b4fa25"
}
]
}
]
SomeFile2.json
[
{
"InstanceId": "i-0886ed703de64028a",
"State": "InService"
},
{
"InstanceId": "i-0561634c4g3b4fa25",
"State": "InService"
}
]
I want the result as below:
[
{
"DNSName": "CLB-test-112a877451.ap-northeast-1.elb.amazonaws.com",
"Instances": [
{
"InstanceId": "i-0886ed703de64028a"
"State": "InService"
}
]
},
{
"DNSName": "CLB-test1-156925981.ap-northeast-1.elb.amazonaws.com",
"Instances": [
{
"InstanceId": "i-0561634c4g3b4fa25"
"State": "InService"
}
]
}
]
I'm processing in bash shell via jq. But, unsuccessful.
Since the contents of the second file are evidently intended to define a mapping from InstanceId to State, let's start by hypothesizing the following invocation of jq:
jq --argfile dict SomeFile2.json -f program.jq SomeFile1.json
Next, let's create a suitable dictionary:
reduce $dict[] as $x ({}; . + ($x|{(.InstanceId): .State}))) as $d
Now the rest is easy:
map(.Instances |= map(. + {State: $d[.InstanceId]}))
Putting the pieces together in program.jq:
(reduce $dict[] as $x ({}; . + ($x|{(.InstanceId): .State}))) as $d
| map(.Instances |= map(. + {State: $d[.InstanceId]}))
Alternatives
The dictionary as above can be constructed without using reduce, as follows:
($dict | map( {(.InstanceId): .State}) | add) as $d
Another alternative is to use INDEX/2:
(INDEX($dict[]; .InstanceId) | map_values(.State))) as $d
If your jq does not have INDEX/2 you can snarf its def from
https://raw.githubusercontent.com/stedolan/jq/master/src/builtin.jq
Since I find jq pretty hard, I started in a procedural way: using ruby's json module:
ruby -rjson -e '
states = JSON.parse(File.read(ARGV.shift)).map {|o| [o["InstanceId"], o["State"]]}.to_h
data = JSON.parse(File.read(ARGV.shift))
data.each do |obj|
obj["Instances"].each do |instance|
instance["State"] = states[instance["InstanceId"]] || "unknown"
end
end
puts JSON.pretty_generate data
' SomeFile2.json SomeFile1.json
But we want jq, so after some trial and error, and finding this in the manual: https://stedolan.github.io/jq/manual/#Complexassignments -- (note, I changed the state for one of the instances so I could verify the output better)
$ cat SomeFile2.json
[
{
"InstanceId": "i-0886ed703de64028a",
"State": "InService"
},
{
"InstanceId": "i-0561634c4g3b4fa25",
"State": "NOTInService"
}
]
First, extract the states into an object mapping the id to the state:
$ state_map=$( jq -c 'map({"key":.InstanceId, "value":.State}) | from_entries' SomeFile2.json )
$ echo "$state_map"
{"i-0886ed703de64028a":"InService","i-0561634c4g3b4fa25":"NOTInService"}
Then, update the instances in the first file:
jq --argjson states "$state_map" '.[].Instances[] |= . + {"State": ($states[.InstanceId] // "unknown")}' SomeFile1.json
[
{
"DNSName": "CLB-test-112a877451.ap-northeast-1.elb.amazonaws.com",
"Instances": [
{
"InstanceId": "i-0886ed703de64028a",
"State": "InService"
}
]
},
{
"DNSName": "CLB-test1-156925981.ap-northeast-1.elb.amazonaws.com",
"Instances": [
{
"InstanceId": "i-0561634c4g3b4fa25",
"State": "NOTInService"
}
]
}
]