Azure pipelines -Add new element in json array VSTS pipelines (Appsettings.json) - json

Is this possible to add a new element in an array of appsetting.json in Azure Release Pipeline?
In appsetting.json I have array variable which I need to fill with another element during deployment through Azure Pipeline.
"Array": [
{
"Name": "AD1",
"IsDefault": "true",
"IdPEntityId": "URL1",
"Metadata": "XMLpath1"
},
{
"Name": "AD2",
"IsDefault": "false",
"IdPEntityId": "URL2",
"Metadata": "XMLPath2"
}
]
Here in the above JSON array I need to add another one elemental last position (array-Index:2).

[CmdletBinding()]
param(
[string] $AdName,
[bool] $AdIsDefault,
[string] $AdIdPEntityId,
[string] $AdMetadata,
[string] $AppSettingFilePath
)
clear-Host
Write-Host 'Updating appsettings.json...' -ForegroundColor Yellow
function Format-Json([Parameter(Mandatory, ValueFromPipeline)][String] $json) {
$indent = 0;
($json -Split '\n' |
% {
if ($_ -match '[\}\]]') {
# This line contains ] or }, decrement the indentation level
$indent--
}
$line = (' ' * $indent * 2) + $_.TrimStart().Replace(': ', ': ')
if ($_ -match '[\{\[]') {
# This line contains [ or {, increment the indentation level
$indent++
}
$line
}) -Join "`n"
}
$JsonDataAdd=#"
{
"Name":"$AdName",
"IsDefault": "$AdIsDefault",
"IdPEntityId":"$AdIdPEntityId",
"Metadata": "$AdMetadata"
}
"#
Write-Host ' Active directory details :' -ForegroundColor Yellow
Write-Host `n $JsonDataAdd -ForegroundColor Green
$jsonData = Get-Content "$AppSettingFilePath" | Out-String | ConvertFrom-Json -ErrorAction Stop
$jsonData.IdentitySettings.ExternalProviders.Saml2Providers += (ConvertFrom-Json $JsonDataAdd)
$jsonData | ConvertTo-Json -Depth 10 | Format-Json | Set-Content "$AppSettingFilePath" -Encoding UTF8
Write-Host 'Successfully Updated -appSettings.json !' -ForegroundColor Yellow

You could use JSON variable substitution. This feature substitutes values in the JSON configuration files. It overrides the values in the specified JSON configuration files (for example, appsettings.json) with the values matching names of release pipeline and stage variables.
When in "Deploy Azure App Service" release task you should see a "File Transforms and Variable Substitution" section. In here you will supply the path to the json file you want to swap variable values.
[![enter image description here][1]][1]
Then you just need to define the required substitution values in release pipeline or stage variables. From here you can add the json property you want to modify as a variable.
[![enter image description here][2]][2]
Finally after the transformation, the JSON will contain new. Azure DevOps will then swap out these values for you when deploying.
More details you could refer our official tutorial here: [File transforms and variable substitution reference][3]
Update:
It only works to adjust existing entries in the appsettings.json files, it doesn't seem to be able to add any new one. You could also take a look at the JSON variable substitution notes
Variable substitution is applied for only the JSON keys predefined in
the object hierarchy. It does not create new keys.
As a workaround, you could choose to use the File Creator extension:https://marketplace.visualstudio.com/items?itemName=eliostruyf.build-task to push the whole new appsettings.json file in the pipeline.
Update2
OP finally moved with PS script written by him to add new elements in Arrays of Appsettings.json

Related

I want to create json file by substituting values from environment variables in a json template file

One requirement of mine is - Using windows, not use any tools not already available as part of aws cli or windows
For example, I have this json file test.json with below content:
"My number is $myvar"
I read this into a powershell variable like so:
$myobj=(get-content .\test.json | convertfrom-json)
$myvar=1
From here, I would like to do something with this $myobj which will enable me to get this output:
$myobj | tee json_with_values_from_environment.json
My number is 1
I got some limited success with iex, but not sure if it can be made to work for this example
You can use $ExecutionContext.InvokeCommand.ExpandString()
$myobj = '{test: "My number is $myvar"}' | ConvertFrom-Json
$myvar = 1
$ExecutionContext.InvokeCommand.ExpandString($myobj.test)
Output
My number is 1
Here is one way to do it using the Parser to find all VariableExpressionAst and replace them with the values in your session.
Given the following test.json:
{
"test1": "My number is $myvar",
"test2": {
"somevalue": "$env:myothervar",
"someothervalue": "$anothervar !!"
}
}
We want to find and replace $myvar, $myothervar and $anothervar with their corresponding values defined in the current session, so the code looks like this (note that we do the replacement before converting the Json string into an object, this way is much easier):
using namespace System.Management.Automation.Language
$isCore7 = $PSVersionTable.PSVersion -ge '7.2'
# Define the variables here
$myvar = 10
$env:myothervar = 'hello'
$anothervar = 'world'
# Read the Json
$json = Get-Content .\test.json -Raw
# Now parse it
$ast = [Parser]::ParseInput($json, [ref] $null, [ref] $null)
# Find all variables in it, and enumerate them
$ast.FindAll({ $args[0] -is [VariableExpressionAst] }, $true) |
Sort-Object { $_.Extent.Text } -Unique | ForEach-Object {
# now replace the text with the actual value
if($isCore7) {
# in PowerShell Core is very easy
$json = $json.Replace($_.Extent.Text, $_.SafeGetValue($true))
return
}
# in Windows PowerShell not so much
$varText = $_.Extent.Text
$varPath = $_.VariablePath
# find the value of the var (here we use the path)
$value = $ExecutionContext.SessionState.PSVariable.GetValue($varPath.UserPath)
if($varPath.IsDriveQualified) {
$value = $ExecutionContext.SessionState.InvokeProvider.Item.Get($varPath.UserPath).Value
}
# now replace the text with the actual value
$json = $json.Replace($varText, $value)
}
# now we can safely convert the string to an object
$json | ConvertFrom-Json
If we were to convert it back to Json to see the result:
{
"test1": "My number is 10",
"test2": {
"somevalue": "hello",
"someothervalue": "world !!"
}
}

How can we create dynamic json object in powershell

I am creating powershell script which generation json file where i want to create json object runtime based on mentioned name in $env variable and put some value inside it as json.
$env="prd,test,dev,..,...." # there are total 15 env & $env has 1 and more env.
Desire JSON :
{
"prd" : {
"Sid": "xxxxxx" }
"test" : {
"Sid": "xxx" }
"So on" : {}
}
Trying to write script:
$env="prd,test,..,...,.."
$env=$env - split ","
For ( $i = 0 $i -le ($env.length -1) ;$i+=1){
$env[$i] = New-object System.Collection.ArrayList
$env[$i].add("anydata10")
}
But this approach not working well since $env variable has any 1 or more env value.
In the powershell, can we create dynamic json object at runtime and any other approch to archive it ?
For the root-level object ({"prd":{...}, "test":{...}}), I'd suggest an ordered dictionary, then create a custom object with the Sid property for each property:
# prepare env keys
$environments = "prd,test,dev,stg,qa" -split ","
# create ordered dictionary
$jsonObject = [ordered]#{}
foreach($envName in $environments){
# create 1 property, the value of which will be an object with a `Sid` property
$jsonObject[$envName] = [pscustomobject]#{
# resolve the correct `Sid` value from somewhere
Sid = Get-SidForEnv $envName
}
}
# convert the whole thing to json
$jsonObject |ConvertTo-Json -Depth 3
Based on your comments about getting the correct sid value "from somewhere", I assume you already have a function that translates prd to prodsid1 and so forth, so simply replace Get-SidForEnv with whatever function that is :)
I'm not sure I understand what you're trying to achieve. You can access environment variables using $env. So, if you have those values prd,test,... in some environment variable and you want to generate new environment variables based on those, you can do it like this:
# $env:foo contains 'prd,test'
$env:foo -split ',' | % {
New-Item -Path Env: -Name $_ -Value (#("anydata10") | ConvertTo-Json)
}

Extract multiline regex from extra large files in Powershell

I have extra large log file in CSV format which includes JSON formatted data inside. What I'm trying to do is extract JSON parts from the data and store it in a separate file.
The real problem is that the file size is almost 70Gb which causes some interesting problems to tackle.
The file size makes it impossible to read the whole file in one chunk. With Powershell's Get-Content combined with -ReadCount and Foreach-Object I can take smaller chunks and run regex pattern over them, chunk by chunk.
$Path = <pathToFile>
$outPath = <pathToOutput>
Out-File -Encoding utf8 -FilePath $outPath
$JsonRegex = "(?smi)\{.*?\}"
Get-Content -Path $Path -ReadCount 100000 | Foreach-Object {
( "$_" | Select-String -Pattern $JsonRegex -AllMatches | Foreach-Object { $_.Matches } | Foreach-Object { $_.Value } ) | Add-Content $outPath
}
But here what happens is, every 100k lines the ReadCount is in the middle of a JSON object thus skipping said object and continuing from next object.
Here is an example how this log data looks like. It includes some columns on first row and then JSON formatted data which is not consistent so I cannot use any fixed ReadCount value to avoid being in the middle of a JSON object.
"5","5","9/10/2019 12:00:46 AM","2","some","data","removed","comment","{
"message": "comment",
"level": "Information",
"logType": "User",
"timeStamp": "2019-09-10T03:00:46.5573047+03:00",
"fingerprint": "some",
}","11"
"5","5","9/10/2019 12:00:46 AM","2","some","data","removed","comment","{
"message": "comment",
"level": "Information",
"logType": "User",
"timeStamp": "2019-09-10T03:00:46.5672713+03:00",
"fingerprint": "some",
"windowsIdentity": "LOCAL\\WinID",
"machineName": "TK-141",
"processVersion": "1.0.71",
"jobId": "24a8",
"machineId": 11
}","11"
Is there any way to accomplish this without missing any data rows from the gigantous logfile?
Use a switch statement with the -Regex and -File parameters to efficiently (by PowerShell standards) read the file line by line and keep state across multiple lines.
For efficient writing to a file, use a .NET API, namely a System.IO.StreamWriter instance.
The following code assumes:
Each JSON string spans multiple lines and is non-nested.
On a given line, an opening { / closing } unambiguously marks the start / end of a (multi-line) JSON string.
# Input file path
$path = '...'
# Output file path
# Important: specify a *full* path
$outFileStream = [System.IO.StreamWriter] "$PWD/out.txt"
$json = ''
switch -Regex -File $path {
'\{.*' { $json = $Matches[0]; continue }
'.*\}' {
$json += "`n" + $Matches[0]
$outFileStream.WriteLine($json)
$json = ''
continue
}
default { if ($json) { $json += "`n" + $_ } }
}
$outFileStream.Close()
If you can further assume that no part of the JSON string follows the opening { / precedes the closing } on the same line, as your sample data suggest, you can simplify (and speed up) the switch statement:
$json = ''
switch -Regex -File $path {
'\{$' { $json ='{'; continue }
'^\}' { $outFileStream.WriteLine(($json + "`n}")); $json = ''; continue }
default { if ($json) { $json += "`n" + $_ } }
}
$outFileStream.Close()
Doug Maurer had a solution attempt involving a System.Text.StringBuilder instance so as to optimize the iterative concatenation of the parts making up each JSON string:
However, at least with an input file crafted from many repetitions of the sample data, I saw only a small performance gain in my informal tests.
For the sake of completeness, here's the System.Text.StringBuilder solution:
$json = [System.Text.StringBuilder]::new(512) # tweak the buffer size as needed
switch -Regex -File $path {
'\{$' { $null = $json.Append('{'); continue }
'^\}' { $outFileStream.WriteLine($json.Append("`n}").ToString()); $null = $json.Clear(); continue }
default { if ($json.Length) { $null = $json.Append("`n").Append($_) } }
}
$outFileStream.Close()

Reading JSON objects in Powershell

I need to integrate a JSON file which contains paths of the different objects in a PS script that generates and compares the hash files of source and destination. The paths in the JSON file are written in the format that I have stated below. I want to use the paths in that manner and pipe them into Get-FileHash in PowerShell. I can't figure out how to integrate my current PowerShell script with the JSON file that contains the information (File Name, full path) etc.
I have two scripts that I have tested and they work fine. One generates the MD5 hashes of two directories (source and destination) and stores them in a csv file. The other compares the MD5 hashes from the two CSV files and generates a new one, showing the result(whether a file is absent from source or destination).
Now, I need to integrate these scripts in another one, which is basically a Powershell Installer. The installer saves the configurations (path, ports, new files to be made etc etc) in a JSON format. In my original scripts, the user would type the path of the source and destination which needed to be compared. However, I now need to take the path from the JSON configuration files. For example, the JSON file below is of the similar nature as the one I have.
{
"destinatiopath": "C:\\Destination\\Mobile Phones\\"
"sourcepath": "C:\\Source\\Mobile Phones\\"
"OnePlus" : {
"files": [
{
"source": "6T",
"destination: "Model\\6T",
]
}
"Samsung" : {
"files": [
{
"source": "S20",
"destination": "Galaxy\\S20",
}
]
This is just a snippet of the JSON code. It's supposed to have the destination and source files. So for instance if the destination path is: C:\\Destination\\Mobile Phones\\ and the source path is C:\\Source\\Mobile Phones\\ and OnePlus has 6T as source and Model\\6T as destination, that means that the Powershell Installer will have the full path C:\\Destination\\Mobile Phones\\Model\\6T as the destination, and C:\\Source\\Mobile Phones\\6T as the source. The same will happen for Samsung and others.
For now, the MD5 hash comparison PS script just generates the CSV files in the two desired directories and compares them. However, I need to check the source and destination of each object in this case. I can't figure out how I can integrate it here. I'm pasting my MD5 hash generation code below.
Generating hash
#$p is the path. In this case, I'm running the script twice in order to get the hashes of both source and destination.
#$csv is the path where the csv will be exported.
Get-ChildItem $p -Recurse | ForEach-Object{ Get-FileHash $_.FullName -Algorithm MD5 -ErrorAction SilentlyContinue} | Select-Object Hash,
#{
Name = "FileName";
Expression = { [string]::Join("\", ($_.Path -split "\\" | Select-Object -Skip ($number))) }
} | Export-Csv -Path $csv
I want to use the paths in that manner and pipe them into Get-FileHash in PowerShell.
As the first step I would reorganize the JSON to be easier to handle. This will make a big difference on the rest of the script.
{
"source": "C:\\Source\\Mobile Phones",
"destination": "C:\\Destination\\Mobile Phones",
"phones": [
{
"name": "OnePlus",
"source": "6T",
"destination": "Model\\6T"
},
{
"name": "Samsung",
"source": "S20",
"destination": "Galaxy\\S20"
}
]
}
Now it's very easy to get all the paths no matter how many "phone" entries there are. You don't even really need an intermediary CSV file.
$config = Get-Content config.json -Encoding UTF8 -Raw | ConvertFrom-Json
$config.phones | ForEach-Object {
$source_path = Join-Path $config.source $_.source
$destination_path = Join-Path $config.destination $_.destination
$source_hashes = Get-ChildItem $source_path -File -Recurse | Get-FileHash -Algorithm MD5
$destination_hashes = Get-ChildItem $destination_path -File -Recurse | Get-FileHash -Algorithm MD5
# the combination of relative path and file hash needs to be unique, so let's combine them
$source_relative = $source_hashes | ForEach-Object {
[pscustomobject]#{
Path = $_.Path
PathHash = $_.Path.Replace($source_path, "") + '|' + $_.Hash
}
}
$destination_relative = $destination_hashes | ForEach-Object {
[pscustomobject]#{
Path = $_.Path
PathHash = $_.Path.Replace($destination_path, "") + '|' + $_.Hash
}
}
# Compare-Object finds the difference between both lists
$diff = Compare-Object $source_relative $destination_relative -Property PathHash, Path
Write-Host $diff
$diff | ForEach-Object {
# work with $_.Path and $_.SideIndicator
}
}

How to replace a property in a json file in powershell

I'm having a difficult time reading in a JSON file in powershell, replacing a value and writing back to the file or a file.
Given the following:
[Object]$QuickJson = #'
{
"swagger": "1.0",
"info": {
"version": "0.0.1",
"title": "this is a title",
"description": "this is a description"
},
"basePath": "/test",
"paths" : {
"/GetSomething": {
"get": {
"name" : "test01"
}
},
"/GetSomethingElse" : {
"get": {
"name" : "test02"
}
},
"/GetAnotherThing": {
"get": {
"name" : "test03"
}
}
}
}
'#
What I'm interested in here is replacing the values in the "paths" object with something else. I read the file and can access the object however with what I'm trying:
[object]$MyPSJson = ConvertFrom-Json -InputObject $QuickJson
foreach($info in $MyPSJson.paths.PSObject.Properties)
{
$path = $info.Name
Write-Host "path: $path"
$info.Name = "$path?code=fsfsfsfsfsfdfds"
Write-Host $info.Name
}
I don't know what those "paths" will be, I need to take the existing value and append a code value to it so I need to be able to iterate through all the paths and do this replacement. When I try the above I get an error:
path: /GetSomething
-- 'Name' is a ReadOnly property. At C:\scripts\test.ps1:44 char:5
+ $info.Name = "$path?code=fsfsfsfsfsfdfds"
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : InvalidOperation: (:) [], RuntimeException
+ FullyQualifiedErrorId : PropertyAssignmentException
I've tried several different things and yet to come up with a good, or workable for that matter, solution. Any help or pointing in the right direction is greatly appreciated.
You could do something very similar to what you have now, just instead of changing the existing properties you record the initial properties, add new ones with the modifications you want, then set $MyPSJson.paths to itself excluding the old properties, so all it has are the new properties.
#Find initial paths
$Paths=$MyPSJson.paths.psobject.Properties.Name
#Add new paths with the modification to the name, and set the value to the same as the old path
$Paths|%{Add-Member -InputObject $MyPSJson.paths -NotePropertyName "$_`?code=fsfsfsfsfsfdfds" -NotePropertyValue $MyPSJson.paths.$_}
#Set the paths object to itself, excluding the original paths
$MyPSJson.paths = $MyPSJson.paths|Select * -ExcludeProperty $Paths
Try the following (PSv3+):
$MyPSJson.paths = $MyPSJson.paths.psobject.properties |
ForEach-Object { $renamedProps = [ordered] #{} } {
$renamedProps[$_.Name + '?foo=bar&baz=bam'] = $_.Value
} { [pscustomobject] $renamedProps }
Of technically necessity, this recreates the .paths object with modified property names.
It does so by enumerating the original properties, adding their values under the modified name to an ordered hashtable, which on completion of the pipeline is converted to a custom object ([pscustomobject]).
As for what you tried:
A given object's properties cannot be renamed - only its properties' values can be changed.
Therefore, you must create a new object with the desired new property names and the same values as the original.
As an aside: an [object] cast is pointless in PowerShell - it is an effective no-op.
By contrast, a [pscustomobject] cast can be used to create [pscustomobject] instances from [ordered] hashtables, which is the technique used above; casting from other types is again a virtual no-op.
In addition to the accepted answer which worked for me I also found another way which was initially suggested which was to use a hashtable:
function Convert-JsonFileToHashTable([string]$file) {
$text = [IO.File]::ReadAllText($file)
$parser = New-Object Web.Script.Serialization.JavaScriptSerializer
$parser.MaxJsonLength = $text.length
$hashtable = $parser.Deserialize($text, #{}.GetType())
return $hashtable
}
$hashtable = Convert-JsonFileToHashTable (Resolve-Path -Path $DefinitionFile)
$keys = New-Object System.Collections.ArrayList($null)
$keys.AddRange($hashtable["paths"].Keys)
foreach ($key in $keys)
{
$newPath = $key + "?code=$Code"
$hashtable["paths"].$newPath = $hashtable["paths"].$key
$hashtable["paths"].Remove($key) | Out-Null
}
$json = $hashtable | ConvertTo-Json -Depth 20
Write-Output $json | Out-File $newFile -Encoding utf8