Reading JSON objects in Powershell - json

I need to integrate a JSON file which contains paths of the different objects in a PS script that generates and compares the hash files of source and destination. The paths in the JSON file are written in the format that I have stated below. I want to use the paths in that manner and pipe them into Get-FileHash in PowerShell. I can't figure out how to integrate my current PowerShell script with the JSON file that contains the information (File Name, full path) etc.
I have two scripts that I have tested and they work fine. One generates the MD5 hashes of two directories (source and destination) and stores them in a csv file. The other compares the MD5 hashes from the two CSV files and generates a new one, showing the result(whether a file is absent from source or destination).
Now, I need to integrate these scripts in another one, which is basically a Powershell Installer. The installer saves the configurations (path, ports, new files to be made etc etc) in a JSON format. In my original scripts, the user would type the path of the source and destination which needed to be compared. However, I now need to take the path from the JSON configuration files. For example, the JSON file below is of the similar nature as the one I have.
{
"destinatiopath": "C:\\Destination\\Mobile Phones\\"
"sourcepath": "C:\\Source\\Mobile Phones\\"
"OnePlus" : {
"files": [
{
"source": "6T",
"destination: "Model\\6T",
]
}
"Samsung" : {
"files": [
{
"source": "S20",
"destination": "Galaxy\\S20",
}
]
This is just a snippet of the JSON code. It's supposed to have the destination and source files. So for instance if the destination path is: C:\\Destination\\Mobile Phones\\ and the source path is C:\\Source\\Mobile Phones\\ and OnePlus has 6T as source and Model\\6T as destination, that means that the Powershell Installer will have the full path C:\\Destination\\Mobile Phones\\Model\\6T as the destination, and C:\\Source\\Mobile Phones\\6T as the source. The same will happen for Samsung and others.
For now, the MD5 hash comparison PS script just generates the CSV files in the two desired directories and compares them. However, I need to check the source and destination of each object in this case. I can't figure out how I can integrate it here. I'm pasting my MD5 hash generation code below.
Generating hash
#$p is the path. In this case, I'm running the script twice in order to get the hashes of both source and destination.
#$csv is the path where the csv will be exported.
Get-ChildItem $p -Recurse | ForEach-Object{ Get-FileHash $_.FullName -Algorithm MD5 -ErrorAction SilentlyContinue} | Select-Object Hash,
#{
Name = "FileName";
Expression = { [string]::Join("\", ($_.Path -split "\\" | Select-Object -Skip ($number))) }
} | Export-Csv -Path $csv

I want to use the paths in that manner and pipe them into Get-FileHash in PowerShell.
As the first step I would reorganize the JSON to be easier to handle. This will make a big difference on the rest of the script.
{
"source": "C:\\Source\\Mobile Phones",
"destination": "C:\\Destination\\Mobile Phones",
"phones": [
{
"name": "OnePlus",
"source": "6T",
"destination": "Model\\6T"
},
{
"name": "Samsung",
"source": "S20",
"destination": "Galaxy\\S20"
}
]
}
Now it's very easy to get all the paths no matter how many "phone" entries there are. You don't even really need an intermediary CSV file.
$config = Get-Content config.json -Encoding UTF8 -Raw | ConvertFrom-Json
$config.phones | ForEach-Object {
$source_path = Join-Path $config.source $_.source
$destination_path = Join-Path $config.destination $_.destination
$source_hashes = Get-ChildItem $source_path -File -Recurse | Get-FileHash -Algorithm MD5
$destination_hashes = Get-ChildItem $destination_path -File -Recurse | Get-FileHash -Algorithm MD5
# the combination of relative path and file hash needs to be unique, so let's combine them
$source_relative = $source_hashes | ForEach-Object {
[pscustomobject]#{
Path = $_.Path
PathHash = $_.Path.Replace($source_path, "") + '|' + $_.Hash
}
}
$destination_relative = $destination_hashes | ForEach-Object {
[pscustomobject]#{
Path = $_.Path
PathHash = $_.Path.Replace($destination_path, "") + '|' + $_.Hash
}
}
# Compare-Object finds the difference between both lists
$diff = Compare-Object $source_relative $destination_relative -Property PathHash, Path
Write-Host $diff
$diff | ForEach-Object {
# work with $_.Path and $_.SideIndicator
}
}

Related

I want to create json file by substituting values from environment variables in a json template file

One requirement of mine is - Using windows, not use any tools not already available as part of aws cli or windows
For example, I have this json file test.json with below content:
"My number is $myvar"
I read this into a powershell variable like so:
$myobj=(get-content .\test.json | convertfrom-json)
$myvar=1
From here, I would like to do something with this $myobj which will enable me to get this output:
$myobj | tee json_with_values_from_environment.json
My number is 1
I got some limited success with iex, but not sure if it can be made to work for this example
You can use $ExecutionContext.InvokeCommand.ExpandString()
$myobj = '{test: "My number is $myvar"}' | ConvertFrom-Json
$myvar = 1
$ExecutionContext.InvokeCommand.ExpandString($myobj.test)
Output
My number is 1
Here is one way to do it using the Parser to find all VariableExpressionAst and replace them with the values in your session.
Given the following test.json:
{
"test1": "My number is $myvar",
"test2": {
"somevalue": "$env:myothervar",
"someothervalue": "$anothervar !!"
}
}
We want to find and replace $myvar, $myothervar and $anothervar with their corresponding values defined in the current session, so the code looks like this (note that we do the replacement before converting the Json string into an object, this way is much easier):
using namespace System.Management.Automation.Language
$isCore7 = $PSVersionTable.PSVersion -ge '7.2'
# Define the variables here
$myvar = 10
$env:myothervar = 'hello'
$anothervar = 'world'
# Read the Json
$json = Get-Content .\test.json -Raw
# Now parse it
$ast = [Parser]::ParseInput($json, [ref] $null, [ref] $null)
# Find all variables in it, and enumerate them
$ast.FindAll({ $args[0] -is [VariableExpressionAst] }, $true) |
Sort-Object { $_.Extent.Text } -Unique | ForEach-Object {
# now replace the text with the actual value
if($isCore7) {
# in PowerShell Core is very easy
$json = $json.Replace($_.Extent.Text, $_.SafeGetValue($true))
return
}
# in Windows PowerShell not so much
$varText = $_.Extent.Text
$varPath = $_.VariablePath
# find the value of the var (here we use the path)
$value = $ExecutionContext.SessionState.PSVariable.GetValue($varPath.UserPath)
if($varPath.IsDriveQualified) {
$value = $ExecutionContext.SessionState.InvokeProvider.Item.Get($varPath.UserPath).Value
}
# now replace the text with the actual value
$json = $json.Replace($varText, $value)
}
# now we can safely convert the string to an object
$json | ConvertFrom-Json
If we were to convert it back to Json to see the result:
{
"test1": "My number is 10",
"test2": {
"somevalue": "hello",
"someothervalue": "world !!"
}
}

Extract multiline regex from extra large files in Powershell

I have extra large log file in CSV format which includes JSON formatted data inside. What I'm trying to do is extract JSON parts from the data and store it in a separate file.
The real problem is that the file size is almost 70Gb which causes some interesting problems to tackle.
The file size makes it impossible to read the whole file in one chunk. With Powershell's Get-Content combined with -ReadCount and Foreach-Object I can take smaller chunks and run regex pattern over them, chunk by chunk.
$Path = <pathToFile>
$outPath = <pathToOutput>
Out-File -Encoding utf8 -FilePath $outPath
$JsonRegex = "(?smi)\{.*?\}"
Get-Content -Path $Path -ReadCount 100000 | Foreach-Object {
( "$_" | Select-String -Pattern $JsonRegex -AllMatches | Foreach-Object { $_.Matches } | Foreach-Object { $_.Value } ) | Add-Content $outPath
}
But here what happens is, every 100k lines the ReadCount is in the middle of a JSON object thus skipping said object and continuing from next object.
Here is an example how this log data looks like. It includes some columns on first row and then JSON formatted data which is not consistent so I cannot use any fixed ReadCount value to avoid being in the middle of a JSON object.
"5","5","9/10/2019 12:00:46 AM","2","some","data","removed","comment","{
"message": "comment",
"level": "Information",
"logType": "User",
"timeStamp": "2019-09-10T03:00:46.5573047+03:00",
"fingerprint": "some",
}","11"
"5","5","9/10/2019 12:00:46 AM","2","some","data","removed","comment","{
"message": "comment",
"level": "Information",
"logType": "User",
"timeStamp": "2019-09-10T03:00:46.5672713+03:00",
"fingerprint": "some",
"windowsIdentity": "LOCAL\\WinID",
"machineName": "TK-141",
"processVersion": "1.0.71",
"jobId": "24a8",
"machineId": 11
}","11"
Is there any way to accomplish this without missing any data rows from the gigantous logfile?
Use a switch statement with the -Regex and -File parameters to efficiently (by PowerShell standards) read the file line by line and keep state across multiple lines.
For efficient writing to a file, use a .NET API, namely a System.IO.StreamWriter instance.
The following code assumes:
Each JSON string spans multiple lines and is non-nested.
On a given line, an opening { / closing } unambiguously marks the start / end of a (multi-line) JSON string.
# Input file path
$path = '...'
# Output file path
# Important: specify a *full* path
$outFileStream = [System.IO.StreamWriter] "$PWD/out.txt"
$json = ''
switch -Regex -File $path {
'\{.*' { $json = $Matches[0]; continue }
'.*\}' {
$json += "`n" + $Matches[0]
$outFileStream.WriteLine($json)
$json = ''
continue
}
default { if ($json) { $json += "`n" + $_ } }
}
$outFileStream.Close()
If you can further assume that no part of the JSON string follows the opening { / precedes the closing } on the same line, as your sample data suggest, you can simplify (and speed up) the switch statement:
$json = ''
switch -Regex -File $path {
'\{$' { $json ='{'; continue }
'^\}' { $outFileStream.WriteLine(($json + "`n}")); $json = ''; continue }
default { if ($json) { $json += "`n" + $_ } }
}
$outFileStream.Close()
Doug Maurer had a solution attempt involving a System.Text.StringBuilder instance so as to optimize the iterative concatenation of the parts making up each JSON string:
However, at least with an input file crafted from many repetitions of the sample data, I saw only a small performance gain in my informal tests.
For the sake of completeness, here's the System.Text.StringBuilder solution:
$json = [System.Text.StringBuilder]::new(512) # tweak the buffer size as needed
switch -Regex -File $path {
'\{$' { $null = $json.Append('{'); continue }
'^\}' { $outFileStream.WriteLine($json.Append("`n}").ToString()); $null = $json.Clear(); continue }
default { if ($json.Length) { $null = $json.Append("`n").Append($_) } }
}
$outFileStream.Close()

Azure pipelines -Add new element in json array VSTS pipelines (Appsettings.json)

Is this possible to add a new element in an array of appsetting.json in Azure Release Pipeline?
In appsetting.json I have array variable which I need to fill with another element during deployment through Azure Pipeline.
"Array": [
{
"Name": "AD1",
"IsDefault": "true",
"IdPEntityId": "URL1",
"Metadata": "XMLpath1"
},
{
"Name": "AD2",
"IsDefault": "false",
"IdPEntityId": "URL2",
"Metadata": "XMLPath2"
}
]
Here in the above JSON array I need to add another one elemental last position (array-Index:2).
[CmdletBinding()]
param(
[string] $AdName,
[bool] $AdIsDefault,
[string] $AdIdPEntityId,
[string] $AdMetadata,
[string] $AppSettingFilePath
)
clear-Host
Write-Host 'Updating appsettings.json...' -ForegroundColor Yellow
function Format-Json([Parameter(Mandatory, ValueFromPipeline)][String] $json) {
$indent = 0;
($json -Split '\n' |
% {
if ($_ -match '[\}\]]') {
# This line contains ] or }, decrement the indentation level
$indent--
}
$line = (' ' * $indent * 2) + $_.TrimStart().Replace(': ', ': ')
if ($_ -match '[\{\[]') {
# This line contains [ or {, increment the indentation level
$indent++
}
$line
}) -Join "`n"
}
$JsonDataAdd=#"
{
"Name":"$AdName",
"IsDefault": "$AdIsDefault",
"IdPEntityId":"$AdIdPEntityId",
"Metadata": "$AdMetadata"
}
"#
Write-Host ' Active directory details :' -ForegroundColor Yellow
Write-Host `n $JsonDataAdd -ForegroundColor Green
$jsonData = Get-Content "$AppSettingFilePath" | Out-String | ConvertFrom-Json -ErrorAction Stop
$jsonData.IdentitySettings.ExternalProviders.Saml2Providers += (ConvertFrom-Json $JsonDataAdd)
$jsonData | ConvertTo-Json -Depth 10 | Format-Json | Set-Content "$AppSettingFilePath" -Encoding UTF8
Write-Host 'Successfully Updated -appSettings.json !' -ForegroundColor Yellow
You could use JSON variable substitution. This feature substitutes values in the JSON configuration files. It overrides the values in the specified JSON configuration files (for example, appsettings.json) with the values matching names of release pipeline and stage variables.
When in "Deploy Azure App Service" release task you should see a "File Transforms and Variable Substitution" section. In here you will supply the path to the json file you want to swap variable values.
[![enter image description here][1]][1]
Then you just need to define the required substitution values in release pipeline or stage variables. From here you can add the json property you want to modify as a variable.
[![enter image description here][2]][2]
Finally after the transformation, the JSON will contain new. Azure DevOps will then swap out these values for you when deploying.
More details you could refer our official tutorial here: [File transforms and variable substitution reference][3]
Update:
It only works to adjust existing entries in the appsettings.json files, it doesn't seem to be able to add any new one. You could also take a look at the JSON variable substitution notes
Variable substitution is applied for only the JSON keys predefined in
the object hierarchy. It does not create new keys.
As a workaround, you could choose to use the File Creator extension:https://marketplace.visualstudio.com/items?itemName=eliostruyf.build-task to push the whole new appsettings.json file in the pipeline.
Update2
OP finally moved with PS script written by him to add new elements in Arrays of Appsettings.json

Create a Specific JSON file from csv in PowerShell

I have no experience with PowerShell and I was asked to create this script as a favor for a friend of mine. The script is supposed to read a csv file (These files have different columns except for time and host, which are common among all files), and output its content into a JSON file of the following format:
CSV file contains columns:
host| message | time | severity | source |
{
"time": 1437522387,
"host": "dataserver992.example.com",
"event": {
"message": "Something happened",
"severity": "INFO",
"source": "testapp"
#...All columns except for time and host should be under "event"
}
}
*The only guaranteed columns are time and host. All other column headers vary from file to file.
This is part of what I have so far:
$csvFile = Import-Csv $filePath
function jsonConverter($file)
{
#Currently not in use
$eventString = $file| select * -ExcludeProperty time, host
$file | Foreach-Object {
Write-Host '{'
Write-Host '"host":"'$_.host'",'
Write-Host '"time":"'$_.time'",'
Write-Host '"event":{'
#TODO: Put all other columns (key, values) under event - Except for
time and host
Write-Host '}'
}
}
jsonConverter($csvFile)
Any ideas of how I could extract only the remaining columns, row by row, outputting its content to a key, value JSON format like the example above?
Thank you!
Provided your csv looks like this:
"host","message","time","severity","source"
"dataserver992.example.com","Something happened","1437522387","INFO","testapp"
this script:
$filepath = '.\input.csv'
$csvData = Import-Csv $filePath
$NewCsvData = foreach($Row in $csvData){
[PSCustomObject]#{
time = $Row.time
host = $Row.host
event = ($Row| Select-Object -Property * -ExcludeProperty time,host)
}
}
$NewCsvData | ConvertTo-Json
will output this Json:
{
"time": "1437522387",
"host": "dataserver992.example.com",
"event": {
"message": "Something happened",
"severity": "INFO",
"source": "testapp"
}
}
If your powershell version is 3.0 or higher (it should):
Import-CSV $filepath | ConvertTo-JSON
Done!

Rename Files & Folders Keywords - Using a CSV Look Up File

I would like to rename files and folders based on keywords found in a CSV file.
The CSV holds the search and replace keywords that will make up file and folder names.
Search | Replace
Document | DTX
Processing | PRX
Implementation | IMX
...
Not all the file names include each word in the file name.
Not all the folders will include each word in the folder name
Powershell will have to search the child item ie the folder and file
names.
If it finds the word (match) - Substitute from the CSV
I have looked at these threads to help me:
Using Powershell to recursively rename directories using a lookup file
powershell script to rename all files in directory
http://code.adonline.id.au/batch-rename-files/
I have only managed below snippet
$folder = "C:\Folders" #target folder containing files
$csv = "C:\FileNameKeywords.csv" #path to CSV file
cd ($folder);
Import-Csv ($csv) | foreach {
Rename-Item -Path $_.Path -NewName $_.Filename
}
It only replaces one at a time.
Question:
How can I recursively search and replace in file and Folder Names using a CSV as a look up or reference file.
When you have the need to look up values by other values the usual go-to data structure is a dictionary, or in PowerShell terms a hashtable. Read your CSV into a dictionary like this:
$keywords = #{}
Import-Csv $csv | ForEach-Object {
$keywords[$_.Search] = $_.Replace
}
Then traverse your folder tree and build the new filenames by replacing each key with its respective value:
Get-ChildItem $folder -Recurse | ForEach-Object {
$newname = $_.Name
foreach ($word in $keywords.Keys) {
$newname = $newname.Replace($word, $keywords[$word])
}
if ($_.Name -ne $newname) {
Rename-Item -Path $_.FullName -NewName $newname
}
}
ill give it a shot. I'm assuming search and replace are your headers in this scenario. this in addition to your $folder and $csv variables.
$csvobject=import-csv $csv
Foreach($obj in $csvobject){
$search=$obj.search
$replace=$obj.replace
get-childitem path $folder |where{$_.name -like "$($obj.search)"} | rename-item -newname {$_.name -replace "$search", "$replace"} }
The replace handles regex so u will need to make sure Any special characters are properly escaped.