I'm really struggling with what seems to be a simple thing. Any help is appreciated...
tldr; i'm trying to find and replace blank or NULL values from powershell
output to "No Data"
I am using the following powershell script to obtain installed application information
Get-ItemProperty HKLM:\Software\Microsoft\Windows\CurrentVersion\Uninstall\*
Below is sample output of the above script (filtered to my liking). I am taking this data, exporting it to CSV, in order to import into another application for analysis at a later time.
Host : Computer1
DisplayName : AutoDWG DWG DXF Converter 2015
Version :
Publisher :
InstallDate :
arrival_datetime : 2015-11-03T09:42:18
Host : Computer2
DisplayName : Beyond Compare Version 3.1.11
Version :
Publisher : Scooter Software
InstallDate : 20150218
arrival_datetime : 2015-11-03T09:42:18
the CSV export puts the data in the correct format, but where items have no data for version, publisher, etc..., it represents it as ",," (see below)
"Computer1","AutoDWG DWG DXF Converter 2015",,,,"2015-11-03T09:54:21"
"Computer2","Beyond Compare Version 3.1.11",,"Scooter Software","20150218","2015-11-03T09:54:21"
When importing the CSV, the blank values are re-interpreted as NULL, which throws the program for a loop, as it is expecting a string. I am attempting to change those to the string "No Data" but am having lots of trouble...
What would be the best way to handle this?
Using Select-Object would be your best bet. Just pipe the data to Select-Object, but customize each desired property as follows:
Get-ItemProperty HKLM:\Software\Microsoft\Windows\CurrentVersion\Uninstall\* |
Select-Object Host, DisplayName, #{
Label = "Version"
Expression = { if ($_.Version) { $_.Version } else { "No Data" } }
}, Other Properties
You could inspect the property values as they are piped from the Import-Csv cmdlet, and change them. Example:
Get-ItemProperty HKLM:\Software\Microsoft\Windows\CurrentVersion\Uninstall\* |
ForEach-Object {
foreach ($p in $_.PSObject.Properties)
{
if ($p.Value -eq [string]::Empty)
{
$p.Value = 'No Data'
}
}
Write-Output $_
}
Building off of this answer, you could generate the calculated properties like this:
$SelectProperties = "Host","DisplayName"
$NoDataFields = "Version","Publisher","InstallDate"
$SelectProperties += $NoDataFields|ForEach-Object {
#{
Label = $_
Expression = [scriptblock]::Create("if(`$_.""$_""){ `$_.""$_"" } else { ""No Data"" }")
}
}
$Data = Import-Csv -Path "C:\SomePath.csv" |
Select-Object $SelectProperties
Do your columns have headers? This was how I did it, when I exported a CSV of groups and their owners. For any group that didn't have an owner in the "ManagedBy" column, it fills the field with "No Owner" instead of a blank space:
$CSVData = Import-Csv $TempCSV -Header "samAccountName", "ManagedBy"
$CSVData | %{
if($_.ManagedBy -eq "") {$_.ManagedBy="No Owner"}
}
$CSVData | Export-Csv $Filename -NoTypeInformation
You could just change the "ManagedBy" to your header name and the "$_.ManagedBy" to what you need, then obviously "No Owner" would be "No Data".
Another option that might work:
$TargetFile = "C:\temp\replaceNull.csv"
$File = Get-Content $TargetFile
$Output = #()
foreach ($Line in $File) {
$Output += $Line -replace ",,",",No Data,"
}
$Output | Set-Content $TargetFile
Related
One requirement of mine is - Using windows, not use any tools not already available as part of aws cli or windows
For example, I have this json file test.json with below content:
"My number is $myvar"
I read this into a powershell variable like so:
$myobj=(get-content .\test.json | convertfrom-json)
$myvar=1
From here, I would like to do something with this $myobj which will enable me to get this output:
$myobj | tee json_with_values_from_environment.json
My number is 1
I got some limited success with iex, but not sure if it can be made to work for this example
You can use $ExecutionContext.InvokeCommand.ExpandString()
$myobj = '{test: "My number is $myvar"}' | ConvertFrom-Json
$myvar = 1
$ExecutionContext.InvokeCommand.ExpandString($myobj.test)
Output
My number is 1
Here is one way to do it using the Parser to find all VariableExpressionAst and replace them with the values in your session.
Given the following test.json:
{
"test1": "My number is $myvar",
"test2": {
"somevalue": "$env:myothervar",
"someothervalue": "$anothervar !!"
}
}
We want to find and replace $myvar, $myothervar and $anothervar with their corresponding values defined in the current session, so the code looks like this (note that we do the replacement before converting the Json string into an object, this way is much easier):
using namespace System.Management.Automation.Language
$isCore7 = $PSVersionTable.PSVersion -ge '7.2'
# Define the variables here
$myvar = 10
$env:myothervar = 'hello'
$anothervar = 'world'
# Read the Json
$json = Get-Content .\test.json -Raw
# Now parse it
$ast = [Parser]::ParseInput($json, [ref] $null, [ref] $null)
# Find all variables in it, and enumerate them
$ast.FindAll({ $args[0] -is [VariableExpressionAst] }, $true) |
Sort-Object { $_.Extent.Text } -Unique | ForEach-Object {
# now replace the text with the actual value
if($isCore7) {
# in PowerShell Core is very easy
$json = $json.Replace($_.Extent.Text, $_.SafeGetValue($true))
return
}
# in Windows PowerShell not so much
$varText = $_.Extent.Text
$varPath = $_.VariablePath
# find the value of the var (here we use the path)
$value = $ExecutionContext.SessionState.PSVariable.GetValue($varPath.UserPath)
if($varPath.IsDriveQualified) {
$value = $ExecutionContext.SessionState.InvokeProvider.Item.Get($varPath.UserPath).Value
}
# now replace the text with the actual value
$json = $json.Replace($varText, $value)
}
# now we can safely convert the string to an object
$json | ConvertFrom-Json
If we were to convert it back to Json to see the result:
{
"test1": "My number is 10",
"test2": {
"somevalue": "hello",
"someothervalue": "world !!"
}
}
So i'm trying to convert XML files on an Azure Storage Container to JSON in the same container.
This way I'm able to read the information into an Azure SQL Database via an Azure Datafactory.
I'd like to stay clear from using Logic apps if able.
The JSON files need to be formatted.
And all this through the use of PowerShell scripting.
What i've got so far after some searching on the interwebs and shamelessly copying and pasting powershell code:
#Connect-AzAccount
# Helper function that converts a *simple* XML document to a nested hashtable
# with ordered keys.
function ConvertFrom-Xml {
param([parameter(Mandatory, ValueFromPipeline)] [System.Xml.XmlNode] $node)
process {
if ($node.DocumentElement) { $node = $node.DocumentElement }
$oht = [ordered] #{}
$name = $node.Name
if ($node.FirstChild -is [system.xml.xmltext]) {
$oht.$name = $node.FirstChild.InnerText
} else {
$oht.$name = New-Object System.Collections.ArrayList
foreach ($child in $node.ChildNodes) {
$null = $oht.$name.Add((ConvertFrom-Xml $child))
}
}
$oht
}
}
function Format-Json
{
<#
.SYNOPSIS
Prettifies JSON output.
.DESCRIPTION
Reformats a JSON string so the output looks better than what ConvertTo-Json outputs.
.PARAMETER Json
Required: [string] The JSON text to prettify.
.PARAMETER Minify
Optional: Returns the json string compressed.
.PARAMETER Indentation
Optional: The number of spaces (1..1024) to use for indentation. Defaults to 4.
.PARAMETER AsArray
Optional: If set, the output will be in the form of a string array, otherwise a single string is output.
.EXAMPLE
$json | ConvertTo-Json | Format-Json -Indentation 2
#>
[CmdletBinding(DefaultParameterSetName = 'Prettify')]
Param(
[Parameter(Mandatory = $true, Position = 0, ValueFromPipeline = $true)]
[string]$Json,
[Parameter(ParameterSetName = 'Minify')]
[switch]$Minify,
[Parameter(ParameterSetName = 'Prettify')]
[ValidateRange(1, 1024)]
[int]$Indentation = 4,
[Parameter(ParameterSetName = 'Prettify')]
[switch]$AsArray
)
if ($PSCmdlet.ParameterSetName -eq 'Minify')
{
return ($Json | ConvertFrom-Json) | ConvertTo-Json -Depth 100 -Compress
}
# If the input JSON text has been created with ConvertTo-Json -Compress
# then we first need to reconvert it without compression
if ($Json -notmatch '\r?\n')
{
$Json = ($Json | ConvertFrom-Json) | ConvertTo-Json -Depth 100
}
$indent = 0
$regexUnlessQuoted = '(?=([^"]*"[^"]*")*[^"]*$)'
$result = $Json -split '\r?\n' |
ForEach-Object {
# If the line contains a ] or } character,
# we need to decrement the indentation level unless it is inside quotes.
if ($_ -match "[}\]]$regexUnlessQuoted")
{
$indent = [Math]::Max($indent - $Indentation, 0)
}
# Replace all colon-space combinations by ": " unless it is inside quotes.
$line = (' ' * $indent) + ($_.TrimStart() -replace ":\s+$regexUnlessQuoted", ': ')
# If the line contains a [ or { character,
# we need to increment the indentation level unless it is inside quotes.
if ($_ -match "[\{\[]$regexUnlessQuoted")
{
$indent += $Indentation
}
$line
}
if ($AsArray) { return $result }
return $result -Join [Environment]::NewLine
}
# Storage account details
$resourceGroup = "insert resource group here"
$storageAccountName = "insert storage account name here"
$container = "insert container here"
$storageAccountKey = (Get-AzStorageAccountKey -ResourceGroupName $resourceGroup -Name $storageAccountName).Value[0]
$storageAccount = Get-AzStorageAccount -ResourceGroupName $resourceGroup -Name $storageAccountName
# Creating Storage context for Source, destination and log storage accounts
#$context = New-AzStorageContext -StorageAccountName $storageAccountName -StorageAccountKey $storageAccountKey
$context = New-AzStorageContext -ConnectionString "insert connection string here"
$blob_list = Get-AzStorageBlob -Container $container -Context $context
foreach($blob_iterator in $blob_list){
[XML](Get-AzStorageBlobContent $blob_iterator.name -Container $container -Context $context) | ConvertFrom-Xml | ConvertTo-Json -Depth 11 | Format-Json | Set-Content ($blob_iterator.name + '.json')
}
Output =
Cannot convert value "Microsoft.WindowsAzure.Commands.Common.Storage.ResourceModel.AzureStorageBlob" to type "System.Xml.XmlDocument". Error: "The specified node cannot
be inserted as the valid child of this node, because the specified node is the wrong type."
At C:\Users\.....\Convert XML to JSON.ps1:116 char:6
+ [XML](Get-AzStorageBlobContent $blob_iterator.name -Container $c ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : InvalidArgument: (:) [], RuntimeException
+ FullyQualifiedErrorId : InvalidCastToXmlDocument
When I run the code the script asks me if I want to download the xml file to a local folder on my laptop.
This is not what I want, I want the conversion to be done in Azure on the Storage container.
And I think that I'm adding ".json" to the .xml file name.
So the output would become something like filename.xml.json instead of just filename.json
What's going wrong here?
And how can it be fixed?
Thank you in advance for your help.
So I have this script that goes out and finds all the software versions installed on machines and lets people know what software and when it was installed across several VMs.
I want to put this on a Dashboard provider we use but they have a specific format in which to use it.
it does produce a valid JSON however I just found out it's not in the format the company wishes.
which would be:
{"table": [["header1", "header2"], ["row1column1", "row1column2"], ["row2column1", "row2column2"]]}
My first thought would be to produce a header row as a beginning variable and then individual variables for each component but that feels very tedious and laborious to create variables for each individual row of data (Date, Name of Software, etc). then at the end combine them into 1 and convert to json
My script is this:
[CmdletBinding()]
Param (
[Parameter(ValueFromPipeline = $true,
ValueFromPipelinebyPropertyName = $true)]
[Alias("Servers")]
[string[]]$Name = (Get-Content "c:\utils\servers.txt")
)
Begin {
}
Process {
$AllComputers = #()
#Gather all computer names before processing
ForEach ($Computer in $Name) {
$AllComputers += $Computer
}
}
End {
ForEach ($Computer in $AllComputers) {
write-output "Checking $computer"
if ($computer -like "*x86*") {
$data = Invoke-Command -cn $computer -ScriptBlock {Get-ItemProperty HKLM:\Software\Microsoft\Windows\CurrentVersion\Uninstall\* | Select-Object #{Label = "ServerName"; Expression = {$env:computername}}, DisplayName, Publisher, DisplayVersion, InstallDate | Where-object { $_.Publisher -match "Foobar" }}
$jsondata += $data
}
else {
$data = Invoke-Command -cn $computer -ScriptBlock { Get-ItemProperty HKLM:\Software\Wow6432Node\Microsoft\Windows\CurrentVersion\Uninstall\* | Select-Object #{Label = "ServerName"; Expression = {$env:computername}}, DisplayName, Publisher, DisplayVersion, InstallDate | Where-object { $_.Publisher -match "foobar" } }
$jsondata += $data
}
}
$jsondata | ConvertTo-Json -depth 100 | Out-File "\\servername\C$\Utils\InstalledApps.json"
}
From the sample output format provided I would conclude that you are looking for an array of array. There is a "bug" using ConvertTo-Json when trying to do this but since we need it inside a table object anyway. I will show an example using your code but just on my local computer. Integrating this into your code should not be an issue.
# gather the results
$results = Get-ItemProperty HKLM:\Software\Microsoft\Windows\CurrentVersion\Uninstall\* | Where-object { $_.Publisher -match "The" } | Select-Object #{Label = "ServerName"; Expression = {$env:computername}}, DisplayName, Publisher, DisplayVersion, InstallDate
# Prepare an array of arrays for the output.
$outputToBeConverted = #()
# build the header
$header = ($results | Get-Member -MemberType NoteProperty).Name
$outputToBeConverted += ,$header
# Add the rows
Foreach($item in $results){
# Create a string array by calling each property individually
$outputToBeConverted += ,[string[]]($header | ForEach-Object{$item."$_"})
}
[pscustomobject]#{table=$outputToBeConverted} | ConvertTo-Json -Depth 5
Basically it is making a jagged array of arrays where the first member is your "header" and each row is built manually from the items in the $results collection.
You will see the unary operator , used above. That is done to prevent PowerShell from unrolling the array. Without that you could end up with one long array in the output.
I'm trying to append a column by seeing if the value of the column from CSV file2 is contained in CSV file1.
I have a CSV file1 (test1.csv):
csv1ColumnOne,csv1ColumnTwo
1,dF3aWv
2,
3,ka21p
4,NAE31
5,dsafl
6,nv02k
7,qng02
8,xcw3r
9,dF3aW
I have a CSV file2 (test2.csv):
csv2ColumnOne,csv2ColumnTwo
bbetfe,DF3AW
asdf,dsafl
qwer,
zxcv,NAE31
poiu,nbrwp1
Given the following code...
$hashTable = #()
Import-Csv C:\path\test1.csv | ForEach-Object {
$hashTable[$_.csv1ColumnOne] = $_.csv1ColumnTwo
}
(Import-Csv C:\path\test2.csv) |
Select-Object -Property *, #{n='csv1ColumnThree';e={
if ($hashTable.ContainsKey($_.csv2ColumnTwo)) {
$_.csv2ColumnTwo
} elseif (-not ($_.csv2ColumnTwo)) {
'No value found from csv file2'
} else {
'No value found from csv file1'
}
}} | Export-Csv "C:\path\testresults.csv" -NoType
The results look like this:
csv2ColumnOne,csv2ColumnTwo,csv1ColumnThree
bbetfe,DF3AW,"No value found from csv file1"
asdf,dsafl,dsafl
qwer,,No value found from csv file2
zxcv,NAE31,NAE31
poiu,nbrwp1,"No value found from csv file1"
When instead it should look like this:
csv2ColumnOne,csv2ColumnTwo,csv1ColumnThree
bbetfe,DF3AW,dF3aW
asdf,dsafl,dsafl
qwer,,"No value found from csv file2"
zxcv,NAE31,NAE31
poiu,nbrwp1,"No value found from csv file1"
The reason I see bbetfe,DF3AW,"No value found from csv file1" ins't bbetfe,DF3AW,dF3aW is because of the case sensitivity of the value. Anyway to ignore the case sensitivity with alpha-numeric values?
Lookups with ContainsKey() already are case-insensitive. You're just using the wrong data structure, and using it in the wrong way, too.
If you want to look up a key in a hashtable you need to actually use the data you want to look up as the key of the hashtable:
$hashTable[$_.csv1ColumnTwo] = $_.csv1ColumnOne
For looking up something in the values of a hashtable use ContainsValue().
However, since you just want to check if the second column of the first CSV contains a the value from the second column of the second CSV you don't need a hashtable in the first place. A simple array will suffice.
$list = Import-Csv 'C:\path\test1.csv' | Select-Object -Expand csv1ColumnTwo
Import-Csv 'C:\path\test2.csv' |
Select-Object -Property *, #{n='csv1ColumnThree';e={
if ($list -contains $_.csv2ColumnTwo) {
$_.csv2ColumnTwo
} elseif (-not ($_.csv2ColumnTwo)) {
'No value found from csv file2'
} else {
'No value found from csv file1'
}
}} | Export-Csv 'C:\path\testresults.csv' -NoType
If you don't want empty strings "found" in the second CSV simply exclude the element from $list:
$list = Import-Csv 'C:\path\test1.csv' |
Select-Object -Expand csv1ColumnTwo |
Where-Object { $_ } # allow only non-empty values
Not every problem is a nail, so don't try to fix everything with a hammer.
To avoid having to convert the string to lower case, just use the -icontains comparison operator (the "i" means case insenstive comparison):
So instead of
If ($hashTable.ContainsKey($_.csv2ColumnTwo)){
try this:
If ($hashTable.keys -icontains $_.csv2ColumnTwo){
can you just make them all lowercase?
$a = ipcsv 'C:\path\test1.csv'
$a | % {$_.csv1columntwo = $_.csv1columntwo.tolower()}
$a
$b = ipcsv 'C:\path\test2.csv'
$b | % {$_.csv2ColumnOne = $_.csv2ColumnOne.tolower(); $_.csv2ColumnTwo = $_.csv2ColumnTwo.tolower()}
$b
Ansgar basically had the right answer, but with a bug. it was printing the row in the second file as qwer,, when instead it should have printed qwer,,No value found from csv file2. There is another condition that needed to be added in the first if statement as shown below.
$list = Import-Csv 'C:\path\test1.csv' | Select-Object -Expand csv1ColumnTwo
Import-Csv 'C:\path\test2.csv' |
Select-Object -Property *, #{n='csv1ColumnThree';e={
if (($list -contains $_.csv2ColumnTwo) -and ($_.csv2ColumnTwo)) {
$_.csv2ColumnTwo
} elseif (-not ($_.csv2ColumnTwo)) {
'No value found from csv file2'
} else {
'No value found from csv file1'
}
}} | Export-Csv 'C:\path\testresults.csv' -NoType
The empty values in the 2nd file were being checked as true, so the elseif was never being reached.
I have a powershell script to find particular instances and then export them to CSV. Here's an example of the way the code works
$items = "Hello Tim", "Hola Bob", "Hello Susan"
$filter = $items | Select-String -Pattern "Hello"
$filter | Select-Object Line, Matches | Export-Csv "C:\log.csv"
Invoke-Item "C:\log.csv"
When I run the Select-Object in PS, it's nicely formatted info like this:
However, when I export to CSV, it exports the whole object and writes it as the following string: System.Text.RegularExpressions.Match[]
How can I get it to export just the first match or a listing of all matches into a single field when writing to CSV?
Here is one way using a PSObject:
$items = "Hello Tim", "Hola Bob", "Hello Susan"
$filter = $items | Select-String -Pattern "Hello"
$filter | % {New-Object PSObject -property #{
Line = $_.Line
Matches = $_.Matches.Groups[0].Value}
} | Export-Csv "C:\log.csv" -NoTypeInformation
Quickly note that Matches is an array which may create issues exporting to a csv.
Try joining the array into a string with a chosen delimiter. I used "::" in my example.
$filter | Select Line, #{Expression={$_.Matches -join "::"}; Label="Matches"} | Export-Csv "C:\log.csv"