Firstly i would apologize if i was asking a very noob question, i'm a beginner powershell users and i don't have much exp in scripting, i have some issue to output html report using powershell. I wanted to change the colour of each row according to the utilization % wheter it is above or below 90%. However i only manage to change the colour of the first row.
i have 2 file which i use to output the report.
output from agg.csv
Aggregate_01_Data1,10.01TB,90%,96.93TB,online,86.92TB,
Aggregate_02_Data1,9.03TB,91%,96.93TB,online,87.90TB,
root_Aggregate_01,17.85GB,85%,368.4GB,online,350.6GB,
root_Aggregate_02,17.85GB,95%,368.4GB,online,350.6GB,
output from aggregate.csv
Aggregate,Size,Available,Used,Utilized,State
Aggregate_01_Data1,96.93TB,10.01TB,86.92TB,90,online
Aggregate_02_Data1,96.93TB,9.03TB,87.90TB,91,online
root_Aggregate_01,368.4GB,17.85GB,350.6GB,85,online
root_Aggregate_02,368.4GB,17.85GB,350.6GB,95,online
my code to generate the html report.
$a1=gc $TPATH\agg.csv
$b1 =$a1
for ($i1 = 0;$i1 -lt $a1.count;$i1++)
{
$c1=$b1[$i1].split(",",10)
$c = $c1[2].Length
$c = $c - 1
$d1=$c1[0] + "," + $c1[3] + "," + $c1[1] + "," + $c1[5] + "," + $c1[2].remove($c) + "," + $c1[4]
$d1 | Out-File ("$TPATH\aggregate.csv") -Encoding UTF8 -append
$bodyB = (import-csv $TPATH\aggregate.csv) |Select Aggregate,Size,Available,Used,Utilized,State -OutVariable ag | ConvertTo-Html -Fragment |
foreach { if($ag.Utilized -gt 90) {$_ -replace "<tr>", "<tr bgcolor=#FE0808>"}else{$_ -replace "<tr>", "<tr bgcolor=#E5FCA1>"}} |Out-String
}
convertTo-HTML -Body "<br> $bodyB" |out-file $PATH\Netapp.html
my final output look like this
how can i get the output to be like this, i want to have the colour change to Red if the threshold is above 90% in the Utilized column and Green if the threshold is below 90%.
Thank you in advance for those whom willing to help, really appreciate your help in this.
Thank You.
Replace -OutVariable ag with -PipelineVariable ag, so that $ag in your foreach (ForEach-Object) script block refers only to the single input object at hand.
The purpose of the common -PipelineVariable (-pv) parameter is to store the current object being output in a self-chosen variable, so that it can be accesses in a script block later in the same pipeline.
By contrast, the purpose of the common -OutVariable (-ov) parameter is to collect all output from a command in a self-chosen variable, for later processing in a different, subsequent statement, not in the same pipeline.
If you do access such a variable in a script block in the same pipeline, you'll see the output objects accumulated so far (which explains the behavior you saw - see below).
As for what you tried:
Due to the mistaken use of -OutVariable ag, your script block saw the Import-Csv output objects collected so far in variable $ag rather than the output object at hand.
Thus, starting with the second output object, $ag was in effect an array of objects, and accessing its . Utilized property resulted in member-access enumeration, and therefore returning an array of utilization values.
Since -gt, like all PowerShell comparison operators, acts as a filter when its LHS is an array, $ag.Utilized -gt 90 started to generate a return value (the subarray of matching values) as soon as at least one element in the array of values satisfied this condition.
In a Boolean context such as an if-statement conditional, a non-empty array is interpreted as $true.
Therefore, in effect ($ag.Utilized -gt 90) returned $true as soon as the first value above 90 was part of the return array, and invariably remained true.
Related
I'm out of luck finding information...
This powershell script collecting cert info in LocalMachine:
$cert_days = get-childitem cert:LocalMAchine -recurse |
select #{Name="{#CERTINFO}"; Expression={($_.FriendlyName)}} |
Sort "{#CERTINFO}"
write-host "{"
write-host " `"data`":`n"
convertto-json $cert_days
write-host
write-host "}"
I can't exclude Nulls or empty items like " ".
Using -ne $Null i get boolean results like true or false...
I would appreciate to hear Yours advice how to eliminate nulls or empty entries
To exclude empty entries, you could add a filter to remove those, preferably before the Sort-Object call., e.g.
$certs = ls Cert:\LocalMachine\ -Recurse |
Select #{Name = '{#CertInfo}'; Expression = {$_.FriendlyName}} |
Where { $_.'{#CertInfo}' } |
Sort '{#CertInfo}'
Robert Westerlund's helpful answer shows one way of filtering out $null and '' (empty-string) values, using the Where-Object cmdlet, which coerces the output from the script block to a Boolean, causing both $null and '' evaluate to $False and thus causing them to be filtered out.
This answer shows an alternative approach and discusses other aspects of the question.
tl;dr:
#{
data = #((Get-ChildItem -Recurse Cert:\LocalMachine).FriendlyName) -notlike '' |
Sort-Object | Select-Object #{ n='{#CERTINFO}'; e={ $_ } }
} | ConvertTo-Json
Using -ne $Null i get boolean results like true or false...
You only get a Boolean if the LHS is a scalar rather than an array - in the case of an array, the matching array elements are returned.
To ensure that the LHS (or any expression or command output) is an array, wrap it in #(...) the array-subexpression operator (the following uses PSv3+ syntax ):
#((Get-ChildItem -Recurse Cert:\LocalMachine).FriendlyName) -notlike ''
Note the use of -notlike '' to weed out both $null and '' values: -notlike forces the LHS to a string, and $null is converted to ''.
By contrast, if you wanted to use -ne $null, you'd have to use -ne '' too so as to also eliminate empty strings (though, in this particular case you could get away with just -ne '', because ConvertTo-Json would simply ignore $null values in its input).
Calling .FriendlyName on the typically array-valued output of Get-ChildItem directly is a PSv3+ feature called member-access enumeration: the .FriendlyName property access is applied to each element of the array, and the results are returned as a new array.
Filtering and sorting the values before constructing the wrapper objects with the {#CERTINFO} property not only simplifies the command, but is also more efficient.
Further thoughts:
Do not use Write-Host to output data: Write-Host bypasses PowerShell's (success) output stream; instead, use Write-Output, which you rarely need to call explicitly however, because its use is implied.
Instead of write-host "{", use write-output "{" or - preferably - simply "{" by itself.
PowerShell supports multi-line strings (see Get-Help about_Quoting_Rules), so there's no need to output the result line by line:
#"
{
"data":
$(<your ConvertTo-Json pipeline>)
}
"#
However, given that you're invoking ConvertTo-Json anyway, it's simpler to provide the data wrapper as a PowerShell object (in the simplest form as a hashtable) to ConvertTo-Json, as shown above.
I'm trying to dynamically parse & build-up a data structure of some incoming JSON files I'm to be supplied with (that'll be in non-standard structure) via Powershell to then process the data in those files & hand them over to the next step.
As part of that, I'm trying to build up the data structure of the JSON file into essentially a list of of data-paths for me to parse through & grab the data out of, so that I can cope with arrays, nested JSON objects and so on. So far so good.
Where I fall into some sort of Powershell peculiarity is in handling 2+ levels of depth via a variable. Let me give you a nice code-block to demonstrate the problem...
# Generate a Quick JSON file with different data types & levels
[object]$QuickJson = #'
{
"Name" : "I am a JSON",
"Version" : "1.2.3.4",
"SomeBool" : true,
"NULLValue" : null,
"ArrayOfVersions" : [1.0,2.0,3.0],
"MyInteger" : 69,
"NestedJSON" : {
"Version" : 5.0,
"IsReady" : false
},
"DoubleNestedJSON" : {
"FirstLevel" : 1,
"DataValue" : "I am at first nested JSON level!",
"Second_JSON_Level" : {
"SecondLevel" : 2,
"SecondDataValue" : "I am on the 2nd nested level"
}
}
}
'#
# Import our JSON file into Powershell
[object]$MyPSJson = ConvertFrom-Json -InputObject $QuickJson
# Two quick string variables to access our JSON data paths
[string]$ShortJsonPath = "Name"
[string]$NestedJsonPath = "NestedJson.Version"
# Long string to access a double-nested JSON object
[string]$LongNestedJsonPath = "DoubleNestedJSON.Second_JSON_Level.SecondDataValue"
# Both of these work fine
Write-Host ("JSON Name (Direct) ==> " + $MyPSJson.Name)
Write-Host ("JSON Name (via Variable) ==> " + $MyPSJson.$ShortJsonPath)
# The following way to access a single nested Json Path works fine
Write-Host ("Nested JSON Version (via direct path) ==> " + $MyPSJson.NestedJson.Version)
# And THIS returns an empty line / is where I fall afoul of something in Powershell
Write-Host ("Nested JSON Version (via variable) ==> " + $MyPSJson.$NestedJsonPath)
# Other things I tried -- all returning an empty line / failing in effect
Write-Host ("Alternate Nested JSON Version ==> " + $($MyPSJson.$NestedJsonPath))
Write-Host ("Alternate Nested JSON Version ==> " + $MyPSJson.$($NestedJsonPath))
Write-Host ("Alternate Nested JSON Version ==> " + $($MyPSJson).$($NestedJsonPath))
# Similarly, while THIS works...
$MyPSJson | select-object -Property NestedJSON
# This will fail / return me nothing
$MyPSJson | select-object -Property NestedJSON.Version
... in doing a bunch of research around this, I came across a suggestion to transform this into a Hashtable -- but that has the same problem, sadly. So with the above code-snippet, the following will transform the JSON object into a hashtable.
# Same problem with a hash-table if constructed from the JSON file...
[hashtable]$MyHash = #{}
# Populate $MyHash with the data from our quickie JSON file...
$QuickJson | get-member -MemberType NoteProperty | Where-Object{ -not [string]::IsNullOrEmpty($QuickJson."$($_.name)")} | ForEach-Object {$MyHash.add($_.name, $QuickJson."$($_.name)")}
# ... and even then -- $MyHash."$($NestedJsonPath)" -- fails, while a single level deep string works fine in the variable! :(
So it's pretty clear that I'm running into "something" of a Powershell internal logic problem, but I can't get Powershell to be overly helpful in WHY that is. Adding a '-debug' or similar in an attempt to increase verbosity hasn't helped shed light on this.
I suspect it's something akin to the items raised in this article here ( https://blogs.technet.microsoft.com/heyscriptingguy/2011/10/16/dealing-with-powershell-hash-table-quirks/ ) but just specific with variables.
I've not had any luck in finding anything obvious in the Powershell language specification (3.0 still being the latest from here as far as I can tell -- https://www.microsoft.com/en-usdownload/details.aspx?id=36389 ) either. It may be in there, I may just miss it.
Any advice in how to get Powershell to play nice with this would be greatly appreciated. I'm not sure how / why Powershell is fine with a simple string but seems to have issues with a 'something.somethingelse' type string here.
Thank you.
Further notes & addenda to the original:
It seems there are several issues to attack. One is "dealing with a single nested level". The "quick fix" for that seems to be using "Invoke-Expression" to resolve the statement, so for instance (IMPORTANT - take note of the back-tick with the first variable!):
iex "`$MyPSJson.$NestedJsonPath"
That use of Invoke-Expression also works with multi-nested situations:
iex "`$MyPSJson.$LongNestedJsonPath"
An alternative approach that was mentioned is the use of multiple select statements ... but I've not been able to get that to work with multi-nested objects (Powershell seems to not resolve those properly for some reason).
So for instance in this scenario:
($MyComp | select $_.DoubleNestedJSON | select FirstLevel)
Powershell returns
FirstLevel
----------
... instead of the actual data value. So - for now, it seems that selects won't work with multi-level nested objects due to Powershell apparently not resolving them?
When you write something like
$MyPSJson.Name
this will attempt to retrieve the member named Name from the object $MyPSJson. If there is no such member, you'll get $null.
Now, when you do that with variables for the member name:
$MyPSJson.$ShortJsonPath
this works pretty much identical in that the member with the name stored in $ShortJsonPath is looked up and its value retrieved. No surprises here.
When you try that with a member that doesn't exist on the object, such as
$MyPSJson.$NestedJsonPath
# equivalent to
# $MyPSJson.'NestedJSON.Version'
you'll get $null, as detailed before. The . operator will only ever access a member of the exact object that is the result of its left-hand-side expression. It will never go through a member hierarchy in the way you seem to expect it to do. To be frank, I'm not aware of a language that works that way.
The reason it works with Invoke-Expression is, that you effectively converting the $NestedJsonPath string into part of an expression resulting in:
$MyPSJson.NestedJSON.Version
which Invoke-Expression then evaluates.
You can, of course, define your own function that works that way (and I'd much prefer that instead of using Invoke-Expression, a cmdlet that should rarely, if ever, used (heck, it's eval for PowerShell – few languages with eval advocate its use)):
function Get-DeepProperty([object] $InputObject, [string] $Property) {
$path = $Property -split '\.'
$obj = $InputObject
$path | %{ $obj = $obj.$_ }
$obj
}
PS> Get-DeepProperty $MyPSJson NestedJson.Version
5,0
You could even make it a filter, so you can use it more naturally on the pipeline:
filter Get-DeepProperty([string] $Property) {
$path = $Property -split '\.'
$obj = $_
$path | %{ $obj = $obj.$_ }
$obj
}
PS> $MyPSJson | Get-DeepProperty nestedjson.version
5,0
Why this doesn't work
When you provide the properties that you'd like within a string, like this
[string]$NestedJsonPath = "NestedJson.Version"
Powershell looks for a property called NestedJSon.Version. It's not actually traversing the properties, but looking for a string literal which contains a period. In fact, if I add a property like that to your JSON like so.
[object]$QuickJson = #'
{
"Name" : "I am a JSON",
"Version" : "1.2.3.4",
"SomeBool" : true,
"NULLValue" : null,
"ArrayOfVersions" : [1.0,2.0,3.0],
"MyInteger" : 69,
"NestedJSON.Version" : 69,
"NestedJSON" : {
"Version" : 5.0,
"IsReady" : false
}
}
I now get a value back, like so:
>$MyPSJson.$NestedJsonPath
69
The best way to get your values back is to use two separate variables, like this.
$NestedJson = "NestedJson"
$property = "Version"
>$MyPSJson.$NestedJson.$property
5.0
Or, alternatively, you could use select statements, as seen in the original answer below.
$MyPSJson | select $_.NestedJSON | select Version
Version
-------
1.2.3.4
If you use multiple Select-Object statements, they'll discard the other properties and allow you to more easily drill down to the value you'd like.
I followed Joey's filter example. However, I found it did not support accessing arrays.
Sharing the code that I got to work for this. Hopefully it will help others as well. Awesome thread!
filter Get-DeepProperty([string] $Property) {
$path = $Property -split '\.'
$obj = $_
foreach($node in $path){
if($node -match '.*\[\d*\]'){
$keyPieces = $node -split ('\[')
$arrayKey = $keyPieces[0]
$arrayIndex = $keyPieces[1] -replace ('\]','')
$obj = $obj.$arrayKey[$arrayIndex]
} else {
$obj = $obj.$node
}
}
$obj
}
Example usage:
$path = "nested.nestedtwo.steps[2]"
$payload | Get-DeepProperty $path
I had the same problem, so I wrote a function that does the trick.
It enables accessing any level of the json by variable path (string):
function getNestedJsonValue() {
param(
[Parameter(Mandatory = $true, ValueFromPipeline)] [PSCustomObject] $inputObj,
[Parameter(Mandatory = $true)] [string] $valuePath
)
if (($valuePath -eq $null) -or ($valuePath.length -eq 0) -or ($inputObj -eq $null)) {
return $inputObj
}
[System.Array] $nodes = "$valuePath" -split '\.'
foreach ($node in $nodes) {
if (($node -ne $null) -and ($node.length -gt 0) -and ($inputObj -ne $null)) {
$inputObj = $inputObj.$node
} else {
return $inputObj
}
}
return $inputObj
}
Usage: getNestedJsonValue -valuePath $nestedValuePath -inputObj $someJson
Pipe usage: $someJson | getNestedJsonValue -valuePath $nestedValuePath
An example nestedValuePath would be $nestedValuePath="some.nested.path"
Credit to wOxxOm for getting things on the right track.
Invoke-Expression does seem to work perfectly for this situation (if somewhat expensive, but that's fine in my personal example & situation), and it can cope with multiple levels of nesting.
So as examples for the above code snippet, the following will resolve just fine (Key point - pay attention to the initial back-tick. That caught me off guard):
Write-Host ("Single level JSON test ==> " + (iex "`$MyPSJson.$NestedJsonPath"))
Write-Host ("Double level JSON test ==> " + (iex "`$MyPSJson.$LongNestedJsonPath"))
That'll return our desired results:
Single level JSON test ==> 5.0
Double level JSON test ==> I am on the 2nd nested level
FoxDeploy's answer of using multi-level selects doesn't seem to work with 2+ levels of nesting, unfortunately for some bizarre reason.
Using:
($MyPSJson | select $_.DoubleNestedJSON | select FirstLevel)
We get the following back from Powershell:
FirstLevel
----------
... it seems that Powershell doesn't resolve nested objects in its entirety? We get a similar results if we intentionally use something that doesn't exist:
($MyPSJson | select $_.DoubleNestedJSON | select Doesnotexist)
... also simply returns:
Doesnotexist
------------
So - for now - it seems as if "Invoke-Expression" works most reliably (and most easily, as it's just a case of handing it a variable with the path'ed string).
I still can't explain the WHY of any of this so far (since I've used 'dotwalk'-ing with multiple variables through arrays quite happily), but at least there's a solution for now ... and that is Invoke-Expression !
The best (/least bad?) explanations for Invoke-Expression I've found so far are here (Microsoft's own description of the cmdlet doesn't really make a great job of hinting that it'd help in situations such as this):
http://ss64.com/ps/invoke-expression.html
https://www.adminarsenal.com/powershell/invoke-expression/
I'm trying to dynamically parse & build-up a data structure of some incoming JSON files I'm to be supplied with (that'll be in non-standard structure) via Powershell to then process the data in those files & hand them over to the next step.
As part of that, I'm trying to build up the data structure of the JSON file into essentially a list of of data-paths for me to parse through & grab the data out of, so that I can cope with arrays, nested JSON objects and so on. So far so good.
Where I fall into some sort of Powershell peculiarity is in handling 2+ levels of depth via a variable. Let me give you a nice code-block to demonstrate the problem...
# Generate a Quick JSON file with different data types & levels
[object]$QuickJson = #'
{
"Name" : "I am a JSON",
"Version" : "1.2.3.4",
"SomeBool" : true,
"NULLValue" : null,
"ArrayOfVersions" : [1.0,2.0,3.0],
"MyInteger" : 69,
"NestedJSON" : {
"Version" : 5.0,
"IsReady" : false
},
"DoubleNestedJSON" : {
"FirstLevel" : 1,
"DataValue" : "I am at first nested JSON level!",
"Second_JSON_Level" : {
"SecondLevel" : 2,
"SecondDataValue" : "I am on the 2nd nested level"
}
}
}
'#
# Import our JSON file into Powershell
[object]$MyPSJson = ConvertFrom-Json -InputObject $QuickJson
# Two quick string variables to access our JSON data paths
[string]$ShortJsonPath = "Name"
[string]$NestedJsonPath = "NestedJson.Version"
# Long string to access a double-nested JSON object
[string]$LongNestedJsonPath = "DoubleNestedJSON.Second_JSON_Level.SecondDataValue"
# Both of these work fine
Write-Host ("JSON Name (Direct) ==> " + $MyPSJson.Name)
Write-Host ("JSON Name (via Variable) ==> " + $MyPSJson.$ShortJsonPath)
# The following way to access a single nested Json Path works fine
Write-Host ("Nested JSON Version (via direct path) ==> " + $MyPSJson.NestedJson.Version)
# And THIS returns an empty line / is where I fall afoul of something in Powershell
Write-Host ("Nested JSON Version (via variable) ==> " + $MyPSJson.$NestedJsonPath)
# Other things I tried -- all returning an empty line / failing in effect
Write-Host ("Alternate Nested JSON Version ==> " + $($MyPSJson.$NestedJsonPath))
Write-Host ("Alternate Nested JSON Version ==> " + $MyPSJson.$($NestedJsonPath))
Write-Host ("Alternate Nested JSON Version ==> " + $($MyPSJson).$($NestedJsonPath))
# Similarly, while THIS works...
$MyPSJson | select-object -Property NestedJSON
# This will fail / return me nothing
$MyPSJson | select-object -Property NestedJSON.Version
... in doing a bunch of research around this, I came across a suggestion to transform this into a Hashtable -- but that has the same problem, sadly. So with the above code-snippet, the following will transform the JSON object into a hashtable.
# Same problem with a hash-table if constructed from the JSON file...
[hashtable]$MyHash = #{}
# Populate $MyHash with the data from our quickie JSON file...
$QuickJson | get-member -MemberType NoteProperty | Where-Object{ -not [string]::IsNullOrEmpty($QuickJson."$($_.name)")} | ForEach-Object {$MyHash.add($_.name, $QuickJson."$($_.name)")}
# ... and even then -- $MyHash."$($NestedJsonPath)" -- fails, while a single level deep string works fine in the variable! :(
So it's pretty clear that I'm running into "something" of a Powershell internal logic problem, but I can't get Powershell to be overly helpful in WHY that is. Adding a '-debug' or similar in an attempt to increase verbosity hasn't helped shed light on this.
I suspect it's something akin to the items raised in this article here ( https://blogs.technet.microsoft.com/heyscriptingguy/2011/10/16/dealing-with-powershell-hash-table-quirks/ ) but just specific with variables.
I've not had any luck in finding anything obvious in the Powershell language specification (3.0 still being the latest from here as far as I can tell -- https://www.microsoft.com/en-usdownload/details.aspx?id=36389 ) either. It may be in there, I may just miss it.
Any advice in how to get Powershell to play nice with this would be greatly appreciated. I'm not sure how / why Powershell is fine with a simple string but seems to have issues with a 'something.somethingelse' type string here.
Thank you.
Further notes & addenda to the original:
It seems there are several issues to attack. One is "dealing with a single nested level". The "quick fix" for that seems to be using "Invoke-Expression" to resolve the statement, so for instance (IMPORTANT - take note of the back-tick with the first variable!):
iex "`$MyPSJson.$NestedJsonPath"
That use of Invoke-Expression also works with multi-nested situations:
iex "`$MyPSJson.$LongNestedJsonPath"
An alternative approach that was mentioned is the use of multiple select statements ... but I've not been able to get that to work with multi-nested objects (Powershell seems to not resolve those properly for some reason).
So for instance in this scenario:
($MyComp | select $_.DoubleNestedJSON | select FirstLevel)
Powershell returns
FirstLevel
----------
... instead of the actual data value. So - for now, it seems that selects won't work with multi-level nested objects due to Powershell apparently not resolving them?
When you write something like
$MyPSJson.Name
this will attempt to retrieve the member named Name from the object $MyPSJson. If there is no such member, you'll get $null.
Now, when you do that with variables for the member name:
$MyPSJson.$ShortJsonPath
this works pretty much identical in that the member with the name stored in $ShortJsonPath is looked up and its value retrieved. No surprises here.
When you try that with a member that doesn't exist on the object, such as
$MyPSJson.$NestedJsonPath
# equivalent to
# $MyPSJson.'NestedJSON.Version'
you'll get $null, as detailed before. The . operator will only ever access a member of the exact object that is the result of its left-hand-side expression. It will never go through a member hierarchy in the way you seem to expect it to do. To be frank, I'm not aware of a language that works that way.
The reason it works with Invoke-Expression is, that you effectively converting the $NestedJsonPath string into part of an expression resulting in:
$MyPSJson.NestedJSON.Version
which Invoke-Expression then evaluates.
You can, of course, define your own function that works that way (and I'd much prefer that instead of using Invoke-Expression, a cmdlet that should rarely, if ever, used (heck, it's eval for PowerShell – few languages with eval advocate its use)):
function Get-DeepProperty([object] $InputObject, [string] $Property) {
$path = $Property -split '\.'
$obj = $InputObject
$path | %{ $obj = $obj.$_ }
$obj
}
PS> Get-DeepProperty $MyPSJson NestedJson.Version
5,0
You could even make it a filter, so you can use it more naturally on the pipeline:
filter Get-DeepProperty([string] $Property) {
$path = $Property -split '\.'
$obj = $_
$path | %{ $obj = $obj.$_ }
$obj
}
PS> $MyPSJson | Get-DeepProperty nestedjson.version
5,0
Why this doesn't work
When you provide the properties that you'd like within a string, like this
[string]$NestedJsonPath = "NestedJson.Version"
Powershell looks for a property called NestedJSon.Version. It's not actually traversing the properties, but looking for a string literal which contains a period. In fact, if I add a property like that to your JSON like so.
[object]$QuickJson = #'
{
"Name" : "I am a JSON",
"Version" : "1.2.3.4",
"SomeBool" : true,
"NULLValue" : null,
"ArrayOfVersions" : [1.0,2.0,3.0],
"MyInteger" : 69,
"NestedJSON.Version" : 69,
"NestedJSON" : {
"Version" : 5.0,
"IsReady" : false
}
}
I now get a value back, like so:
>$MyPSJson.$NestedJsonPath
69
The best way to get your values back is to use two separate variables, like this.
$NestedJson = "NestedJson"
$property = "Version"
>$MyPSJson.$NestedJson.$property
5.0
Or, alternatively, you could use select statements, as seen in the original answer below.
$MyPSJson | select $_.NestedJSON | select Version
Version
-------
1.2.3.4
If you use multiple Select-Object statements, they'll discard the other properties and allow you to more easily drill down to the value you'd like.
I followed Joey's filter example. However, I found it did not support accessing arrays.
Sharing the code that I got to work for this. Hopefully it will help others as well. Awesome thread!
filter Get-DeepProperty([string] $Property) {
$path = $Property -split '\.'
$obj = $_
foreach($node in $path){
if($node -match '.*\[\d*\]'){
$keyPieces = $node -split ('\[')
$arrayKey = $keyPieces[0]
$arrayIndex = $keyPieces[1] -replace ('\]','')
$obj = $obj.$arrayKey[$arrayIndex]
} else {
$obj = $obj.$node
}
}
$obj
}
Example usage:
$path = "nested.nestedtwo.steps[2]"
$payload | Get-DeepProperty $path
I had the same problem, so I wrote a function that does the trick.
It enables accessing any level of the json by variable path (string):
function getNestedJsonValue() {
param(
[Parameter(Mandatory = $true, ValueFromPipeline)] [PSCustomObject] $inputObj,
[Parameter(Mandatory = $true)] [string] $valuePath
)
if (($valuePath -eq $null) -or ($valuePath.length -eq 0) -or ($inputObj -eq $null)) {
return $inputObj
}
[System.Array] $nodes = "$valuePath" -split '\.'
foreach ($node in $nodes) {
if (($node -ne $null) -and ($node.length -gt 0) -and ($inputObj -ne $null)) {
$inputObj = $inputObj.$node
} else {
return $inputObj
}
}
return $inputObj
}
Usage: getNestedJsonValue -valuePath $nestedValuePath -inputObj $someJson
Pipe usage: $someJson | getNestedJsonValue -valuePath $nestedValuePath
An example nestedValuePath would be $nestedValuePath="some.nested.path"
Credit to wOxxOm for getting things on the right track.
Invoke-Expression does seem to work perfectly for this situation (if somewhat expensive, but that's fine in my personal example & situation), and it can cope with multiple levels of nesting.
So as examples for the above code snippet, the following will resolve just fine (Key point - pay attention to the initial back-tick. That caught me off guard):
Write-Host ("Single level JSON test ==> " + (iex "`$MyPSJson.$NestedJsonPath"))
Write-Host ("Double level JSON test ==> " + (iex "`$MyPSJson.$LongNestedJsonPath"))
That'll return our desired results:
Single level JSON test ==> 5.0
Double level JSON test ==> I am on the 2nd nested level
FoxDeploy's answer of using multi-level selects doesn't seem to work with 2+ levels of nesting, unfortunately for some bizarre reason.
Using:
($MyPSJson | select $_.DoubleNestedJSON | select FirstLevel)
We get the following back from Powershell:
FirstLevel
----------
... it seems that Powershell doesn't resolve nested objects in its entirety? We get a similar results if we intentionally use something that doesn't exist:
($MyPSJson | select $_.DoubleNestedJSON | select Doesnotexist)
... also simply returns:
Doesnotexist
------------
So - for now - it seems as if "Invoke-Expression" works most reliably (and most easily, as it's just a case of handing it a variable with the path'ed string).
I still can't explain the WHY of any of this so far (since I've used 'dotwalk'-ing with multiple variables through arrays quite happily), but at least there's a solution for now ... and that is Invoke-Expression !
The best (/least bad?) explanations for Invoke-Expression I've found so far are here (Microsoft's own description of the cmdlet doesn't really make a great job of hinting that it'd help in situations such as this):
http://ss64.com/ps/invoke-expression.html
https://www.adminarsenal.com/powershell/invoke-expression/
I have large CSV files that 0.5-2gb+ files I am trying to import with Powershell.
Data looks like so:
Name, Date, Value
"Joe, John", 2016-08-01, "value"
"Smith, Jane", 2016-08-01, "value"
...
I have this function
$elapsed = [System.Diagnostics.Stopwatch]::StartNew()
$reader = new-object System.IO.StreamReader($csv)
while (($line = $reader.ReadLine()) -ne $null) {
# Use RegEx to only split on (,) outside quotes and remove quoted strings
$row = ($line -split ',(?=(?:[^"]|"[^"]*")*$)').Replace("`"","")
# Row Indicator
$i++;
if (($i % 50000) -eq 0) {
Write-Host "$i rows have been processed in $($elapsed.Elapsed.ToString())."
}
}
Splitting the line by a comma "," works perfect as I get ~16K a second, but I need to only split outside of any quotes, so I implemented the regular expression, however the performance tanks to 900 rows a second.
I am looking for a more efficient way to loop through a CSV file that is comma delimited but has commas in the quotes that need to be excluded.
Import-Csv, as noted in the comments above, does not load everything into memory unless you ask it to. Like the example in the question it implements a stream reader and pushes the content it's read off to the output pipeline.
You will see significant memory usage if you do something like this:
$var = Import-Csv thefile.csv
After all, the content of the CSV has to go somewhere.
Whereas if you do something with the output pipeline there's less impact. e.g.
Import-Csv thefile.csv | ForEach-Object {
Do-Something
}
Finally, Import-Csv really doesn't work for you I have a CSV reader class along with a side-by-side implementation of Import-Csv called Indented.Text.Csv on github. This implementation provides a public class with a number of features I needed so I could process CSV files very quickly.
I'm able to fill a datatable, $devices from a MySql query (for example):
PS C:\Users\MKANET\Desktop\dev\lab> $devices.table[10]
name ip mac vendor
---- -- --- ------
ComputerTest 10.51.18.6 fd1969ff4cb9 HewlettP
I'd like to convert that datatable type into a custom PSObject; where, the mac column is converted to PSObject NoteProperty name "Mac-Address"; and, respective values converted to 00-00-00-00 string format:
PS C:\Users\MKANET\Desktop\dev\lab> $devices[1]
name ip MAC-Address vendor
------- -- ----------- ------------
ComputerTest 10.51.18.6 fd-19-69-ff-4c-b9 HewlettP
Considering these datatables may be relatively large (may have a couple hundred rows), I need the fastest processing method to do this in Powershell.
If it'll make the converting/modifying process significantly faster, I'd be happy with keeping $Devices as a Datatable; only modifying/processing the values for the "mac" column to: 00-00-00-00-00-00 text format, permanently.
Using Select-Object, it should auto-convert a DataRow / DataTable to a PSCustomObject in Powershell. Here's what I would run to do the conversion to PSCustomObject and handle the MAC address formatting in one line. This can be applied to a single row at specific index (like the example code) or against the entire DataTable to convert to an array of PSObjects.
The MAC address formatting removes any non-alphanumeric characters in the original, casts to lowercase, then inserts the hyphens at the appropriate indexes.
$devices.table[10] | Select-Object name,ip,#{N="MAC-Address";E={((((($_.mac -replace '[^a-zA-Z0-9]','').ToLower().insert(2,"-")).insert(5,"-")).insert(8,"-")).insert(11,"-")).insert(14,"-")}},vendor
Just to be clear for some readers: within PowerShell, a DataTable is a PS object, just one with a [System.Data.DataTable] type. This type (class) provides a number of methods for manipulating the object such as adding a new column (shown in the answer provided by #e-z-hart).
Converting an object to a generic 'PSObject' effectively means stripping away all the additional methods, etc. associated with the [DataTable] class and leaving just a simple object. You could do this as follows:
$obj = $devices.table[10] | foreach {
$macaddress = ($_.mac -replace '(..)','$1-').TrimEnd("-")
New-Object -TypeName PSObject -Property #{
"name = [System.String] $_.name
"ip" = [System.Net.IPAddress] $_.ip
"mac-address" = [System.String] $macaddress
"vendor" = [System.String] $_.vendor
}
}
Hopefully the -replace operator with a regex is a little more elegant (cf. answer from #e-z-hart) than having to chop the mac address up with substring functions. Someone may be able to improve on the regex so that it isn't necessary to Trim the trailing "-" at the end.
This should be pretty fast for the "couple of hundred" rows you mention.
Note that, for bonus points, the 'ip' column is converted to a [System.Net.IPAddress] type, which includes a ToString() method to return the address in the dotted format we usually see them in. I suppose this is overkill so you'd probably want this to be a [System.String] throughout.
Finally, if you really want the MAC-Address column to be a NoteProperty, then I'd exclude that column from the list of properties in the New-Object cmdlet and use Add-Member. You'll could assign the output of New-Object to a variable and then pass that to Add-Member or just try to do it in one go with the output of New-Object pipelined to Add-Member:
New-Object ... | Add-Member -MemberType NoteProperty -Name "Mac-Address" -Value $macaddress
powershell datatable object psobject
You can just add a formatted version of the MAC column to your datatable object using an expression column. There's probably a more elegant way to write the expression, but this will work (I'm assuming $devices is a DataTable):
$maColumn = $devices.Columns.Add('MAC-Address')
$maColumn.Expression = "SUBSTRING(mac, 1, 2) + '-' + SUBSTRING(mac, 3, 2) + '-' + SUBSTRING(mac, 5, 2) + '-' + SUBSTRING(mac, 7, 2) + '-' + SUBSTRING(mac, 9, 2) + '-' + SUBSTRING(mac, 11, 2)"