Fastest way to convert/process Datatable to Custom PSObject - mysql

I'm able to fill a datatable, $devices from a MySql query (for example):
PS C:\Users\MKANET\Desktop\dev\lab> $devices.table[10]
name ip mac vendor
---- -- --- ------
ComputerTest 10.51.18.6 fd1969ff4cb9 HewlettP
I'd like to convert that datatable type into a custom PSObject; where, the mac column is converted to PSObject NoteProperty name "Mac-Address"; and, respective values converted to 00-00-00-00 string format:
PS C:\Users\MKANET\Desktop\dev\lab> $devices[1]
name ip MAC-Address vendor
------- -- ----------- ------------
ComputerTest 10.51.18.6 fd-19-69-ff-4c-b9 HewlettP
Considering these datatables may be relatively large (may have a couple hundred rows), I need the fastest processing method to do this in Powershell.
If it'll make the converting/modifying process significantly faster, I'd be happy with keeping $Devices as a Datatable; only modifying/processing the values for the "mac" column to: 00-00-00-00-00-00 text format, permanently.

Using Select-Object, it should auto-convert a DataRow / DataTable to a PSCustomObject in Powershell. Here's what I would run to do the conversion to PSCustomObject and handle the MAC address formatting in one line. This can be applied to a single row at specific index (like the example code) or against the entire DataTable to convert to an array of PSObjects.
The MAC address formatting removes any non-alphanumeric characters in the original, casts to lowercase, then inserts the hyphens at the appropriate indexes.
$devices.table[10] | Select-Object name,ip,#{N="MAC-Address";E={((((($_.mac -replace '[^a-zA-Z0-9]','').ToLower().insert(2,"-")).insert(5,"-")).insert(8,"-")).insert(11,"-")).insert(14,"-")}},vendor

Just to be clear for some readers: within PowerShell, a DataTable is a PS object, just one with a [System.Data.DataTable] type. This type (class) provides a number of methods for manipulating the object such as adding a new column (shown in the answer provided by #e-z-hart).
Converting an object to a generic 'PSObject' effectively means stripping away all the additional methods, etc. associated with the [DataTable] class and leaving just a simple object. You could do this as follows:
$obj = $devices.table[10] | foreach {
$macaddress = ($_.mac -replace '(..)','$1-').TrimEnd("-")
New-Object -TypeName PSObject -Property #{
"name = [System.String] $_.name
"ip" = [System.Net.IPAddress] $_.ip
"mac-address" = [System.String] $macaddress
"vendor" = [System.String] $_.vendor
}
}
Hopefully the -replace operator with a regex is a little more elegant (cf. answer from #e-z-hart) than having to chop the mac address up with substring functions. Someone may be able to improve on the regex so that it isn't necessary to Trim the trailing "-" at the end.
This should be pretty fast for the "couple of hundred" rows you mention.
Note that, for bonus points, the 'ip' column is converted to a [System.Net.IPAddress] type, which includes a ToString() method to return the address in the dotted format we usually see them in. I suppose this is overkill so you'd probably want this to be a [System.String] throughout.
Finally, if you really want the MAC-Address column to be a NoteProperty, then I'd exclude that column from the list of properties in the New-Object cmdlet and use Add-Member. You'll could assign the output of New-Object to a variable and then pass that to Add-Member or just try to do it in one go with the output of New-Object pipelined to Add-Member:
New-Object ... | Add-Member -MemberType NoteProperty -Name "Mac-Address" -Value $macaddress
powershell datatable object psobject

You can just add a formatted version of the MAC column to your datatable object using an expression column. There's probably a more elegant way to write the expression, but this will work (I'm assuming $devices is a DataTable):
$maColumn = $devices.Columns.Add('MAC-Address')
$maColumn.Expression = "SUBSTRING(mac, 1, 2) + '-' + SUBSTRING(mac, 3, 2) + '-' + SUBSTRING(mac, 5, 2) + '-' + SUBSTRING(mac, 7, 2) + '-' + SUBSTRING(mac, 9, 2) + '-' + SUBSTRING(mac, 11, 2)"

Related

how to change column colour base on a condition in powershell html

Firstly i would apologize if i was asking a very noob question, i'm a beginner powershell users and i don't have much exp in scripting, i have some issue to output html report using powershell. I wanted to change the colour of each row according to the utilization % wheter it is above or below 90%. However i only manage to change the colour of the first row.
i have 2 file which i use to output the report.
output from agg.csv
Aggregate_01_Data1,10.01TB,90%,96.93TB,online,86.92TB,
Aggregate_02_Data1,9.03TB,91%,96.93TB,online,87.90TB,
root_Aggregate_01,17.85GB,85%,368.4GB,online,350.6GB,
root_Aggregate_02,17.85GB,95%,368.4GB,online,350.6GB,
output from aggregate.csv
Aggregate,Size,Available,Used,Utilized,State
Aggregate_01_Data1,96.93TB,10.01TB,86.92TB,90,online
Aggregate_02_Data1,96.93TB,9.03TB,87.90TB,91,online
root_Aggregate_01,368.4GB,17.85GB,350.6GB,85,online
root_Aggregate_02,368.4GB,17.85GB,350.6GB,95,online
my code to generate the html report.
$a1=gc $TPATH\agg.csv
$b1 =$a1
for ($i1 = 0;$i1 -lt $a1.count;$i1++)
{
$c1=$b1[$i1].split(",",10)
$c = $c1[2].Length
$c = $c - 1
$d1=$c1[0] + "," + $c1[3] + "," + $c1[1] + "," + $c1[5] + "," + $c1[2].remove($c) + "," + $c1[4]
$d1 | Out-File ("$TPATH\aggregate.csv") -Encoding UTF8 -append
$bodyB = (import-csv $TPATH\aggregate.csv) |Select Aggregate,Size,Available,Used,Utilized,State -OutVariable ag | ConvertTo-Html -Fragment |
foreach { if($ag.Utilized -gt 90) {$_ -replace "<tr>", "<tr bgcolor=#FE0808>"}else{$_ -replace "<tr>", "<tr bgcolor=#E5FCA1>"}} |Out-String
}
convertTo-HTML -Body "<br> $bodyB" |out-file $PATH\Netapp.html
my final output look like this
how can i get the output to be like this, i want to have the colour change to Red if the threshold is above 90% in the Utilized column and Green if the threshold is below 90%.
Thank you in advance for those whom willing to help, really appreciate your help in this.
Thank You.
Replace -OutVariable ag with -PipelineVariable ag, so that $ag in your foreach (ForEach-Object) script block refers only to the single input object at hand.
The purpose of the common -PipelineVariable (-pv) parameter is to store the current object being output in a self-chosen variable, so that it can be accesses in a script block later in the same pipeline.
By contrast, the purpose of the common -OutVariable (-ov) parameter is to collect all output from a command in a self-chosen variable, for later processing in a different, subsequent statement, not in the same pipeline.
If you do access such a variable in a script block in the same pipeline, you'll see the output objects accumulated so far (which explains the behavior you saw - see below).
As for what you tried:
Due to the mistaken use of -OutVariable ag, your script block saw the Import-Csv output objects collected so far in variable $ag rather than the output object at hand.
Thus, starting with the second output object, $ag was in effect an array of objects, and accessing its . Utilized property resulted in member-access enumeration, and therefore returning an array of utilization values.
Since -gt, like all PowerShell comparison operators, acts as a filter when its LHS is an array, $ag.Utilized -gt 90 started to generate a return value (the subarray of matching values) as soon as at least one element in the array of values satisfied this condition.
In a Boolean context such as an if-statement conditional, a non-empty array is interpreted as $true.
Therefore, in effect ($ag.Utilized -gt 90) returned $true as soon as the first value above 90 was part of the return array, and invariably remained true.

How to get a lengthy string environment variable in full?

The command gci env:ApiSecret | ConvertTo-Json works to return a long string, the API secret for Twitter, which is truncated without the pipe to JSON.
However, the JSON is rather spammy.
Is there a "goldilocks" way to get the lengthy string value without the extraneous details?
(Unfortunately, gci env: truncates the key)
Get-ChildItem is for retrieving all or a subset of items from a container. Note that it outputs an object with Name and Value properties (substituting Path as another lengthy environment variable value)...
PS> gci env:Path
Name Value
---- -----
Path C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0\;C:\WINDO...
Get-Item yields the same result...
PS> gi env:Path
Name Value
---- -----
Path C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0\;C:\WINDO...
Either way, the object retrieved is a DictionaryEntry...
PS> gi env:Path | Get-Member
TypeName: System.Collections.DictionaryEntry
Name MemberType Definition
---- ---------- ----------
Name AliasProperty Name = Key
Equals Method bool Equals(System.Object obj)
GetHashCode Method int GetHashCode()
GetType Method type GetType()
ToString Method string ToString()
PSDrive NoteProperty PSDriveInfo PSDrive=Env
PSIsContainer NoteProperty bool PSIsContainer=False
PSPath NoteProperty string PSPath=Microsoft.PowerShell.Core\Environment::path
PSProvider NoteProperty ProviderInfo PSProvider=Microsoft.PowerShell.Core\Environment
Key Property System.Object Key {get;set;}
Value Property System.Object Value {get;set;}
...and when you pipe that to ConvertTo-Json it will include all kinds of undesirable properties from that class.
In short, don't use ConvertTo-Json for this. Since you know the exact item you want, just retrieve it directly using variable syntax...
PS> $env:Path
C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0\;...
Equivalent code using the .NET API would be...
PS> [Environment]::GetEnvironmentVariable('Path')
C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0\;...
If you really wanted to use a Get-*Item cmdlet you'd just need to specify that it's the Value property you want using property syntax...
PS> (gi env:Path).Value
C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0\;...
...or Select-Object...
PS> gi env:Path | Select-Object -ExpandProperty 'Value'
C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0\;...
All of the above commands will output only a [String] containing the entirety of that environment variable value. I inserted trailing ellipses since showing my entire Path value is not useful here; in practice, those commands will output the entire environment variable with no truncation.
The simplest way to inspect the value of environment variables in full is to use the $env:<varName> (namespace variable notation) syntax, which in your case means: $env:ApiSecret (if the variable name contains special characters, enclose everything after the $ in {...}; e.g., ${env:ApiSecret(1)})
That way, environment-variable values (which are invariably strings) that are longer than your terminal's (console's) width simply continue on subsequent lines.
To demonstrate:
# Simulate a long value (200 chars.)
$env:ApiSecret = 'x' * 199 + '!'
# Output the long value
$env:ApiSecret
With an 80-char. wide terminal, you'd see output as follows:
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx!
If you do want to use Get-Item (or Get-ChildItem, which acts the same in this case), you have two options:
# Format-List shows each property on its own line,
# with values wrapping across multiple lines
Get-Item env:ApiSecret | Format-List
# Format-Table -Wrap causes values to wrap as well.
Get-Item env:ApiSecret | Format-Table -Wrap
Your statement does not strip anything away. However, for console display purpose, it truncate the output that you view in the console.
If you assign the result to a variable or pipe to a file, nothing will be truncated.
Therefore, my assumption on your question is that you want to view the result in the console without the console truncating your stuff there.
For that, you could write the results to the host yourself.
Here's a simple example that do just that.
$envvars = gci env:
$Max = ($envvars.name| Measure-Object -Property length -Maximum).Maximum + 3
$envvars | % {Write-Host $_.name.padright($Max,' ') -ForegroundColor Cyan -NoNewline;Write-Host $_.value}
Result — As you can see, the path variable value is no longer truncated.

Access JSON nested property using dynamic proprty name [duplicate]

I'm trying to dynamically parse & build-up a data structure of some incoming JSON files I'm to be supplied with (that'll be in non-standard structure) via Powershell to then process the data in those files & hand them over to the next step.
As part of that, I'm trying to build up the data structure of the JSON file into essentially a list of of data-paths for me to parse through & grab the data out of, so that I can cope with arrays, nested JSON objects and so on. So far so good.
Where I fall into some sort of Powershell peculiarity is in handling 2+ levels of depth via a variable. Let me give you a nice code-block to demonstrate the problem...
# Generate a Quick JSON file with different data types & levels
[object]$QuickJson = #'
{
"Name" : "I am a JSON",
"Version" : "1.2.3.4",
"SomeBool" : true,
"NULLValue" : null,
"ArrayOfVersions" : [1.0,2.0,3.0],
"MyInteger" : 69,
"NestedJSON" : {
"Version" : 5.0,
"IsReady" : false
},
"DoubleNestedJSON" : {
"FirstLevel" : 1,
"DataValue" : "I am at first nested JSON level!",
"Second_JSON_Level" : {
"SecondLevel" : 2,
"SecondDataValue" : "I am on the 2nd nested level"
}
}
}
'#
# Import our JSON file into Powershell
[object]$MyPSJson = ConvertFrom-Json -InputObject $QuickJson
# Two quick string variables to access our JSON data paths
[string]$ShortJsonPath = "Name"
[string]$NestedJsonPath = "NestedJson.Version"
# Long string to access a double-nested JSON object
[string]$LongNestedJsonPath = "DoubleNestedJSON.Second_JSON_Level.SecondDataValue"
# Both of these work fine
Write-Host ("JSON Name (Direct) ==> " + $MyPSJson.Name)
Write-Host ("JSON Name (via Variable) ==> " + $MyPSJson.$ShortJsonPath)
# The following way to access a single nested Json Path works fine
Write-Host ("Nested JSON Version (via direct path) ==> " + $MyPSJson.NestedJson.Version)
# And THIS returns an empty line / is where I fall afoul of something in Powershell
Write-Host ("Nested JSON Version (via variable) ==> " + $MyPSJson.$NestedJsonPath)
# Other things I tried -- all returning an empty line / failing in effect
Write-Host ("Alternate Nested JSON Version ==> " + $($MyPSJson.$NestedJsonPath))
Write-Host ("Alternate Nested JSON Version ==> " + $MyPSJson.$($NestedJsonPath))
Write-Host ("Alternate Nested JSON Version ==> " + $($MyPSJson).$($NestedJsonPath))
# Similarly, while THIS works...
$MyPSJson | select-object -Property NestedJSON
# This will fail / return me nothing
$MyPSJson | select-object -Property NestedJSON.Version
... in doing a bunch of research around this, I came across a suggestion to transform this into a Hashtable -- but that has the same problem, sadly. So with the above code-snippet, the following will transform the JSON object into a hashtable.
# Same problem with a hash-table if constructed from the JSON file...
[hashtable]$MyHash = #{}
# Populate $MyHash with the data from our quickie JSON file...
$QuickJson | get-member -MemberType NoteProperty | Where-Object{ -not [string]::IsNullOrEmpty($QuickJson."$($_.name)")} | ForEach-Object {$MyHash.add($_.name, $QuickJson."$($_.name)")}
# ... and even then -- $MyHash."$($NestedJsonPath)" -- fails, while a single level deep string works fine in the variable! :(
So it's pretty clear that I'm running into "something" of a Powershell internal logic problem, but I can't get Powershell to be overly helpful in WHY that is. Adding a '-debug' or similar in an attempt to increase verbosity hasn't helped shed light on this.
I suspect it's something akin to the items raised in this article here ( https://blogs.technet.microsoft.com/heyscriptingguy/2011/10/16/dealing-with-powershell-hash-table-quirks/ ) but just specific with variables.
I've not had any luck in finding anything obvious in the Powershell language specification (3.0 still being the latest from here as far as I can tell -- https://www.microsoft.com/en-usdownload/details.aspx?id=36389 ) either. It may be in there, I may just miss it.
Any advice in how to get Powershell to play nice with this would be greatly appreciated. I'm not sure how / why Powershell is fine with a simple string but seems to have issues with a 'something.somethingelse' type string here.
Thank you.
Further notes & addenda to the original:
It seems there are several issues to attack. One is "dealing with a single nested level". The "quick fix" for that seems to be using "Invoke-Expression" to resolve the statement, so for instance (IMPORTANT - take note of the back-tick with the first variable!):
iex "`$MyPSJson.$NestedJsonPath"
That use of Invoke-Expression also works with multi-nested situations:
iex "`$MyPSJson.$LongNestedJsonPath"
An alternative approach that was mentioned is the use of multiple select statements ... but I've not been able to get that to work with multi-nested objects (Powershell seems to not resolve those properly for some reason).
So for instance in this scenario:
($MyComp | select $_.DoubleNestedJSON | select FirstLevel)
Powershell returns
FirstLevel
----------
... instead of the actual data value. So - for now, it seems that selects won't work with multi-level nested objects due to Powershell apparently not resolving them?
When you write something like
$MyPSJson.Name
this will attempt to retrieve the member named Name from the object $MyPSJson. If there is no such member, you'll get $null.
Now, when you do that with variables for the member name:
$MyPSJson.$ShortJsonPath
this works pretty much identical in that the member with the name stored in $ShortJsonPath is looked up and its value retrieved. No surprises here.
When you try that with a member that doesn't exist on the object, such as
$MyPSJson.$NestedJsonPath
# equivalent to
# $MyPSJson.'NestedJSON.Version'
you'll get $null, as detailed before. The . operator will only ever access a member of the exact object that is the result of its left-hand-side expression. It will never go through a member hierarchy in the way you seem to expect it to do. To be frank, I'm not aware of a language that works that way.
The reason it works with Invoke-Expression is, that you effectively converting the $NestedJsonPath string into part of an expression resulting in:
$MyPSJson.NestedJSON.Version
which Invoke-Expression then evaluates.
You can, of course, define your own function that works that way (and I'd much prefer that instead of using Invoke-Expression, a cmdlet that should rarely, if ever, used (heck, it's eval for PowerShell – few languages with eval advocate its use)):
function Get-DeepProperty([object] $InputObject, [string] $Property) {
$path = $Property -split '\.'
$obj = $InputObject
$path | %{ $obj = $obj.$_ }
$obj
}
PS> Get-DeepProperty $MyPSJson NestedJson.Version
5,0
You could even make it a filter, so you can use it more naturally on the pipeline:
filter Get-DeepProperty([string] $Property) {
$path = $Property -split '\.'
$obj = $_
$path | %{ $obj = $obj.$_ }
$obj
}
PS> $MyPSJson | Get-DeepProperty nestedjson.version
5,0
Why this doesn't work
When you provide the properties that you'd like within a string, like this
[string]$NestedJsonPath = "NestedJson.Version"
Powershell looks for a property called NestedJSon.Version. It's not actually traversing the properties, but looking for a string literal which contains a period. In fact, if I add a property like that to your JSON like so.
[object]$QuickJson = #'
{
"Name" : "I am a JSON",
"Version" : "1.2.3.4",
"SomeBool" : true,
"NULLValue" : null,
"ArrayOfVersions" : [1.0,2.0,3.0],
"MyInteger" : 69,
"NestedJSON.Version" : 69,
"NestedJSON" : {
"Version" : 5.0,
"IsReady" : false
}
}
I now get a value back, like so:
>$MyPSJson.$NestedJsonPath
69
The best way to get your values back is to use two separate variables, like this.
$NestedJson = "NestedJson"
$property = "Version"
>$MyPSJson.$NestedJson.$property
5.0
Or, alternatively, you could use select statements, as seen in the original answer below.
$MyPSJson | select $_.NestedJSON | select Version
Version
-------
1.2.3.4
If you use multiple Select-Object statements, they'll discard the other properties and allow you to more easily drill down to the value you'd like.
I followed Joey's filter example. However, I found it did not support accessing arrays.
Sharing the code that I got to work for this. Hopefully it will help others as well. Awesome thread!
filter Get-DeepProperty([string] $Property) {
$path = $Property -split '\.'
$obj = $_
foreach($node in $path){
if($node -match '.*\[\d*\]'){
$keyPieces = $node -split ('\[')
$arrayKey = $keyPieces[0]
$arrayIndex = $keyPieces[1] -replace ('\]','')
$obj = $obj.$arrayKey[$arrayIndex]
} else {
$obj = $obj.$node
}
}
$obj
}
Example usage:
$path = "nested.nestedtwo.steps[2]"
$payload | Get-DeepProperty $path
I had the same problem, so I wrote a function that does the trick.
It enables accessing any level of the json by variable path (string):
function getNestedJsonValue() {
param(
[Parameter(Mandatory = $true, ValueFromPipeline)] [PSCustomObject] $inputObj,
[Parameter(Mandatory = $true)] [string] $valuePath
)
if (($valuePath -eq $null) -or ($valuePath.length -eq 0) -or ($inputObj -eq $null)) {
return $inputObj
}
[System.Array] $nodes = "$valuePath" -split '\.'
foreach ($node in $nodes) {
if (($node -ne $null) -and ($node.length -gt 0) -and ($inputObj -ne $null)) {
$inputObj = $inputObj.$node
} else {
return $inputObj
}
}
return $inputObj
}
Usage: getNestedJsonValue -valuePath $nestedValuePath -inputObj $someJson
Pipe usage: $someJson | getNestedJsonValue -valuePath $nestedValuePath
An example nestedValuePath would be $nestedValuePath="some.nested.path"
Credit to wOxxOm for getting things on the right track.
Invoke-Expression does seem to work perfectly for this situation (if somewhat expensive, but that's fine in my personal example & situation), and it can cope with multiple levels of nesting.
So as examples for the above code snippet, the following will resolve just fine (Key point - pay attention to the initial back-tick. That caught me off guard):
Write-Host ("Single level JSON test ==> " + (iex "`$MyPSJson.$NestedJsonPath"))
Write-Host ("Double level JSON test ==> " + (iex "`$MyPSJson.$LongNestedJsonPath"))
That'll return our desired results:
Single level JSON test ==> 5.0
Double level JSON test ==> I am on the 2nd nested level
FoxDeploy's answer of using multi-level selects doesn't seem to work with 2+ levels of nesting, unfortunately for some bizarre reason.
Using:
($MyPSJson | select $_.DoubleNestedJSON | select FirstLevel)
We get the following back from Powershell:
FirstLevel
----------
... it seems that Powershell doesn't resolve nested objects in its entirety? We get a similar results if we intentionally use something that doesn't exist:
($MyPSJson | select $_.DoubleNestedJSON | select Doesnotexist)
... also simply returns:
Doesnotexist
------------
So - for now - it seems as if "Invoke-Expression" works most reliably (and most easily, as it's just a case of handing it a variable with the path'ed string).
I still can't explain the WHY of any of this so far (since I've used 'dotwalk'-ing with multiple variables through arrays quite happily), but at least there's a solution for now ... and that is Invoke-Expression !
The best (/least bad?) explanations for Invoke-Expression I've found so far are here (Microsoft's own description of the cmdlet doesn't really make a great job of hinting that it'd help in situations such as this):
http://ss64.com/ps/invoke-expression.html
https://www.adminarsenal.com/powershell/invoke-expression/

Problems parsing / accessing nested JSON / Hashtable data via variables in Powershell

I'm trying to dynamically parse & build-up a data structure of some incoming JSON files I'm to be supplied with (that'll be in non-standard structure) via Powershell to then process the data in those files & hand them over to the next step.
As part of that, I'm trying to build up the data structure of the JSON file into essentially a list of of data-paths for me to parse through & grab the data out of, so that I can cope with arrays, nested JSON objects and so on. So far so good.
Where I fall into some sort of Powershell peculiarity is in handling 2+ levels of depth via a variable. Let me give you a nice code-block to demonstrate the problem...
# Generate a Quick JSON file with different data types & levels
[object]$QuickJson = #'
{
"Name" : "I am a JSON",
"Version" : "1.2.3.4",
"SomeBool" : true,
"NULLValue" : null,
"ArrayOfVersions" : [1.0,2.0,3.0],
"MyInteger" : 69,
"NestedJSON" : {
"Version" : 5.0,
"IsReady" : false
},
"DoubleNestedJSON" : {
"FirstLevel" : 1,
"DataValue" : "I am at first nested JSON level!",
"Second_JSON_Level" : {
"SecondLevel" : 2,
"SecondDataValue" : "I am on the 2nd nested level"
}
}
}
'#
# Import our JSON file into Powershell
[object]$MyPSJson = ConvertFrom-Json -InputObject $QuickJson
# Two quick string variables to access our JSON data paths
[string]$ShortJsonPath = "Name"
[string]$NestedJsonPath = "NestedJson.Version"
# Long string to access a double-nested JSON object
[string]$LongNestedJsonPath = "DoubleNestedJSON.Second_JSON_Level.SecondDataValue"
# Both of these work fine
Write-Host ("JSON Name (Direct) ==> " + $MyPSJson.Name)
Write-Host ("JSON Name (via Variable) ==> " + $MyPSJson.$ShortJsonPath)
# The following way to access a single nested Json Path works fine
Write-Host ("Nested JSON Version (via direct path) ==> " + $MyPSJson.NestedJson.Version)
# And THIS returns an empty line / is where I fall afoul of something in Powershell
Write-Host ("Nested JSON Version (via variable) ==> " + $MyPSJson.$NestedJsonPath)
# Other things I tried -- all returning an empty line / failing in effect
Write-Host ("Alternate Nested JSON Version ==> " + $($MyPSJson.$NestedJsonPath))
Write-Host ("Alternate Nested JSON Version ==> " + $MyPSJson.$($NestedJsonPath))
Write-Host ("Alternate Nested JSON Version ==> " + $($MyPSJson).$($NestedJsonPath))
# Similarly, while THIS works...
$MyPSJson | select-object -Property NestedJSON
# This will fail / return me nothing
$MyPSJson | select-object -Property NestedJSON.Version
... in doing a bunch of research around this, I came across a suggestion to transform this into a Hashtable -- but that has the same problem, sadly. So with the above code-snippet, the following will transform the JSON object into a hashtable.
# Same problem with a hash-table if constructed from the JSON file...
[hashtable]$MyHash = #{}
# Populate $MyHash with the data from our quickie JSON file...
$QuickJson | get-member -MemberType NoteProperty | Where-Object{ -not [string]::IsNullOrEmpty($QuickJson."$($_.name)")} | ForEach-Object {$MyHash.add($_.name, $QuickJson."$($_.name)")}
# ... and even then -- $MyHash."$($NestedJsonPath)" -- fails, while a single level deep string works fine in the variable! :(
So it's pretty clear that I'm running into "something" of a Powershell internal logic problem, but I can't get Powershell to be overly helpful in WHY that is. Adding a '-debug' or similar in an attempt to increase verbosity hasn't helped shed light on this.
I suspect it's something akin to the items raised in this article here ( https://blogs.technet.microsoft.com/heyscriptingguy/2011/10/16/dealing-with-powershell-hash-table-quirks/ ) but just specific with variables.
I've not had any luck in finding anything obvious in the Powershell language specification (3.0 still being the latest from here as far as I can tell -- https://www.microsoft.com/en-usdownload/details.aspx?id=36389 ) either. It may be in there, I may just miss it.
Any advice in how to get Powershell to play nice with this would be greatly appreciated. I'm not sure how / why Powershell is fine with a simple string but seems to have issues with a 'something.somethingelse' type string here.
Thank you.
Further notes & addenda to the original:
It seems there are several issues to attack. One is "dealing with a single nested level". The "quick fix" for that seems to be using "Invoke-Expression" to resolve the statement, so for instance (IMPORTANT - take note of the back-tick with the first variable!):
iex "`$MyPSJson.$NestedJsonPath"
That use of Invoke-Expression also works with multi-nested situations:
iex "`$MyPSJson.$LongNestedJsonPath"
An alternative approach that was mentioned is the use of multiple select statements ... but I've not been able to get that to work with multi-nested objects (Powershell seems to not resolve those properly for some reason).
So for instance in this scenario:
($MyComp | select $_.DoubleNestedJSON | select FirstLevel)
Powershell returns
FirstLevel
----------
... instead of the actual data value. So - for now, it seems that selects won't work with multi-level nested objects due to Powershell apparently not resolving them?
When you write something like
$MyPSJson.Name
this will attempt to retrieve the member named Name from the object $MyPSJson. If there is no such member, you'll get $null.
Now, when you do that with variables for the member name:
$MyPSJson.$ShortJsonPath
this works pretty much identical in that the member with the name stored in $ShortJsonPath is looked up and its value retrieved. No surprises here.
When you try that with a member that doesn't exist on the object, such as
$MyPSJson.$NestedJsonPath
# equivalent to
# $MyPSJson.'NestedJSON.Version'
you'll get $null, as detailed before. The . operator will only ever access a member of the exact object that is the result of its left-hand-side expression. It will never go through a member hierarchy in the way you seem to expect it to do. To be frank, I'm not aware of a language that works that way.
The reason it works with Invoke-Expression is, that you effectively converting the $NestedJsonPath string into part of an expression resulting in:
$MyPSJson.NestedJSON.Version
which Invoke-Expression then evaluates.
You can, of course, define your own function that works that way (and I'd much prefer that instead of using Invoke-Expression, a cmdlet that should rarely, if ever, used (heck, it's eval for PowerShell – few languages with eval advocate its use)):
function Get-DeepProperty([object] $InputObject, [string] $Property) {
$path = $Property -split '\.'
$obj = $InputObject
$path | %{ $obj = $obj.$_ }
$obj
}
PS> Get-DeepProperty $MyPSJson NestedJson.Version
5,0
You could even make it a filter, so you can use it more naturally on the pipeline:
filter Get-DeepProperty([string] $Property) {
$path = $Property -split '\.'
$obj = $_
$path | %{ $obj = $obj.$_ }
$obj
}
PS> $MyPSJson | Get-DeepProperty nestedjson.version
5,0
Why this doesn't work
When you provide the properties that you'd like within a string, like this
[string]$NestedJsonPath = "NestedJson.Version"
Powershell looks for a property called NestedJSon.Version. It's not actually traversing the properties, but looking for a string literal which contains a period. In fact, if I add a property like that to your JSON like so.
[object]$QuickJson = #'
{
"Name" : "I am a JSON",
"Version" : "1.2.3.4",
"SomeBool" : true,
"NULLValue" : null,
"ArrayOfVersions" : [1.0,2.0,3.0],
"MyInteger" : 69,
"NestedJSON.Version" : 69,
"NestedJSON" : {
"Version" : 5.0,
"IsReady" : false
}
}
I now get a value back, like so:
>$MyPSJson.$NestedJsonPath
69
The best way to get your values back is to use two separate variables, like this.
$NestedJson = "NestedJson"
$property = "Version"
>$MyPSJson.$NestedJson.$property
5.0
Or, alternatively, you could use select statements, as seen in the original answer below.
$MyPSJson | select $_.NestedJSON | select Version
Version
-------
1.2.3.4
If you use multiple Select-Object statements, they'll discard the other properties and allow you to more easily drill down to the value you'd like.
I followed Joey's filter example. However, I found it did not support accessing arrays.
Sharing the code that I got to work for this. Hopefully it will help others as well. Awesome thread!
filter Get-DeepProperty([string] $Property) {
$path = $Property -split '\.'
$obj = $_
foreach($node in $path){
if($node -match '.*\[\d*\]'){
$keyPieces = $node -split ('\[')
$arrayKey = $keyPieces[0]
$arrayIndex = $keyPieces[1] -replace ('\]','')
$obj = $obj.$arrayKey[$arrayIndex]
} else {
$obj = $obj.$node
}
}
$obj
}
Example usage:
$path = "nested.nestedtwo.steps[2]"
$payload | Get-DeepProperty $path
I had the same problem, so I wrote a function that does the trick.
It enables accessing any level of the json by variable path (string):
function getNestedJsonValue() {
param(
[Parameter(Mandatory = $true, ValueFromPipeline)] [PSCustomObject] $inputObj,
[Parameter(Mandatory = $true)] [string] $valuePath
)
if (($valuePath -eq $null) -or ($valuePath.length -eq 0) -or ($inputObj -eq $null)) {
return $inputObj
}
[System.Array] $nodes = "$valuePath" -split '\.'
foreach ($node in $nodes) {
if (($node -ne $null) -and ($node.length -gt 0) -and ($inputObj -ne $null)) {
$inputObj = $inputObj.$node
} else {
return $inputObj
}
}
return $inputObj
}
Usage: getNestedJsonValue -valuePath $nestedValuePath -inputObj $someJson
Pipe usage: $someJson | getNestedJsonValue -valuePath $nestedValuePath
An example nestedValuePath would be $nestedValuePath="some.nested.path"
Credit to wOxxOm for getting things on the right track.
Invoke-Expression does seem to work perfectly for this situation (if somewhat expensive, but that's fine in my personal example & situation), and it can cope with multiple levels of nesting.
So as examples for the above code snippet, the following will resolve just fine (Key point - pay attention to the initial back-tick. That caught me off guard):
Write-Host ("Single level JSON test ==> " + (iex "`$MyPSJson.$NestedJsonPath"))
Write-Host ("Double level JSON test ==> " + (iex "`$MyPSJson.$LongNestedJsonPath"))
That'll return our desired results:
Single level JSON test ==> 5.0
Double level JSON test ==> I am on the 2nd nested level
FoxDeploy's answer of using multi-level selects doesn't seem to work with 2+ levels of nesting, unfortunately for some bizarre reason.
Using:
($MyPSJson | select $_.DoubleNestedJSON | select FirstLevel)
We get the following back from Powershell:
FirstLevel
----------
... it seems that Powershell doesn't resolve nested objects in its entirety? We get a similar results if we intentionally use something that doesn't exist:
($MyPSJson | select $_.DoubleNestedJSON | select Doesnotexist)
... also simply returns:
Doesnotexist
------------
So - for now - it seems as if "Invoke-Expression" works most reliably (and most easily, as it's just a case of handing it a variable with the path'ed string).
I still can't explain the WHY of any of this so far (since I've used 'dotwalk'-ing with multiple variables through arrays quite happily), but at least there's a solution for now ... and that is Invoke-Expression !
The best (/least bad?) explanations for Invoke-Expression I've found so far are here (Microsoft's own description of the cmdlet doesn't really make a great job of hinting that it'd help in situations such as this):
http://ss64.com/ps/invoke-expression.html
https://www.adminarsenal.com/powershell/invoke-expression/

Import only select data from json to powershell

I want to Import selected data from Json url and so I can convert it to XML.
I am using following code to import.
(Invoke-RestMethod -URI "http://www.broadbandmap.gov/broadbandmap/broadband/dec2013/wireline?latitude=29.488412&longitude=-98.550208&format=json").Results.wirelineServices.providerName | Select-Object | Format-Table –AutoSize
so I am using .Results.wirelineServices.providerName to pull selected columns from one branch/table.
how can I pull data from .Results.broadbandSource.stateFips also at same time?
Thanks bunch.
Json code screenshot.
follow up question
I think that you should separate out your steps a bit:
$r = Invoke-RestMethod -URI "http://www.broadbandmap.gov/broadbandmap/broadband/dec2013/wireline?latitude=29.488412&longitude=-98.550208&format=json"
$providers = $r.Results.wirelineServices.providerName
$stateFips = $r.Results.broadbandSource.stateFips
Note that in your example, your call to Select-Object is redundant (you're not selecting anything, so it's not changing the input object).
Also, a very important point about Format-Table (and any Format- cmdlet) is that those are for display only so they should always be the last thing you do, if in fact they're needed at all.
The code I've given gives you the information in objects, which you can then work with, filter or, display as needed. I'm not sure how you wanted to use/display it, but since there are multiple providers and only one stateFips value, I might assume that you would apply the Fips value to each provider. Here's an example of that which uses the $stateFips variable we created:
$r.Results.wirelineServices | Select-Object providerName,#{Name='stateFips' ; Expression={ $stateFips }}
And here's an example that uses only the original result $r:
$r.Results.wirelineServices | Select-Object providerName,#{Name='stateFips' ; Expression={ $r.Results.broadbandSource.stateFips }}
The Select-Object computed column syntax
Note that the second column definition looks a bit wonky. It's actually a hashtable that allows you to specify the name of the column, and an expression (a complete code block) whose return value will be the value of the column. It could be spread over multiple lines like this:
$r.Results.wirelineServices | Select-Object providerName,#{
Name = 'stateFips'
Expression = {
$r.Results.broadbandSource.stateFips
}
}
Or you could even create the hashtable as a variable first:
$computed = #{
Name = 'stateFips'
Expression = {
$r.Results.broadbandSource.stateFips
}
}
$r.Results.wirelineServices | Select-Object providerName,$computed
XML?
#Stephen Connolly's answer reminded me that you wanted to make XML out of this. Let's take the above code and assign it to a variable:
$computed = #{
Name = 'stateFips'
Expression = {
$r.Results.broadbandSource.stateFips
}
}
$data = $r.Results.wirelineServices | Select-Object providerName,$computed
Because $data is still an object and wasn't sent through a Format- command, we can still use it!
$xml = $data | ConvertTo-Xml -NoTypeInformation
As his comment also suggested though, we don't know how you wanted the resultant XML to be formatted.
So here's another approach:
Forget the JSON
$r = Invoke-RestMethod -URI "http://www.broadbandmap.gov/broadbandmap/broadband/dec2013/wireline?latitude=29.488412&longitude=-98.550208&format=xml"
Now $r contains XML already. You can filter it out and modify it using XPATH. I won't get into that at the moment unless you think that way would work better for you.
Hope this helps, let me know if I've misunderstood what you're trying to do here.
If you want a composite object try something like
$results = (Invoke-RestMethod -URI "http://www.broadbandmap.gov/broadbandmap/broadband/dec2013/wireline?latitude=29.488412&longitude=-98.550208&format=json")
$obj = $results.Results.wirelineServices
$obj | add-member -type noteproperty -Name StateFips -Value $($results.Results.broadbandSource.stateFips) -PassThru
$obj | convertto-xml -as string