Select-Object Data Failure From Succesful Invoke-RestMethod - json

I have a working call to a rest service using Invoke-RestMethod -Uri https://.... The call's result, in JSON, can be piped the data to Format-Table -Property ... and data is shown.
But when Select-Object -Property ... is used after the invoke with the same parameters, the PSObject has the columns but no data for them. If I use a different webservice the call will work.
What could be causing the PSObject to not show any values?
Working Example on public rest web services
Invoke-RestMethod -Uri https://jsonplaceholder.typicode.com/todos/1 |
Select-Object -Property title
Result
#{title=delectus aut autem}
New Failure Different API
Invoke-RestMethod -Uri https://cat-fact.herokuapp.com/facts | Select-Object -Property text

You've stumbled upon an unholy combination of two PowerShell oddities when converting JSON arrays:
Invoke-RestMethod and ConvertFrom-Json send converted-from-JSON arrays as a whole through the pipeline, instead of element by element, as usual:
Note: In PowerShell (Core) 7.0, ComvertFrom-Json's behavior was changed to align with the usual enumeration-of-elements behavior, and a -NoEnumerate switch was added as an opt-in to the old behavior. For the discussion that led to this change, see GitHub issue #3424.
However, as of this writing (PowerShell (Core 7.2) Invoke-RestMethod still exhibits this unexpected behavior, which is discussed in GitHub issue #15272.
Select-Object does not perform member-access enumeration, so it looks for the specified property (e.g., text) directly on the array, where it doesn't exist.
To demonstrate the problem with a simple example:
# Windows PowerShell only:
# Because ConvertFrom-Json sends an *array* (of 2 custom objects) through
# the pipeline, Select-Object looks for property .text on the *array* -
# and can't find it.
# The same applies to Invoke-RestMethod (also still in
# PowerShell (Core) as of v7.2)
PS> ConvertFrom-Json '[{ "text": "a" }, { "text": "b" }]' | Select-Object text
text
----
# NO VALUES
A simple workaround is to enclose the ConvertFrom-Json / Invoke-RestMethod call in (...), which forces enumeration of the array, causing Select-Object to work as expected.:
# (...) forces enumeration
PS> (ConvertFrom-Json '[{ "text": "a" }, { "text": "b" }]') | Select-Object text
text
----
a
b
Note that a command such as Select-Object -Property text (without -ExpandProperty) still output custom objects that have a .text property, not the .text property values.
If all you're interested in is property values, the solution is simpler, because you can use the above-mentioned member-access enumeration directly on the array:
# Using .<propName> on an array (collection) implicitly returns the
# property values from the *elements* of that collection (member-access enumeration).
PS> (ConvertFrom-Json '[{ "text": "a" }, { "text": "b" }]').text
a
b
Note how the output now has no text header, because it is mere string values that are being output, not custom objects.

your problem in the 2nd example is that there is NO prop named text. [grin]
the only prop is all and that contains an array of objects that DO contain a prop named text. so you need something that can get that deeper prop. one way is to use two Select-Object calls. something like this ...
$Url = 'https://cat-fact.herokuapp.com/facts'
$RawIRM = Invoke-RestMethod -Uri $Url
$SO_IRM = $RawIRM |
Select-Object -ExpandProperty all |
Select-Object -Property text
the $SO_IRM var now has an array of 178 strings about cats. [grin]

Related

How to parse/access JSON returned by Invoke-WebRequest in Powershell 4

I have the following call to an API in a powershell script (Powershell 4.0):
$Json = Invoke-WebRequest -Uri $RequestURL -UseBasicParsing -Headers $headers -ContentType 'application/json; charset=utf-8' -Method POST -Body $postParams -TimeoutSec 40
...and the content of the response (which is a string in JSON format) is written to a file:
Set-Content $path -Value $Json.Content
An example of a typical response...
{
"MyArray": [{
"MyField": "A1",
"MyField2": "A2"
}, {
"MyField": "B1",
"MyField2": "B2"
}]
}
All well and good, but now I have a requirement to parse the returned content as JSON and query some properties from within this Powershell script.
I presume I need to convert my string to 'proper' JSON and then to a powershell object in order to access the properties...so I have tried combinations of ConvertTo-Json and ConvertFrom-Json but can't ever seem to access it in anything other than a string. For example...
$x = $Json.Content | ConvertTo-Json
Write-Host $x.MyArray[0].MyField
$y = $x | ConvertFrom-Json
Write-Host $y[0].MyArray[0].MyField
In both cases above I get an error "Cannot index into a null array" suggesting that MyArray is null.
How do I convert my $Json response object into an object I can drill down into?
See ConvertFrom-Json
Converts a JSON-formatted string to a custom object or a hash table.
The ConvertFrom-Json cmdlet converts a JavaScript Object Notation
(JSON) formatted string to a custom PSCustomObject object that has a
property for each field in the JSON string.
Once you get the response converted to custom object or a hash table, you can access the individual properties
The link includes coding examples
This seems to work...
[void][System.Reflection.Assembly]::LoadWithPartialName("System.Web.Extensions")
$x = (New-Object -TypeName System.Web.Script.Serialization.JavaScriptSerializer -Property #{MaxJsonLength=67108864}).DeserializeObject($Json.Content)
Write-Host $x.MyArray[0].MyField
...although not sure why yet.

Using Powershell to convert a file's contents into a string that can be transferred using JSON

How does one convert a text file's contents into a string and then insert this string into a JSON file?
For example, if a file contains:
this
is
a
sample
file
The script would generate:
"this\r\nis\r\na\r\nsample\r\nfile"
To insert into a JSON template:
"something":"<insertPoint>"
To produce:
"something":"this\r\nis\r\na\r\nsample\r\nfile"
I'm using Powershell 5 and have managed to load the file, generate some JSON and insert it by running:
# get contents and convert to JSON
$contentToInsert = Get-Content $sourceFilePath -raw | ConvertTo-Json
# write in output file
(Get-Content $outputFile -Raw).replace('<insertPoint>', $contentToInsert) | Set-Content $outputFile
However, a lot of other, unwanted fields are also added.
"something":"{
"value": "this\r\nis\r\na\r\nsample\r\nfile"
"PSPath": "C:\\src\\intro.md",
"PSParentPath": "C:\\src",
"PSChildName": "intro.md",
etc...
Ultimately, I'm trying to send small rich text segments to a web page via JSON but want to edit and store them locally using Markdown. If this doesn't make sense and there's a better way of sending these then please let me know also.
iRon's answer helpfully suggests not using string manipulation to create JSON in PowerShell, but to use hashtables (or custom objects) to construct the data and then convert it to JSON.
However, that alone does not solve your problem:
PS> #{ something = Get-Content -Raw $sourceFilePath } | ConvertTo-Json
{
"something": {
"value": "this\nis\na\nsample\nfile\n",
"PSPath": "/Users/mklement/Desktop/pg/lines.txt",
# ... !! unwanted properties are still there
}
The root cause of the problem is that Get-Content decorates the strings it outputs with metadata in the form of NoteProperty properties, and ConvertTo-Json currently invariably includes these.
A proposal to allow opting out of this decoration when calling Get-Content can be found in GitHub issue #7537.
Complementarily, GitHub issue #5797 suggests that ConvertTo-Json should ignore the extra properties for primitive .NET types such as strings.
The simplest workaround is to access the underlying .NET instance with .psobject.baseobject, which bypasses the invisible wrapper object PowerShell uses to supply the extra properties:
PS> #{ something = (Get-Content -Raw $sourceFilePath).psobject.baseobject } |
ConvertTo-Json
{
"something": "this\nis\na\nsample\nfile\n"
}
Just a general recommendation apart from the actually issue described by #mklement0 and metadata added to the Get-Content results:
Do not poke (replace, insert, etc.) in any Json content.
Instead, modify the object (if necessary, use ConvertFrom-Json to restore the object) prior converting it into (ConvertTo-Json) a Json file.
In this example, I would use a hash-table with a here-string for this:
#{'something' = #'
this
is
a
sample
file
'#
} | ConvertTo-Json
Result:
{
"something": "this\nis\na\nsample\nfile"
}
You can use the Out-String cmdlet to coerce the output of Get-Content into a flat string first:
#{ "something" = (Get-Content lines.txt | Out-String) } | ConvertTo-Json
This produces:
{
"something": "this\r\nis\r\na\r\nsample\r\nfile\r\n"
}

Why does ConvertFrom-Json produce an Object which turns into a PSCustomObject after assignment? And how do I shortcut that?

I have a simple json file as follows:
[
{"ClientName": "Test Site 1", "ClientID": "000001"},
{"ClientName": "Test Site 2", "ClientID": "000002"},
{"ClientName": "Test Site 3", "ClientID": "000003"}
]
When I use the following PowerShell command:
ConvertFrom-Json (Get-Content TestSites.json -Raw)
I get back a System.Object[]. This doesn't allow me to pipe the output to another function I have which accepts "ClientName" and "ClientID" parameters.
However, when I assign that object to another variable, like this:
$myobj = ConvertFrom-Json (Get-Content TestSites.json -Raw)
$myobj is actually a System.Management.Automation.PSCustomObject which is capable of being passed to my function.
How can I just pipe the results of the original command without having to assign it to another variable first?
I hope that makes sense.
Your JSON is an array correct? PowerShell will unroll arrays in the pipeline unless you explicity change that behavior. Assuming your JSON is stored in the variable $json as a single string consider the following examples.
ConvertFrom-Json $json | ForEach-Object{$_.gettype().fullname}
System.Object[]
(convertFrom-Json $json) | ForEach-Object{$_.gettype().fullname}
System.Management.Automation.PSCustomObject
System.Management.Automation.PSCustomObject
System.Management.Automation.PSCustomObject
You should be able to wrap the expression in brackets to change the outcome. In the second example it should be sending the 3 objects individually down the pipe. In the first it is being sent as a single object array.
My explanation needs work but I am sure of the cause is how PowerShell deals with arrays and the pipeline. Unrolling being a common word used to describe it.
So depending on your use case you might just be able to wrap the expression in brackets so it gets processed before the pipe to ForEach in my example.
If you have an array such as System.Object[] you could try piping via foreach:
ConvertFrom-Json (Get-Content TestSites.json -Raw) | %{ $_ | your-function }
If you want to pass the whole array down the pipe as-is, you can try adding a comma (aka a unary comma before the variable:
,$myobj | whatever
You can probably see how the latter works by comparing the following:
$myobj | Get-Member # Shows the type of the elements of the array
Get-Member -InputObject $myobj # Shows the type of the array
,$myobj | Get-Member # Shows the type of the array

Expand Multi-level Powershell nested hash tables from JSON api call

I am having an issue with a JSON API call using Powershell, that returns a multi-level nested array. Using the standard convertfrom-json & convertto-json doesn't give me the full results. I just get a ...
I have done a lot of research on this and found the following link on stack for expanding the nested hash tables, and this is about the best format that I have gotten. For reference, I am trying to insert employee records into SQL Server. I have a JSON function for doing so, and need to get the JSON response in the expanded format. There are some security aspects, that won't allow me to post the actual detail so I am trying to provide an example of some issues that I am having. Keep in mind the response is for 1 employee record out of hundreds overall, and in SQL will be one table with a row per employee.
$json1 = Invoke-webRequest -Uri $Workeruri -Certificate $cert -Headers $WorkerHeader | convertfrom-json
$workers = $json1.workers | select -property *
$workers | format-table Name, #{n='Value'; e={
if ($_.Value -is [Hashtable]) {
$ht = $_.Value
$a = $ht.keys | sort | % { '{0}={1}' -f $_, $ht[$_] }
'{{{0}}}' -f ($a -join ', ')
} else {
$_.Value
}
}}
$workers
I have also tried:
$workers | convertto-json
and
$workers = json1.workers | foreach-object {$_}
The data returned comes back like this:
associateOID : XXXXXXXXXXXXXX
workerID : #{idValue=XXXXXXX}
person : #{governmentIDs=System.Object[]; legalName=; birthDate=0000-00-00; legalAddress=; genderCode=; maritalStatusCode=; socialInsurancePrograms=System.Object[]; tobaccoUserIndicator=False; raceCode=; customFieldGroup=}
workerDates : #{originalHireDate=2015-03-09; terminationDate=2016-03-18}
workerStatus : #{statusCode=}
businessCommunication : #{emails=System.object[]}
workAssignments : {#{itemID=xxxxxx; occupationalclassification=System.Object[]; etc}}
I need it to come back with all of the columns on the left side, utilizing the "AssociateOID" as the key identifier for the individual. I have previously gotten the JSON response to come back completely expanded using this format but it wasn't working with the import into SQL Server or looking very nice like the Google Postman response:
$json1 = Invoke-webRequest -Uri $Workeruri -Certificate $cert -Headers $WorkerHeader | format-list -property *
You mean you want to get a JSON file that all objects/properties are expanded, right? Use the -Depth option of ConvertTo-Json to write a JSON file properly. Default value of -Depth option is 2.
$workers | ConvertTo-Json -Depth 100
Specify the appropriate depth according to your JSON files.

Import only select data from json to powershell

I want to Import selected data from Json url and so I can convert it to XML.
I am using following code to import.
(Invoke-RestMethod -URI "http://www.broadbandmap.gov/broadbandmap/broadband/dec2013/wireline?latitude=29.488412&longitude=-98.550208&format=json").Results.wirelineServices.providerName | Select-Object | Format-Table –AutoSize
so I am using .Results.wirelineServices.providerName to pull selected columns from one branch/table.
how can I pull data from .Results.broadbandSource.stateFips also at same time?
Thanks bunch.
Json code screenshot.
follow up question
I think that you should separate out your steps a bit:
$r = Invoke-RestMethod -URI "http://www.broadbandmap.gov/broadbandmap/broadband/dec2013/wireline?latitude=29.488412&longitude=-98.550208&format=json"
$providers = $r.Results.wirelineServices.providerName
$stateFips = $r.Results.broadbandSource.stateFips
Note that in your example, your call to Select-Object is redundant (you're not selecting anything, so it's not changing the input object).
Also, a very important point about Format-Table (and any Format- cmdlet) is that those are for display only so they should always be the last thing you do, if in fact they're needed at all.
The code I've given gives you the information in objects, which you can then work with, filter or, display as needed. I'm not sure how you wanted to use/display it, but since there are multiple providers and only one stateFips value, I might assume that you would apply the Fips value to each provider. Here's an example of that which uses the $stateFips variable we created:
$r.Results.wirelineServices | Select-Object providerName,#{Name='stateFips' ; Expression={ $stateFips }}
And here's an example that uses only the original result $r:
$r.Results.wirelineServices | Select-Object providerName,#{Name='stateFips' ; Expression={ $r.Results.broadbandSource.stateFips }}
The Select-Object computed column syntax
Note that the second column definition looks a bit wonky. It's actually a hashtable that allows you to specify the name of the column, and an expression (a complete code block) whose return value will be the value of the column. It could be spread over multiple lines like this:
$r.Results.wirelineServices | Select-Object providerName,#{
Name = 'stateFips'
Expression = {
$r.Results.broadbandSource.stateFips
}
}
Or you could even create the hashtable as a variable first:
$computed = #{
Name = 'stateFips'
Expression = {
$r.Results.broadbandSource.stateFips
}
}
$r.Results.wirelineServices | Select-Object providerName,$computed
XML?
#Stephen Connolly's answer reminded me that you wanted to make XML out of this. Let's take the above code and assign it to a variable:
$computed = #{
Name = 'stateFips'
Expression = {
$r.Results.broadbandSource.stateFips
}
}
$data = $r.Results.wirelineServices | Select-Object providerName,$computed
Because $data is still an object and wasn't sent through a Format- command, we can still use it!
$xml = $data | ConvertTo-Xml -NoTypeInformation
As his comment also suggested though, we don't know how you wanted the resultant XML to be formatted.
So here's another approach:
Forget the JSON
$r = Invoke-RestMethod -URI "http://www.broadbandmap.gov/broadbandmap/broadband/dec2013/wireline?latitude=29.488412&longitude=-98.550208&format=xml"
Now $r contains XML already. You can filter it out and modify it using XPATH. I won't get into that at the moment unless you think that way would work better for you.
Hope this helps, let me know if I've misunderstood what you're trying to do here.
If you want a composite object try something like
$results = (Invoke-RestMethod -URI "http://www.broadbandmap.gov/broadbandmap/broadband/dec2013/wireline?latitude=29.488412&longitude=-98.550208&format=json")
$obj = $results.Results.wirelineServices
$obj | add-member -type noteproperty -Name StateFips -Value $($results.Results.broadbandSource.stateFips) -PassThru
$obj | convertto-xml -as string