I am having an issue with a JSON API call using Powershell, that returns a multi-level nested array. Using the standard convertfrom-json & convertto-json doesn't give me the full results. I just get a ...
I have done a lot of research on this and found the following link on stack for expanding the nested hash tables, and this is about the best format that I have gotten. For reference, I am trying to insert employee records into SQL Server. I have a JSON function for doing so, and need to get the JSON response in the expanded format. There are some security aspects, that won't allow me to post the actual detail so I am trying to provide an example of some issues that I am having. Keep in mind the response is for 1 employee record out of hundreds overall, and in SQL will be one table with a row per employee.
$json1 = Invoke-webRequest -Uri $Workeruri -Certificate $cert -Headers $WorkerHeader | convertfrom-json
$workers = $json1.workers | select -property *
$workers | format-table Name, #{n='Value'; e={
if ($_.Value -is [Hashtable]) {
$ht = $_.Value
$a = $ht.keys | sort | % { '{0}={1}' -f $_, $ht[$_] }
'{{{0}}}' -f ($a -join ', ')
} else {
$_.Value
}
}}
$workers
I have also tried:
$workers | convertto-json
and
$workers = json1.workers | foreach-object {$_}
The data returned comes back like this:
associateOID : XXXXXXXXXXXXXX
workerID : #{idValue=XXXXXXX}
person : #{governmentIDs=System.Object[]; legalName=; birthDate=0000-00-00; legalAddress=; genderCode=; maritalStatusCode=; socialInsurancePrograms=System.Object[]; tobaccoUserIndicator=False; raceCode=; customFieldGroup=}
workerDates : #{originalHireDate=2015-03-09; terminationDate=2016-03-18}
workerStatus : #{statusCode=}
businessCommunication : #{emails=System.object[]}
workAssignments : {#{itemID=xxxxxx; occupationalclassification=System.Object[]; etc}}
I need it to come back with all of the columns on the left side, utilizing the "AssociateOID" as the key identifier for the individual. I have previously gotten the JSON response to come back completely expanded using this format but it wasn't working with the import into SQL Server or looking very nice like the Google Postman response:
$json1 = Invoke-webRequest -Uri $Workeruri -Certificate $cert -Headers $WorkerHeader | format-list -property *
You mean you want to get a JSON file that all objects/properties are expanded, right? Use the -Depth option of ConvertTo-Json to write a JSON file properly. Default value of -Depth option is 2.
$workers | ConvertTo-Json -Depth 100
Specify the appropriate depth according to your JSON files.
Related
looking for some help.
I'm newbie to powershell and looking to call an API which has JSON output (encapsulated in single array).
Expected Output (couldn't get all columns in excel to show):
Below is the response from API:
{
"data": [
{
"Records": "[
{
'RecordId':'A8E00212-BE7B-4AA8-A87B-121E7018EAE0',
'RecordNumber':'1999',
'FormName':'Costco Boneless Center Cut Loin (Strap On, Strap Off, Super Trim, Premium)',
'FormVersion':'22',
'FormType':'Check',
'LocationName':'Beardstown (85B)',
'ResourceName':'Loins-Costco Boneless CC Loin',
'SubmittedUserName':'Maria Nava-Garcia',
'SubmittedDate':'2020-11-14T16:54:35.920',
'SubmittedDateLocal':'2020-11-14T16:54:35.920',
'SubmittedTimeZone':'America/Chicago',
'ProcessedDate':'2020-11-14T17:41:23.873',
'ModifiedDate':'2020-11-14T16:54:35.920',
'FieldGroups': [
{
'FieldGroupName':'' ,
'Fields': [
{'Date':'11/14/2020'},
{'Shift':'1'},
{'Product Code':'55620'},
{'Number of Pieces Audited':'8'},
{'Incorrect Bagging Orientation, S-Shaped Bags':'0'},
{'Incorrect Pieces in Bag':'0'},
{'Incorrect Boxing Placement':'0'},
{'Bag Tails':'0'},
{'Dating Over Bag Print':'0'},
{'Bones/Cartilage':'1'},
{'Number of Leakers':'0'},
{'Foreign Material/Contamination':'0'},
{'PSE':'0'},
{'Incorrect Code Dating and Est. #':'0'},
{'Blood Splash':'0'},
{'Dark Bruise':'0'},
{'Scores':'0'},
{'Sirloin End Not Square':'0'},
{'Pin Bone Removal':'0'},
{'Shoulder End Eye Diameter':'0'},
{'Gouges':'0'},
{'Fat':'2'},
{'Remaining Strap (Strap Off Codes Only)':'0'},
{'Belly Strap (Strap On Codes Only)':'0'},
{'Ham Patch Present':'0'},
{'False Lean Present':'1'},
{'Tag Ends':'0'},
{'Packaging Defects Total':'0'},
{'% Packaging Defects':'0'},
{'Critical Defects Total':'1'},
{'% Critical Defects':'12.5'},
{'Other Defects Total':'3'},
{'% Other Defects':'37.5'},
{'Total Defects':'4'},
{'Audit Acceptable':'No'},
{'What Did You Do?':'Tagged'},
{'How Many Boxes?':'15'},
{'Comments':'Notified Dave Long Hector Rodriguez and Jeff Harris'},]},]}
]"
}
]
}
And here's the code I have written so far,
Invoke-WebRequest -Uri 'https://abcd.com/api' | ConvertFrom-Json | Select-Object -ExpandProperty data |ConvertTo-Csv -NoTypeInformation | Set-Content C:\OUTPUT\OUTPUT.csv
The output in excel file is single row record.
How can I split it into all the columns in excel file.
Thanks for all the help !!
As Records field is another JSON embedded in a string value, you have to parse JSON twice:
$response = Invoke-WebRequest -Uri 'https://abcd.com/api'
# Parse outer JSON
$json = $response | ConvertFrom-Json
# Parse embedded JSON
$recJson = $json.data.records | ConvertFrom-Json
This is just to get you started. You can't directly convert $recJson to CSV because of the nested FieldGroups array. You have to process it separately. If you have specific issues with that, it's better asked as a new question.
So I'm trying to pull the number of hours worked and the date worked from a table in my companies database to make a chart in Power BI through a streaming data set. I'm using powershell to parse a JSON file
Here's a JSON sample:
{"COUNT":"334","DISPLAY_LIST_START":"1","DISPLAY_LIST_STOP":"334","STOP":"334","RECORD":[{"SESSION_ID":"c_a7FdTFicmxBJh9kln4V6gKxz_QErcufE7URF9m","FIELD":["6",["04/23/2018"]]},{"SESSION_ID":"c_a7FdTFicmxBJh9kln4V6gKxz_QErcufE7URF9m","FIELD":["6",["04/24/2018"]]},{"SESSION_ID":"c_a7FdTFicmxBJh9kln4V6gKxz_QErcufE7URF9m","FIELD":["6",["04/26/2018"]]},{"SESSION_ID":"c_a7FdTFicmxBJh9kln4V6gKxz_QErcufE7URF9m","FIELD":["6",["04/30/2018"]]},{"SESSION_ID":"c_a7FdTFicmxBJh9kln4V6gKxz_QErcufE7URF9m","FIELD":["6",["05/01/2018"]]},{"SESSION_ID":"c_a7FdTFicmxBJh9kln4V6gKxz_QErcufE7URF9m","FIELD":["4",["05/02/2018"]]},{"SESSION_ID":"c_a7FdTFicmxBJh9kln4V6gKxz_QErcufE7URF9m","FIELD":["6",["05/03/2018"]]},{"SESSION_ID":"c_a7FdTFicmxBJh9kln4V6gKxz_QErcufE7URF9m","FIELD":["6",["05/07/2018"]]},
I know it's not the best in terms of organization, but it's all I have to work with.
Here's the powershell code I have so far:
Invoke-WebRequest -Uri "http://wya.works/rta_develop/xmlServlet?&command=retrieve&sql=select%20%5B%24Hours%5D%2C%20%5B%24Date%20Worked%5D%20from%20%5B%21HOURS%5D%20&attributesOnly=Date%20Worked%2C%20Hours&contentType=JSON&referer=&time=1595443368507&key=696a6768"
$endpoint = "https://api.powerbi.com/beta/d6cdaa23-930e-49c1-9d2a-0fbe648551b2/datasets/91466553-d719-420c-9e3e-73e748379263/rows?noSignUpCheck=1&key=SU5GRBBWuuEIDSjqHW5hdgJzSMCQ3qUQ9mGrBDanjgpExv6woY1Sa1c3PC1Wk3WHHn1N%2FEpIuVgzHHcw0JXwYw%3D%3D"
$json.RECORD | Foreach-Object {
Write-Output "Checking Records"
$hours = 0
$date = ""
$json.FIELD | Foreach-Object{
Write-Output "Checking Field"
if ($_ -match '\d{1,2}/\d{1,2}\/d{4}'){
$date = $_
}
else {
$hours = $_
}
}
$payload = #{
"Hours" = $hours
"Date Worked" =$date
}
}
Invoke-RestMethod -Method Post -Uri "$endpoint" jlk-Body (ConvertTo-Json #($payload))
I need to parse through each record and pull the values of the hours (the numeric value in the JSON) and the Date (the date value).
When I run the code I don't get any errors, but it doesn't seem to be reaching the -match or the else statements. I tried logging the output on both and it returns nothing.
Is there something wrong with my loops?
I'm brand new to powershell and most of this code I got from help from other people, but I understand what its doing for the most part.
Also, anyone who knows about streaming datasets, will pulling this this way even give me what I want?
Store the Invoke-Webrequest values in your $json first. You missed that point; that's why you are getting Null.I don't know it is a typo or a miss.
Secondly, you $json.RECORD is wrong because it doesnt have any Record in the response. What you are looking for is basically the content. $json.content is going to give you the content of numbers.
$json=Invoke-WebRequest -Uri "http://wya.works/rta_develop/xmlServlet?&command=retrieve&sql=select%20%5B%24Hours%5D%2C%20%5B%24Date%20Worked%5D%20from%20%5B%21HOURS%5D%20&attributesOnly=Date%20Worked%2C%20Hours&contentType=JSON&referer=&time=1595443368507&key=696a6768"
Your endpoint and invoke-restmethod has nothing to do with your json parsing. First handle the response in the loop and see what is the outcome you are getting. I have structured it but I have not worked on the JSON sample data as if now:
$json=Invoke-WebRequest -Uri "http://wya.works/rta_develop/xmlServlet?&command=retrieve&sql=select%20%5B%24Hours%5D%2C%20%5B%24Date%20Worked%5D%20from%20%5B%21HOURS%5D%20&attributesOnly=Date%20Worked%2C%20Hours&contentType=JSON&referer=&time=1595443368507&key=696a6768"
$json.content | Foreach-Object {
Write-Output "Checking Records"
$hours = 0
$date = ""
$json.FIELD | Foreach-Object{
Write-Output "Checking Field"
if ($_ -match '\d{1,2}/\d{1,2}\/d{4}'){
$date = $_
}
else {
$hours = $_
}
}
$payload = #{
"Hours" = $hours
"Date Worked" =$date
}
}
$endpoint = "https://api.powerbi.com/beta/d6cdaa23-930e-49c1-9d2a-0fbe648551b2/datasets/91466553-d719-420c-9e3e-73e748379263/rows?noSignUpCheck=1&key=SU5GRBBWuuEIDSjqHW5hdgJzSMCQ3qUQ9mGrBDanjgpExv6woY1Sa1c3PC1Wk3WHHn1N%2FEpIuVgzHHcw0JXwYw%3D%3D"
Invoke-RestMethod -Method Post -Uri "$endpoint" jlk-Body (ConvertTo-Json #($payload))
The root of your problem is this:
$json.RECORD | Foreach-Object {
$json.FIELD | Foreach-Object{
...
}
}
Your outer foreach-object is looping over each item in the RECORD array, but the inner foreach-object is trying to loop over top-level FIELD properties that don't actually exist!
However, I think you'll hit more problems further along your code if you try to troubleshoot what you've already got, and there's an easier way to do what you're trying to do...
First, your sample json isn't quite valid - there's a trailing comma and some unclosed brackets so I'm going to reformat it with some line breaks so it's easier to read, and then fix it.
Note - you'll get the text from your call to Invoke-WebRequest, or as #Lee_Dailey suggested, Invoke-RestMethod.
# reformat the json, fix it, and assign it to a variable using a "Here-String"
# (see https://learn.microsoft.com/en-us/powershell/module/microsoft.powershell.core/about/about_quoting_rules?view=powershell-7#here-strings)
$text = #"
{
"COUNT":"334",
"DISPLAY_LIST_START":"1",
"DISPLAY_LIST_STOP":"334",
"STOP":"334",
"RECORD":[
{"SESSION_ID":"c_a7FdTFicmxBJh9kln4V6gKxz_QErcufE7URF9m","FIELD":["6",["04/23/2018"]]},
{"SESSION_ID":"c_a7FdTFicmxBJh9kln4V6gKxz_QErcufE7URF9m","FIELD":["6",["04/24/2018"]]},
{"SESSION_ID":"c_a7FdTFicmxBJh9kln4V6gKxz_QErcufE7URF9m","FIELD":["6",["04/26/2018"]]},
{"SESSION_ID":"c_a7FdTFicmxBJh9kln4V6gKxz_QErcufE7URF9m","FIELD":["6",["04/30/2018"]]},
{"SESSION_ID":"c_a7FdTFicmxBJh9kln4V6gKxz_QErcufE7URF9m","FIELD":["6",["05/01/2018"]]},
{"SESSION_ID":"c_a7FdTFicmxBJh9kln4V6gKxz_QErcufE7URF9m","FIELD":["4",["05/02/2018"]]},
{"SESSION_ID":"c_a7FdTFicmxBJh9kln4V6gKxz_QErcufE7URF9m","FIELD":["6",["05/03/2018"]]},
{"SESSION_ID":"c_a7FdTFicmxBJh9kln4V6gKxz_QErcufE7URF9m","FIELD":["6",["05/07/2018"]]}
]
}
"#
Then, we'll parse it - i.e. convert it from a string into an object model with well-defined properties:
# parse the json text
$json = $text | ConvertFrom-Json
And then process each record in turn:
$json.RECORD | ForEach-Object {
# get the number of hours worked. this is the first value in the FIELD array
# (note - the array starts at index 0 because it's zero-indexed)
$hours = $_.FIELD[0] # e.g. "6"
# get the "date worked". we need to get the second value (index 1) in
# the FIELD array, but this is a nested array, so once we've got the
# inner array we need to get the first value (index 0 again) from that
$date = $_.FIELD[1][0] # e.g. "04/23/2018"
# now we can build the payload...
$payload = #{
"Hours" = $hours
"Date Worked" = $date
}
# ...and invoke the api for this record
$endpoint = "https://api.powerbi.com/beta/d6cdaa23-930e-49c1-9d2a-0fbe648551b2/datasets/91466553-d719-420c-9e3e-73e748379263/rows?noSignUpCheck=1&key=SU5GRBBWuuEIDSjqHW5hdgJzSMCQ3qUQ9mGrBDanjgpExv6woY1Sa1c3PC1Wk3WHHn1N%2FEpIuVgzHHcw0JXwYw%3D%3D"
Invoke-RestMethod -Method Post -Uri "$endpoint" -Body (ConvertTo-Json #($payload))
}
Note the api call is inside the foreach loop, otherwise you end up calculating the $payload for each RECORD but only ever actually calling the api for the last one.
(I've also removed a spurious "jlk" from your final line, which is probably a typo).
I have a working call to a rest service using Invoke-RestMethod -Uri https://.... The call's result, in JSON, can be piped the data to Format-Table -Property ... and data is shown.
But when Select-Object -Property ... is used after the invoke with the same parameters, the PSObject has the columns but no data for them. If I use a different webservice the call will work.
What could be causing the PSObject to not show any values?
Working Example on public rest web services
Invoke-RestMethod -Uri https://jsonplaceholder.typicode.com/todos/1 |
Select-Object -Property title
Result
#{title=delectus aut autem}
New Failure Different API
Invoke-RestMethod -Uri https://cat-fact.herokuapp.com/facts | Select-Object -Property text
You've stumbled upon an unholy combination of two PowerShell oddities when converting JSON arrays:
Invoke-RestMethod and ConvertFrom-Json send converted-from-JSON arrays as a whole through the pipeline, instead of element by element, as usual:
Note: In PowerShell (Core) 7.0, ComvertFrom-Json's behavior was changed to align with the usual enumeration-of-elements behavior, and a -NoEnumerate switch was added as an opt-in to the old behavior. For the discussion that led to this change, see GitHub issue #3424.
However, as of this writing (PowerShell (Core 7.2) Invoke-RestMethod still exhibits this unexpected behavior, which is discussed in GitHub issue #15272.
Select-Object does not perform member-access enumeration, so it looks for the specified property (e.g., text) directly on the array, where it doesn't exist.
To demonstrate the problem with a simple example:
# Windows PowerShell only:
# Because ConvertFrom-Json sends an *array* (of 2 custom objects) through
# the pipeline, Select-Object looks for property .text on the *array* -
# and can't find it.
# The same applies to Invoke-RestMethod (also still in
# PowerShell (Core) as of v7.2)
PS> ConvertFrom-Json '[{ "text": "a" }, { "text": "b" }]' | Select-Object text
text
----
# NO VALUES
A simple workaround is to enclose the ConvertFrom-Json / Invoke-RestMethod call in (...), which forces enumeration of the array, causing Select-Object to work as expected.:
# (...) forces enumeration
PS> (ConvertFrom-Json '[{ "text": "a" }, { "text": "b" }]') | Select-Object text
text
----
a
b
Note that a command such as Select-Object -Property text (without -ExpandProperty) still output custom objects that have a .text property, not the .text property values.
If all you're interested in is property values, the solution is simpler, because you can use the above-mentioned member-access enumeration directly on the array:
# Using .<propName> on an array (collection) implicitly returns the
# property values from the *elements* of that collection (member-access enumeration).
PS> (ConvertFrom-Json '[{ "text": "a" }, { "text": "b" }]').text
a
b
Note how the output now has no text header, because it is mere string values that are being output, not custom objects.
your problem in the 2nd example is that there is NO prop named text. [grin]
the only prop is all and that contains an array of objects that DO contain a prop named text. so you need something that can get that deeper prop. one way is to use two Select-Object calls. something like this ...
$Url = 'https://cat-fact.herokuapp.com/facts'
$RawIRM = Invoke-RestMethod -Uri $Url
$SO_IRM = $RawIRM |
Select-Object -ExpandProperty all |
Select-Object -Property text
the $SO_IRM var now has an array of 178 strings about cats. [grin]
I am trying to get JSON details from an api using powershell.
I do not have any expereince with power shell, but I guess Invoke-RestMethod would be helpful.
the api I am trying to use is :
api.spotcrime.com/crimes.json?lat=30.639155&lon=-96.3647937&radius=0.02&key=spotcrime-private-api-key
Please read the online documentation: Invoke-RestMethod
Here is a sample showing how to access the data returned from this API:
$uri = 'api.spotcrime.com/crimes.json'
$params = #{
lat = '30.639155'
lon = '-96.3647937'
radius = '0.02'
key = 'spotcrime-private-api-key'
}
$crimes = Invoke-RestMethod -Uri $uri -Body $params |
Select-Object -ExpandProperty crimes
$crimes | Format-Table -AutoSize
In this dataset we have an object named crimes which contains an array of records. Expanding the crimes object and storing it to a $crimes variable will allow us to work with the records more easily.
Notice how Invoke-RestMethod automatically detects and converts JSON into objects. No need to use ConvertFrom-Json.
I want to Import selected data from Json url and so I can convert it to XML.
I am using following code to import.
(Invoke-RestMethod -URI "http://www.broadbandmap.gov/broadbandmap/broadband/dec2013/wireline?latitude=29.488412&longitude=-98.550208&format=json").Results.wirelineServices.providerName | Select-Object | Format-Table –AutoSize
so I am using .Results.wirelineServices.providerName to pull selected columns from one branch/table.
how can I pull data from .Results.broadbandSource.stateFips also at same time?
Thanks bunch.
Json code screenshot.
follow up question
I think that you should separate out your steps a bit:
$r = Invoke-RestMethod -URI "http://www.broadbandmap.gov/broadbandmap/broadband/dec2013/wireline?latitude=29.488412&longitude=-98.550208&format=json"
$providers = $r.Results.wirelineServices.providerName
$stateFips = $r.Results.broadbandSource.stateFips
Note that in your example, your call to Select-Object is redundant (you're not selecting anything, so it's not changing the input object).
Also, a very important point about Format-Table (and any Format- cmdlet) is that those are for display only so they should always be the last thing you do, if in fact they're needed at all.
The code I've given gives you the information in objects, which you can then work with, filter or, display as needed. I'm not sure how you wanted to use/display it, but since there are multiple providers and only one stateFips value, I might assume that you would apply the Fips value to each provider. Here's an example of that which uses the $stateFips variable we created:
$r.Results.wirelineServices | Select-Object providerName,#{Name='stateFips' ; Expression={ $stateFips }}
And here's an example that uses only the original result $r:
$r.Results.wirelineServices | Select-Object providerName,#{Name='stateFips' ; Expression={ $r.Results.broadbandSource.stateFips }}
The Select-Object computed column syntax
Note that the second column definition looks a bit wonky. It's actually a hashtable that allows you to specify the name of the column, and an expression (a complete code block) whose return value will be the value of the column. It could be spread over multiple lines like this:
$r.Results.wirelineServices | Select-Object providerName,#{
Name = 'stateFips'
Expression = {
$r.Results.broadbandSource.stateFips
}
}
Or you could even create the hashtable as a variable first:
$computed = #{
Name = 'stateFips'
Expression = {
$r.Results.broadbandSource.stateFips
}
}
$r.Results.wirelineServices | Select-Object providerName,$computed
XML?
#Stephen Connolly's answer reminded me that you wanted to make XML out of this. Let's take the above code and assign it to a variable:
$computed = #{
Name = 'stateFips'
Expression = {
$r.Results.broadbandSource.stateFips
}
}
$data = $r.Results.wirelineServices | Select-Object providerName,$computed
Because $data is still an object and wasn't sent through a Format- command, we can still use it!
$xml = $data | ConvertTo-Xml -NoTypeInformation
As his comment also suggested though, we don't know how you wanted the resultant XML to be formatted.
So here's another approach:
Forget the JSON
$r = Invoke-RestMethod -URI "http://www.broadbandmap.gov/broadbandmap/broadband/dec2013/wireline?latitude=29.488412&longitude=-98.550208&format=xml"
Now $r contains XML already. You can filter it out and modify it using XPATH. I won't get into that at the moment unless you think that way would work better for you.
Hope this helps, let me know if I've misunderstood what you're trying to do here.
If you want a composite object try something like
$results = (Invoke-RestMethod -URI "http://www.broadbandmap.gov/broadbandmap/broadband/dec2013/wireline?latitude=29.488412&longitude=-98.550208&format=json")
$obj = $results.Results.wirelineServices
$obj | add-member -type noteproperty -Name StateFips -Value $($results.Results.broadbandSource.stateFips) -PassThru
$obj | convertto-xml -as string