looking for some help.
I'm newbie to powershell and looking to call an API which has JSON output (encapsulated in single array).
Expected Output (couldn't get all columns in excel to show):
Below is the response from API:
{
"data": [
{
"Records": "[
{
'RecordId':'A8E00212-BE7B-4AA8-A87B-121E7018EAE0',
'RecordNumber':'1999',
'FormName':'Costco Boneless Center Cut Loin (Strap On, Strap Off, Super Trim, Premium)',
'FormVersion':'22',
'FormType':'Check',
'LocationName':'Beardstown (85B)',
'ResourceName':'Loins-Costco Boneless CC Loin',
'SubmittedUserName':'Maria Nava-Garcia',
'SubmittedDate':'2020-11-14T16:54:35.920',
'SubmittedDateLocal':'2020-11-14T16:54:35.920',
'SubmittedTimeZone':'America/Chicago',
'ProcessedDate':'2020-11-14T17:41:23.873',
'ModifiedDate':'2020-11-14T16:54:35.920',
'FieldGroups': [
{
'FieldGroupName':'' ,
'Fields': [
{'Date':'11/14/2020'},
{'Shift':'1'},
{'Product Code':'55620'},
{'Number of Pieces Audited':'8'},
{'Incorrect Bagging Orientation, S-Shaped Bags':'0'},
{'Incorrect Pieces in Bag':'0'},
{'Incorrect Boxing Placement':'0'},
{'Bag Tails':'0'},
{'Dating Over Bag Print':'0'},
{'Bones/Cartilage':'1'},
{'Number of Leakers':'0'},
{'Foreign Material/Contamination':'0'},
{'PSE':'0'},
{'Incorrect Code Dating and Est. #':'0'},
{'Blood Splash':'0'},
{'Dark Bruise':'0'},
{'Scores':'0'},
{'Sirloin End Not Square':'0'},
{'Pin Bone Removal':'0'},
{'Shoulder End Eye Diameter':'0'},
{'Gouges':'0'},
{'Fat':'2'},
{'Remaining Strap (Strap Off Codes Only)':'0'},
{'Belly Strap (Strap On Codes Only)':'0'},
{'Ham Patch Present':'0'},
{'False Lean Present':'1'},
{'Tag Ends':'0'},
{'Packaging Defects Total':'0'},
{'% Packaging Defects':'0'},
{'Critical Defects Total':'1'},
{'% Critical Defects':'12.5'},
{'Other Defects Total':'3'},
{'% Other Defects':'37.5'},
{'Total Defects':'4'},
{'Audit Acceptable':'No'},
{'What Did You Do?':'Tagged'},
{'How Many Boxes?':'15'},
{'Comments':'Notified Dave Long Hector Rodriguez and Jeff Harris'},]},]}
]"
}
]
}
And here's the code I have written so far,
Invoke-WebRequest -Uri 'https://abcd.com/api' | ConvertFrom-Json | Select-Object -ExpandProperty data |ConvertTo-Csv -NoTypeInformation | Set-Content C:\OUTPUT\OUTPUT.csv
The output in excel file is single row record.
How can I split it into all the columns in excel file.
Thanks for all the help !!
As Records field is another JSON embedded in a string value, you have to parse JSON twice:
$response = Invoke-WebRequest -Uri 'https://abcd.com/api'
# Parse outer JSON
$json = $response | ConvertFrom-Json
# Parse embedded JSON
$recJson = $json.data.records | ConvertFrom-Json
This is just to get you started. You can't directly convert $recJson to CSV because of the nested FieldGroups array. You have to process it separately. If you have specific issues with that, it's better asked as a new question.
Related
Taking an example from JQs official site, I am trying to parse the jsonArray in below example screenshot.
I am trying to get both jsonObjects, so that I can do something with them for my problem statement. However, when I try to do this in PowerShell using below code:
$inputJson = '[{"name":"JSON", "good":true}, {"name":"XML", "good":false}]'
$items = $inputJson | jq '.[]'
I get not two $items objects but 8 $items psobjects. As seen in the second screenshot.
Any pointers, on what changes I can make to get only two jsonObjects here?
The question is confusing. All jq does is output strings. Powershell will make each line a new string array element. The only way I see it doing what you want, is with jq's --compact-output option, which outputs one object (to jq) per line. That seems to be the way that online example was run, anyway. But to powershell, these are two strings. They would be objects to powershell only after using convertfrom-json.
$items = $inputJson | jq '.[]' --compact-output
$items
{"name":"JSON","good":true}
{"name":"XML","good":false}
Although still dont know how to do it with jq, here is another way of achieving what I wanted using Powershell.
$inputJson = '[{"name":"JSON", "good":true}, {"name":"XML", "good":false}]' | ConvertFrom-Json
write-host "`nNr of jsonObjects:"
write-host $inputJson.Length
write-host "`nfirst jsonObject:"
Write-Host $inputJson[0]
write-host "`njsonObjects value:"
$value = $inputJson[0] | ConvertTo-Json
Write-Host $value
Result is:
Nr of jsonObjects:
2
first jsonObject:
#{name=JSON; good=True}
jsonObjects value:
{
"name": "JSON",
"good": true
}
I have a working call to a rest service using Invoke-RestMethod -Uri https://.... The call's result, in JSON, can be piped the data to Format-Table -Property ... and data is shown.
But when Select-Object -Property ... is used after the invoke with the same parameters, the PSObject has the columns but no data for them. If I use a different webservice the call will work.
What could be causing the PSObject to not show any values?
Working Example on public rest web services
Invoke-RestMethod -Uri https://jsonplaceholder.typicode.com/todos/1 |
Select-Object -Property title
Result
#{title=delectus aut autem}
New Failure Different API
Invoke-RestMethod -Uri https://cat-fact.herokuapp.com/facts | Select-Object -Property text
You've stumbled upon an unholy combination of two PowerShell oddities when converting JSON arrays:
Invoke-RestMethod and ConvertFrom-Json send converted-from-JSON arrays as a whole through the pipeline, instead of element by element, as usual:
Note: In PowerShell (Core) 7.0, ComvertFrom-Json's behavior was changed to align with the usual enumeration-of-elements behavior, and a -NoEnumerate switch was added as an opt-in to the old behavior. For the discussion that led to this change, see GitHub issue #3424.
However, as of this writing (PowerShell (Core 7.2) Invoke-RestMethod still exhibits this unexpected behavior, which is discussed in GitHub issue #15272.
Select-Object does not perform member-access enumeration, so it looks for the specified property (e.g., text) directly on the array, where it doesn't exist.
To demonstrate the problem with a simple example:
# Windows PowerShell only:
# Because ConvertFrom-Json sends an *array* (of 2 custom objects) through
# the pipeline, Select-Object looks for property .text on the *array* -
# and can't find it.
# The same applies to Invoke-RestMethod (also still in
# PowerShell (Core) as of v7.2)
PS> ConvertFrom-Json '[{ "text": "a" }, { "text": "b" }]' | Select-Object text
text
----
# NO VALUES
A simple workaround is to enclose the ConvertFrom-Json / Invoke-RestMethod call in (...), which forces enumeration of the array, causing Select-Object to work as expected.:
# (...) forces enumeration
PS> (ConvertFrom-Json '[{ "text": "a" }, { "text": "b" }]') | Select-Object text
text
----
a
b
Note that a command such as Select-Object -Property text (without -ExpandProperty) still output custom objects that have a .text property, not the .text property values.
If all you're interested in is property values, the solution is simpler, because you can use the above-mentioned member-access enumeration directly on the array:
# Using .<propName> on an array (collection) implicitly returns the
# property values from the *elements* of that collection (member-access enumeration).
PS> (ConvertFrom-Json '[{ "text": "a" }, { "text": "b" }]').text
a
b
Note how the output now has no text header, because it is mere string values that are being output, not custom objects.
your problem in the 2nd example is that there is NO prop named text. [grin]
the only prop is all and that contains an array of objects that DO contain a prop named text. so you need something that can get that deeper prop. one way is to use two Select-Object calls. something like this ...
$Url = 'https://cat-fact.herokuapp.com/facts'
$RawIRM = Invoke-RestMethod -Uri $Url
$SO_IRM = $RawIRM |
Select-Object -ExpandProperty all |
Select-Object -Property text
the $SO_IRM var now has an array of 178 strings about cats. [grin]
How does one convert a text file's contents into a string and then insert this string into a JSON file?
For example, if a file contains:
this
is
a
sample
file
The script would generate:
"this\r\nis\r\na\r\nsample\r\nfile"
To insert into a JSON template:
"something":"<insertPoint>"
To produce:
"something":"this\r\nis\r\na\r\nsample\r\nfile"
I'm using Powershell 5 and have managed to load the file, generate some JSON and insert it by running:
# get contents and convert to JSON
$contentToInsert = Get-Content $sourceFilePath -raw | ConvertTo-Json
# write in output file
(Get-Content $outputFile -Raw).replace('<insertPoint>', $contentToInsert) | Set-Content $outputFile
However, a lot of other, unwanted fields are also added.
"something":"{
"value": "this\r\nis\r\na\r\nsample\r\nfile"
"PSPath": "C:\\src\\intro.md",
"PSParentPath": "C:\\src",
"PSChildName": "intro.md",
etc...
Ultimately, I'm trying to send small rich text segments to a web page via JSON but want to edit and store them locally using Markdown. If this doesn't make sense and there's a better way of sending these then please let me know also.
iRon's answer helpfully suggests not using string manipulation to create JSON in PowerShell, but to use hashtables (or custom objects) to construct the data and then convert it to JSON.
However, that alone does not solve your problem:
PS> #{ something = Get-Content -Raw $sourceFilePath } | ConvertTo-Json
{
"something": {
"value": "this\nis\na\nsample\nfile\n",
"PSPath": "/Users/mklement/Desktop/pg/lines.txt",
# ... !! unwanted properties are still there
}
The root cause of the problem is that Get-Content decorates the strings it outputs with metadata in the form of NoteProperty properties, and ConvertTo-Json currently invariably includes these.
A proposal to allow opting out of this decoration when calling Get-Content can be found in GitHub issue #7537.
Complementarily, GitHub issue #5797 suggests that ConvertTo-Json should ignore the extra properties for primitive .NET types such as strings.
The simplest workaround is to access the underlying .NET instance with .psobject.baseobject, which bypasses the invisible wrapper object PowerShell uses to supply the extra properties:
PS> #{ something = (Get-Content -Raw $sourceFilePath).psobject.baseobject } |
ConvertTo-Json
{
"something": "this\nis\na\nsample\nfile\n"
}
Just a general recommendation apart from the actually issue described by #mklement0 and metadata added to the Get-Content results:
Do not poke (replace, insert, etc.) in any Json content.
Instead, modify the object (if necessary, use ConvertFrom-Json to restore the object) prior converting it into (ConvertTo-Json) a Json file.
In this example, I would use a hash-table with a here-string for this:
#{'something' = #'
this
is
a
sample
file
'#
} | ConvertTo-Json
Result:
{
"something": "this\nis\na\nsample\nfile"
}
You can use the Out-String cmdlet to coerce the output of Get-Content into a flat string first:
#{ "something" = (Get-Content lines.txt | Out-String) } | ConvertTo-Json
This produces:
{
"something": "this\r\nis\r\na\r\nsample\r\nfile\r\n"
}
I have a simple json file as follows:
[
{"ClientName": "Test Site 1", "ClientID": "000001"},
{"ClientName": "Test Site 2", "ClientID": "000002"},
{"ClientName": "Test Site 3", "ClientID": "000003"}
]
When I use the following PowerShell command:
ConvertFrom-Json (Get-Content TestSites.json -Raw)
I get back a System.Object[]. This doesn't allow me to pipe the output to another function I have which accepts "ClientName" and "ClientID" parameters.
However, when I assign that object to another variable, like this:
$myobj = ConvertFrom-Json (Get-Content TestSites.json -Raw)
$myobj is actually a System.Management.Automation.PSCustomObject which is capable of being passed to my function.
How can I just pipe the results of the original command without having to assign it to another variable first?
I hope that makes sense.
Your JSON is an array correct? PowerShell will unroll arrays in the pipeline unless you explicity change that behavior. Assuming your JSON is stored in the variable $json as a single string consider the following examples.
ConvertFrom-Json $json | ForEach-Object{$_.gettype().fullname}
System.Object[]
(convertFrom-Json $json) | ForEach-Object{$_.gettype().fullname}
System.Management.Automation.PSCustomObject
System.Management.Automation.PSCustomObject
System.Management.Automation.PSCustomObject
You should be able to wrap the expression in brackets to change the outcome. In the second example it should be sending the 3 objects individually down the pipe. In the first it is being sent as a single object array.
My explanation needs work but I am sure of the cause is how PowerShell deals with arrays and the pipeline. Unrolling being a common word used to describe it.
So depending on your use case you might just be able to wrap the expression in brackets so it gets processed before the pipe to ForEach in my example.
If you have an array such as System.Object[] you could try piping via foreach:
ConvertFrom-Json (Get-Content TestSites.json -Raw) | %{ $_ | your-function }
If you want to pass the whole array down the pipe as-is, you can try adding a comma (aka a unary comma before the variable:
,$myobj | whatever
You can probably see how the latter works by comparing the following:
$myobj | Get-Member # Shows the type of the elements of the array
Get-Member -InputObject $myobj # Shows the type of the array
,$myobj | Get-Member # Shows the type of the array
I am having an issue with a JSON API call using Powershell, that returns a multi-level nested array. Using the standard convertfrom-json & convertto-json doesn't give me the full results. I just get a ...
I have done a lot of research on this and found the following link on stack for expanding the nested hash tables, and this is about the best format that I have gotten. For reference, I am trying to insert employee records into SQL Server. I have a JSON function for doing so, and need to get the JSON response in the expanded format. There are some security aspects, that won't allow me to post the actual detail so I am trying to provide an example of some issues that I am having. Keep in mind the response is for 1 employee record out of hundreds overall, and in SQL will be one table with a row per employee.
$json1 = Invoke-webRequest -Uri $Workeruri -Certificate $cert -Headers $WorkerHeader | convertfrom-json
$workers = $json1.workers | select -property *
$workers | format-table Name, #{n='Value'; e={
if ($_.Value -is [Hashtable]) {
$ht = $_.Value
$a = $ht.keys | sort | % { '{0}={1}' -f $_, $ht[$_] }
'{{{0}}}' -f ($a -join ', ')
} else {
$_.Value
}
}}
$workers
I have also tried:
$workers | convertto-json
and
$workers = json1.workers | foreach-object {$_}
The data returned comes back like this:
associateOID : XXXXXXXXXXXXXX
workerID : #{idValue=XXXXXXX}
person : #{governmentIDs=System.Object[]; legalName=; birthDate=0000-00-00; legalAddress=; genderCode=; maritalStatusCode=; socialInsurancePrograms=System.Object[]; tobaccoUserIndicator=False; raceCode=; customFieldGroup=}
workerDates : #{originalHireDate=2015-03-09; terminationDate=2016-03-18}
workerStatus : #{statusCode=}
businessCommunication : #{emails=System.object[]}
workAssignments : {#{itemID=xxxxxx; occupationalclassification=System.Object[]; etc}}
I need it to come back with all of the columns on the left side, utilizing the "AssociateOID" as the key identifier for the individual. I have previously gotten the JSON response to come back completely expanded using this format but it wasn't working with the import into SQL Server or looking very nice like the Google Postman response:
$json1 = Invoke-webRequest -Uri $Workeruri -Certificate $cert -Headers $WorkerHeader | format-list -property *
You mean you want to get a JSON file that all objects/properties are expanded, right? Use the -Depth option of ConvertTo-Json to write a JSON file properly. Default value of -Depth option is 2.
$workers | ConvertTo-Json -Depth 100
Specify the appropriate depth according to your JSON files.