Powershell script not parsing correctly - json

Question

So I'm trying to pull the number of hours worked and the date worked from a table in my companies database to make a chart in Power BI through a streaming data set. I'm using powershell to parse a JSON file
Here's a JSON sample:
{"COUNT":"334","DISPLAY_LIST_START":"1","DISPLAY_LIST_STOP":"334","STOP":"334","RECORD":[{"SESSION_ID":"c_a7FdTFicmxBJh9kln4V6gKxz_QErcufE7URF9m","FIELD":["6",["04/23/2018"]]},{"SESSION_ID":"c_a7FdTFicmxBJh9kln4V6gKxz_QErcufE7URF9m","FIELD":["6",["04/24/2018"]]},{"SESSION_ID":"c_a7FdTFicmxBJh9kln4V6gKxz_QErcufE7URF9m","FIELD":["6",["04/26/2018"]]},{"SESSION_ID":"c_a7FdTFicmxBJh9kln4V6gKxz_QErcufE7URF9m","FIELD":["6",["04/30/2018"]]},{"SESSION_ID":"c_a7FdTFicmxBJh9kln4V6gKxz_QErcufE7URF9m","FIELD":["6",["05/01/2018"]]},{"SESSION_ID":"c_a7FdTFicmxBJh9kln4V6gKxz_QErcufE7URF9m","FIELD":["4",["05/02/2018"]]},{"SESSION_ID":"c_a7FdTFicmxBJh9kln4V6gKxz_QErcufE7URF9m","FIELD":["6",["05/03/2018"]]},{"SESSION_ID":"c_a7FdTFicmxBJh9kln4V6gKxz_QErcufE7URF9m","FIELD":["6",["05/07/2018"]]},
I know it's not the best in terms of organization, but it's all I have to work with.
Here's the powershell code I have so far:
Invoke-WebRequest -Uri "http://wya.works/rta_develop/xmlServlet?&command=retrieve&sql=select%20%5B%24Hours%5D%2C%20%5B%24Date%20Worked%5D%20from%20%5B%21HOURS%5D%20&attributesOnly=Date%20Worked%2C%20Hours&contentType=JSON&referer=&time=1595443368507&key=696a6768"
$endpoint = "https://api.powerbi.com/beta/d6cdaa23-930e-49c1-9d2a-0fbe648551b2/datasets/91466553-d719-420c-9e3e-73e748379263/rows?noSignUpCheck=1&key=SU5GRBBWuuEIDSjqHW5hdgJzSMCQ3qUQ9mGrBDanjgpExv6woY1Sa1c3PC1Wk3WHHn1N%2FEpIuVgzHHcw0JXwYw%3D%3D"
$json.RECORD | Foreach-Object {
Write-Output "Checking Records"
$hours = 0
$date = ""
$json.FIELD | Foreach-Object{
Write-Output "Checking Field"
if ($_ -match '\d{1,2}/\d{1,2}\/d{4}'){
$date = $_
}
else {
$hours = $_
}
}
$payload = #{
"Hours" = $hours
"Date Worked" =$date
}
}
Invoke-RestMethod -Method Post -Uri "$endpoint" jlk-Body (ConvertTo-Json #($payload))
I need to parse through each record and pull the values of the hours (the numeric value in the JSON) and the Date (the date value).
When I run the code I don't get any errors, but it doesn't seem to be reaching the -match or the else statements. I tried logging the output on both and it returns nothing.
Is there something wrong with my loops?
I'm brand new to powershell and most of this code I got from help from other people, but I understand what its doing for the most part.
Also, anyone who knows about streaming datasets, will pulling this this way even give me what I want?

Answer 1

Store the Invoke-Webrequest values in your $json first. You missed that point; that's why you are getting Null.I don't know it is a typo or a miss.
Secondly, you $json.RECORD is wrong because it doesnt have any Record in the response. What you are looking for is basically the content. $json.content is going to give you the content of numbers.
$json=Invoke-WebRequest -Uri "http://wya.works/rta_develop/xmlServlet?&command=retrieve&sql=select%20%5B%24Hours%5D%2C%20%5B%24Date%20Worked%5D%20from%20%5B%21HOURS%5D%20&attributesOnly=Date%20Worked%2C%20Hours&contentType=JSON&referer=&time=1595443368507&key=696a6768"
Your endpoint and invoke-restmethod has nothing to do with your json parsing. First handle the response in the loop and see what is the outcome you are getting. I have structured it but I have not worked on the JSON sample data as if now:
$json=Invoke-WebRequest -Uri "http://wya.works/rta_develop/xmlServlet?&command=retrieve&sql=select%20%5B%24Hours%5D%2C%20%5B%24Date%20Worked%5D%20from%20%5B%21HOURS%5D%20&attributesOnly=Date%20Worked%2C%20Hours&contentType=JSON&referer=&time=1595443368507&key=696a6768"
$json.content | Foreach-Object {
Write-Output "Checking Records"
$hours = 0
$date = ""
$json.FIELD | Foreach-Object{
Write-Output "Checking Field"
if ($_ -match '\d{1,2}/\d{1,2}\/d{4}'){
$date = $_
}
else {
$hours = $_
}
}
$payload = #{
"Hours" = $hours
"Date Worked" =$date
}
}
$endpoint = "https://api.powerbi.com/beta/d6cdaa23-930e-49c1-9d2a-0fbe648551b2/datasets/91466553-d719-420c-9e3e-73e748379263/rows?noSignUpCheck=1&key=SU5GRBBWuuEIDSjqHW5hdgJzSMCQ3qUQ9mGrBDanjgpExv6woY1Sa1c3PC1Wk3WHHn1N%2FEpIuVgzHHcw0JXwYw%3D%3D"
Invoke-RestMethod -Method Post -Uri "$endpoint" jlk-Body (ConvertTo-Json #($payload))

Answer 2

The root of your problem is this:
$json.RECORD | Foreach-Object {
$json.FIELD | Foreach-Object{
...
}
}
Your outer foreach-object is looping over each item in the RECORD array, but the inner foreach-object is trying to loop over top-level FIELD properties that don't actually exist!
However, I think you'll hit more problems further along your code if you try to troubleshoot what you've already got, and there's an easier way to do what you're trying to do...
First, your sample json isn't quite valid - there's a trailing comma and some unclosed brackets so I'm going to reformat it with some line breaks so it's easier to read, and then fix it.
Note - you'll get the text from your call to Invoke-WebRequest, or as #Lee_Dailey suggested, Invoke-RestMethod.
# reformat the json, fix it, and assign it to a variable using a "Here-String"
# (see https://learn.microsoft.com/en-us/powershell/module/microsoft.powershell.core/about/about_quoting_rules?view=powershell-7#here-strings)
$text = #"
{
"COUNT":"334",
"DISPLAY_LIST_START":"1",
"DISPLAY_LIST_STOP":"334",
"STOP":"334",
"RECORD":[
{"SESSION_ID":"c_a7FdTFicmxBJh9kln4V6gKxz_QErcufE7URF9m","FIELD":["6",["04/23/2018"]]},
{"SESSION_ID":"c_a7FdTFicmxBJh9kln4V6gKxz_QErcufE7URF9m","FIELD":["6",["04/24/2018"]]},
{"SESSION_ID":"c_a7FdTFicmxBJh9kln4V6gKxz_QErcufE7URF9m","FIELD":["6",["04/26/2018"]]},
{"SESSION_ID":"c_a7FdTFicmxBJh9kln4V6gKxz_QErcufE7URF9m","FIELD":["6",["04/30/2018"]]},
{"SESSION_ID":"c_a7FdTFicmxBJh9kln4V6gKxz_QErcufE7URF9m","FIELD":["6",["05/01/2018"]]},
{"SESSION_ID":"c_a7FdTFicmxBJh9kln4V6gKxz_QErcufE7URF9m","FIELD":["4",["05/02/2018"]]},
{"SESSION_ID":"c_a7FdTFicmxBJh9kln4V6gKxz_QErcufE7URF9m","FIELD":["6",["05/03/2018"]]},
{"SESSION_ID":"c_a7FdTFicmxBJh9kln4V6gKxz_QErcufE7URF9m","FIELD":["6",["05/07/2018"]]}
]
}
"#
Then, we'll parse it - i.e. convert it from a string into an object model with well-defined properties:
# parse the json text
$json = $text | ConvertFrom-Json
And then process each record in turn:
$json.RECORD | ForEach-Object {
# get the number of hours worked. this is the first value in the FIELD array
# (note - the array starts at index 0 because it's zero-indexed)
$hours = $_.FIELD[0] # e.g. "6"
# get the "date worked". we need to get the second value (index 1) in
# the FIELD array, but this is a nested array, so once we've got the
# inner array we need to get the first value (index 0 again) from that
$date = $_.FIELD[1][0] # e.g. "04/23/2018"
# now we can build the payload...
$payload = #{
"Hours" = $hours
"Date Worked" = $date
}
# ...and invoke the api for this record
$endpoint = "https://api.powerbi.com/beta/d6cdaa23-930e-49c1-9d2a-0fbe648551b2/datasets/91466553-d719-420c-9e3e-73e748379263/rows?noSignUpCheck=1&key=SU5GRBBWuuEIDSjqHW5hdgJzSMCQ3qUQ9mGrBDanjgpExv6woY1Sa1c3PC1Wk3WHHn1N%2FEpIuVgzHHcw0JXwYw%3D%3D"
Invoke-RestMethod -Method Post -Uri "$endpoint" -Body (ConvertTo-Json #($payload))
}
Note the api call is inside the foreach loop, otherwise you end up calculating the $payload for each RECORD but only ever actually calling the api for the last one.
(I've also removed a spurious "jlk" from your final line, which is probably a typo).