Powershell Invoke-WebRequest and character encoding - json

I am trying to get information from the Spotify database through their Web API.
However, I'm facing issues with accented vowels (ä,ö,ü etc.)
Lets take Tiësto as an example.
Spotify's API Browser can display the information correctly:
https://developer.spotify.com/web-api/console/get-artist/?id=2o5jDhtHVPhrJdv3cEQ99Z
If I make a API call with Invoke-Webrequest I get
Ti??sto
as name:
function Get-Artist {
param($ArtistID = '2o5jDhtHVPhrJdv3cEQ99Z',
$AccessToken = 'MyAccessToken')
$URI = "https://api.spotify.com/v1/artists/{0}" -f $ArtistID
$JSON = Invoke-WebRequest -Uri $URI -Headers #{"Authorization"= ('Bearer ' + $AccessToken)}
$JSON = $JSON | ConvertFrom-Json
return $JSON
}
How can I get the correct name?

Update: PowerShell (Core) 7.3+ now defaults to UTF-8 in the absence of a (valid) charset attribute in the HTTP response header, so the problem no longer arises there.
Jeroen Mostert, in a comment on the question, explains the problem well:
The problem is that Spotify is (unwisely) not returning the encoding it's using in its headers. PowerShell obeys the standard by assuming ISO-8859-1, but unfortunately the site is using UTF-8. (PowerShell ought to ignore standards here and assume UTF-8, but that's just like, my opinion, man.) More details here, along with the follow-up ticket.
A workaround that doesn't require the use of temporary files:
Assuming that function ConvertTo-BodyWithEncoding has been defined (see below), you can use the following:
# ConvertTo-BodyWithEncoding defaults to UTF-8.
$JSON = Invoke-WebRequest -Uri $URI ... | ConvertTo-BodyWithEncoding
Convenience function ConvertTo-BodyWithEncoding:
Note:
The function manually decodes the raw bytes that make up the given response's body, as UTF-8 by default, or with the given encoding, which can be specified as a [System.Text.Encoding] instance, a code-page number (e.g. 1251), or an encoding name (e.g., 'utf-16le).
The function is also available as an MIT-licensed Gist, and only the latter will be maintained going forward. Assuming you have looked at the linked code to ensure that it is safe (which I can personally assure you of, but you should always check), you can define it directly as follows (instructions for how to make the function available in future sessions or to convert it to a script will be displayed):
irm https://gist.github.com/mklement0/209a9506b8ba32246f95d1cc238d564d/raw/ConvertTo-BodyWithEncoding.ps1 | iex
function ConvertTo-BodyWithEncoding {
[CmdletBinding(PositionalBinding=$false)]
param(
[Parameter(Mandatory, ValueFromPipeline)]
[Microsoft.PowerShell.Commands.WebResponseObject] $InputObject,
# The encoding to use; defaults to UTF-8
[Parameter(Position=0)]
$Encoding = [System.Text.Encoding]::Utf8
)
begin {
if ($Encoding -isnot [System.Text.Encoding]) {
try {
$Encoding = [System.Text.Encoding]::GetEncoding($Encoding)
}
catch {
throw
}
}
}
process {
$Encoding.GetString(
$InputObject.RawContentStream.ToArray()
)
}
}

Issue solved with the workaround provided by Jeron Mostert.
You have to save it in a file and explicit tell Powershell which Encoding it should use.
This workaround works for me because my program can take whatever time it needs (regarding read/write IO)
function Invoke-SpotifyAPICall {
param($URI,
$Header = $null,
$Body = $null
)
if($Header -eq $null) {
Invoke-WebRequest -Uri $URI -Body $Body -OutFile ".\SpotifyAPICallResult.txt"
} elseif($Body -eq $null) {
Invoke-WebRequest -Uri $URI -Headers $Header -OutFile ".\SpotifyAPICallResult.txt"
}
$JSON = Get-Content ".\SpotifyAPICallResult.txt" -Encoding UTF8 -Raw | ConvertFrom-JSON
Remove-Item ".\SpotifyAPICallResult.txt" -Force
return $JSON
}
function Get-Artist {
param($ArtistID = '2o5jDhtHVPhrJdv3cEQ99Z',
$AccessToken = 'MyAccessToken')
$URI = "https://api.spotify.com/v1/artists/{0}" -f $ArtistID
return (Invoke-SpotifyAPICall -URI $URI -Header #{"Authorization"= ('Bearer ' + $AccessToken)})
}
Get-Artist

Related

Azure Runbook test with JSON input parameter? [duplicate]

I am badly struck by this problem. I request you to answer or give a hint. I am running out of options.
I am calling an azure runbook upon high CPU utilization via a WebHook. My problem is inside runbook data is not getting decoded properly. For example, the below line is not printing anything.
Write-Output $WebHookData.RequestHeader
Wheras IF i try to explictly convert the data to JSON, like this
*$WebhookData = ConvertFrom-Json $WebhookData*
then it is a throwing error.
ConvertFrom-Json : Invalid JSON primitive: . At line:6 char:31 +
$WebhookData = $WebhookData | ConvertFrom-Json
By the way, I am trying to use the runbook available on Azure gallery {Vertically scale up an Azure Resource Manager VM with Azure Automation}
My Webhook is called from alert created on VM.
A very strange observation:
Working WebHood Example (found in an example) {"WebhookName":"test1","RequestBody":" [\r\n {\r\n \"Message\": \"Test Message\"\r\n }\r\n****]****"
Not Working(the data sent upon calling runbook from VM):
{"WebhookName":"test2","RequestBody":" {\"schemaId\":\"AzureMonitorMetricAlert\"}}
Thanks
I was getting the same error. From my testing, it appears that when performing a "Test" of the runbook, the Webhook data is received as plain text, but when invoked remotely it comes through already formatted as JSON. Here was my solution to cover both scenarios and so far has been working well...
Param (
[object] $WebhookData
)
# Structure Webhook Input Data
If ($WebhookData.WebhookName) {
$WebhookName = $WebhookData.WebhookName
$WebhookHeaders = $WebhookData.RequestHeader
$WebhookBody = $WebhookData.RequestBody
} ElseIf ($WebhookData) {
$WebhookJSON = ConvertFrom-Json -InputObject $WebhookData
$WebhookName = $WebhookJSON.WebhookName
$WebhookHeaders = $WebhookJSON.RequestHeader
$WebhookBody = $WebhookJSON.RequestBody
} Else {
Write-Error -Message 'Runbook was not started from Webhook' -ErrorAction stop
}
I tried with a webhook, the script Write-Output $WebHookData.RequestHeader should work fine.
And if I use ConvertFrom-Json $WebhookData, I can reproduce your issue, not sure why it occurred, according to the doc, the $WebhookData is also in a JSON format, if it is accepted, you could use ConvertFrom-Json -InputObject $WebhookData.RequestBody, it will work fine.
My runbook:
param
(
[Parameter (Mandatory = $false)]
[object] $WebhookData
)
if ($WebhookData) {
Write-Output $WebhookData.RequestHeader
$Body = ConvertFrom-Json -InputObject $WebhookData.RequestBody
Write-Output $Body
} else
{
Write-Output "Missing information";
exit;
}
The powershell script I used to send a webhook:
$uri = "https://s5events.azure-automation.net/webhooks?token=xxxxxxxxxxxx"
$vms = #(
#{ Name="vm01";ResourceGroup="vm01"},
#{ Name="vm02";ResourceGroup="vm02"}
)
$body = ConvertTo-Json -InputObject $vms
$header = #{ message="StartedbyContoso"}
$response = Invoke-WebRequest -Method Post -Uri $uri -Body $body -Headers $header
$jobid = (ConvertFrom-Json ($response.Content)).jobids[0]
Output:
I had the same problem use following to get webhookdata if using test pane with Alert json as input
if(-Not $WebhookData.RequestBody){
$WebhookData = (ConvertFrom-Json -InputObject $WebhookData)
}
$RequestBody = ConvertFrom-JSON -InputObject $WebhookData.RequestBody

Parsing a JSON Byte Stream Returned from a Web Request using Powershell without writing to a file

I'm using the following Powershell to call a URL that returns a JSON file. (If you were to paste the URL into the browser it would download a file, rather than display the raw JSON):
$fileName = "output.json"
$url = "https://some-url-that-returns-a-json-file-byte-stream"
$jsonFileResponse = Invoke-WebRequest -Method GET -Uri $url -UseDefaultCredentials -OutFile "output.json"
$parsedData = Get-Content -Path $fileName | ConvertFrom-Json
This works fine, and I get my parsed data at the end.
I was wondering, is there a way of doing this without involving the file system, i.e. without having the step of outputting to output.json first, and instead handling this in-memory and without any disk i/o?
The question is less one of converting to 'JSON' and more of just converting to text.
Using [System.Text.Encoding]::ASCII.GetString seems to do the trick.
$url = "https://some-url-that-returns-a-json-file-byte-stream"
$jsonFileResponse = Invoke-WebRequest -Method GET -Uri $url -UseDefaultCredentials
$parsedData = [System.Text.Encoding]::ASCII.GetString($jsonFileResponse.Content) | ConvertFrom-Json

Powershell script not parsing correctly

So I'm trying to pull the number of hours worked and the date worked from a table in my companies database to make a chart in Power BI through a streaming data set. I'm using powershell to parse a JSON file
Here's a JSON sample:
{"COUNT":"334","DISPLAY_LIST_START":"1","DISPLAY_LIST_STOP":"334","STOP":"334","RECORD":[{"SESSION_ID":"c_a7FdTFicmxBJh9kln4V6gKxz_QErcufE7URF9m","FIELD":["6",["04/23/2018"]]},{"SESSION_ID":"c_a7FdTFicmxBJh9kln4V6gKxz_QErcufE7URF9m","FIELD":["6",["04/24/2018"]]},{"SESSION_ID":"c_a7FdTFicmxBJh9kln4V6gKxz_QErcufE7URF9m","FIELD":["6",["04/26/2018"]]},{"SESSION_ID":"c_a7FdTFicmxBJh9kln4V6gKxz_QErcufE7URF9m","FIELD":["6",["04/30/2018"]]},{"SESSION_ID":"c_a7FdTFicmxBJh9kln4V6gKxz_QErcufE7URF9m","FIELD":["6",["05/01/2018"]]},{"SESSION_ID":"c_a7FdTFicmxBJh9kln4V6gKxz_QErcufE7URF9m","FIELD":["4",["05/02/2018"]]},{"SESSION_ID":"c_a7FdTFicmxBJh9kln4V6gKxz_QErcufE7URF9m","FIELD":["6",["05/03/2018"]]},{"SESSION_ID":"c_a7FdTFicmxBJh9kln4V6gKxz_QErcufE7URF9m","FIELD":["6",["05/07/2018"]]},
I know it's not the best in terms of organization, but it's all I have to work with.
Here's the powershell code I have so far:
Invoke-WebRequest -Uri "http://wya.works/rta_develop/xmlServlet?&command=retrieve&sql=select%20%5B%24Hours%5D%2C%20%5B%24Date%20Worked%5D%20from%20%5B%21HOURS%5D%20&attributesOnly=Date%20Worked%2C%20Hours&contentType=JSON&referer=&time=1595443368507&key=696a6768"
$endpoint = "https://api.powerbi.com/beta/d6cdaa23-930e-49c1-9d2a-0fbe648551b2/datasets/91466553-d719-420c-9e3e-73e748379263/rows?noSignUpCheck=1&key=SU5GRBBWuuEIDSjqHW5hdgJzSMCQ3qUQ9mGrBDanjgpExv6woY1Sa1c3PC1Wk3WHHn1N%2FEpIuVgzHHcw0JXwYw%3D%3D"
$json.RECORD | Foreach-Object {
Write-Output "Checking Records"
$hours = 0
$date = ""
$json.FIELD | Foreach-Object{
Write-Output "Checking Field"
if ($_ -match '\d{1,2}/\d{1,2}\/d{4}'){
$date = $_
}
else {
$hours = $_
}
}
$payload = #{
"Hours" = $hours
"Date Worked" =$date
}
}
Invoke-RestMethod -Method Post -Uri "$endpoint" jlk-Body (ConvertTo-Json #($payload))
I need to parse through each record and pull the values of the hours (the numeric value in the JSON) and the Date (the date value).
When I run the code I don't get any errors, but it doesn't seem to be reaching the -match or the else statements. I tried logging the output on both and it returns nothing.
Is there something wrong with my loops?
I'm brand new to powershell and most of this code I got from help from other people, but I understand what its doing for the most part.
Also, anyone who knows about streaming datasets, will pulling this this way even give me what I want?
Store the Invoke-Webrequest values in your $json first. You missed that point; that's why you are getting Null.I don't know it is a typo or a miss.
Secondly, you $json.RECORD is wrong because it doesnt have any Record in the response. What you are looking for is basically the content. $json.content is going to give you the content of numbers.
$json=Invoke-WebRequest -Uri "http://wya.works/rta_develop/xmlServlet?&command=retrieve&sql=select%20%5B%24Hours%5D%2C%20%5B%24Date%20Worked%5D%20from%20%5B%21HOURS%5D%20&attributesOnly=Date%20Worked%2C%20Hours&contentType=JSON&referer=&time=1595443368507&key=696a6768"
Your endpoint and invoke-restmethod has nothing to do with your json parsing. First handle the response in the loop and see what is the outcome you are getting. I have structured it but I have not worked on the JSON sample data as if now:
$json=Invoke-WebRequest -Uri "http://wya.works/rta_develop/xmlServlet?&command=retrieve&sql=select%20%5B%24Hours%5D%2C%20%5B%24Date%20Worked%5D%20from%20%5B%21HOURS%5D%20&attributesOnly=Date%20Worked%2C%20Hours&contentType=JSON&referer=&time=1595443368507&key=696a6768"
$json.content | Foreach-Object {
Write-Output "Checking Records"
$hours = 0
$date = ""
$json.FIELD | Foreach-Object{
Write-Output "Checking Field"
if ($_ -match '\d{1,2}/\d{1,2}\/d{4}'){
$date = $_
}
else {
$hours = $_
}
}
$payload = #{
"Hours" = $hours
"Date Worked" =$date
}
}
$endpoint = "https://api.powerbi.com/beta/d6cdaa23-930e-49c1-9d2a-0fbe648551b2/datasets/91466553-d719-420c-9e3e-73e748379263/rows?noSignUpCheck=1&key=SU5GRBBWuuEIDSjqHW5hdgJzSMCQ3qUQ9mGrBDanjgpExv6woY1Sa1c3PC1Wk3WHHn1N%2FEpIuVgzHHcw0JXwYw%3D%3D"
Invoke-RestMethod -Method Post -Uri "$endpoint" jlk-Body (ConvertTo-Json #($payload))
The root of your problem is this:
$json.RECORD | Foreach-Object {
$json.FIELD | Foreach-Object{
...
}
}
Your outer foreach-object is looping over each item in the RECORD array, but the inner foreach-object is trying to loop over top-level FIELD properties that don't actually exist!
However, I think you'll hit more problems further along your code if you try to troubleshoot what you've already got, and there's an easier way to do what you're trying to do...
First, your sample json isn't quite valid - there's a trailing comma and some unclosed brackets so I'm going to reformat it with some line breaks so it's easier to read, and then fix it.
Note - you'll get the text from your call to Invoke-WebRequest, or as #Lee_Dailey suggested, Invoke-RestMethod.
# reformat the json, fix it, and assign it to a variable using a "Here-String"
# (see https://learn.microsoft.com/en-us/powershell/module/microsoft.powershell.core/about/about_quoting_rules?view=powershell-7#here-strings)
$text = #"
{
"COUNT":"334",
"DISPLAY_LIST_START":"1",
"DISPLAY_LIST_STOP":"334",
"STOP":"334",
"RECORD":[
{"SESSION_ID":"c_a7FdTFicmxBJh9kln4V6gKxz_QErcufE7URF9m","FIELD":["6",["04/23/2018"]]},
{"SESSION_ID":"c_a7FdTFicmxBJh9kln4V6gKxz_QErcufE7URF9m","FIELD":["6",["04/24/2018"]]},
{"SESSION_ID":"c_a7FdTFicmxBJh9kln4V6gKxz_QErcufE7URF9m","FIELD":["6",["04/26/2018"]]},
{"SESSION_ID":"c_a7FdTFicmxBJh9kln4V6gKxz_QErcufE7URF9m","FIELD":["6",["04/30/2018"]]},
{"SESSION_ID":"c_a7FdTFicmxBJh9kln4V6gKxz_QErcufE7URF9m","FIELD":["6",["05/01/2018"]]},
{"SESSION_ID":"c_a7FdTFicmxBJh9kln4V6gKxz_QErcufE7URF9m","FIELD":["4",["05/02/2018"]]},
{"SESSION_ID":"c_a7FdTFicmxBJh9kln4V6gKxz_QErcufE7URF9m","FIELD":["6",["05/03/2018"]]},
{"SESSION_ID":"c_a7FdTFicmxBJh9kln4V6gKxz_QErcufE7URF9m","FIELD":["6",["05/07/2018"]]}
]
}
"#
Then, we'll parse it - i.e. convert it from a string into an object model with well-defined properties:
# parse the json text
$json = $text | ConvertFrom-Json
And then process each record in turn:
$json.RECORD | ForEach-Object {
# get the number of hours worked. this is the first value in the FIELD array
# (note - the array starts at index 0 because it's zero-indexed)
$hours = $_.FIELD[0] # e.g. "6"
# get the "date worked". we need to get the second value (index 1) in
# the FIELD array, but this is a nested array, so once we've got the
# inner array we need to get the first value (index 0 again) from that
$date = $_.FIELD[1][0] # e.g. "04/23/2018"
# now we can build the payload...
$payload = #{
"Hours" = $hours
"Date Worked" = $date
}
# ...and invoke the api for this record
$endpoint = "https://api.powerbi.com/beta/d6cdaa23-930e-49c1-9d2a-0fbe648551b2/datasets/91466553-d719-420c-9e3e-73e748379263/rows?noSignUpCheck=1&key=SU5GRBBWuuEIDSjqHW5hdgJzSMCQ3qUQ9mGrBDanjgpExv6woY1Sa1c3PC1Wk3WHHn1N%2FEpIuVgzHHcw0JXwYw%3D%3D"
Invoke-RestMethod -Method Post -Uri "$endpoint" -Body (ConvertTo-Json #($payload))
}
Note the api call is inside the foreach loop, otherwise you end up calculating the $payload for each RECORD but only ever actually calling the api for the last one.
(I've also removed a spurious "jlk" from your final line, which is probably a typo).

How to send UTF8 to the WordPress REST-API?

Malformed UTF-8 characters, possibly incorrectly encoded
Disclaimer: Character encodings confuse me a lot.
I want to create a lot of posts automated via the WordPress REST-API. I can create posts without an issue - unless I got special characters in my content. Emojis for example.
$params = #{
title = $Title
content = $Content # "foobar" would work just fine;
# "👎äöü" will give me an error
}
Invoke-RestMethod -Uri "https://my.blog.foo/wp-json/v2/posts" `
-Method POST `
-Headers #{ Authorization = ("Basic {0}" -f $ACCESS_TOKEN) } `
-ContentType "application/json" `
-UseBasicParsing `
-Body $($params | ConvertTo-Json)
I assume this actually isn't a WordPress related issue. Anyhow I'm fiddling aroundd for quite a while now and I just can't figure out a way to send my content without malforming ("ö" gets "?") it.
Use -ContentType "application/json; charset=utf-8"; instead.
If it doesn't work, You might also need to encode your body to UTF8 too.
$body = [System.Text.Encoding]::UTF8.GetBytes($body);
Complete example
$params = #{
title = $Title
content = $Content # "foobar" would work just fine;
# "👎äöü" will give me an error
}
$Body = $params | ConvertTo-Json
$Body = [System.Text.Encoding]::UTF8.GetBytes($body)
$Splat_Post= #{
ContentType = "application/json; charset=utf-8"
UseBasicParsing = $true
Method = 'POST'
Uri = "https://my.blog.foo/wp-json/v2/posts"
Headers = #{ Authorization = ("Basic {0}" -f $ACCESS_TOKEN) }
}
Invoke-RestMethod #Splat_Post-Body $Body
Note: I reworked the code to demonstrate splatting .
Should you need to call that method with all the same parameters each time except body, you could call it like that :
Invoke-RestMethod #Splat_Post -Body $Body then just changing the $Body to the content of your desires each time.
You could try and encode the body as UTF8 before posting it
-Body ([System.Text.Encoding]::UTF8.GetBytes(($params | ConvertTo-Json)))

Passing parameter values to a PowerShell script

The following is a snippet from my PowerShell script where the values for the parameters $book and $author are not getting substituted. Please suggest some techniques that I can apply to fix it or share some code that can help me out.
$body = #{
version = '1.0'
inactive = 'false'
yml = { "Service1:\n book: $book\n author: $author\n "} | ConvertFrom-Json
} | ConvertTo-Json
$request = Invoke-WebRequest -UseBasicParsing -Method Post -Uri $uri -Body
$body -Headers $headers -ContentType $contentType
$response = ConvertFrom-Json -InputObject $request.Content
You have some weird stuff going on in this line
... yml = { "Service1:\n book: $book\n author: $author\n "} | ConvertFrom-Json } | ConvertTo-Json
Because it says "do a script block with this body, and try to convert the script block to JSON".
So, if you want to have a JSON string in yml field, you have two options.
Write the proper JSON string yourself:
#{...put the rest of your elements here...; yml = "{Service1:'', book:'$book', author: '$author'}"
Populate a hashtable first and then convert it to JSON string:
#{...put the rest of your elements here...; yml = #{Service1=''; book='$book'; author='$author'} } | ConvertTo-Json