Malformed UTF-8 characters, possibly incorrectly encoded
Disclaimer: Character encodings confuse me a lot.
I want to create a lot of posts automated via the WordPress REST-API. I can create posts without an issue - unless I got special characters in my content. Emojis for example.
$params = #{
title = $Title
content = $Content # "foobar" would work just fine;
# "👎äöü" will give me an error
}
Invoke-RestMethod -Uri "https://my.blog.foo/wp-json/v2/posts" `
-Method POST `
-Headers #{ Authorization = ("Basic {0}" -f $ACCESS_TOKEN) } `
-ContentType "application/json" `
-UseBasicParsing `
-Body $($params | ConvertTo-Json)
I assume this actually isn't a WordPress related issue. Anyhow I'm fiddling aroundd for quite a while now and I just can't figure out a way to send my content without malforming ("ö" gets "?") it.
Use -ContentType "application/json; charset=utf-8"; instead.
If it doesn't work, You might also need to encode your body to UTF8 too.
$body = [System.Text.Encoding]::UTF8.GetBytes($body);
Complete example
$params = #{
title = $Title
content = $Content # "foobar" would work just fine;
# "👎äöü" will give me an error
}
$Body = $params | ConvertTo-Json
$Body = [System.Text.Encoding]::UTF8.GetBytes($body)
$Splat_Post= #{
ContentType = "application/json; charset=utf-8"
UseBasicParsing = $true
Method = 'POST'
Uri = "https://my.blog.foo/wp-json/v2/posts"
Headers = #{ Authorization = ("Basic {0}" -f $ACCESS_TOKEN) }
}
Invoke-RestMethod #Splat_Post-Body $Body
Note: I reworked the code to demonstrate splatting .
Should you need to call that method with all the same parameters each time except body, you could call it like that :
Invoke-RestMethod #Splat_Post -Body $Body then just changing the $Body to the content of your desires each time.
You could try and encode the body as UTF8 before posting it
-Body ([System.Text.Encoding]::UTF8.GetBytes(($params | ConvertTo-Json)))
Related
I have the following PowerShell API script:
$VMname = "abcd"
$IP_address = '2.2.2.2'
$url = "https://ansibletower.xyz.com/api/v2/job_templates/12321/launch/"
$token = "9998980fsfdfsdfdf"
$headers = #{Authorization = "Bearer $token"}
$contentType = "application/json"
$method = "POST"
#### $body = '{"extra_vars": {"vm_name": "abc", "vm_ip_address": "2.2.2.2"}}'
$body = '{"extra_vars":{"vm_name":'$VMname', "vm_ip_address":'$IP_address'}}'
Invoke-RestMethod -ContentType "$contentType" -Uri $url -Method $method -Headers $headers -Body $body
When I try it with manually predefined values in the body (see the commented body line above) - it works. But when I try it with variables $VMname and $IP_address, I get the error:
Unexpected token '$VMname', "vm_ip_address":'$IP_address'}}''
expression or statement.
And if I remove single quotes before and after variables VMname and IP_address I get:
{"detail":"JSON parse error - Expecting value: line 1
column...Possible cause: trailing comma."}
It is obviously a problem with syntax and / or formatting but I cannot find the solution. Does anyone know?
Use ConvertTo-Json for this:
$VMname = "abcd"
$IP_address = '2.2.2.2'
$body = ConvertTo-Json -Compress -Depth 9 #{
extra_vars = #{
vm_name = $VMname
vm_ip_address = $IP_address
}
}
$Body
{"extra_vars":{"vm_name":"abcd","vm_ip_address":"2.2.2.2"}}
Btw, it is probably not even required to serialize your body to Json as Invoke-Restmethod is supposed to take care of that:
-body
When the input is a POST request and the body is a String, the value to the left of the first equals sign (=) is set as a key in the form data and the remaining text is set as the value. To specify multiple keys, use an IDictionary object, such as a hash table, for the Body.
I have a script that successfully parses data from a URL featuring JSON data.
However, when I add a new URL, that includes a colon (":") followed by two forward slashes ("//"), the code errors out. I feel it's due to how the script is interpreting ("://") in the URL.
I tried to replace ("://") with (":%2F%2F") in the URL parameter but, that did not work either.
Here's the script:
$cred = "[MY TOKEN]"
$headers = #{ Authorization = "Bearer $cred"; Accept = "application/json" }
$Output = Invoke-RestMethod `
-Method Get `
-Headers $headers `
-ContentType: "application/json" `
-Uri "https://www.example.com/id_user://234kjh-2234jh-45645l"
$result = $Output | ConvertTo-Json
$result
The error is as follows:
Invoke-RestMethod : {"baseType":"error","code":"ServerError","message":"users.com
\"234kjh-2234jh-45645l\" not found","status":500,"type":"error"}
At line:4 char:13
+ $Output = Invoke-RestMethod `
last year I created a PS script, that took care of automatically creating users and adding them to our servicedesk, we use a special user creation account for this, the credentials are locally saved in a text file. It all worked fine, however the script doesn't seem to work anymore, did the JIRA API change?
I get following error message: Invoke-Rest-Method: The remote server returned an error (401) Unauthorized at response = Invoke-Rest-Method -Uri...etc
I checked and our user creation account still has all the permissions to create users, I can manually create them and the log shows that the user also logs in normally through the script.
Hopefully somebody can help with my problem!
Here's the code:
$jiraCredentials = Get-Content -Raw -Path "C:\PowerShellScripts\New-AdUser\credentials.json" |ConvertFrom-Json
$bytes = [System.Text.Encoding]::ASCII.GetBytes("${$jiraCredentials.username}:${$jiraCredentials.password}")
$base64 = [System.Convert]::ToBase64String($bytes)
$basicAuthValue = "Basic $base64"
$headers = New-Object "System.Collections.Generic.Dictionary[[String],[String]]"
$headers.add("Authorization", $basicAuthValue)
$headers.add("X-Experimentalapi", "opt-in")
$headers.add("Content-Type", "application/json")
#$headers.add("X-Atlassian-Token", "nocheck")
$JSON = #"
{
"fullName": "$emailAddressClean",
"email": "$emailAddressClean"
}
"#
$response = Invoke-RestMethod -Uri https://jira.dilax.com/rest/servicedeskapi/customer -Method POST -Body $JSON -ContentType "application/json" -Headers $headers
#add customer to servicedesk
$JSON2 = #"
{
"usernames":[
"$emailAddressClean"
]
}
"#
$response2 = Invoke-RestMethod -Uri https://jira.dilax.com/rest/servicedeskapi/servicedesk/9/customer -Method POST -Body $JSON2 -ContentType "application/json" -Headers $headers
managed to fix it because the log in credentials didn't get transmitted correctly:
$jiraCredentials = Get-Content -Raw -Path "C:\PowerShellScripts\New-AdUser\credentials.json" |ConvertFrom-Json
$user = $jiraCredentials.username
$pass = $jiraCredentials.password
$pair = "${user}:${pass}"
$bytes = [System.Text.Encoding]::ASCII.GetBytes($pair)
$base64 = [System.Convert]::ToBase64String($bytes)
$basicAuthValue = "Basic $base64"
I am trying to get information from the Spotify database through their Web API.
However, I'm facing issues with accented vowels (ä,ö,ü etc.)
Lets take Tiësto as an example.
Spotify's API Browser can display the information correctly:
https://developer.spotify.com/web-api/console/get-artist/?id=2o5jDhtHVPhrJdv3cEQ99Z
If I make a API call with Invoke-Webrequest I get
Ti??sto
as name:
function Get-Artist {
param($ArtistID = '2o5jDhtHVPhrJdv3cEQ99Z',
$AccessToken = 'MyAccessToken')
$URI = "https://api.spotify.com/v1/artists/{0}" -f $ArtistID
$JSON = Invoke-WebRequest -Uri $URI -Headers #{"Authorization"= ('Bearer ' + $AccessToken)}
$JSON = $JSON | ConvertFrom-Json
return $JSON
}
How can I get the correct name?
Update: PowerShell (Core) 7.3+ now defaults to UTF-8 in the absence of a (valid) charset attribute in the HTTP response header, so the problem no longer arises there.
Jeroen Mostert, in a comment on the question, explains the problem well:
The problem is that Spotify is (unwisely) not returning the encoding it's using in its headers. PowerShell obeys the standard by assuming ISO-8859-1, but unfortunately the site is using UTF-8. (PowerShell ought to ignore standards here and assume UTF-8, but that's just like, my opinion, man.) More details here, along with the follow-up ticket.
A workaround that doesn't require the use of temporary files:
Assuming that function ConvertTo-BodyWithEncoding has been defined (see below), you can use the following:
# ConvertTo-BodyWithEncoding defaults to UTF-8.
$JSON = Invoke-WebRequest -Uri $URI ... | ConvertTo-BodyWithEncoding
Convenience function ConvertTo-BodyWithEncoding:
Note:
The function manually decodes the raw bytes that make up the given response's body, as UTF-8 by default, or with the given encoding, which can be specified as a [System.Text.Encoding] instance, a code-page number (e.g. 1251), or an encoding name (e.g., 'utf-16le).
The function is also available as an MIT-licensed Gist, and only the latter will be maintained going forward. Assuming you have looked at the linked code to ensure that it is safe (which I can personally assure you of, but you should always check), you can define it directly as follows (instructions for how to make the function available in future sessions or to convert it to a script will be displayed):
irm https://gist.github.com/mklement0/209a9506b8ba32246f95d1cc238d564d/raw/ConvertTo-BodyWithEncoding.ps1 | iex
function ConvertTo-BodyWithEncoding {
[CmdletBinding(PositionalBinding=$false)]
param(
[Parameter(Mandatory, ValueFromPipeline)]
[Microsoft.PowerShell.Commands.WebResponseObject] $InputObject,
# The encoding to use; defaults to UTF-8
[Parameter(Position=0)]
$Encoding = [System.Text.Encoding]::Utf8
)
begin {
if ($Encoding -isnot [System.Text.Encoding]) {
try {
$Encoding = [System.Text.Encoding]::GetEncoding($Encoding)
}
catch {
throw
}
}
}
process {
$Encoding.GetString(
$InputObject.RawContentStream.ToArray()
)
}
}
Issue solved with the workaround provided by Jeron Mostert.
You have to save it in a file and explicit tell Powershell which Encoding it should use.
This workaround works for me because my program can take whatever time it needs (regarding read/write IO)
function Invoke-SpotifyAPICall {
param($URI,
$Header = $null,
$Body = $null
)
if($Header -eq $null) {
Invoke-WebRequest -Uri $URI -Body $Body -OutFile ".\SpotifyAPICallResult.txt"
} elseif($Body -eq $null) {
Invoke-WebRequest -Uri $URI -Headers $Header -OutFile ".\SpotifyAPICallResult.txt"
}
$JSON = Get-Content ".\SpotifyAPICallResult.txt" -Encoding UTF8 -Raw | ConvertFrom-JSON
Remove-Item ".\SpotifyAPICallResult.txt" -Force
return $JSON
}
function Get-Artist {
param($ArtistID = '2o5jDhtHVPhrJdv3cEQ99Z',
$AccessToken = 'MyAccessToken')
$URI = "https://api.spotify.com/v1/artists/{0}" -f $ArtistID
return (Invoke-SpotifyAPICall -URI $URI -Header #{"Authorization"= ('Bearer ' + $AccessToken)})
}
Get-Artist
The following is a snippet from my PowerShell script where the values for the parameters $book and $author are not getting substituted. Please suggest some techniques that I can apply to fix it or share some code that can help me out.
$body = #{
version = '1.0'
inactive = 'false'
yml = { "Service1:\n book: $book\n author: $author\n "} | ConvertFrom-Json
} | ConvertTo-Json
$request = Invoke-WebRequest -UseBasicParsing -Method Post -Uri $uri -Body
$body -Headers $headers -ContentType $contentType
$response = ConvertFrom-Json -InputObject $request.Content
You have some weird stuff going on in this line
... yml = { "Service1:\n book: $book\n author: $author\n "} | ConvertFrom-Json } | ConvertTo-Json
Because it says "do a script block with this body, and try to convert the script block to JSON".
So, if you want to have a JSON string in yml field, you have two options.
Write the proper JSON string yourself:
#{...put the rest of your elements here...; yml = "{Service1:'', book:'$book', author: '$author'}"
Populate a hashtable first and then convert it to JSON string:
#{...put the rest of your elements here...; yml = #{Service1=''; book='$book'; author='$author'} } | ConvertTo-Json