How do I get a html post request to use a specific charset? - html

I am trying to post a form using an encoding that was previously chosen by the recipient.
The problem I am facing is that I can't seem to get the encoding of the swedish characters åäö to be translated correctly. Consider the following form:
<form accept-charset="ISO-8859-1" action="http://httpbin.org/post" method="post" id="aForm">
<input type="hidden" name="hej" value="köttfärssås"/>
<input type="submit"/>
</form>
Posting this using Chrome gives me the following result:
{
"args": {},
"data": "",
"files": {},
"form": {
"hej": "k\ufffdttf\ufffdrss\ufffds"
},
"headers": {
"Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8",
"Accept-Encoding": "gzip, deflate",
"Accept-Language": "sv-SE,sv;q=0.8,en-US;q=0.6,en;q=0.4",
"Cache-Control": "max-age=0",
"Content-Length": "21",
"Content-Type": "application/x-www-form-urlencoded",
"Host": "httpbin.org",
"Origin": "null",
"User-Agent": "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/40.0.2214.93 Safari/537.36"
},
"json": null,
"origin": "85.119.130.112",
"url": "http://httpbin.org/post"
}
Not only are the åäö characters jumbled, there are no headers here showing which encoding is being used. What am I missing here?
I have also tried using <meta charset="ISO-8859-1"> and <meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1">, but with no success.
So, how do I force the form to use the encoding of my choice?

Related

can not get X-AppEngine info from req header in 2nd generation AppEngine

I build 2 functions in GoogleCloudFunctions today, with the same code and different environment (1st gen and 2nd gen).
res.json(req.headers)
the 1st generation with X-Appengine-* in headers but 2nd generation don't have it.
1st generation res:
{
"host": "xxxxx",
"user-agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:106.0) Gecko/20100101 Firefox/106.0",
"accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8",
"accept-encoding": "gzip, deflate, br",
"accept-language": "zh-CN,zh;q=0.8,zh-TW;q=0.7,zh-HK;q=0.5,en-US;q=0.3,en;q=0.2",
"forwarded": "for=\"xx.xx.xx.xx\";proto=https",
"function-execution-id": "z93h1p8dezsd",
"sec-fetch-dest": "document",
"sec-fetch-mode": "navigate",
"sec-fetch-site": "none",
"sec-fetch-user": "?1",
"traceparent": "00-c6fbb14b26a1999f7203caa810bd82bd-f4e8995eb815facc-01",
"upgrade-insecure-requests": "1",
"x-appengine-appversionid": "h~j3d1cb91ff7bbe64cp-tp/a8aa78b2557e1c923250159d17af9995:2.447902922919183830",
"x-appengine-city": "?",
"x-appengine-citylatlong": "0.000000,0.000000",
"x-appengine-country": "HK",
"x-appengine-default-version-hostname": "j3d1cb91ff7bbe64cp-tp.appspot.com",
"x-appengine-https": "on",
"x-appengine-region": "?",
"x-appengine-request-log-id": "63745ec000ff00ff035e0349af4c0001687e6a33643163623931666637626265363463702d7470000161386161373862323535376531633932333235303135396431376166393939353a32000100",
"x-appengine-timeout-ms": "599999",
"x-appengine-user-ip": "xx.xx.xx.xx",
"x-cloud-trace-context": "c6fbb14b26a1999f7203caa810bd82bd/17647523771945712332;o=1",
"x-forwarded-for": "xx.xx.xx.xx",
"x-forwarded-proto": "https",
"connection": "close"
}
2nd generation res:
{
"host": "xxxxx.run.app",
"user-agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:106.0) Gecko/20100101 Firefox/106.0",
"accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8",
"accept-language": "zh-CN,zh;q=0.8,zh-TW;q=0.7,zh-HK;q=0.5,en-US;q=0.3,en;q=0.2",
"upgrade-insecure-requests": "1",
"sec-fetch-dest": "document",
"sec-fetch-mode": "navigate",
"sec-fetch-site": "none",
"sec-fetch-user": "?1",
"x-cloud-trace-context": "7adcb5fe556c66fc94f83b6dfe62dfe6/5799981330522755859;o=1",
"traceparent": "00-7adcb5fe556c66fc94f83b6dfe62dfe6-507dac4a5db4d313-01",
"x-forwarded-for": "xxx.xxx.xxx.xxx",
"x-forwarded-proto": "https",
"forwarded": "for=\"xxx.xxx.xxx.xxx\";proto=https",
"accept-encoding": "gzip, deflate, br"
}
Could I get X-Appengine-* from 2nd generation Functions?
Thank you
App Engine-specific headers is not supported for second generation environment.Because Cloud Functions (2nd gen) is built on Cloud Run,you can refer to this document
As mentioned by john hanley in above comment If you want those headers, deploy on gen1.
If you think it’s a valid feature request you may raise here with a clear description.Good feature requests will solve common problems or enable new use cases.

Requesting Json

when i put this link : https://www.sofascore.com/basketball/livescore/json?_=159726531 in google
i got JSON data. So I request this link:
import json
import time
import requests
url = "https://www.sofascore.com/basketball/livescore/json?_=159729590"
headers = {
"referer": "https://www.google.pl/",
"user-agent": 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/84.0.4147.89 Safari/537.36',
"accept-encoding": "gzip, deflate, br",
"accept-language": "pl-PL,pl;q=0.9,en-US;q=0.8,en;q=0.7",
"authority": "www.sofascore.com",
"x-requested-with": "XMLHttpRequest",
"method": "GET",
"scheme": "https",
"accept": "*/*",
}
page = requests.get(url, headers=headers)
#dict_obj = json.loads(page.text)
#print(dict_obj)
I didn't get even text but i got response 200. What i did wrong?

How can I test a CSP report-uri endpoint?

I've added a Content Security Policy to my website and made a report-uri endpoint with AWS API Gateway, Lambda, and DynamoDB. I've tested it with Postman using the following JSON
{
"resource": "/",
"path": "/",
"requestContext": {
"resourcePath": "/",
"httpMethod": "POST",
"path": "/latest"
},
"headers": {
"accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9",
"accept-encoding": "gzip, deflate, br",
"Host": "70ixmpl4fl.execute-api.us-east-2.amazonaws.com",
"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.132 Safari/537.36",
"X-Amzn-Trace-Id": "Root=1-5e66d96f-7491f09xmpl79d18acf3d050"
},
"multiValueHeaders": {
"accept": [
"text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9"
],
"accept-encoding": [
"gzip, deflate, br"
]
},
"queryStringParameters": null,
"multiValueQueryStringParameters": null,
"pathParameters": null,
"stageVariables": null,
"body": {
"csp-report": {
"document-uri": "https://example.com/signup.html",
"referrer": "",
"blocked-uri": "https://example.com/css/style.css",
"violated-directive": "style-src cdn.example.com",
"original-policy": "default-src 'none'; style-src cdn.example.com; report-uri /_/csp-reports"
}
},
"isBase64Encoded": false
}
It seems to work properly when using Postman. But when I added the endpoint to my Content Security Policy and attempt to violate the policy, I can't seem to get it to report correctly.
Does the JSON above adequately demonstrate what a real CSP violation would look like?
I've looked around and haven't seen much about developing your own endpoint. Any resources or other advice is greatly appreciated.
It depends on the browser
some browser sends the Json and CSP violation report some sends thec csp-report.
Few browser sends the request base64 encoded.
Specifically in chrome browser it sends base64 encoded and as Json format. In your payload.
Hopefully it answers your question.

jq - insert values into subkey

I have a httpie session file that looks like this:
{
"headers": {
"Host": "34.213.0.202",
"Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
"User-Agent": "Mozilla/5.0 (X11; Linux x86_64; rv:67.0) Gecko/20100101 Firefox/67.0",
"Accept-Language": "en-US,en;q=0.5",
"Accept-Encoding": "gzip, deflate",
"DNT": "1",
"Connection": "keep-alive",
"Upgrade-Insecure-Requests": "1",
"Cache-Control": "max-age=0"
}
}
I would like to insert an item into headers (think '.headers + {"Cookie": "xyz"}') but I would like the output to be the whole JSON. Using jq '.headers + {"Cookie": "xyz"}' only prints out the .headers value. What I am looking to get is the whole entire JSON, but with the cookie subkey included.
What's the way to get that?
Use assignment operators:
.headers += { Cookie: "xyz" }
.headers.Cookie = "xyz"
The latter one changes a bit when there are multiple fields to be added:
.headers |= (.Cookies = "xyz" | .Referer = "xyz")

Raw body payload in AWS API Gateway Body Mapping Template

For some reason I'm having a hard time getting the raw body from within the event. It's logging the $input.body as json for a application/json content-type. The docs say that that should contain the raw payload.
Here my Integration Request Body Mapping Template:
{
"body" : $input.json('$'),
"rawBody": $input.body,
"headers": {
#foreach($header in $input.params().header.keySet())
"$header": "$util.escapeJavaScript($input.params().header.get($header))" #if($foreach.hasNext),#end
#end
},
"method": "$context.httpMethod",
"params": {
#foreach($param in $input.params().path.keySet())
"$param": "$util.escapeJavaScript($input.params().path.get($param))" #if($foreach.hasNext),#end
#end
},
"query": {
#foreach($queryParam in $input.params().querystring.keySet())
"$queryParam": "$util.escapeJavaScript($input.params().querystring.get($queryParam))" #if($foreach.hasNext),#end
#end
}
}
Here's the payload example:
{
"event": {
"body": {
"hello": "meow"
},
"rawBody": {
"hello": "meow"
},
"headers": {
"Accept": "*/*",
"Accept-Encoding": "gzip, deflate",
"Accept-Language": "en-US",
"Cache-Control": "no-cache",
"CloudFront-Forwarded-Proto": "https",
"CloudFront-Is-Desktop-Viewer": "true",
"CloudFront-Is-Mobile-Viewer": "false",
"CloudFront-Is-SmartTV-Viewer": "false",
"CloudFront-Is-Tablet-Viewer": "false",
"CloudFront-Viewer-Country": "US",
"Content-Type": "application/json",
"Host": "7nuy7lymef.execute-api.us-east-1.amazonaws.com",
"Origin": "file://",
"Postman-Token": "0ce7c6f4-3864-c9b4-f2db-739737b2ba49",
"User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_4) AppleWebKit/537.36 (KHTML, like Gecko) Postman/4.2.2 Chrome/47.0.2526.73 Electron/0.36.2 Safari/537.36",
"Via": "1.1 1eea0bca59557555878da4d9775c509f.cloudfront.net (CloudFront)",
"X-Amz-Cf-Id": "SDjaGcuJ5eVkOMMCn6M3vGaVicA1fuA7h0bUYE4ARlKupO60eeYNFA==",
"X-Forwarded-For": "206.71.230.14, 205.251.250.135",
"X-Forwarded-Port": "443",
"X-Forwarded-Proto": "https",
"x_example_header": "my awesome header"
},
"method": "POST",
"params": {},
"query": {
"example_param": "myawesomeparam"
}
},
"context": {
"callbackWaitsForEmptyEventLoop": false,
"logGroupName": "/aws/lambda/reggi-log-post",
"logStreamName": "2016/06/08/[$LATEST]aad04e0e46614c288ac8ca43d0a95076",
"functionName": "reggi-log-post",
"memoryLimitInMB": "128",
"functionVersion": "$LATEST",
"invokeid": "6e4e1e13-2dc1-11e6-a1f7-4dad3a8eb122",
"awsRequestId": "6e4e1e13-2dc1-11e6-a1f7-4dad3a8eb122",
"invokedFunctionArn": "arn:aws:lambda:us-east-1:562508364089:function:reggi-log-post"
}
}
Is there any way to access the raw body from this request?
Is there any way to change the content-type to accept all types?
The following blog post explains in detail how to get around this problem. https://nicholasjackson.io/2016/12/13/using-graphql-with-aws-lambda/
It's written specifically in the context of GraphQL, but it will work for any content type. In short:
Go to the Binary Support section. Enable binary support for your chosen media type and save.
Return to the your method in Resources section and open Integration Request. Add/edit the body mapping template for your chosen content type and put the following:
"rawBody": "$util.escapeJavaScript($util.base64Decode($input.body))"
Save and redeploy the API.
Adding binary support encodes the request as a base64 string. The body mapping template decodes it.
$input.body contains the raw payload. You need to put quotes around it like "rawBody": "$input.body". Otherwise the body will be interpreted as part of the json document.