I have a httpie session file that looks like this:
{
"headers": {
"Host": "34.213.0.202",
"Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
"User-Agent": "Mozilla/5.0 (X11; Linux x86_64; rv:67.0) Gecko/20100101 Firefox/67.0",
"Accept-Language": "en-US,en;q=0.5",
"Accept-Encoding": "gzip, deflate",
"DNT": "1",
"Connection": "keep-alive",
"Upgrade-Insecure-Requests": "1",
"Cache-Control": "max-age=0"
}
}
I would like to insert an item into headers (think '.headers + {"Cookie": "xyz"}') but I would like the output to be the whole JSON. Using jq '.headers + {"Cookie": "xyz"}' only prints out the .headers value. What I am looking to get is the whole entire JSON, but with the cookie subkey included.
What's the way to get that?
Use assignment operators:
.headers += { Cookie: "xyz" }
.headers.Cookie = "xyz"
The latter one changes a bit when there are multiple fields to be added:
.headers |= (.Cookies = "xyz" | .Referer = "xyz")
Related
I build 2 functions in GoogleCloudFunctions today, with the same code and different environment (1st gen and 2nd gen).
res.json(req.headers)
the 1st generation with X-Appengine-* in headers but 2nd generation don't have it.
1st generation res:
{
"host": "xxxxx",
"user-agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:106.0) Gecko/20100101 Firefox/106.0",
"accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8",
"accept-encoding": "gzip, deflate, br",
"accept-language": "zh-CN,zh;q=0.8,zh-TW;q=0.7,zh-HK;q=0.5,en-US;q=0.3,en;q=0.2",
"forwarded": "for=\"xx.xx.xx.xx\";proto=https",
"function-execution-id": "z93h1p8dezsd",
"sec-fetch-dest": "document",
"sec-fetch-mode": "navigate",
"sec-fetch-site": "none",
"sec-fetch-user": "?1",
"traceparent": "00-c6fbb14b26a1999f7203caa810bd82bd-f4e8995eb815facc-01",
"upgrade-insecure-requests": "1",
"x-appengine-appversionid": "h~j3d1cb91ff7bbe64cp-tp/a8aa78b2557e1c923250159d17af9995:2.447902922919183830",
"x-appengine-city": "?",
"x-appengine-citylatlong": "0.000000,0.000000",
"x-appengine-country": "HK",
"x-appengine-default-version-hostname": "j3d1cb91ff7bbe64cp-tp.appspot.com",
"x-appengine-https": "on",
"x-appengine-region": "?",
"x-appengine-request-log-id": "63745ec000ff00ff035e0349af4c0001687e6a33643163623931666637626265363463702d7470000161386161373862323535376531633932333235303135396431376166393939353a32000100",
"x-appengine-timeout-ms": "599999",
"x-appengine-user-ip": "xx.xx.xx.xx",
"x-cloud-trace-context": "c6fbb14b26a1999f7203caa810bd82bd/17647523771945712332;o=1",
"x-forwarded-for": "xx.xx.xx.xx",
"x-forwarded-proto": "https",
"connection": "close"
}
2nd generation res:
{
"host": "xxxxx.run.app",
"user-agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:106.0) Gecko/20100101 Firefox/106.0",
"accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8",
"accept-language": "zh-CN,zh;q=0.8,zh-TW;q=0.7,zh-HK;q=0.5,en-US;q=0.3,en;q=0.2",
"upgrade-insecure-requests": "1",
"sec-fetch-dest": "document",
"sec-fetch-mode": "navigate",
"sec-fetch-site": "none",
"sec-fetch-user": "?1",
"x-cloud-trace-context": "7adcb5fe556c66fc94f83b6dfe62dfe6/5799981330522755859;o=1",
"traceparent": "00-7adcb5fe556c66fc94f83b6dfe62dfe6-507dac4a5db4d313-01",
"x-forwarded-for": "xxx.xxx.xxx.xxx",
"x-forwarded-proto": "https",
"forwarded": "for=\"xxx.xxx.xxx.xxx\";proto=https",
"accept-encoding": "gzip, deflate, br"
}
Could I get X-Appengine-* from 2nd generation Functions?
Thank you
App Engine-specific headers is not supported for second generation environment.Because Cloud Functions (2nd gen) is built on Cloud Run,you can refer to this document
As mentioned by john hanley in above comment If you want those headers, deploy on gen1.
If you think it’s a valid feature request you may raise here with a clear description.Good feature requests will solve common problems or enable new use cases.
I've added a Content Security Policy to my website and made a report-uri endpoint with AWS API Gateway, Lambda, and DynamoDB. I've tested it with Postman using the following JSON
{
"resource": "/",
"path": "/",
"requestContext": {
"resourcePath": "/",
"httpMethod": "POST",
"path": "/latest"
},
"headers": {
"accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9",
"accept-encoding": "gzip, deflate, br",
"Host": "70ixmpl4fl.execute-api.us-east-2.amazonaws.com",
"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.132 Safari/537.36",
"X-Amzn-Trace-Id": "Root=1-5e66d96f-7491f09xmpl79d18acf3d050"
},
"multiValueHeaders": {
"accept": [
"text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9"
],
"accept-encoding": [
"gzip, deflate, br"
]
},
"queryStringParameters": null,
"multiValueQueryStringParameters": null,
"pathParameters": null,
"stageVariables": null,
"body": {
"csp-report": {
"document-uri": "https://example.com/signup.html",
"referrer": "",
"blocked-uri": "https://example.com/css/style.css",
"violated-directive": "style-src cdn.example.com",
"original-policy": "default-src 'none'; style-src cdn.example.com; report-uri /_/csp-reports"
}
},
"isBase64Encoded": false
}
It seems to work properly when using Postman. But when I added the endpoint to my Content Security Policy and attempt to violate the policy, I can't seem to get it to report correctly.
Does the JSON above adequately demonstrate what a real CSP violation would look like?
I've looked around and haven't seen much about developing your own endpoint. Any resources or other advice is greatly appreciated.
It depends on the browser
some browser sends the Json and CSP violation report some sends thec csp-report.
Few browser sends the request base64 encoded.
Specifically in chrome browser it sends base64 encoded and as Json format. In your payload.
Hopefully it answers your question.
I try to convert a JSON file to a csv file. I have checked most of the delivered solutions on iterative blogs and here at stackoverflow. But the things I tried, did not work out...
I am collecting requests and response headers from different websites and I am using 'json' to save the incoming data.
For analyzing purposes, I would like to convert those data into either csv or excel, optionally SQLite.
Collecting data in this case means, that the headers are not known and may change over time.
I tried some with pandas, but without any success.
My Json data looks like:
[
{
"request_headers": {
"User-Agent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/76.0.3809.71 Safari/537.36",
"Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3",
"Connection": "keep-alive",
"Upgrade-Insecure-Requests": "1",
"Cookie": "NID=189=rWuM7h_22U5ed7lJX9ewHyxOHJbRB0pgrlemwaGWLZtxDxweyi6e97D6kJ8ONYTz7OwtTl7SWj7qKED8Sspu7VfOnm9uZztJenCp68GjWIsMqOXrueIByIFgi-sLWjzLoDr7UnR1svQFa9tgHPDguKxVoV9-5uRpJsWQIJ30iZs; 1P_JAR=2019-10-13-16; CONSENT=WP.27f3c1",
"DNT": "1",
"Accept-Encoding": "gzip, deflate",
"Accept-Language": "it-IT,de;q=0.9,en-US;q=0.8,en;q=0.7"
},
"status_code": 200,
"url": "https://www.google.com/?gws_rd=ssl",
"headers": {
"Set-Cookie": "1P_JAR=2019-10-13-16; expires=Tue, 12-Nov-2019 16:11:15 GMT; path=/; domain=.google.com; SameSite=none, NID=189=K1LFDX7rwYGGQ74fbrLC7oPToRGnrEjLM-oFj2m5p4uYbHtKlmEaDmBW0qb7PZeYOXq7SFvdYlUCmqZmAU2Rrd1rMPqN2uKs0Iid30HiVFJKReQl8Dkjpm9iAgHjgbxgb0ouEQKo6dSCNSbLOyQTMnRLBqhvSavOsOT4DtmeCrs; expires=Mon, 13-Apr-2020 16:11:15 GMT; path=/; domain=.google.com; HttpOnly",
"Transfer-Encoding": "chunked",
"Content-Type": "text/html; charset=UTF-8",
"P3P": "CP=\"This is not a P3P policy! See g.co/p3phelp for more info.\"",
"Server": "gws",
"X-Frame-Options": "SAMEORIGIN",
"Strict-Transport-Security": "max-age=31536000",
"Alt-Svc": "quic=\":443\"; ma=2592000; v=\"46,43\",h3-Q048=\":443\"; ma=2592000,h3-Q046=\":443\"; ma=2592000,h3-Q043=\":443\"; ma=2592000",
"Date": "Sun, 13 SEP 2018 16:11:15 GMT",
"X-XSS-Protection": "0",
"Expires": "-1",
"Content-Encoding": "gzip",
"Cache-Control": "private, max-age=0"
}
},
I would like to get a converter, which reads any json file. The keys in the first place are considered to be the headers for each column (e.g. User-Agent, Transfer-Encoding, Cache-Control, Date a.s.o), and the given values should be within the corresponding column/row.
so you have
User-Agent | Date | Url | status_code
Firefox | Sun,... | www.google.com | 200
Chrome | Sat,... | www.google.com | 301
Edge | Wed,...
.
.
Thanks in advance!
PS: If you know any link, which helps me to find a proper solution, I am also happy.
This is my solution for now:
import pandas as pd
df = pd.read_json('file.json')
df.to_csv('file.csv')
For some reason I'm having a hard time getting the raw body from within the event. It's logging the $input.body as json for a application/json content-type. The docs say that that should contain the raw payload.
Here my Integration Request Body Mapping Template:
{
"body" : $input.json('$'),
"rawBody": $input.body,
"headers": {
#foreach($header in $input.params().header.keySet())
"$header": "$util.escapeJavaScript($input.params().header.get($header))" #if($foreach.hasNext),#end
#end
},
"method": "$context.httpMethod",
"params": {
#foreach($param in $input.params().path.keySet())
"$param": "$util.escapeJavaScript($input.params().path.get($param))" #if($foreach.hasNext),#end
#end
},
"query": {
#foreach($queryParam in $input.params().querystring.keySet())
"$queryParam": "$util.escapeJavaScript($input.params().querystring.get($queryParam))" #if($foreach.hasNext),#end
#end
}
}
Here's the payload example:
{
"event": {
"body": {
"hello": "meow"
},
"rawBody": {
"hello": "meow"
},
"headers": {
"Accept": "*/*",
"Accept-Encoding": "gzip, deflate",
"Accept-Language": "en-US",
"Cache-Control": "no-cache",
"CloudFront-Forwarded-Proto": "https",
"CloudFront-Is-Desktop-Viewer": "true",
"CloudFront-Is-Mobile-Viewer": "false",
"CloudFront-Is-SmartTV-Viewer": "false",
"CloudFront-Is-Tablet-Viewer": "false",
"CloudFront-Viewer-Country": "US",
"Content-Type": "application/json",
"Host": "7nuy7lymef.execute-api.us-east-1.amazonaws.com",
"Origin": "file://",
"Postman-Token": "0ce7c6f4-3864-c9b4-f2db-739737b2ba49",
"User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_4) AppleWebKit/537.36 (KHTML, like Gecko) Postman/4.2.2 Chrome/47.0.2526.73 Electron/0.36.2 Safari/537.36",
"Via": "1.1 1eea0bca59557555878da4d9775c509f.cloudfront.net (CloudFront)",
"X-Amz-Cf-Id": "SDjaGcuJ5eVkOMMCn6M3vGaVicA1fuA7h0bUYE4ARlKupO60eeYNFA==",
"X-Forwarded-For": "206.71.230.14, 205.251.250.135",
"X-Forwarded-Port": "443",
"X-Forwarded-Proto": "https",
"x_example_header": "my awesome header"
},
"method": "POST",
"params": {},
"query": {
"example_param": "myawesomeparam"
}
},
"context": {
"callbackWaitsForEmptyEventLoop": false,
"logGroupName": "/aws/lambda/reggi-log-post",
"logStreamName": "2016/06/08/[$LATEST]aad04e0e46614c288ac8ca43d0a95076",
"functionName": "reggi-log-post",
"memoryLimitInMB": "128",
"functionVersion": "$LATEST",
"invokeid": "6e4e1e13-2dc1-11e6-a1f7-4dad3a8eb122",
"awsRequestId": "6e4e1e13-2dc1-11e6-a1f7-4dad3a8eb122",
"invokedFunctionArn": "arn:aws:lambda:us-east-1:562508364089:function:reggi-log-post"
}
}
Is there any way to access the raw body from this request?
Is there any way to change the content-type to accept all types?
The following blog post explains in detail how to get around this problem. https://nicholasjackson.io/2016/12/13/using-graphql-with-aws-lambda/
It's written specifically in the context of GraphQL, but it will work for any content type. In short:
Go to the Binary Support section. Enable binary support for your chosen media type and save.
Return to the your method in Resources section and open Integration Request. Add/edit the body mapping template for your chosen content type and put the following:
"rawBody": "$util.escapeJavaScript($util.base64Decode($input.body))"
Save and redeploy the API.
Adding binary support encodes the request as a base64 string. The body mapping template decodes it.
$input.body contains the raw payload. You need to put quotes around it like "rawBody": "$input.body". Otherwise the body will be interpreted as part of the json document.
I am trying to post a form using an encoding that was previously chosen by the recipient.
The problem I am facing is that I can't seem to get the encoding of the swedish characters åäö to be translated correctly. Consider the following form:
<form accept-charset="ISO-8859-1" action="http://httpbin.org/post" method="post" id="aForm">
<input type="hidden" name="hej" value="köttfärssås"/>
<input type="submit"/>
</form>
Posting this using Chrome gives me the following result:
{
"args": {},
"data": "",
"files": {},
"form": {
"hej": "k\ufffdttf\ufffdrss\ufffds"
},
"headers": {
"Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8",
"Accept-Encoding": "gzip, deflate",
"Accept-Language": "sv-SE,sv;q=0.8,en-US;q=0.6,en;q=0.4",
"Cache-Control": "max-age=0",
"Content-Length": "21",
"Content-Type": "application/x-www-form-urlencoded",
"Host": "httpbin.org",
"Origin": "null",
"User-Agent": "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/40.0.2214.93 Safari/537.36"
},
"json": null,
"origin": "85.119.130.112",
"url": "http://httpbin.org/post"
}
Not only are the åäö characters jumbled, there are no headers here showing which encoding is being used. What am I missing here?
I have also tried using <meta charset="ISO-8859-1"> and <meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1">, but with no success.
So, how do I force the form to use the encoding of my choice?