Replicating a Post action from a HTTP header - html

I am trying to send a remote post action to a website to change from one state to another automatically at specific intervals, however, I am unable to decipher the HTTP header information to get the desired result. Every time I send a post the website doesn't accept it, so obviously I have parsed the post incorrectly.
The working HTTP Header information I have captured is as follows:
http://URLXXX.com/p/9998812/update_availability
POST /p/9998812/update_availability HTTP/1.1
Host: URLXXX.com
User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:39.0) Gecko/20100101 Firefox/39.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate
DNT: 1
Referer: http://URLXXX.com/p/9998812/s/fwkA-irHT-2kMfS
Cookie:
Connection: keep-alive
Content-Type: application/x-www-form-urlencoded
Content-Length: 138
utf8=%E2%9C%93&_method=put&authenticity_token=xMiaIdT%2Fnw%2FPbsYq%2BmVaLFnH362HIvIdXQQX3D%2F4uEo%3D&product%5Bstate%5D=active&commit=Save
HTTP/1.1 302 Found
Server: nginx/1.8.0
Date: Mon, 20 Jul 2015 06:00:16 GMT
Content-Type: text/html; charset=utf-8
Content-Length: 93
Connection: keep-alive
Status: 302 Found
Location: http://URLXXX.com/p/9998812
Set-Cookie: makara-force-master=master; expires=Mon, 20-Jul-2015 06:00:21 GMT
Set-Cookie: csrf-param=authenticity_token; path=/
Set-Cookie: _ssn=c8a813425bc34cd850277f5745ff957e; domain=.URLXXX.com; path=/; expires=Mon, 20-Jul-2015 06:30:16 GMT; HttpOnly
X-UA-Compatible: IE=Edge,chrome=1
Cache-Control: no-cache
X-Request-Id: 3bcc2b5f06cdd5215a613e01726559d9
X-Runtime: 0.160565
X-Served-By: app102.c1.prod
----------------------------------------------------------
http://URLXXX.com/p/9998812
GET /p/9998812 HTTP/1.1
Host: URLXXX.com
User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:39.0) Gecko/20100101 Firefox/39.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate
DNT: 1
Referer: http://URLXXX.com/p/9998812/s/fwkA-irHT-2kMfS
Cookie:
Connection: keep-alive
HTTP/1.1 301 Moved Permanently
Server: nginx/1.8.0
Date: Mon, 20 Jul 2015 06:00:16 GMT
Content-Type: text/html; charset=utf-8
Content-Length: 127
Connection: keep-alive
Status: 301 Moved Permanently
Vary: User-Agent
Location: http://URLXXX.com/p/9998812/productdetail
X-UA-Compatible: IE=Edge,chrome=1
Cache-Control: no-cache
Set-Cookie: _ssn=c8a813425bc34cd850277f5745ff957e; domain=.URLXXX.com; path=/; expires=Mon, 20-Jul-2015 06:30:16 GMT; HttpOnly
X-Request-Id: 8b68dac5cb24355d19aa46c9ac22df61
X-Runtime: 0.020239
X-Served-By: app103.c1.prod
----------------------------------------------------------
http://URLXXX.com/p/9998812/productdetail
GET /p/9998812/productdetail HTTP/1.1
Host: URLXXX.com
User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:39.0) Gecko/20100101 Firefox/39.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate
DNT: 1
Referer: http://URLXXX.com/p/9998812/s/fwkA-irHT-2kMfS
Cookie:
Connection: keep-alive
HTTP/1.1 200 OK
Server: nginx/1.8.0
Date: Mon, 20 Jul 2015 06:00:17 GMT
Content-Type: text/html; charset=utf-8
Transfer-Encoding: chunked
Connection: keep-alive
Status: 200 OK
Vary: User-Agent
X-UA-Compatible: IE=Edge,chrome=1
Etag: W/"2e41a435d3ea497f97654949f587fb46"
Cache-Control: max-age=0, private, must-revalidate
Set-Cookie: csrf-param=authenticity_token; path=/
Set-Cookie: _ssn=c8a813425bc34cd850277f5745ff957e; domain=.URLXXX.com; path=/; expires=Mon, 20-Jul-2015 06:30:17 GMT; HttpOnly
X-Request-Id: 7334d318bb69997260fd1b24f2d290de
X-Runtime: 0.369774
X-Served-By: app101.c1.prod
Content-Encoding: gzip
----------------------------------------------------------
Any help someone can provide me would be great. I really just want to understand how the pass the same parameters to the site so I can replicate the function.
Thanks

You need to send the request line, the headers, a blank line, and then the data (in the case of a POST request). For the example above this should look like this:
POST /p/9998812/update_availability HTTP/1.1
Host: URLXXX.com
User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:39.0) Gecko/20100101 Firefox/39.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate
DNT: 1
Referer: http://URLXXX.com/p/9998812/s/fwkA-irHT-2kMfS
Cookie:
Connection: keep-alive
Content-Type: application/x-www-form-urlencoded
Content-Length: 138
utf8=%E2%9C%93&_method=put&authenticity_token=xMiaIdT%2Fnw%2FPbsYq%2BmVaLFnH362HIvIdXQQX3D%2F4uEo%3D&product%5Bstate%5D=active&commit=Save
You didn't say what you are using to make the request to the server. There are several very powerful tools (such as curl for Unix/Linux systems), and lots of very powerful and friendly libraries (for instance requests for Python) for almost every language, which you can use to make HTTP requests. These do a lot of the work of handling the details of the protocol for you.
If you are writing your own HTTP client, using lower-level networking libraries, you should seriously consider using one of these tools. If you have a good reason not to do this, you should look at the RFC which specifies HTTP: http://www.w3.org/Protocols/rfc2616/rfc2616.html or at a comprehensive resource on HTTP.

Related

Cookies and <object> tags

I have a web page that displays other web pages within an tag. Some of the web pages that used to work now no longer work. I've tracked it down to a single cookie missing that is there when the web page is accessed directly, but missing when accessed via the tag.
Object:
<object type='text/html' data='<redacted>' />"
Resultant request headers:
Accept-Encoding: gzip, deflate
Accept-Language: en-US,en;q=0.9
Cache-Control: no-cache
Connection: keep-alive
Content-Length: 1235
Content-Type: multipart/form-data; boundary=----WebKitFormBoundary2tSNTMAmp3EFR4Yc
Host: <redacted>
Origin: <redacted>
Pragma: no-cache
Referer: <redacted>
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/101.0.4951.64 Safari/537.36
X-CSRFToken: IjkyZGFlYjliZGNhNTcyZTEzZjFiYjcxOGU5NDg1NjgyMmI5Yzc5ODEi.Ynwfmg.Lb6-klXrom4sh5Q1nxyXln2aF60
Request headers when page loaded directly:
Accept-Encoding: gzip, deflate
Accept-Language: en-US,en;q=0.9
Cache-Control: no-cache
Connection: keep-alive
Content-Length: 1235
Content-Type: multipart/form-data; boundary=----WebKitFormBoundaryHBNTMELe1hRhD51s
Cookie: session=.eJwdy1sKwjAQQNG9zHebKGnTNlspUiaPMWJoSjKgIu7d6N-Fy3nDRiXUCIYw1dCBq4U2zvewgwG9zKiURpomq_XoWipnB1T-ZMdhOU-zR0U0QgcpO0yhmQY7OPAatnirnMsLzAqR-TBSFn729cGpJ3S_JWh3Yg8s_lp6rNFmLF6mRiVcPl_nPTTK.YnwfwA.oNQn5zWoK0-4tFOkYteXl7Je-aU
Host: <redacted>
Origin: <redacted>
Pragma: no-cache
Referer: <redacted>
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/101.0.4951.64 Safari/537.36
X-CSRFToken: IjY5OGEzMzZhZjc3YjY2NWMzNmEzY2I0YTNkMGI1NDkxNzhkYTNmZjUi.Ynwfvw.hvXCEyOv5gPFh36DP1Gi3O_vas8
The only difference is the cookie in the request headers.

Rest API response headers for json data?

This is likely a duplicate but I wasn't able to word my query in a way that yielded a situation like mine. Thanks for any help answering or refining the question.
I've implemented a simple REST api in java / apache camel. When I invoke an endpoint with a GET request from my browser, I get the response I expect:
From my browser:
GET http://localhost:8080/app/pipeline?text=hello
{"input":"hello"}
Request Headers:
Accept:text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8
Accept-Encoding:gzip, deflate, sdch, br
Accept-Language:en-US,en;q=0.8
Cache-Control:no-cache
Connection:keep-alive
Host:localhost:8080
Pragma:no-cache
Upgrade-Insecure-Requests:1
User-Agent:Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.2924.87 Safari/537.36
Response Headers:
Accept:text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8
Accept-Encoding:gzip, deflate, sdch, br
Accept-Language:en-US,en;q=0.8
Access-Control-Allow-Headers:Origin, Accept, X-Requested-With, Content-Type,
Access-Control-Request-Method, Access-Control-Request-Headers
Access-Control-Allow-Methods:GET, HEAD, POST, PUT, DELETE, TRACE, OPTIONS, CONNECT, PATCH
Access-Control-Allow-Origin:*
Access-Control-Max-Age:3600
Connection:keep-alive
Content-Length:31
Content-Type:application/json
Upgrade-Insecure-Requests:1
User-Agent:Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.2924.87 Safari/537.36
From Curl:
Now the problem comes in when I hit the same endpoint via curl:
$ curl -i 'http://localhost:8080/app/pipeline?text=hi'
HTTP/1.1 200 OK
Content-Length: 42
Accept: */*
Access-Control-Allow-Headers: Origin, Accept, X-Requested-With, Content-Type, Access-Control-Request-Method, Access-Control-Request-Headers
Access-Control-Allow-Methods:
GET,HEAD,POST,PUT,DELETE,TRACE,OPTIONS,CONNECT,PATCH
Access-Control-Allow-Origin: *
Access-Control-Max-Age: 3600
text: hi
User-Agent: Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident/5.0)
Content-Type: application/json
Connection: keep-alive
"eyJpbnB1dCI6ImhpIiwibWV0YWRhdGEiOnt9fQ=="
I get this weird encoded response above. I'm learning about the Accept and Encoding type headers, so I set the accept and content-encoding header like my request from chrome had:
$ curl -i -sH 'Accept: application/json' -sH 'Content-Encoding: gzip, deflate, sdch, br' 'http://localhost:8080/app/pipeline?text=hi'
HTTP/1.1 200 OK
Content-Length: 42
Accept: application/json
Access-Control-Allow-Headers: Origin, Accept, X-Requested-With, Content-Type, Access-Control-Request-Method, Access-Control-Request-Headers
Access-Control-Allow-Methods: GET, HEAD, POST, PUT, DELETE, TRACE, OPTIONS, CONNECT, PATCH
Access-Control-Allow-Origin: *
Access-Control-Max-Age: 3600
Content-Encoding: gzip, deflate, sdch, br
text: hi
User-Agent: Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident/5.0)
Content-Type: application/json
Connection: keep-alive
"eyJpbnB1dCI6ImhpIiwibWV0YWRhdGEiOnt9fQ=="
What the heck? It turns out that unless I specify the Accept header exactly how it was in the chrome request, I get this weird output. With that header set I get the desired response:
$ curl -i -sH 'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8' -sH 'Content-Encoding: gzip, deflate, sdch, br' 'http://localhost:8080/app/pipeline?text=hi'
HTTP/1.1 200 OK
Content-Length: 28
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8
Access-Control-Allow-Headers: Origin, Accept, X-Requested-With, Content-Type, Access-Control-Request-Method, Access-Control-Request-Headers
Access-Control-Allow-Methods: GET, HEAD, POST, PUT, DELETE, TRACE, OPTIONS, CONNECT, PATCH
Access-Control-Allow-Origin: *
Access-Control-Max-Age: 3600
Content-Encoding: gzip, deflate, sdch, br
text: hi
User-Agent: Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident/5.0)
Content-Type: application/json
Connection: keep-alive
{"input":"hi","metadata":{}}
The Question
I'm returning application/json from my service, so why does my request not work unless I specify the header like this: Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8 ? I assume */* includes everything, but it doesn't work if I just supply */* in the header, or if I just supply application/json.
Any leads in the right direction to understanding are much appreciated.

httpd-2.4.18 mod_http2 works with curl and nghttp but doesn't work with the browser

I have installed httpd-2.4.18 with nghttp 1.6.0 and curl 7.46 to work with an http2 server. It seems that the server works with http2 when I test it with Curl and nghttp comands (as you can see below), but when I use the browser( Google Chrome 47.0.2526.106) the response headers are http1 instead of http2, and the Spdy indicator is grey (should be blue). Does anybody know why?
Commands Used
Curl command used that says me that http2 works properly:
eloy#eloy-OptiPlex-745:/usr/local/apache2/logs$ curl --http2 -I http://localhost
HTTP/1.1 101 Switching Protocols
Upgrade: h2c
Connection: Upgrade
HTTP/2.0 200
date:Thu, 07 Jan 2016 21:38:06 GMT
server:Apache/2.4.18 (Unix) OpenSSL/1.0.2e
last-modified:Mon, 11 Jun 2007 18:53:14 GMT
etag:"2d-432a5e4a73a80"
accept-ranges:bytes
content-length:45
content-type:text/html
The same with nghttp2, it seems that http2 server is working properly with the following command:
eloy#eloy-OptiPlex-745:/usr/local/apache2/logs$ nghttp -uv http://localhost
[ 0.000] Connected
[ 0.000] HTTP Upgrade request
GET / HTTP/1.1
host: localhost
connection: Upgrade, HTTP2-Settings
upgrade: h2c
http2-settings: AAMAAABkAAQAAP__
accept: */*
user-agent: nghttp2/1.6.0
[ 0.001] HTTP Upgrade response
HTTP/1.1 101 Switching Protocols
Upgrade: h2c
Connection: Upgrade
[ 0.001] HTTP Upgrade success
[ 0.001] recv SETTINGS frame <length=6, flags=0x00, stream_id=0>
Response headers from browser:
HTTP/1.1 304 Not Modified
Date: Thu, 07 Jan 2016 21:49:40 GMT
Server: Apache/2.4.18 (Unix) OpenSSL/1.0.2e
Connection: Upgrade, Keep-Alive
Keep-Alive: timeout=5, max=100
ETag: "2d-432a5e4a73a80"
Request Headers from browser:
GET / HTTP/1.1
Host: localhost
Connection: keep-alive
Cache-Control: max-age=0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8
Upgrade-Insecure-Requests: 1
User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/47.0.2526.106 Safari/537.36
Accept-Encoding: gzip, deflate, sdch
Accept-Language: es-ES,es;q=0.8
If-None-Match: "2d-432a5e4a73a80"
If-Modified-Since: Mon, 11 Jun 2007 18:53:14 GMT
Browsers do not support HTTP/1.1 to HTTP/2 upgrade requests.
The only way to use HTTP/2 from browsers is via TLS and ALPN.
Having said that, your "Request headers from browser" above are actually response headers and viceversa, so it's difficult to tell what you are actually doing. The request headers lack the necessary upgrade bits.
If you make a clear-text request from a browser (i.e. using the http scheme), then the browser will not try to upgrade and you will stay in HTTP/1.1 mode.

Chrome is not caching JS and CSS

I am running an expressjs server and tested caching in Chrome and Firefox. The headers are shown after the questions.
Could anyone tell me why Chrome is not caching JS and CSS files?
How to make Chrome cache the files?
Chrome request headers (initial and repeated):
GET /js/app.js HTTP/1.1
Host: 10.64.30.105
Connection: keep-alive
Accept: */*
User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/46.0.2490.80 Safari/537.36
Referer: https://10.64.30.105/
Accept-Encoding: gzip, deflate, sdch
Accept-Language: en-US,en;q=0.8,zh-CN;q=0.6,zh;q=0.4
Chrome response headers (initial and repeated):
HTTP/1.1 200 OK
Access-Control-Allow-Origin:
Access-Control-Allow-Credentials:
Access-Control-Allow-Methods:
Access-Control-Allow-Headers:
Accept-Ranges: bytes
Date: Thu, 12 Nov 2015 22:16:57 GMT
Cache-Control: public, max-age=31536000
Last-Modified: Thu, 12 Nov 2015 16:02:47 GMT
ETag: W/"XsMH2eh+CkXmU96uopajGg=="
Content-Type: application/javascript
Vary: Accept-Encoding
Content-Encoding: gzip
Connection: keep-alive
Transfer-Encoding: chunked
The same request is cached in Firefox.
Firefox initial request headers:
GET /js/app.js HTTP/1.1
Host: 10.64.30.105
User-Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:42.0) Gecko/20100101 Firefox/42.0
Accept: */*
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate
Referer: https://10.64.30.105/
Connection: keep-alive
Pragma: no-cache
Cache-Control: no-cache
Firefox initial response headers:
HTTP/1.1 200 OK
Accept-Ranges: bytes
Date: Thu, 12 Nov 2015 22:30:27 GMT
Cache-Control: public, max-age=31536000
Last-Modified: Thu, 12 Nov 2015 16:02:47 GMT
Etag: W/"XsMH2eh+CkXmU96uopajGg=="
Content-Type: application/javascript
Vary: Accept-Encoding
Content-Encoding: gzip
Connection: keep-alive
Transfer-Encoding: chunked
There is a problem with gzipped resources like .js .css and the Vary: Accept-encoding Header with Chrome.
Please check my Anwser given here: https://stackoverflow.com/a/40726246/135785
This solved the problem for me:
<FilesMatch "(\.js\.gz|\.css\.gz)$">
# Serve correct encoding type.
Header set Content-Encoding gzip
# Force proxies to cache gzipped & non-gzipped css/js files separately.
BrowserMatch "Chrome" ChromeFound
Header append Vary Accept-Encoding env=!ChromeFound
</FilesMatch>
Check your Apache Config for "Header append Vary Accept-Encoding"

Getting duplicate http requests when setting content-type to utf-8 inside http-equiv meta tag

I've found in my site that I'm getting the http requests twice. I'm using an apache2 server. For example I visit index.php and I get 2 different header requests for index.php (images and CSS files are only requested once), so the page is served twice and any database operation is done twice.
I've found that this is being caused by the meta tag, http-equiv. When I set the content-type attribute to UTF-8 I get this behaviour, removing the tag or setting it to another encoding type (such as ISO-8859-1) eliminates this issue.
This is the html code for that meta-tag:
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8"/>
Here are the sent and received headers caught by Http Headers Live plugin, that show the duplicate request:
http://oposiziones.dev/
GET / HTTP/1.1
Host: oposiziones.dev
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:7.0.1) Gecko/20100101 Firefox/7.0.1
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: es-es,en-us;q=0.7,en;q=0.3
Accept-Encoding: gzip, deflate
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
Connection: keep-alive
Referer: http://oposiziones.dev/error-53_q0.html
Cookie: PHPSESSID=jeup12fp5lpoo5t9k052qt7tl7
HTTP/1.1 200 OK
Date: Mon, 21 Nov 2011 11:53:25 GMT
Server: Apache/2.2.20 (Ubuntu)
X-Powered-By: PHP/5.3.6-13ubuntu3.2
Expires: Thu, 19 Nov 1981 08:52:00 GMT
Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0
Pragma: no-cache
Vary: Accept-Encoding
Content-Encoding: gzip
Content-Length: 6496
Keep-Alive: timeout=5, max=100
Connection: Keep-Alive
Content-Type: text/html
http://oposiziones.dev/
GET / HTTP/1.1
Host: oposiziones.dev
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:7.0.1) Gecko/20100101 Firefox/7.0.1
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: es-es,en-us;q=0.7,en;q=0.3
Accept-Encoding: gzip, deflate
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
Connection: keep-alive
Referer: http://oposiziones.dev/error-53_q0.html
Cookie: PHPSESSID=jeup12fp5lpoo5t9k052qt7tl7
HTTP/1.1 200 OK
Date: Mon, 21 Nov 2011 11:53:26 GMT
Server: Apache/2.2.20 (Ubuntu)
X-Powered-By: PHP/5.3.6-13ubuntu3.2
Expires: Thu, 19 Nov 1981 08:52:00 GMT
Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0
Pragma: no-cache
Vary: Accept-Encoding
Content-Encoding: gzip
Content-Length: 6385
Keep-Alive: timeout=5, max=99
Connection: Keep-Alive
Content-Type: text/html
Anybody with an idea on how to solve this? I need to keep the UTF-8 encoding because my database data is set to UTF-8, and everything should be encoded to UTF-8.
I guess this is an apache encoding issue, but have no idea why this happens.
Thanks in advance!
I didn't find why is happening this but, I solved the issue by adding this directive to the apache configuration file.
Added to config file /etc/apache2/conf.d/charset
AddDefaultCharset UTF-8
This option overrides any http-equiv charset meta tag, so the content is always sent in utf-8. This is no problem if all your content should be sent in that encoding, but won't be a solution if you use several types of encoding.
You can move this configuration directive to your .htaccess so it doesn't affect the whole server, just the folder/site you want to.