The requirement is to make a POST request to a particular URL with a given JSON payload. The URL will only respond if the payload is correct and the request is via POST.
This is my code:
request1 = HTTPRequest()
control = HTTPPluginControl.getConnectionDefaults()
httpUtilities = HTTPPluginControl.getHTTPUtilities()
control.setProxyServer("proxy.example.com", 1234)
payload = JSONObject({
"uaaURL": "https://com-example.something.com",
"sampleID": "admin",
"sampleSecret": "password",
"sampleID2": "example-sample-el",
"sampleSecret2": "ssenjsoemal/+11=",
"username": "test",
"someAttributes": {
"Groups": [
"example_com-abc"
],
"attribute": [
"value1"
]
}
})
payload = str(payload)
url = "https://example-something.com:6443/getvalues"
headers = [
NVPair('Content-Type', 'application/json'),
NVPair('Charset', 'UTF-8'),]
class TestRunner:
def __call__(self):
result = request1.POST(url, payload, headers)
print payload, headers
Now the issue with this is that my POST request gives me a 403 forbidden. However, when I use the same payload and send the request using DHC, it gives me a 200. So I'm sure of the payload and the link I'm connecting to. The proxy also I've tested in another script and works fine. Besides, if the proxy didn't work, I wouldn't get a 403 either.
Lastly, I'm parsing it as a string because POST requires the second argument to be string that it will internally convert into byte[].
I'm really not able to understand what's happening so any insight would be immensely helpful. Thanks in advance
EDIT: Fiddler's catch of DHC's Request
POST https://example-something.com:6443/getvalues HTTP/1.1
Host: example-something.com:6444
Connection: keep-alive
Content-Length: 688
Origin: chrome-extension://aejoelaoggembcahagimdiliamlcdmfm
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/53.0.2785.116 Safari/537.36
Content-Type: application/json
Accept: */*
Accept-Encoding: gzip, deflate, br
Accept-Language: en-US,en;q=0.8
{
"uaaURL": "https://com-example.something.com",
"sampleID": "admin",
"sampleSecret": "password",
"sampleID2": "example-sample-el",
"sampleSecret2": "ssenjsoemal/+11=",
"username": "test",
"someAttributes": {
"Groups": [
"example_com-abc"
],
"attribute": [
"value1"
]
}
}
I even edited my Grinder request headers to so
headers = (
NVPair('Content-Type', 'application/json'),
NVPair('Charset','UTF-8'),
NVPair('Accept', '*/*'),
NVPair('Accept-Encoding', 'gzip, deflate, br'),
NVPair('Accept-Language', 'en-US,en;q=0.8'),
NVPair('Connection', 'keep-alive'),
)
Best guess? You are likely missing a header with some credential information to pass a proxy/firewall/access gate to the application. Your REST Client, DHC, likely is passing this additional data but grinder is not. Grab a proxy (Fiddler, Charles, etc...) and check out the handshake from DHC to your destination and then match that with Grinder. My guess is you will find the delta.
The issue was that Grinder was not able to access the port. It had nothing to do with the JSON. The URL, which runs on port 6443 was the problem and Grinder couldn't access that (don't know why). I changed my URL itself to run on the default port 8080 and instantly my script worked. Thanks for the help!
Related
I'm successfully making a GET request from Postman to this URI and it returns a valid response. The headers used are:
Host: asunnot.oikotie.fi
User-Agent: Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:56.0) Gecko/20100101 Firefox/56.0 Waterfox/56.3
Accept: application/json
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate, br
OTA-token: 8552f1e070ca1c843bfdd22df45057d0111f1740411359b1209feea8c0b43b98
OTA-loaded: 1586629618
OTA-cuid: 5f97229e372238054461b228c11ae28de0f691b6
Referer: https://asunnot.oikotie.fi/myytavat-asunnot?conditionType%5B%5D=1&conditionType%5B%5D=2&locations=%5B%5B1669,4,%22Lauttasaari,%20Helsinki%22%5D,%5B14714,5,%2200340,%20Helsinki%22%5D%5D&lotOwnershipType%5B%5D=1&price%5Bmax%5D=600000&price%5Bmin%5D=150000&size%5Bmin%5D=35&roomCount%5B%5D=3&cardType=100
Cookie: AWSALB=TTLrvKn+28GOvkXt/3Mcen9O9n5kK68AngdNerJ312R8jh3zoc2XydYaah9p7Niu7rRWrIGys9lONCpG87rTTM9ba/OIK4jYhnRxqEYivrTq8Op+tG4oq9B0F4Il; AWSALBCORS=TTLrvKn+28GOvkXt/3Mcen9O9n5kK68AngdNerJ312R8jh3zoc2XydYaah9p7Niu7rRWrIGys9lONCpG87rTTM9ba/OIK4jYhnRxqEYivrTq8Op+tG4oq9B0F4Il; cardType=100; instance=9a1337a3128a8b73ae9beeff3fe2c1a7; PHPSESSID=a4112077ed3095154257d8b1759c8af5; user_id=5f97229e372238054461b228c11ae28de0f691b6
Connection: keep-alive
If-Modified-Since: Sat, 11 Apr 2020 18:25:00 GMT
Cache-Control: max-age=0
Now, when I try to replicate the same request with google apps script, I get a 401 unauthorized request (to the same URI with the same header). What gives?
This is the code used (using encodeURI because otherwise I get an 'Exception: Invalid argument'):
function testGetJSON(){
var url = 'https://asunnot.oikotie.fi/api/cards?cardType=100&conditionType[]=1&conditionType[]=2&limit=24&locations=[[1669,4,"Lauttasaari,+Helsinki"],[14714,5,"00340,+Helsinki"]]&lotOwnershipType[]=1&offset=0&price[max]=600000&price[min]=150000&roomCount[]=3&size[min]=35&sortBy=published_sort_desc';
var res = encodeURI(url);
var opt = {
"method": "GET",
"muteHttpExceptions": true,
// "escaping":false,
headers: {
"Host": "asunnot.oikotie.fi",
"User-Agent": "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:56.0) Gecko/20100101 Firefox/56.0 Waterfox/56.3",
"Accept": "application/json",
"Accept-Language": "en-US,en;q=0.5",
"Accept-Encoding": "gzip, deflate, br",
"OTA-token": "8552f1e070ca1c843bfdd22df45057d0111f1740411359b1209feea8c0b43b98",
"OTA-loaded": "1586629618",
"OTA-cuid": "5f97229e372238054461b228c11ae28de0f691b6",
"Referer": "https://asunnot.oikotie.fi/myytavat-asunnot?conditionType%5B%5D=1&conditionType%5B%5D=2&locations=%5B%5B1669,4,%22Lauttasaari,%20Helsinki%22%5D,%5B14714,5,%2200340,%20Helsinki%22%5D%5D&lotOwnershipType%5B%5D=1&price%5Bmax%5D=600000&price%5Bmin%5D=150000&size%5Bmin%5D=35&roomCount%5B%5D=3&cardType=100",
"Cookie": "AWSALB=TTLrvKn+28GOvkXt/3Mcen9O9n5kK68AngdNerJ312R8jh3zoc2XydYaah9p7Niu7rRWrIGys9lONCpG87rTTM9ba/OIK4jYhnRxqEYivrTq8Op+tG4oq9B0F4Il; AWSALBCORS=TTLrvKn+28GOvkXt/3Mcen9O9n5kK68AngdNerJ312R8jh3zoc2XydYaah9p7Niu7rRWrIGys9lONCpG87rTTM9ba/OIK4jYhnRxqEYivrTq8Op+tG4oq9B0F4Il; cardType=100; instance=9a1337a3128a8b73ae9beeff3fe2c1a7; PHPSESSID=a4112077ed3095154257d8b1759c8af5; user_id=5f97229e372238054461b228c11ae28de0f691b6",
"Connection": "keep-alive",
"If-Modified-Since": "Sat, 11 Apr 2020 18:25:00 GMT",
"Cache-Control": "max-age=0",
}
};
var str = UrlFetchApp.fetch(res,opt);
Logger.log(str.getContentText());
}
Where am I going wrong?
Bonus: What is the easiest way to turn the block of header parameters above (colon separated, without quotes or commas) to the one needed in GAS? (without manually adding all the quotes and commas)
A way to get around this issue is to simply use the option 'copy as... > fetch' within the browser inspector (in my case Opera), which gives out the following (somewhat different headers than the ones sent by the browser):
fetch("https://asunnot.oikotie.fi/api/cards?cardType=100&conditionType%5B%5D=1&conditionType%5B%5D=2&limit=24&locations=%5B%5B1669,4,%22Lauttasaari,+Helsinki%22%5D,%5B14714,5,%2200340,+Helsinki%22%5D%5D&lotOwnershipType%5B%5D=1&offset=0&price%5Bmax%5D=600000&price%5Bmin%5D=150000&roomCount%5B%5D=3&size%5Bmin%5D=35&sortBy=published_sort_desc", {"credentials":"omit","headers":{"accept":"application/json","ota-cuid":"fd2a3a03d52a2721f9a9aa844ddf7eef2ac66ed6","ota-loaded":"1586685082","ota-token":"ab7e9f830a7dff3a9b01fbdcbc899ed7bfa659a4793103f1943e83ef5f938b16","sec-fetch-dest":"empty"},"referrer":"https://asunnot.oikotie.fi/myytavat-asunnot?conditionType%5B%5D=1&conditionType%5B%5D=2&locations=%5B%5B1669,4,%22Lauttasaari,%20Helsinki%22%5D,%5B14714,5,%2200340,%20Helsinki%22%5D%5D&lotOwnershipType%5B%5D=1&price%5Bmax%5D=600000&price%5Bmin%5D=150000&size%5Bmin%5D=35&roomCount%5B%5D=3&cardType=100","referrerPolicy":"no-referrer-when-downgrade","body":null,"method":"GET","mode":"cors"});
Which I used in GAS in the following way:
function testGetJSON(){
var str = UrlFetchApp.fetch("https://asunnot.oikotie.fi/api/cards?cardType=100&conditionType%5B%5D=1&conditionType%5B%5D=2&limit=24&locations=%5B%5B1669,4,%22Lauttasaari,+Helsinki%22%5D,%5B14714,5,%2200340,+Helsinki%22%5D%5D&lotOwnershipType%5B%5D=1&offset=0&price%5Bmax%5D=600000&price%5Bmin%5D=150000&roomCount%5B%5D=3&size%5Bmin%5D=35&sortBy=published_sort_desc",
{"credentials":"omit",
"headers":{"accept":"application/json","ota-cuid":"fd2a3a03d52a2721f9a9aa844ddf7eef2ac66ed6","ota-loaded":"1586685082","ota-token":"ab7e9f830a7dff3a9b01fbdcbc899ed7bfa659a4793103f1943e83ef5f938b16","sec-fetch-dest":"empty"},"referrer":"https://asunnot.oikotie.fi/myytavat-asunnot?conditionType%5B%5D=1&conditionType%5B%5D=2&locations=%5B%5B1669,4,%22Lauttasaari,%20Helsinki%22%5D,%5B14714,5,%2200340,%20Helsinki%22%5D%5D&lotOwnershipType%5B%5D=1&price%5Bmax%5D=600000&price%5Bmin%5D=150000&size%5Bmin%5D=35&roomCount%5B%5D=3&cardType=100","referrerPolicy":"no-referrer-when-downgrade","body":null,"method":"GET","mode":"cors"});
Logger.log(str.getContentText());
}
I am making a python module that interacts with Carousell using the requests module. Now I am trying to send a post request with a JSON payload, but I keep getting HTTP error code 422(UNPROCESSABLE ENTITY). I don't know what's wrong with my JSON payload, python dict(before it's converted to JSON) or perhaps I am missing something in my request headers.
I tried taking the raw json string(from the POST request that I captured using Chrome dev tools) converting it dict and copy that dict(printed out) and try to use it in the program. It didn't work.
login_session = requests.session()
login_session.headers.update({"DNT":"1", "User-Agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.157 Safari/537.36", "Origin":"https://sg.carousell.com"})
login_payload = {'requests': {'g0': {'resource': 'sso', 'operation': 'create', 'params': {'loginToken': cookies["login-token"]}, 'body': {}}}, 'context': {'_csrf': cookies["_csrf"]}}
login_cookies = {"__cfduid": cookies["__cfduid"], "_csrf": cookies["_csrf"], "gtkprId": cookies["gtkprId"], "login-token": cookies["login-token"], "redirect":"redirect"}
login_headers = {'accept':'*/*','accept-encoding':'gzip, deflate, br','accept-language':'en-GB,en;q=0.9,zh-CN;q=0.8,zh;q=0.7,en-US;q=0.6', 'x-requested-with': 'XMLHttpRequest', 'content-type': 'application/json'}
login_data = login_session.post(query_url, cookies=login_cookies, data=json.dumps(login_payload), headers=login_header)
Heres the output from debugging logger
DEBUG:urllib3.connectionpool:https://sg.carousell.com:443 "POST /ui/iso?_csrf=TNZTMZpBdQYgRFFouCF4ELVB HTTP/1.1" 422 0
Edit:
Heres the JSON payload which was sent to the server. I am trying to replicate it.
{"requests":{"g0":{"resource":"sso","operation":"create","params":{"loginToken":"eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpYXQiOjE1NTg1MDQyMjQsImlzcyI6ImxvZ2luLmNhcm91c2VsbC5jb20iLCJzc29pZCI6IkRacG1rd1l1SXAxdDF5U3A2M1RXWExPUTJnWmRFRzBOSHd3d0ZGSm9PSkFvVFFOdGFyNWt0MDMzNm5EVHRudHoiLCJ1c2VyaWQiOiIxNDczMjI3NCJ9.x7YxdLLk1ID6_jWy4trtLzbrPnZZ0eI7g_cQN1BilF8"},"body":{}}},"context":{"_csrf":"hPPhgajp-1GMLSbgjZBNBD7z2EGPVGCuA_mU"}}
Note that login token and _csrf are data from the cookies.
I got a problem which I dont understand.
I try to post data to my API in a form using the following code
formSubmit() {
const req =this.http.post('http://[ip]/api/login', {
id: '7',
username: 'PostTest',
password: 'studp123lan',
matrikelnr: 'winf303666',
email: 'winf303666#example.de',
email_verified: '1'
})
.subscribe(
res => {
console.log(res);
},
err => {
console.log("Error occured");
}
When I inspect it in the Chrome Developter tools, this is what I get:
Failed to load http://[ip]/api/login: Response for preflight has
invalid HTTP status code 404
register.component.ts:42 Error occured
And this is what I get in the network tab:
General:
Request URL:http://[ip]/api/login
Request Method:OPTIONS
Status Code:404 Not Found
Remote Address:[ip]:80
Referrer Policy:no-referrer-when-downgrade
Response Header:
Access-Control-Allow-Headers:Content-Type
Access-Control-Allow-Methods:GET, POST, PUT, DELETE, OPTIONS
Access-Control-Allow-Origin:*
Content-Length:0
Date:Thu, 21 Dec 2017 09:00:35 GMT
Server:Kestrel
X-Powered-By:ASP.NET
Request Header:
Accept:*/*
Accept-Encoding:gzip, deflate
Accept-Language:de-DE,de;q=0.9,en-US;q=0.8,en;q=0.7
Access-Control-Request-Headers:content-type
Access-Control-Request-Method:POST
Connection:keep-alive
Host:[ip]
Origin:http://localhost:4200
User-Agent:Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36
(KHTML, like Gecko) Chrome/63.0.3239.108 Safari/537.36
Somehow, this doesn't work. But when I Post the same data via Postman Post Request to the same URL, it works like a charm.
Can anyone explain and help?
Thanks.
404 is a page not found error : this means your endpoint isn't available at this address.
Sure, you make a postman call and it works : but did you create your postman call by hand, or used the interceptor to make it ? (The interceptor is a Chomr plugin that allows you to register all calls made by Chrome, into postman).
There must be something you have forgotten. Could you post your postman call, and if you can, try with the interceptor ?
I have runing bitcoind on ubuntu. bitcoin-cli works fine. I can not get working json rpc protocol
bitcoin.conf file:
testnet=0
rpcuser="bitcoinrpc"
rpcpassword="xxxxx"
rpcport=8332
rpcallowip="*"
server=1
http post request with url='http://bitcoinrpc:xxxxx#127.0.0.1:8332/' fails with 401 error.
request headers:
Accept:*/*
Accept-Encoding:gzip, deflate
Accept-Language:en-US,en;q=0.8,ru;q=0.6,de;q=0.4,sr;q=0.2
Authorization:Basic Yml0Y29pbnJwYzp4eHh4eA==
Cache-Control:no-cache
Connection:keep-alive
Content-Length:53
Content-Type:text/plain
DNT:1
Host:127.0.0.1:8332
Origin:chrome-extension://fhjcajmcbmldlhcimfajhfbgofnpcjmb
Pragma:no-cache
User-Agent:Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Ubuntu Chromium/51.0.2704.79 Chrome/51.0.2704.79 Safari/537.36
request post payload:
{jsonrpc: "2.0", method: "getinfo", params: []}
What is correct way for bitcoind json rpc autentification?
For future googlers: a possible problem is that the password should not contain the pound sign (#) as this is treated as a comment!
My attempt is to extract a table from this website.
The web is in Chinese, but basically, you can type your log-in details within those boxes lying above the big blue button in the middle of the web page. After logged-in, the table will appear in the middle of the page. NOTE: in /articlenew.html, only USERNAME and PASSWORD are required for log-in. Nothing else.
After authentication, the headers of the webs are shown as below:
Request URL:http://www.sxcoal.com/user/login.aspx
Request Method:POST
Status Code:302 Found
Request Headersview source
Accept:text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8
Accept-Encoding:gzip,deflate,sdch
Accept-Language:en,en-GB;q=0.8,zh;q=0.6,zh-CN;q=0.4
Connection:keep-alive
Content-Length:39
Content-Type:application/x-www-form-urlencoded
Cookie:the_cookies
Host:www.sxcoal.com
Origin:http://www.sxcoal.com
Referer:http://www.sxcoal.com/coal/3478186/articlenew.html
User-Agent:Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/30.0.1599.101 Safari/537.36
Form Dataview sourceview URL encoded
username:myusername
password:mypassword
Response Headersview source
Cache-Control:private
Content-Length:167
Content-Type:text/html; charset=gb2312
Date:Thu, 14 Nov 2013 01:06:00 GMT
Location:http://www.sxcoal.com/coal/3478186/articlenew.html
Server:Microsoft-IIS/7.0
Set-Cookie:s_info=zhuhaiqinfa|15816; domain=sxcoal.com; path=/
X-AspNet-Version:2.0.50727
X-Powered-By:ASP.NET
I have tried to use the method shown by Gergely Daróczi. However, R couldn't log-in for some reasons. My guess is that /login.aspx (http:[DELETE]//www.[DELETE]sxcoal.[DELETE]com/user/login.[DELETE]aspx
)[sorry I haven't got enough 'reputation' to post more links.] nested in the /articlenew.html actually requires more things than just a username and a corresponding password. I put the headers of the /login.aspx at the end of the question.
Here is the code I used,
library(RCurl)
mycurl <- getCurlHandle()
agent <- "Mozilla/5.0"
curlSetOpt(cookiejar = "", followlocation = TRUE, useragent = agent, autoreferer = TRUE, curl = mycurl)
html <- getURL('http://www.sxcoal.com/user/login.aspx', curl = mycurl)
viewstate <- as.character(sub('.*id="__VIEWSTATE" value="([0-9a-zA-Z+/=]*).*', '\\1', html))
eventvalidation <- as.character(sub('.*id="__EVENTVALIDATION" value="([0-9a-zA-Z+/=]*).*', '\\1', html))
##checkcode <- ??????????????? ## can't define it as it changes
params <- list(
"txtuser" = "myusername",
"txtpass" = "mypassword",
"__VIEWSTATE" = viewstate,
"__EVENTVALIDATION" = eventvalidation,
"CheckCode" = checkcode,
"Button2" = ""
)
html <- postForm('http://www.sxcoal.com/user/login.aspx', .params = params, curl = mycurl)
The CheckCode is a validation code shown by a pic(http://www.sxcoal.com/CheckCode/CheckCode.aspx). Unlike the __VIEWSTATE and __EVENTVALIDATION, the CheckCode changes every time when you refresh the page.
And there is something every complicated as I don't know anything about website coding. It seems to me that the log-in details required by /login.aspx nested within /articlenew.html differ from those required by /login.aspx itself. Are there any methods that can fix the log-in details required by the web such that I don't need to deal with the validation code shown by a random picture? If not, can anyone know how I can take care of the validation picture?
Thanks in advance.
Request URL:http://www.sxcoal.com/user/login.aspx
Request Method:POST
Status Code:302 Found
Request Headersview source
Accept:text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8
Accept-Encoding:gzip,deflate,sdch
Accept-Language:en,en-GB;q=0.8,zh;q=0.6,zh-CN;q=0.4
Connection:keep-alive
Content-Length:234
Content-Type:application/x-www-form-urlencoded
Cookie:the_cookies
Host:www.sxcoal.com
Origin:http://www.sxcoal.com
Referer:http://www.sxcoal.com/user/login.aspx
User-Agent:Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/30.0.1599.101 Safari/537.36
Form Dataview sourceview URL encoded
__VIEWSTATE:whatever_it_is
txtuser:myusername
txtpass:mypassword
CheckCode:04854
Button2:
__EVENTVALIDATION:whatever_it_it_2
Response Headersview source
Cache-Control:private
Content-Length:170
Content-Type:text/html; charset=gb2312
Date:Thu, 14 Nov 2013 01:09:57 GMT
Location:http://www.sxcoal.com/?aspxerrorpath=/user/login.aspx
Server:Microsoft-IIS/7.0
X-AspNet-Version:2.0.50727
X-Powered-By:ASP.NET