hie
i have a html form and while posting a request to a url i also want to pass crendtials using basic-auth
can anybody please share a code example on how to do that?
You could set the url of the form's action attribute to
http://username:password#www.yourserver.com/somewebpage.html
but I don't think this will work with Internet Explorer. There were some spammers abusing this and Microsoft disabled it.
Related
I'm trying to write a spider that will automatically log in to this website. However, when I try using scrapy.FormRequest.from_response in the shell I get the error:
No <form> element found in <200 https://www.athletic.net/account/login/?ReturnUrl=%2Fdefault.aspx>
I can definitely see the form when I inspect element on the site, but it just did not show up in Scrapy when I tried finding it using response.xpath() either. Is it possible for the form content to be hidden from my spider somehow? If so, how do I fix it?
The form is created using Javascript, it's not part of the static HTML source code. Scrapy does not parse Javascript, thus it cannot be found.
The relevant part of the static HTML (where they inject the form using Javascript) is:
<div ng-controller="AppCtrl as appC" class="m-auto pt-3 pb-5 container" style="max-width: 425px;">
<section ui-view></section>
</div>
To find issues like this, I would either:
compare the source code from "View Source Code" and "Inspect" to each other
browse the web page with a browser without Javascript (when I develop scrapers I usually have one browser with Javascript for research and documentations and another one for checking web pages without Javascript)
In this case, you have to manually create your FormRequest for this web page. I was not able to spot any form of CSRF protection on their form, so it might be as simple as:
FormRequest(url='https://www.athletic.net/account/auth.ashx',
formdata={"e": "foo#example.com", "pw": "secret"})
However, I think you cannot use formdata, but instead they expect you to send JSON. Not sure if FormRequest can handle this, I guess you just want to use a standard Request.
Since they heavily use Javascript on their front end, you cannot use the source code of the page to find these parameters either. Instead, I used the developer console of my browser and checked the request/response that happened when I tried to login with invalid credentials.
This gave me:
General:
Request URL: https://www.athletic.net/account/auth.ashx
[...]
Request Payload:
{e: "foo#example.com", pw: "secret"}
Scrapy has a JsonRequest class to help with posting JSON. See here [https://docs.scrapy.org/en/latest/topics/request-response.html]
So something like the below should work
data = {"password": "pword", "username": "user"}
# JSON POST to API login URL
return JsonRequest(
url=url,
callback=self.after_login,
data=data,
)
My ActionHandler.ashx file should be POSTed yet upon entry to ProcessRequest the context.Request.RequestType is always "GET".
Background:
This HttpHandler currently works OK (i.e. clicking a link in an email causes my ActionHandler.ashx to be entered and the querystring is processed correctly). For example:
https://mdwdata/CorporateBrain/ActionHandler.ashx?Action=MarkComplete&ID=1024~nzmewoojgnn&CUID=13
is the URL for the link shown as Mark-Complete in the image just below:
But now I am trying to improve it by following this advice in a previous SO thread :
"In the body of the email, instead of sending a link, include an HTML form that contains a button which performs a postback to your server."
Problem Summary: When I click the Submit button, my handler is entered with verb GET not POST (hence, I have no access to the hidden form data in the Request.Form collection.
Here is a snippet (image) of the email body
If I can get the Submit to post the hidden form variables to my handler, then of course I would remove the links. In the debugger, I verified the form data and it looks good me:
I added this line to my web.config file:
<add path="ActionHandler.ashx" verb="GET,POST" type="System.Web.UI.SimpleHandlerFactory" validate="true" />
Also, my email client is Thunderbird.
What would cause the request to be GET instead of POST?
The short answer to this problem is that Thunderbird does not POST to the URL in the Action attribute of the HTML form tag. Even the newest version of Thunderbird (version 31.2.0) "ignores" the POST and requests the URL via GET.
The construction of the HTML form is properly done and other email clients I have tested work fine:
Microsoft Office 365 Outlook Web App
Google GMail
So, I guess I am doing it "right" but some email clients apparently don't support this (even my favorite which is Thunderbird).
I am trying to submit that form on the site from java. up till now i found the POST-DATA that is send using httpFox. but the problem that the form doesn't include an action that i could use. here is the link
http://tamarod.com/index.php?page=english
the data that is sent on submit is
Member_Name=&National_ID=&governerate_id=0&district_id=&Email=&Date_Of_Birth=&Submit=I+Accept
but i tried adding this line beside the url on the browser to try it but nothing happens..
I figured out the problem. I was missing SetDoInput() and SetDoOutput() in my HttpConnection. so i can POST and GET data.
I have implemented paypal in my web page. Process is 'given inputs are redirect to other page(2 nd page) which have to get that input and redirect to paypal page(third page). Here we submit data on first page. value pass to second page(in this page user interaction not allowed) after pass to third page.It works fine in IE . But Not In Mozila.Send any Solution.
Code sample(second page):
<%string product = Request.QueryString["productName"].ToString();%>
<% string amount = Request.QueryString["price"].ToString(); %>
">
">
document.all.frmpaypal.submit();
Fine in IE, Not In Mozila
document.getElementById("frmpaypal").submit();
document.all is an IE-only non-standard extension. You can use:
document.getElementById("frmpaypal").submit();
Which will work on both browsers. Better yet, use something like jQuery:
$("#frmpaypal").submit();
(This simple example doesn't really show you the power of jQuery, but you'll love it once you find out everything it can do!)
document.all is non-standard. Add an ID and use document.getElementById.
Have you checked into the possibility of sending values via GET instead of POST in the FORM's action attribute?
A form in my app has the following:
<form action="/faculty/update/agxzdGFuZHJld3NqaHNyDQsSB0ZhY3VsdHkYBww" method="PUT" accept-charset="utf-8">
However, upon submission the request is treated as a GET and is handled by def get() and not def put(). Any help would be appreciated!
Edit:
Thanks for the responses. If I can't use method="PUT" what is the best way of directed the form the the put() method within my handler class? Should I add another handler within main.py?
HTML v4 and XHTML v1 only support the GET and POST request methods within HTML forms.
On the other hand the GET, POST, PUT and DELETE methods are supported via XMLHttpRequest in all modern browsers.
Related Stack Overflow post:
Are the PUT, DELETE, HEAD, etc methods available in most web browsers?
EDIT:
Further to your update, I think your only options would be:
Use the POST method in your form and handle it through the post() handler.
Use AJAX (XMLHttpRequest) to post your form with JavaScript, using the PUT method.
Use HTML5, but this will not work in Internet Explorer.
Browsers only do GET & POST methods. See if your app's platform can simulate PUT methods via a "method" parameter.
I believe GET and POST are the only valid values on a FORM method attribute.