So I'm scraping a website (instacart.com) and it requires a zip code to determine what data it displays. I want to use Python requests to post an arbitrary zip code. The only problem is I don't know what url to post it to and whether it requires any other arguments like an authenticity token or a user cache key. The zip code is entered via an text box that looks like this:
<form data-radium="true">
<input id="postalcode-16749"
name="postal_code"
type="text"
aria-invalid="false"
aria-describedby=""
autocomplete="on"
placeholder=""
data-radium="true"
value="" style=(super long block of css stuff)>
</form>
and then posted via a button that looks like this:
<button type="submit"
data-radium="true"
style="touch-action: manipulation; (long block of more css)">
Continue
</button>
I don't know a lot about web programming, but I was taught in school that HTML forms would look more like this: <form action="/action_page.php" method="get"> and you could use the action attribute to find where it was posting to. Is there a way to use the developer console to find what I'm looking for? How can I post a zip code to this website with Python?
Edit: I did a little more digging and I found that the request payload is {"current_zip_code":"some_zip_code"}, and that it's actually not using POST, it's using PUT. There's still a problem though, the request url looks like this: https://www.instacart.com/v3/bundle?source=web&cache_key= and then there's a different code each time for the cache_key. How do I know what url to post to?
I'm posting this answer in case anyone tries to do a similar thing. I found the url the button posts to and its parameters by looking in the network tab of the developer console and clicking the button. Then I ran into the problem that the url it sends the PUT request to changes every time, always ending in a different cache_key.
The solution was to use a python module called seleniumwire to simulate a browser and then grab all the network traffic. From there I looped through it and found urls containing cache_key= and stored everything after that as a string. Then tacked that string to the end of this url: https://www.instacart.com/v3/bundle?source=web&cache_key= and went back to using requests.
hope this helps someone!
Related
I am trying to replicate Google's I'm feeling lucky button.
But when I click the button it takes me to " Redirect Notice" page.
My code is as below:
<form action="https://google.com/search">
<input type="text" name="q">
<input type="submit" value="Google Search">
<input type="submit" name = btnI value="I'm Feeling Lucky">
</form>
TLDR: Just use DuckDuckGo. You typically don't care about what search engine is doing the redirecting, so long as you land on the top-result for a search term and the page for that result loads (and not a redirect page of the search engine).
https://duckduckgo.com/?q=%5Csoybeans
This URL above will redirect to...
https://en.wikipedia.org/wiki/Soybean
The %5c code above is the indicator for loading the page directly without search results (similar to the BtnI of Google).
Details on the Redirect Page from Linking to Google's "I'm Feeling Lucky"
Oct. 2, 2019: The redirect page was first reported on Google Search Help, the question has since been closed and comments are disabled -- they are probably not going to fix this.
You cannot use an <iframe />. Loading <iframe src="https://www.google.com/" /> has always loaded blank for me for the past ten years.
Likewise, you cannot use XMLHttpRequest, which I just tried, because CORS will block you out, and most browsers do not allow XMLHttepRequest.setHeader("origin", "http://www.google.com/");, otherwise, you could simply get the redirect page, parse for the first URL, and load that page. We know this is the case, because doing this following in a terminal will bypass the redirect page (thanks to Metamorphic for discovering this):
w3m -header "Referer: http://www.google.com/" "https://www.google.com/search?btnI=1&q=soybeans"
Iframes don't work, XmlHttpRequest doesn't work, and the main post on the Google Search forums for this topic has been closed and locked. I think it's obvious you'll need to find a different approach -- i.e., like a different search engine.
The %5c code above is the indicator for loading the page directly
without search results (similar to the BtnI of Google).
I've been looking all over the place for this, thank you!
Hi I am working with Laravel and I have a question about the . Back in time we used to write something like the following when we want to submit data.
<form action="insert.php" method="POST"> </form>
Now, I have seen in Youtube videos and here in stackoverflow many code snips where we use the following:
<form action="{{URL::to('/insert')}}" method="POST">
I would like to know what is the difference? In the second way /insert is pointing to a file or a controller? The first one is wrong? Or it is just an alternative old fashion way?
Also I have seen two ways of inserting token. Which is the best? What are the differences? Both work the same? What will happen if I do not insert a token?
{{csrf_field()}}
#csrf
Thank you for your time!
Well, when you do this:
<form action="insert.php" method="POST"> </form>
What you are doing is telling the html markup explicitly the relative path to post the data, in your case index.php
Say, you were on this url in your app: http://myapp.com/some-page and you were click the submit button, what will happen is that it will post the data to this relative url: http://myapp.com/some-page/index.php
Now, the reason why we use the URL helper facade is to make urls relative to the application's url.
For example, if you've defined in your application config (or .env) that the APP_URL is equal to something like http://myapp.com then when you use this: URL::to('/insert') it will output the following url: http://myapp.com/insert - regardless of which url you are on.
Hope this makes sense. As for this:
{{csrf_field()}}
#csrf
I believe they achieve the same, they generate an hidden input field with your current csrf token in it.
Can someone please tell me what happens behind the scenes in the following case (i.e. explain the whole technical process)?
<form method="get" action="#">
<input type="text" name="d" value="flowers">
<button type="submit">send</button>
</form>
In this case after one has clicks on “send” a new webpage opens saying: "You have searched for "flowers" " and an image of some flowers below.
In the browser tab right after the URL of the newly opened page there is
“/?s=flowers”. What is that?
Thank you in advance for your answers!
When you click Send, the page data specified in the form information and values is passed to the server via HTTP.
The /?s=flowers is the GET data being passed back to the server. Although, based on the form code you've provided, the "name" of that value is d. So the URL would actually have /?d=flowers
The PHP or server side language then handles that information to do specific tasks. It can access the info using the name "d". This method of sending data is called GET, there are also other ways of doing this. The most common, POST, does not display the data in the URL and send the data through HTTP headers.
The code you've shown has an action of "#" which means the HTTP method is being sent the same page. Meaning this page code would have some PHP located in it. This can also be done by using a seperate file, such as action='send.php'
I've checked around and most things I've found are from 2012 and workarounds for a then-existent bug. Unfortunately I'm not understanding Google's documentation on this very well.
I have a script project that is serving a web page to visitors with an HTML form:
<form id="gradingform">
<input type="text" name="name" placeholder="Name">
<input type="number" name="grade" placeholder"100">
<input type="submit" onclick="<this is where I'm having issues>">
</form>
I believe that this needs to be handled like any other time getting a script while serving a web page - by using the google.script.run. With a form specifically, I think that it's supposed to be using a success handler, so for example, something along the lines of
google.script.run.withSuccessHandler(gradeSubmitted).recordGrades()
gradeSubmitted() would be a function that just dispays a message, easy enough by doing some easy div changing. What my real issue is what recordGrades() would be like.
How do I pass the form to this function, and how do I collect the information from the form? From there I will be adding it to a Sheet, which is easy enough once the information is in an array because I can just append it. The documents say the form information should be passed as a Blob, but Google's example is kind of confusing.
This is what you have to do
onclick="google.script.run.withSuccessHandler(gradeSubmitted).recordGrades(this.form)"
And in the code.gs file you will be receiving a json as
[16-03-25 10:51:51:046 IST] {grade=10, name=Anees}
I need to retrieve the news feed for a user, and using the Graph API that returns multiple pages. I'd like to get four pages, and that's pretty slow, so I'd like to batch a request for all pages into one request using batching. I can't figure out how to batch-request multiple pages - clearly each request in the batch is dependent on the previous.
I wrote up a webpage to let me test this all out, containing the following form:
<form method="GET" action="https://graph.facebook.com">
<input type="hidden" name="access_token" value="blahblahblah">
<input type="hidden" name="batch" value="[{'method':'GET', 'name':'getnews',
'omit_response_on_success':false, 'relative_url':'me/home'},{'method':'GET',
'relative_url':'{result=getnews:$.paging.next}'}]">
<input type="hidden" name="method" value="post">
<input type="submit">
</form>
Of course, when I get a response from Facebook that requires paging, the paging.next value is a full URL and the batching functionality wants a relative_url, so my first request works and my second request returns with the paging.next URL in a "body" key.
I found a piece of facebook documentation which states that a request like the following works, where you graph.facebook.com is followed by a full URL specifying a request://graph.facebook.com/http://graph.facebook.com/me/home?_fb_url=me/home&access_token=blahblahblah"
I was surprised to find that this works, but it does when I just make that GET request to the Graph API. Unfortunately, the batching functionality does not allow me to put that full URL in the "relative_url" field - it just does that "body" thing.
Does anybody have a good way to batch requests for multiple pages? kongo09 and I were wondering this over in the facebook dev forum, but I guess that's on its way out... http://forum.developers.facebook.net/viewtopic.php?id=107098
Thanks,
-Karl
I have found a way:
You should use
"relative_url":"me/home?after={result=getnews:$.paging.cursors.after}"