please may I have a little help I'm stuck not being able to google for a solution because of very common words.
There is a web page that uses POST to send data to a page on a subdomain when a button is clicked.
I need to recreate a button and send the same information.
My question is: Is it possible just by looking at the page (and the console??) when you click the button, to observer what happens and recreate/implement the same POST method?
Can I say for example: It does this, therefore I need this code to do the same thing?
Or is it not possible to reverse engineer? Will I have to seek help form the web page developer (not really an option in this case).
It is perfectly possible to inspect the request and reverse engineer this. You can use tools like the developer console in your Chrome/Edge browser (press F12), and tools like Postman to simulate requests. Also inspect the form and eventual javascript events attached to the button.
Related
Forgive me if I don't use the proper terminology. I have a webpage that I'm trying to scrape information from. The problem is that when I view the page source the data I want to scrape is not there. I've encountered this problem before where the main http request triggers other requests and so the information I'm looking for is actually somewhere else which I find using Google chromes inspect - Network feature. I manually search the various documents and xhr files so the one that has the correct information. This is sometimes long and tedious. I can also use google chromes inspect feature to inspect the element that contains the information I want and that brings up the correct source code but it I can't seem to figure out where or how I can use that to quickly find the corresponding HTTP headers.
Restated in a short - can I use the inspect element feature of google chrome and then ask it to show me the corresponding network event (HTTP request) that produced that code?
I'll add the case study I'm working on.
http://www.flashscore.com/tennis/atp-singles/acapulco/results/
shows the different matches that took place at a tennis tournament. I'm trying to scrape the the match hrefs but if you view source of the page you'll see they're not there.
Thanks
Restated in a short - can I use the inspect element feature of google chrome and then ask it to show me the corresponding network event (HTTP request) that produced that code?
No. This isn't something that the browser keeps track of.
In most situations, the HTTP response will pass through a good deal of Javascript code before being eventually turned into elements on the page. Tracing which HTTP response was "responsible" for a given element would involve a great deal of data flow analysis, and is impractical for a browser to do.
One way:
open firefox, install LiveHttpHeaders, then run it, and you will see the expected HEADERS.
There's the same addon for google chrome, but not tested.
I'm using
window.postMessage({message: "Hello !"}, url);
to send a message from a Chrome Extension (i don't know if this is relevant) to a specific page in a window with multiples opened pages. I noticed that sometimes i have TWO pages with the same URL.
I have a simple question:
How can i be sure to which page is postMessage sending the message ?
I want to send the message to only one tab. Can i use anything else apart from the url to identify the it?
Thanks in advance !
Considering that you said you can modify the remote website's code, and I don't see how to fix the postMessage solution, here are a couple of alternatives. I would love to know if there is a way to fix the postMessage approach, as it is the recommended one from the docs!
First off, you will need to coordinate your scripts from a central background page, which can keep track of open tabs.
Custom DOM events
This is an old recommendation from Chrome docs, that was replaced with window.postMessage example. It is described here (disregard the old chrome.extension.connect API) and consists of firing a custom event in shared DOM.
So, a sample architecture would be a background page deciding which tab to post message to, and sending that tab a message via chrome.tabs.sendMessage, to which your content script listens with chrome.runtime.onMessage. The tab's content script can then communicate with the page using the above custom event technique.
One possible approach to keeping tack of tabs: have the tabs permission to be able to enumerate all open tabs with the chrome.tabs API. Your background page can then decide which tab to message based on URL.
Another possible approach, to eliminate need for the scary tabs permission, is to have your content scripts report to the background page with chrome.runtime.connect as soon as they are initialized. The background page then can keep track of all active instances of your script and therefore decide which tab to message.
Webpage connecting to your extension
This is a "modern" way of doing communication with one exact extension.
It is described in the Chrome docs here. You can define your extension as externally connectable from your webpage, and your webpage initiates a port connection with your background script.
Then, as above, you can track live ports and use them for communication, cutting out the content script middleman.
I am working on my first site using google scripts. I have a script embedded that creates a UI and references a Google spreadsheet to link to pages of the same site. Whenever a link is clicked I get an interim page that says:
The previous page is sending you to
https://sites.google.com/site/gchromeat/xxxx.
If you do not want to visit that page, you can return to the previous
page.
Is there a way to get around this?
The behavior can be replicated here: https://sites.google.com/site/gchromeat/home/access
There's no way to go around this, as it seems to be a designed behavior. But there's an issue opened regarding this, you should star it to keep track of updates and kind of vote for it.
I presume you have used an Anchor object in your Google Apps Script. The reason why you get the redirect message is because the anchor object changes the link.
Browsers detect the redirect and ask for user confirmation (browser settings can be changed to avoid this).
Also see issue 1376 - which has been opened for getting rid of some undesirable side effects of this.
I'm quite new to web development, but I'm trying to do some stuff in Python and so I need help.
Sometime ago, I went into a page to debug html forms that allowed me to change the POSTs to GETs in the form request and that really helped me get some info from html forms by just doing everything in the URL instead of having to make my program fill some forms.
Unfortunately I lost that page address and as much as I search in Google I cannot come up with it or anything similar again. Can you direct me to some links that do this?
I tried both Firebug in Firefox and developer tools in chrome, but they don't seem to do this.
Ok, this is probably quite basic for most people, but in Firebug, you just have to go into the "NET" tab, make the request and then analyse it and if you right click it and choose: "Copy location with parameters" you get the formatted URL ready to go, even when using javascript.
If you have: <form name="form1">
You can this JavaScript:
document.form1.type="GET"
Use this in the page before submitting the form.
The web developer tool bar for firefox will do this and much more:
https://addons.mozilla.org/en-US/firefox/addon/web-developer/
EDIT
You can also use firebug and inspect the form element and change it from POST to GET
How can I intercept the post data a page is sending in FF or Chrome via configuration, extension or code? (Code part makes this programming related. ;)
I currently use Wireshark/Ethereal for this, but it's a bit difficult to use.
You could just use the Chrome Developer Tools, if you only need to track requests.
Activate them with Ctrl+Shift+I and select the Network tab.
This works also when Chrome talks HTTPS with another server (and unless you have the HTTPS private key you cannot use Wireshark to sniff that traffic).
(I copied this answer from this related query.)
With Firefox you can use the Network tab (Ctrl+Shift+E or Command+Option+E). The sub-tab "Params" shows the submitted form data.
Reference: https://developer.mozilla.org/en-US/docs/Tools/Network_Monitor/request_details#Params
Alternatively, in the console (Ctrl+Shift+K or Command+Option+K) right click on the big pane and check "Log Request and Response Bodies". Then when the form is submitted, a line with POST <url> will appear. Click on it; it will open a new window with the form data.
As of the time of originally writing this reply, both methods messed up newlines in textarea fields. The former deleted them, the latter converted them to blanks. I haven't checked with a newer version.
Do you have control of the browser POSTing the data?
If you do, then just use Firebug. It's got a lot of usefull features, including this
For Firefox there is also TamperData, and even more powerful and cross-browser is Fiddler.
Programatically, you can do this with dBug - it's a small code module you can integrate into any website.
I use it with CodeIgniter and it works perfectly.
In network tab of Web Developer tools in firefox right click on the PUT, POST or any type of request, you will find "Use as Fetch in Console" option. Here we can seed the data we are passing.
Do the respective steps sequentially.