I have a simple script to use Puppter to crawl a webpage stopped working, so I started to debug.
I found if I launch my script by setting the headless option to true, my script can work as expected.
So I saved the page content and a screenshot, and I found they really are different than when the page is loaded with headless set to false.
I have enabled loading and executing JavaScript. It looks like the page can detect if it is loaded in headless mode and can render/behave differently.
Any idea how to troubleshoot if there's a bug in my script or that web page really can detect headless browser?
When launching the browser in headless mode, the user-agent changes to HeadlessChrome.
Thats what a web page can detect obviously, you need to set the user-agent manually using: page.setUserAgent(userAgent).
Launch headless==true:
Mozilla/5.0 ... AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/80.1.5.115 Safari/537.36
Launch headless==false:
Mozilla/5.0 ... AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.149 Safari/537.36
Related
I've been using Chrome Debugger since forever to test my site's mobile layout. I click the mobile icon in the top left corner and then refresh the page. It updates the User Agent string and it's all good. Last week it started doing this weird behavior where it changes it to this after a page refresh:
Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.130 Safari/537.36
When I click the mobile icon in the Chrome Debugger then it immediately reports the correct user agent. But when I refresh the browser it reverts to the non-mobile user agent listed above and stays there. Note that the browser still shows the page dimensions correctly as being a mobile device, so it does know that I clicked the mobile button, but the user agent doesn't match up anymore.
I have to toggle it off and back on using the Debugger and then it only works until I refresh the page. So it temporarily sets it to the correct user agent string but always loses it on a page refresh. If I click the refresh button enough times it does show the correct user agent, but on the next refresh it immediately goes back to the non-mobile user agent. I'm guessing it's correct about ten percent of the time.
I use the site https://www.whatismybrowser.com/detect/what-is-my-user-agent to test the current setting.
Any ideas?
Thanks.
My extension / add-on is live on multiple browser stores but if I want to direct someone to the extension, I have to provide them 4 different links (for e.g. shown here - http://credizian.gitlab.io/untrackme or in an email, as a list).
I'm trying to figure out how I can create one URL that will auto-redirect user to the relevant browser extension store - aka better user experience.
The process I've thought of is - creating a page that has javascript built-in to redirect users based on browser detection logic. I am thinking to host it in S3 and just put basic javascript only with a blank page for fastest redirect. Is that the best way?
Do browsers flag this from of redirect since it's going to external URLs? I see email links constantly opening a page and multiple redirects happening before the final page loading...not sure if they are doing something to not be considered spam
I've looked at other answers but they are focused on redirection to 'internal pages'
User agents for specified browsers will look like:
Mozilla - Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:15.0) Gecko/20100101 Firefox/15.0.1
Chrome - Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/62.0.3202.9 Safari/537.36
Opera - Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/61.0.3163.100 Safari/537.36 OPR/48.0.2685.52
Safari - Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US) AppleWebKit/533.20.25 (KHTML, like Gecko) Version/5.0.4 Safari/533.20.27
browser_dict = {'Safari': "safari url here",'OPR':"opera url here",'Chrome':"chrome url here", 'Firefox':"firefox url here"}
What you can do is to split user-agent with space:
Here is the pseudo code I am writing:
useragent = user-agent.split(' ') //useragent will be list
again splitting last and last second item of above list using '/':
useragent-last = useragent[-1].split('/')
useragent-lastsecond= useragent[-2].split('/')
now checking if useragent-last or useragent-lastsecond present in browser dictionary
for key, value in browser_dict.items():
if useragent-last == key:
go to url browser_dict[key] //url
elif useragent-lastsecond == key:
go to url browser_dict[key] //url
else:
print (No known browser detected.)
This might help you.
I have created a website that displays different source code depending on whether the user is using a desktop or mobile device.
In Chrome on desktop, I can easily view the source code of the desktop version of any website simply by right-clicking anywhere on the website and then choosing "View Page Source".
In Chrome on desktop, is there a way that I can view the source code of the mobile version of a website? I know that in developer tools I can view websites using the mobile emulator, and I can inspect elements as rendered to the mobile browser. However, I have not yet figured out how to view the raw source code rendered to the mobile browser.
To view raw source of mobile version. From developer tools window, on Network tab, select request that retrieve page of interest.
On right pane, activate Response tab. The original source is in Response payload window.
You can check https://developers.google.com/web/tools/chrome-devtools/debug/remote-debugging/remote-debugging?hl=en
Basically you connect your mobile to you pc/mac and you can use your desktop chrome to inspect the pages you render with your mobile and even control it from there.
I haven't been able to find a reliable solution so just end up using curl in the terminal:
curl -H "user-agent: Mozilla/5.0 (iPhone; CPU iPhone OS 11_0 like Mac OS X) AppleWebKit/604.1.38 (KHTML, like Gecko) Version/11.0 Mobile/15A372 Safari/604.1" https://url/to/inspect
For ease can save as a simple bash script fetch-mob-src.sh:
#!/bin/bash
user_agent="Mozilla/5.0 (iPhone; CPU iPhone OS 11_0 like Mac OS X) AppleWebKit/604.1.38 (KHTML, like Gecko) Version/11.0 Mobile/15A372 Safari/604.1"
curl -XGET -H "user-agent: $user_agent" $1
E.g. ./fetch-mob-src.sh https://url/to/inspect
For iOS google chrome, when a user hits the "Request desktop site" button what does the browser do to try to bring up a desktop site? I imagine some sort of header on the request that sites are looking for, or something similar?
I think the only difference is the User-Agent: header in the request.
Here are the User-Agent headers sent by Chrome on my Android device:
Mozilla/5.0 (Linux; Android 4.0.4; Galaxy Nexus Build/IMM76K) AppleWebKit/535.19 (KHTML, like Gecko) Chrome/18.0.1025.166 Mobile Safari/535.19
Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/535.19 (KHTML, like Gecko) Chrome/18.0.1025.45 Safari/535.19
Notice the word "Mobile' in the first one, and also the mention of Android system and device. Checking these, I see that it also provides false information - namely X11 and x86_64 - to closely match the value sent by the Desktop Linux version of Chrome.
Just wanted to point out that Chrome now not only changes the User-Agent but also ignores the original viewport meta tag if you "Request Desktop Site". Thus it won't be necessary to sniff the User-Agent anymore and you can rely on the viewport change as most responsive sites will automatically do. See this Change for further reference.
One other slight difference is that the request appears to have been to the last intentionally entered URL before any re-directors moved it. For example:
Given: somesite.com sniffs the agent, sees Android, and does a document.location += "/m";
Then: the browser will have a URL of somesite.com/m
But: if you "Request desktop site" it will change the User-Agent and re-request from somesite.com
Unless: you had gone directly in on the mobile URL of somesite.com/m in the first place, in which case it just reloads somesite.com/m.
I would expect that this works with HTTP 301 and 302 redirects, I know it works with document.location changes (at least as described), and would speculate that it works with <meta> refreshes.
This javascript snippet will effectively do the same :
function requestDesktopSite() {
document.getElementsByTagName('meta')['viewport'].content='min-width: 980px;';
}
<button onclick="requestDesktopSite()">Request Desktop Site</button>
This question already has answers here:
User Agent Switcher for Chrome
(7 answers)
Closed 8 years ago.
I'm tinkering with a phpbb mod which adds a skin for mobile phones, and to work on it in my computer I want to mimic an iphone. In Firefox I manage to do this using the "user agent switcher" extension which comes with built-in "iphone" feature:
https://addons.mozilla.org/en-us/firefox/addon/user-agent-switcher/
However, in Chrome (which I prefer) I cannot make this happen. I downloaded what seemed to be the relevant extension:
https://chrome.google.com/extensions/detail/aafciojnlamllgpkpdkbamkfgbofhgcj?hl=he
It doesn't come with a built-in iPhone mode, so I added the following user agent:
"Mozilla/5.0 (iPhone; U; CPU like Mac OS X; en) AppleWebKit/420+ (KHTML, like Gecko) Version/3.0 Mobile/1A543 Safari/419.3"
However, this seems to have absolutely no effect - the pages load normally and not in the mobile skin.
With the latest release in the Chrome Dev Channel (version 18) you can browse to developer tool settings and change the user agent string directly without needing any plugins or command line options.
First download Chrome from the developer channel, restart it, CTRL+SHIFT+I and click settings on bottom right. You should now be able to select the UA string.
You can change chrome user agent by running it from command line:
chrome.exe --user-agent="User Agent String"
check out This Page for more information
The User-Agent Switcher for Chrome works nicely, I just tested it out.
I downloaded the portable version of chrome and created a custom shortcut to it to always have a mobile browser.
F:_SOE\Apps\GoogleChromePortable\GoogleChromePortable.exe --user-agent="Mozilla/5.0 (iPhone; U; CPU like Mac OS X; en) AppleWebKit/420+ (KHTML, like Gecko) Version/3.0 Mobile/1A543 Safari/419.3"
There is now the Ultimate User Agent Switcher in the Chrome Webstore. It's awesome!
I don't think chrome can do that but using one of the proxy out there you would be able to do it like privoxy