Testing with watir chromedriver failed with Sec-Fetch-User in Request Headers - selenium-chromedriver

I am using watir for testing an internal application. The testing recently stops working.
These are the simplified steps of the test:
require 'watir'
Selenium::WebDriver::Chrome.path = 'PATH_TO_CHROME_EXE'
browser = Watir::Browser.new :chrome, :options => {:options => {'useAutomationExtension' => false}}
URL = "TEST_URL"
browser.goto URL
After the goto line executed, the browser failed to navigate to the page. When inspecting the network activity, The request status showing as canceled. I also noticed the "Sec-Fetch-User" field populate in the header:
Sec-Fetch-Mode: navigate
**Sec-Fetch-User: ?1**
Upgrade-Insecure-Requests: 1
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/77.0.3865.90 Safari/537.36
If I directly enter the test URL into the browser, the field is not populated and the login pop-up.
This is my setting:
jruby 9.2.5.0 (2.5.0) 2018-12-06 6d5a228 Java HotSpot(TM) 64-Bit Server VM 25.102-b14 on 1.8.0_102-b14 +jit [mswin32-x86_64]
Version 77.0.3865.90 (Official Build) (64-bit)
Watir 6.16.5
ChromeDriver 77.0.3865.40 (f484704e052e0b556f8030b65b953dce96503217-refs/branch-heads/3865#{#442})
Did anyone run into similar issue? Would it be possible to suppress the header?
Steve

I run into exactly the same problem of Sec-Fetch-User: ?1.
The only way I could get to load the webpage I wanted was to ditch the chromedriver and use the firefox driver instead. Firefox did not send those headers.
I think that, if you really need to use chromedriver, and if you need to change the headers is to use a proxy like https://github.com/lightbody/browsermob-proxy

Related

How do I force Chrome DevTools Audits to use desktop (fullscreen) emulation?

I want to run a performance audit of my website for: desktop, without throttling network, without throttling CPU.
I'm using Google Chrome's DevTools's Audits tab and selecting the following options:
Device: Desktop
Audits: Performance
Throttling: No throttling
Clear storage (checked)
My chrome version:
Google Chrome is up to date
Version 71.0.3578.98 (Official Build) (64-bit)
According to the "Audits" tab, my Lighthouse version in Chrome is 3.2.0
After clicking Run audits button, the report is generated, and contains the following Runtime settings:
URL: https://example.com/
Fetch time: Dec 15, 2018, 11:18 PM PST
Device: Emulated Nexus 5X
Network throttling: 150 ms TCP RTT, 1,638.4 Kbps throughput (Simulated)
CPU throttling: 4x slowdown (Simulated)
User agent (host): Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/71.0.3578.98 Safari/537.36
User agent (network): Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5 Build/MRA58N) AppleWebKit/537.36(KHTML, like Gecko) Chrome/71.0.3559.0 Mobile Safari/537.36
CPU/Memory Power: 998
So how do I force my settings to be used so I can correctly audit for desktop?
EDIT: Verified upstream as Chrome bug:
https://bugs.chromium.org/p/chromium/issues/detail?id=914387
It appears this is a Google Chrome bug, see below for quotes and reference:
We've found the culprit, but it'll require a Chrome-side fix.
Note that it's not just Mobile/Desktop, but the throttling settings and clear storage options are also being ignored at the moment which would explain all of the performance score differences.
-- patrickhulce, paulirish
Ref: https://github.com/GoogleChrome/lighthouse/issues/6772#issuecomment-446626154
Upstream bug: https://bugs.chromium.org/p/chromium/issues/detail?id=914387
Verified the fix on Mac 10.12.6, Windows-10 & Ubuntu 17.10 on Chrome version #73.0.3640.0 as per the comment#0
Approved for Chrome M72 branch, 3626

Fiddler Echo Service error while accessing localhost in my chrome browser

I want to run apache-tomcat-6.0.44 server on my laptop. I have configured Classpath, Path and CATALINA_HOME variables. I am able to run the server from command prompt (using "startup" command). But the problem is, when I am typing "http://localhost:8080" in the url bar of my chrome browser and clicking enter, I am gettig error as:
Fiddler Echo Service
GET / HTTP/1.1 Host: localhost:8080 Proxy-Connection: keep-alive
Accept:
text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,/;q=0.8
Upgrade-Insecure-Requests: 1 User-Agent: Mozilla/5.0 (Windows NT 6.1)
AppleWebKit/537.36 (KHTML, like Gecko) Chrome/45.0.2454.101
Safari/537.36 Accept-Encoding: gzip, deflate, sdch Accept-Language:
en-GB,en-US;q=0.8,en;q=0.6 Cookie: _ga=GA1.1.1441246844.1443724804 To
configure Fiddler as a reverse proxy instead of seeing this page, see
Reverse Proxy Setup You can download the FiddlerRoot certificate**
I have gone through some of the solutions, which are:
1) Fiddler Echo Service blocking oracle homepage in browser
2) Fiddler not capturing traffic from browsers
But I am still not able to fix the issue. Could anyone please help me out with this.
Along with using proxy server, Check if your Lan setting is set to detect Automatically.
Go to below section of chrome
"settings>Advanced settings>Network>Change Proxy settings" You shall get a wizard.
In there click "Lan Settings" and select the checkbox for "Automatically detect settings"
Another possibility is mismatch of crtificate installed.
server name used in Apache (httpd.conf) must be the same as the server name in apache (httpd-ssl.conf) e.g. in Apache (httpd.conf)ServerName localhost:8080 then in apache (httpd-ssl.conf) should be like this ServerName www.example.com:8080

How do I modify Google Chrome user agent string with Robot Framework

I tried to modify the user agent string using Open Browser with desired_capabilities and discovered that Chrome does not support that technique any more.
After much searching and reading I discovered that there was a new version of Selenium2library that had a new keyword Create Webdriver that is supposed to address this issue.
I modified their example to suite my needs. But no matter what I do, it does not modify the user agent string.
I get no errors, no warnings, no nothing, except a perfectly working browser without a modified user agent string.
I tried to modify other options like --start-maximized with the same result, i.e. no result at all.
Excerpt from keyword that opens Google Chrome and (allegedly) modifies the user agent string:
${options}= Evaluate sys.modules['selenium.webdriver'].ChromeOptions() sys, selenium.webdriver
${options.add_argument}= Set Variable user-agent="Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/28.0.1500.95 Safari/537.36 System/ComputerId"
Create WebDriver Chrome chrome_options=${options}
Go To http://www.useragentstring.com
Fashioned after the example given here (at the bottom of the page):
https://github.com/rtomac/robotframework-selenium2library/issues/225
My software setup:
Google Chrome 31.0.1650.59
Selenium 2.39.0
Selenium2library 1.5
Robot Framework 2.8.3
Robot Framework Ride 1.2.2
So what is the problem?
After some more tinkering and reading I managed to figure out a way to get the example working.
Instead of using ${options.add_argument}= I used Call Method ${options} add_argument.
${options}= Evaluate sys.modules['selenium.webdriver'].ChromeOptions() sys, selenium.webdriver
Call Method ${options} add_argument --user-agent=Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/28.0.1500.95 Safari/537.36 System/ComputerId
Create WebDriver Chrome chrome_options=${options}
Go To http://www.useragentstring.com
I tried setting up user agent in chrome using RF and working fine for me below snippet...!
${options}= Evaluate sys.modules['selenium.webdriver'].ChromeOptions() sys, selenium.webdriver
${userAgent}= set variable --user-agent="Mozilla/5.0 (Linux; Android 4.2.1; en-us; Nexus 5 Build/JOP40D) AppleWebKit/535.19 (KHTML, like Gecko) Chrome/18.0.1025.166 Mobile Safari/535.19"
Call Method ${options} add_argument ${userAgent}
Create WebDriver Chrome chrome_options=${options}
I have encountered a similar problem. i tried running your code but no luck in making it work. It just says that the user agent is not defined. I have browsed and came across this code, but unfortunately is written in python:
from selenium import webdriver
import webbrowser
from selenium.webdriver.chrome.options import Options
mobile_emulation = {
"deviceMetrics": { "width": 360, "height": 640, "pixelRatio": 3.0 },
"userAgent": "Mozilla/5.0 (Linux; Android 4.2.1; en-us; Nexus 5 Build/JOP40D) AppleWebKit/535.19 (KHTML, like Gecko) Chrome/18.0.1025.166 Mobile Safari/535.19" }
chrome_options = Options()
chrome_options.add_experimental_option("mobileEmulation", mobile_emulation)
driver = webdriver.Chrome(chrome_options = chrome_options)

Android Chrome, CORS and Play!Framework fails

I've built a web application using CORS for communicating with the API.
The API accept all origin and some headers and is written using Play!Framework.
On every request made to the app, I add these headers :
override def doFilter(action: EssentialAction): EssentialAction = EssentialAction { request =>
action.apply(request).map(_.withHeaders(
"Access-Control-Allow-Origin" -> "*",
"Access-Control-Allow-Methods" -> "GET, POST, PUT, DELETE, OPTIONS",
"Access-Control-Allow-Headers" -> "Origin, X-Requested-With, Content-Type, Accept, Host, Api-Token"
))
}
Everything works great in Firefox and Chrome in the desktop (tested under Windows and Linux), but fails on my Android phone (using Chrome browser and the native browser).
I enabled the Chrome debugging via USB, and I can clearly see that Chrome doesn't go further than the OPTIONS request made to the server. Here's the request made :
Request URL:http://127.0.0.1:9100/auth/login
Request Headersview source
Access-Control-Request-Headers:accept, origin, x-requested-with, content-type
Access-Control-Request-Method:POST
Origin:http://192.168.0.15:9000
Referer:http://192.168.0.15:9000/login
User-Agent:Mozilla/5.0 (Linux; Android 4.1.1; HTC One S Build/JRO03C) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/27.0.1453.90 Mobile Safari/537.36
And it stops here, no response from the server (but the Network tabs indicate "Pending" for this request !).
So now I don't know what is wrong. How can I fix this problem ?
It may be a red herring, but I see in the headers you show the string:
Request URL:http://127.0.0.1:9100/auth/login
It seems to me that it tries to connect to localhost (127.0.0.1) but your mobile doesn't have a running server, so it fails.

Google Chrome issuing Head Request to random domains

Today I was investigating on something with Fiddler, when I noticed that, when I launch Google Chrome, I have always 3 HEAD requests to some domains which seem to be randomly chosen.
Here is a sample :
HEAD http://fkgrekxzgo/ HTTP/1.1
Host: fkgrekxzgo
Proxy-Connection: keep-alive
Content-Length: 0
User-Agent: Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.31 (KHTML, like Gecko) Chrome/26.0.1410.64 Safari/537.31
Accept-Encoding: gzip,deflate,sdch
Do you have any idea of why Google Chrome behaves this way ?
Thanks guys
This is Chrome checking to see if your ISP converts non-resolving domains into your.isp.com/search?q=randomnonresolvingdomain
See https://mikewest.org/2012/02/chrome-connects-to-three-random-domains-at-startup
This algorithm seems unusable with forward proxy servers. Browser definitely asks for random page and proxy definitely returns some page -- error (50x), masked error (50x or 40x) or nice "you are lost" page with HTTP code 200 .