Running Puppeteer on WSL2 - google-chrome

Is there a straightforward way to run Puppeteer on wsl2? I've tried many solutions across the web and read tons of forums looking for answers. So far no success and I've found running Puppeteer very easy in windows but extremely difficult in wsl2.
I've tried downloading chrome and chromium, and specifying the executablePath with no luck.
I know that it works on my windows system, but cannot seem to get Puppeteer to run in wsl2.
I am currently stuck on this error:
Timeout Error: Timed out after 30000 ms while trying to connect to the browser! Only Chrome at revision r1022525 is guaranteed to work.
When looking into node_modules I can see that version 1022525 is in use. I can't understand what Puppeteer's problem is nor how to fix it.
Anyone have any ideas?

As I do not see exactly what are you trying to do , in the moment I can think that this error accused because of your windows defender firewall, which do not allow access of your X-server.
anyway for me I use Puppeteer on wsl2 as following:
I need to install VcXsrc as X-server to display what is going on in the Linux side of the computer. https://sourceforge.net/projects/vcxsrv
notes: you need to allow “Public networks” when the Windows firewall pops up.
You need now to set a DISPLAY environment variable has to be set,becuase wsl has its own IP address, we need to connect to our X-server app using this IP so to do this follow the next steps.
In your .bashrc (or .zshrc if you are using ZSH) add this line exactly to the end:
export DISPLAY=$(cat /etc/resolv.conf | grep nameserver | awk '{print $2; exit;}'):0.0
now executes the content of your bash use source ~/.bashrc (or source ~/.zshrc if you are using ZSH)
to be sure that your DISPLAY environment variable has set correctly , try this command (echo $DISPLAY) the result should be something like this : # 172.16.210.1:0.0 it will be your wsl IP address.
now Open the VcXsrv program in Windows (called XLaunch), allow this options step after step “Multiple windows” and “Start no client”. and in the end tick the option of disable access control Be sure to disable access control.
as last check go to windows defender firewall and be sure that you enable all network connection for VcXSrv.
now you are ready to go, let suppose I am using puppeteer to test inside jest file like the follwoing :
const puppeteer = require('puppeteer');
test('should create an element...etc', async () => {
const browser = await puppeteer.launch({
headless: false,
args: ['--window-size=1920,1080']
});
const page = await browser.newPage();
await page.goto(
'www.google.com'
);
});
start the last code using (jest nameofyourtestfile) and your puppeteer will work as you like, Chromium will open in your windows ...enjoy...if there is any details which is not clear may you ask again. I hope that I helped

Related

how turn off browser sync in sublime text 3?

I'm using Sublime Text 3 and installed browsersync package for live reloading.
When I quit from Sublime and close liveserver app in my browser and try to open new liveserver with react start command I get a message that say: "Something is already running on port 3000." Then I check localhost:3000 and the browser still sync to Sublime I don't know how to turn off liveserver in port 3000 ?
please help me
I assume that this is the plugin you're using. Unfortunately, the plugin has been abandoned for 5 years, and the way it's written, there is no method for killing the node server once you're done with the plugin. From the Windows command line, this:
taskkill /im node-windows.exe /f /t
should do the trick, but you'll have to run it manually. Alternatively, you can use Task Manager to kill any node-windows.exe processes.
Another way to kill the service is going to
Start -> Resource Monitor -> Network
Under "Listening Ports" check for your port number, in this case 3000.
Take the process Id and then look for it under "Processes with Network Activity"
Check the process and right click -> End Process.
This will end the process utilizing the port 3000.
Try using Ctrl+C on your keyboard in your terminal or your command line (depending on whether your operating system is Unix based or Windows based). PowerShell should work too, if on a PC.

SSLKEYLOGFILE environment variable doesn't populate any text file

My operating system is Windows 10 64 bits. I use the latest versions of Firefox and Chrome.
I want to save the pre-master keys in order to use them with WireShark.
For this, I have found many tutorials that all recommend using the SSLKEYLOGFILE environment variable.
However, no matter the path I put in this variable, no file is being created by Firefox (normal and developer editions) or Chrome. This is true even when I restart those browsers or the operating system. This is also true when I clear the cache on the browsers.
I found an alternative way of setting the SSLKEYLOGFILE variable with Chrome by launching the browser with the following argument:
--ssl-key-log-file=PATH
And while this argument does create a log file, it is never populated. It remains completely empty.
What makes this problem even more annoying is that I don't know if there's any debug log that would at least let me know what's happening.
Anyone have ideas on how to solve this issue?
Verified in both Chrome and Firefox
Windows 10 64bit [Version 10.0.17763.379]
I would not use the --ssl-key-log-file flag with Chrome. In my testing, it does not have an effect.
Steps to get SSL keylog file
Change your directory to one that you or your programs have access to. I am using the Desktop folder.
C:\> cd $HOME\Desktop
Set the SSLKEYLOGFILE variable. This sets it for the user (HKCU). To set it for the machine (HKLM), add the /m flag to the end of the command.
PS C:\Users\rj\Desktop> SetX SSLKEYLOGFILE "$(get-location)\ssl.log"
Verify that the variable has been set in a separate powershell window (SetX does not apply to the current window).
PS C:\Users\rj\Desktop> Get-ChildItem ENV: | findstr SSLKEYLOGFILE
SSLKEYLOGFILE C:\Users\rj\Desktop\ssl.log
You can also verify that SSLKEYLOGFILE is a user variable by going to the Control Panel > System and Security > System > Advanced System Settings > Advanced tab > Environment Variables > User Variables. You should see a listing like the powershell example where the value is a directory.
Open Chrome/Firefox and go to an https website like https://stackoverflow.com.
You should see an ssl.log show up on your desktop.
Image of end result
Further Reading
2018-12-07, F5, Decrypting SSL traffic: Great article on doing the above, but cross-platform
2019-03-18, Mozilla, NSS Key Log Format
: The SSLKEYLOGFILE format
2018-08-17, Wireshark Bugzilla, Add TLS 1.3 support: Has good test pcap/SSLKEYLOGFILE contents for TLS1.3 (See Comment 83)
You mentioned that you are using Wireshark. If you using it to export a file from a TLS-encrypted stream in a capture, this article may help you.
One thing I ran into is: Chrome doesn't always fully close when you close the window. Sometimes it stays open in the background. Use the Windows Process Explorer to list all the processes and make cure Chrome is closed. Also, after restarting Chrome, you can double click on it in Process Explorer and select the Environment tab to list all the variables set in it's environment. Make sure SSLKEYLOGFILE is shown in that list. Then Chrome will be sending keys to the specified file.
First off all the problem is that the browsers are protected better and that flaw is patched, thats why you cant capture no ssl. I dont know the details. But chrome with --no-sandbox(linux) option on ,when running will enable you to capture ssl/tls key without any issues from the first try, so you can play with decription of packets. Never tried on windows btw..
This worked for me...
set the SSKEYLOG env: variable as normal in Linux.
just point it to your pefererd save location of the sslkey.log
windows system/advance/env vars/user variables.
add SSLKEYLOGfile with the path to your preferred save location C:/user/sslkey.log
close and your done here.
NSS_ALLOW_SSLKEYLOG=1 is what's missing from the Booleans in Firefox's advanced options.
its been also removed from the general release but is still in the dev ops version. just not listed.
download and install the dev ops version of Firefox
open a terminal start it from its exe if on Linux.
windows just open it from the icon.
go to the about:config
create a new Boolean with NSS_ALLOW_SSLKEYLOG=1 and set it to true.
quit
now create a new symbolic/icon link to the new dev version fire fox version
when you run Firefox from the terminal in Linux the sskeylog should be created and start populating.
run it from the task bar icon in Linux and it doesn't make or update the keylog (just in case you dont want it populating every time you open the browser).
in windows, you will have to disable the env var or it will keep populating every time you open any browser that supports sslkey logging.
on that note opera and vivaldi should work with no changes to the advanced option.
lastly.. open wireshark go to
edit/preferences/protocols/tls
and put the path to the sslkey.log in the pre master secret log box.
click ok.
close everything..
start Wireshark
start your browser (from terminal in linux).
you should start seeing sslkeylog start populating and see extra options for decryption in the tabs of Wireshark.
have fun :)
1st solution:
Restart Chrome
2nd solution:
Use Firefox

Is there an option to enable json-loading from HDD in Chrome or Internet Explorer?

Is there an option to enable Chrome or the Internet Explorer to load (via ajax-get) an JSON file from the local harddisk?
Firefox does so by default. Internet Explorer says 'premission denied' at the line
request.open(method, url, true);
There are no simple cleaner way you can do this, you have to do some server-side coding.
I suggest you node.js this is very handy for this type of IO operations.
It looks very difficult for first time but it is very handy.
Download and install nodejs.
create a simple project to handle this.
In the directory you want files to get (e.x. D:\myfiles), create one file server.js, content of that js file would be following
var connect = require('connect');
var serveStatic = require('serve-static');
connect().use(serveStatic("./")).listen(105);
open command promp from and get to this directory (D:\myfiles)
Run these two commands npm install connect and npm install serve-static (up to this, is one time process)
Run node server.js - That's all your server is ready and running
call http://localhost:105/<file name> and you can get any files within that directory.
For chrome you can use following command line flags, taken from following question
--allow-file-access-from-files
--disable-web-security
As noted in that question you need to restart your all chrome browsers.
For internet explorer, trick is to use ActiveX version of the XMLHttpRequest object.
See following link.
https://social.msdn.microsoft.com/Forums/ie/en-US/9ae077e0-a7b9-433f-835f-2643aa1a7e09/ie11previewcannot-read-local-files-with-ajax-method-of-jquery?forum=iewebdevelopment

Any way to start Google Chrome in headless mode?

I carefully revised the list of switches at http://peter.sh/experiments/chromium-command-line-switches/#chrome-frame and I couldn't find anything that would launch Chrome in a hidden background process.
The closest I was able to is --keep-alive-for-test + custom packaged app, but the app fails to execute any passed code because (the way it reports) "no window - ChromeHidden".
TL;DR
google-chrome --headless --remote-debugging-port=9222 http://example.com
You'd also need --disable-gpu temporarily.
Tutorial:
https://developers.google.com/web/updates/2017/04/headless-chrome
There's a work in progress: https://code.google.com/p/chromium/issues/detail?id=546953
The main deliverables are:
A library which headless applications can link to to.
A sample application which demonstrates the use of headless APIs.
So it would be possible to create a simple application that runs in console without connecting to display.
Update Apr 18 '16: The work is mainly done. There's a public forum now:
https://groups.google.com/a/chromium.org/forum/#!forum/headless-dev
Documentation is being in progress:
https://chromium.googlesource.com/chromium/src/+/master/headless/README.md
Update Sep 20 '16: It looks like chrome will eventually get the "--headless" parameter:
https://bugs.chromium.org/p/chromium/issues/detail?id=612904
There was a presentation on BlinkOn 6 (June 16/17, 2016)
Update Nov 29 '16: Design doc for --headless flag: https://docs.google.com/document/d/1aIJUzQr3eougZQp90bp4mqGr5gY6hdUice8UPa-Ys90/edit#heading=h.qxqfzv2lj12s
Update Dec 13 '16: --headless flag is expected to be available in Canary builds soon
Update Mar 12 '17: Chrome 57 has a --headless flag working. Waiting for Selenium and other tools to catch up. User guide: https://chromium.googlesource.com/chromium/src/+/lkgr/headless/README.md
This guy managed to run Chrome headlessly by using Xvfb (X virtual frame buffer) to trick Chrome into thinking it was displaying a window:
http://e-method.blogspot.fr/2010/11/google-chrome-with-xvfb-headless-server.html
If you're on Linux you could try that.
So basically you need to install X virtual frame buffer and Google Chrome via:
root#localhost: ~# apt-get install xvfb imagemagick
root#localhost: ~# apt-get install google-chrome
Then run the browser on the display:
root#localhost: ~# xvfb-run --server-args='-screen 0, 1024x768x24' \
google-chrome -start-maximized http://www.example.com \
> & /dev/null &
root#localhost: ~# DISPLAY=:99 import -window root myimage.png
Or you can look at PhantomJS project which is a headless WebKit implementation.
You could set up a linux VM and use xvfb in it.
Installation on debian / ubuntu:
sudo aptitude install xvfb
Start Chrome headless and visit http://example.com :
xvfb-run --server-args='-screen 0, 1024x768x16' google-chrome
-start-maximized http://example.com > /dev/null &
Turns out it starts in headless mode if you start it as a child subprocess. Besides that:
nircmd.exe can do win hide on chrome based on its PID
Autohotkey_L can also start Chrome hidden without a taskbar button
The Chromium Embedded Framework project seems like it might fit your usecase. I don't have personal experience with the project, but I've heard good things, and it has a solid API that you should be able to exploit for your purposes.
I don't have enough reputation to comment yet, but want to let you guys know that the chrome headless mode which Vanuan mentions actually works with Selenium webdriver.
In Java you can pass the flag to chrome through chromeDriver with the following code:
ChromeOptions options = new ChromeOptions();
options.addArguments("--headless");
ChromeDriver chromeDriver = new ChromeDriver(options);
I've recently found this article which mentions several commandline options that seem to do it. Using these keywords I googled out this piece of code which seem to confirm that these options exist.
// Does not automatically open a browser window on startup (used when
// launching Chrome for the purpose of hosting background apps).
const char kNoStartupWindow[] = "no-startup-window";
// Causes Chrome to launch without opening any windows by default. Useful if
// one wishes to use Chrome as an ash server.
const char kSilentLaunch[] = "silent-launch";
I managed to successfuly run Chrome with --no-startup-window and indeed it launched without any windows. It looked like it launched properly, it spawned all typical children, but the website I tried to make it load inside didn't seem to be actually visited. It maybe possible that this headless mode is only for running apps and not for visiting sites headless*), but it looks very promising as the normal worker tree is set up, just no windows.
The second option --silent-launch made chrome process very silent. I didn't notice any children spawned and the process exited promptly. I doubt it'll be usable for this case.
After I failed my attempts with these options, I focused on less sophisticated ways. On the bottom of the list there are two options:
// Specify the initial window position: --window-position=x,y
const char kWindowPosition[] = "window-position";
// Specify the initial window size: --window-size=w,h
const char kWindowSize[] = "window-size";
I ran Chrome with options to move it completely out of the working area:
--window-size=800,600 --window-position=-800,0
and as dirty as it feels, sure it's no true headless, but still the window is out of my sight, and everything done just with chrome's startup options, without external tools sending low-level window-hide messages.
*) yes, I know try to do weird things. Essentially I tried to get rid of Chrome window that's kept by Karma during tests. I know I can switch to PhantomJS, but I specifically needed to run them in Chrome, and the window popping out was.. eh.. disturbing in the long run.
It is currently under development, you can read more information about it from here:
https://chromium.googlesource.com/chromium/src/+/master/headless/README.md
Headless Chromium is a library for running Chromium in a
headless/server environment. Expected use cases include loading web
pages, extracting metadata (e.g., the DOM) and generating bitmaps from
page contents -- using all the modern web platform features provided
by Chromium and Blink.
It currently works on Linux, there is a nice presentation.
I was also able to make chrome headless work with NightwatchJS. Here is the config that let me use it:
"chromeHeadless": {
"desiredCapabilities": {
"browserName": "chrome",
"chromeOptions": {
"args": ["--headless"],
"binary": "/Applications/Google Chrome Canary.app/Contents/MacOS/Google Chrome Canary"
}
}
}
Chrome 59 has the ability to create instance as headless.
Find the below tutorial
https://www.automation99.com/2017/07/how-to-use-chrome-headless-using.html?m=1

Validate HTML on local machine

I'm currently trying to learn HTML and Java EE Servlet programming. I have an application server running on my local machine (Orion Application Server) and I'm connecting to web pages I've deployed on this server using a browser running on the same machine, directed to http://localhost/mypage.htm (for example).
I know W3C has a site you can go to that will validate an HTML page (and count how many errors are found for a given doctype), but that has to be a publicly available URL. How do you validate HTML on a locally running setup like I've described above?
many options:
see installation of w3c validation service:
http://validator.w3.org/docs/install.html
Firefox addons:
Firefox addon or other tool to locally validate HTML pages
https://addons.mozilla.org/en-US/firefox/addon/249/
Offline validator:
http://htmlhelp.com/tools/validator/offline/index.html.en
You can download a vnu.jar release for checking HTML5 documents offline:
https://github.com/validator/validator/releases/download/latest/vnu.jar
Alternatively, you can install it using any of the following:
docker run -it --rm -p 8888:8888 ghcr.io/validator/validator:latest
npm install vnu-jar
brew install vnu
pip install html5validator
See https://validator.github.io/validator/ for more details.
If you're using firefox, this plugin is perfect:
http://users.skynet.be/mgueury/mozilla/
I use it all day. When you view source it shows you a list of errors and highlights them for you.
A command line tool for validating a folder of html files: https://github.com/svenkreiss/html5validator
It integrates with CircleCI and TravisCI and can be used for validating Pelican and Jekyll sites.
Perhaps the most straightforward way to do this, is the way I do it all the time. "View source" of the web page, select it all (ctrl+a), choose "copy" (crtl+c), tab over to the validator, and its "direct input" option, and paste it in (ctrl+v). Easy, peasy.
On Mac, install w3validator by homebrew brew install vnu. Then check your local site by vnu http://localhost/site or any local file by vnu path/to/your/file.html (From Bluu answer)
If you're using node you may use package html-validator
const validator = require('html-validator')
const fs = require('fs')
var options = {
format: 'text'
}
fs.readFile( 'file-to-validate.html', 'utf8', (err, html) => {
if (err) {
throw err;
}
options.data = html
validator(options)
.then((data) => {
console.log(data)
})
.catch((error) => {
console.error(error)
})
})
http://validator.w3.org/#validate_by_upload if you don't mind uploading the HTML source file.
http://getfirebug.com/ if you're running Firefox can help with HTML validation issues as well.
if you have internet connection and want to use https://validator.w3.org/ for localhost without installing, you can use ngrok
reference : https://academy.byidmore.com/post/W3C-Markup-Validation-for-Localhost-5bc942eb3704302c0986bd60
You can run the tool on your local with docker just by using the command below.
docker run -it --rm -p 8888:8888 ghcr.io/validator/validator:latest
After running it with docker, when you go to 127.0.0.1:8888 you will see the validator tool. When you try to validate a url and if you get such an error like IO Error (Connection refused) then you can try installing vnu with brew by using the second command below.
brew install vnu
I tried it with docker and I got IO Error. Then I tried it with brew and it was successful. After you install it with brew, now to check a url you should run the command below.
vnu http://localhost/page-to-test/
Just replace the url with the one you want to validate with the tool.