I extensively use this method to locate element inside shadowRoot using Protractor framework.
But after the chromedriver was updated to 96, it has stopped working.
Does anyone knows how to do work around for this in protractor framework
This solution worked for me with Chrome 97:
async function getShadowElement(shadowDomContainer: ElementFinder): Promise<any> {
const shadowRootObject = await browser.executeScript<any>('return arguments[0].shadowRoot', shadowDomContainer);
const rootKey = Object.keys(shadowRootObject)[0];
const rootId = shadowRootObject[rootKey];
return new WebElement(browser, rootId);
}
could thinks working using queryselector:
async expandUsingQuerySelector(sRoot,sElement){
var script="document.querySelector('"+sRoot+"').shadowRoot.querySelector('"+sElement+"')";
try{
return await browser.executeScript("return "+script );
} catch( error ) {
throw error;
}
}
I was working on a flutter package that I made https://github.com/pratikbaid3/flutter_client_sse. This is used to consume server-sent events. This works fine when the response returned is comparatively small. But as soon as the response grows large, I start getting this error flutter: FormatException: Unterminated string
This is the code snippet used to get the sse
while (true) {
try {
_client = http.Client();
var request = new http.Request("GET", Uri.parse(url));
request.headers["Cache-Control"] = "no-cache";
request.headers["Accept"] = "text/event-stream";
request.headers["Cookie"] = token;
Future<http.StreamedResponse> response = _client.send(request);
await for (final data in response.asStream()) {
await for (final d in data.stream) {
final rawData = utf8.decode(d);
final event = rawData.split("\n")[1];
if (event != null && event != '') {
yield SSEModel.fromData(rawData);
}
}
}
} catch (e) {
print('---ERROR---');
print(e);
yield SSEModel(data: '', id: '', event: '');
}
await Future.delayed(Duration(seconds: 1), () {});
}
My guess is that a part of the response is lost due to the large size.
So after hours of debugging, I found that in my current approach when the response is very large, part of it is truncated (Not sure why). I changed the code to read one line at a time and got it to work.
data.stream
..transform(Utf8Decoder())
.transform(LineSplitter())
.listen((dataLine) {
//Do what you want with the data here
}
This seemed to solve the issue.
The complete code can be found at https://github.com/pratikbaid3/flutter_client_sse
I had an MYSQL Dumped database which I converted to SQLite and moved that into the assets folder in Flutter.
I am trying to open the database but it is throwing the following error:
E/SQLiteLog(29199): (8) statement aborts at 1: [PRAGMA user_version = 1] attempt to write a readonly database I/flutter (29199): error DatabaseException(attempt to write a readonly database (code 8 SQLITE_READONLY)) sql 'PRAGMA user_version = 1' args []} during open, closing...
Here is the code:
_initDatabase() async {
var databasePath = await getDatabasesPath();
var path = join(databasePath, "example.db");
var exists = await databaseExists(path);
if (!exists) {
print('creating a new copy from asset!');
try {
await Directory(dirname(path)).create(recursive: true);
} catch (_) {}
ByteData data = await rootBundle.load(join("assets", "example.db"));
List<int> bytes =
data.buffer.asUint8List(data.offsetInBytes, data.lengthInBytes);
await File(path).writeAsBytes(bytes, flush: true);
} else {
print('opening existing database');
}
return await openDatabase(path, version: _dbVersion, readOnly: true);
}
It doesn't throw an error in the above method but when I try to query (a simple SELECT for example).
I have searched similar questions but of no avail. Some are saying it is a permission issue and some are saying something different.
Can anyone please tell me what I am doing wrong?
Specifying a version during openDatabase is to perform database schema migration, hence not for read only mode. Just remove the version parameter and you should be fine:
await openDatabase(path, readOnly: true);
Is there any way to run Headless Chrome/Chromium in a Google Cloud Function? I understand I can include and run statically compiled binaries in GCF. Can I get a statically compiled version of Chrome that would work for this?
The Node.js 8 runtime for Google Cloud Functions now includes all the necessary OS packages to run Headless Chrome.
Here is a code sample of an HTTP function that returns screenshots:
Main index.js file:
const puppeteer = require('puppeteer');
exports.screenshot = async (req, res) => {
const url = req.query.url;
if (!url) {
return res.send('Please provide URL as GET parameter, for example: ?url=https://example.com');
}
const browser = await puppeteer.launch({
args: ['--no-sandbox']
});
const page = await browser.newPage();
await page.goto(url);
const imageBuffer = await page.screenshot();
await browser.close();
res.set('Content-Type', 'image/png');
res.send(imageBuffer);
}
and package.json
{
"name": "screenshot",
"version": "0.0.1",
"dependencies": {
"puppeteer": "^1.6.2"
}
}
I've just deployed a GCF function running headless Chrome. A couple takeways:
you have to statically compile Chromium and NSS on Debian 8
you have to patch environment variables to point to NSS before launching Chromium
performance is much worse than what you'd get on AWS Lambda (3+ seconds)
For 1, you should be able to find plenty of instructions online.
For 2, the code that I'm using is the following:
static executablePath() {
let bin = path.join(__dirname, '..', 'bin', 'chromium');
let nss = path.join(__dirname, '..', 'bin', 'nss', 'Linux3.16_x86_64_cc_glibc_PTH_64_OPT.OBJ');
if (process.env.PATH === undefined) {
process.env.PATH = path.join(nss, 'bin');
} else if (process.env.PATH.indexOf(nss) === -1) {
process.env.PATH = [path.join(nss, 'bin'), process.env.PATH].join(':');
}
if (process.env.LD_LIBRARY_PATH === undefined) {
process.env.LD_LIBRARY_PATH = path.join(nss, 'lib');
} else if (process.env.LD_LIBRARY_PATH.indexOf(nss) === -1) {
process.env.LD_LIBRARY_PATH = [path.join(nss, 'lib'), process.env.LD_LIBRARY_PATH].join(':');
}
if (fs.existsSync('/tmp/chromium') === true) {
return '/tmp/chromium';
}
return new Promise(
(resolve, reject) => {
try {
fs.chmod(bin, '0755', () => {
fs.symlinkSync(bin, '/tmp/chromium'); return resolve('/tmp/chromium');
});
} catch (error) {
return reject(error);
}
}
);
}
You also need to use a few required arguments when starting Chrome, namely:
--disable-dev-shm-usage
--disable-setuid-sandbox
--no-first-run
--no-sandbox
--no-zygote
--single-process
I hope this helps.
As mentioned in the comment, work is being done on a possible solution to running a headless browser in a cloud function. A directly applicable discussion:"headless chrome & aws lambda" can be read on Google Groups.
The question at. had was can you run headless chrome or chromium in Firebase Cloud Functions... the answer is NO! since the node.js project will not have access any chrome/chromium executables and therefore will fail! (TRUST ME - I've Tried!).
A better solutions is to use the Phantom npm package, which uses PhantomJS under the hood:
https://www.npmjs.com/package/phantom
Docs and info can be found here:
http://amirraminfar.com/phantomjs-node/#/
or
https://github.com/amir20/phantomjs-node
The site i was trying to crawl had implemented screen scraping software, the trick is to wait for the page to load by searching for expected string, or regex match, i.e. i do a regex for a , if you need a regex of any complexity made for you - get in touch at https://AppLogics.uk/ - starting at £5 (GPB).
here is a typescript snippet to make the http or https call:
const phantom = require('phantom');
const instance: any = await phantom.create(['--ignore-ssl-errors=yes', '--load-images=no']);
const page: any = await instance.createPage();
const status = await page.open('https://somewebsite.co.uk/');
const content = await page.property('content');
same again in JavaScript:
const phantom = require('phantom');
const instance = yield phantom.create(['--ignore-ssl-errors=yes', '--load-images=no']);
const page = yield instance.createPage();
const status = yield page.open('https://somewebsite.co.uk/');
const content = yield page.property('content');
Thats the easy bit! if its a static page your pretty much done and you can parse the HTML into something like the cheerio npm package: https://github.com/cheeriojs/cheerio - an implementation of core JQuery designed for servers!
However if it is a dynamically loading page, i.e. lazy loading, or even anti-scraping methods, you will need to wait for the page to update by looping and calling the page.property('content') method and running a text search or regex to see if your page has finished loading.
I have created a generic asynchronous function returning the page content (as a string) on success and throws an exception on failure or timeout. It takes as parameters the variables for the page, text (string to search for that indicates success), error (string to indicate failure or null to not check for error), and timeout (number - self explanatory):
TypeScript:
async function waitForPageToLoadStr(page: any, text: string, error: string, timeout: number): Promise<string> {
const maxTime = timeout ? (new Date()).getTime() + timeout : null;
let html: string = '';
html = await page.property('content');
async function loop(): Promise<string>{
async function checkSuccess(): Promise <boolean> {
html = await page.property('content');
if (!isNullOrUndefined(error) && html.includes(error)) {
throw new Error(`Error string found: ${ error }`);
}
if (maxTime && (new Date()).getTime() >= maxTime) {
throw new Error(`Timed out waiting for string: ${ text }`);
}
return html.includes(text)
}
if (await checkSuccess()){
return html;
} else {
return loop();
}
}
return await loop();
}
JavaScript:
function waitForPageToLoadStr(page, text, error, timeout) {
return __awaiter(this, void 0, void 0, function* () {
const maxTime = timeout ? (new Date()).getTime() + timeout : null;
let html = '';
html = yield page.property('content');
function loop() {
return __awaiter(this, void 0, void 0, function* () {
function checkSuccess() {
return __awaiter(this, void 0, void 0, function* () {
html = yield page.property('content');
if (!isNullOrUndefined(error) && html.includes(error)) {
throw new Error(`Error string found: ${error}`);
}
if (maxTime && (new Date()).getTime() >= maxTime) {
throw new Error(`Timed out waiting for string: ${text}`);
}
return html.includes(text);
});
}
if (yield checkSuccess()) {
return html;
}
else {
return loop();
}
});
}
return yield loop();
});
}
I have personally used this function like this:
TypeScript:
try {
const phantom = require('phantom');
const instance: any = await phantom.create(['--ignore-ssl-errors=yes', '--load-images=no']);
const page: any = await instance.createPage();
const status = await page.open('https://somewebsite.co.uk/');
await waitForPageToLoadStr(page, '<div>Welcome to somewebsite</div>', '<h1>Website under maintenance, try again later</h1>', 1000);
} catch (error) {
console.error(error);
}
JavaScript:
try {
const phantom = require('phantom');
const instance = yield phantom.create(['--ignore-ssl-errors=yes', '--load-images=no']);
const page = yield instance.createPage();
yield page.open('https://vehicleenquiry.service.gov.uk/');
yield waitForPageToLoadStr(page, '<div>Welcome to somewebsite</div>', '<h1>Website under maintenance, try again later</h1>', 1000);
} catch (error) {
console.error(error);
}
Happy crawling!
Due to the latest update of chrome (v54) we've noticed our protractor tests failing. We attempted to update to the latest version of gulp-protractor (v3.0.0) which in turn downloads the latest web driver (v2.25) to resolve the issue but unfortunately a new error occurs we've been unable to resolve.
Everything worked fine before chrome's update.
Our protractor configuration is as follows:
exports.config = {
// Capabilities to be passed to the webdriver instance.
capabilities: {
'browserName': 'chrome'
},
onPrepare: function () {
var fs = require('fs');
var testDir = 'testresults/';
if (!fs.existsSync(testDir)) {
fs.mkdirSync(testDir);
}
var jasmineReporters = require('jasmine-reporters');
// returning the promise makes protractor wait for the reporter config before executing tests
return browser.getProcessedConfig().then(function () {
// you could use other properties here if you want, such as platform and version
var browserName = 'browser';
browser.getCapabilities().then(function (caps) {
browserName = caps.caps_.browserName.replace(/ /g, "_");
var junitReporter = new jasmineReporters.JUnitXmlReporter({
consolidateAll: true,
savePath: testDir,
// this will produce distinct xml files for each capability
filePrefix: 'test-protractor-' + browserName,
modifySuiteName: function (generatedSuiteName) {
// this will produce distinct suite names for each capability,
// e.g. 'firefox.login tests' and 'chrome.login tests'
return 'test-protractor-' + browserName + '.' + generatedSuiteName;
}
});
jasmine.getEnv().addReporter(junitReporter);
});
});
},
baseUrl: 'http://localhost:3000',
// Spec patterns are relative to the current working directory when
// protractor is called.
specs: [paths.e2e + '/**/*.js'],
// Options to be passed to Jasmine-node.
jasmineNodeOpts: {
showColors: true,
defaultTimeoutInterval: 30000
}
};
The error is:
[13:27:13] E/launcher - Error: Error
at C:\ws\node_modules\protractor\built\util.js:55:37
at _rejected (C:\ws\node_modules\q\q.js:844:24)
at C:\ws\node_modules\q\q.js:870:30
at Promise.when (C:\ws\node_modules\q\q.js:1122:31)
at Promise.promise.promiseDispatch (C:\ws\node_modules\q\q.js:788:41)
at C:\ws\node_modules\q\q.js:604:44
at runSingle (C:\ws\node_modules\q\q.js:137:13)
at flush (C:\ws\node_modules\q\q.js:125:13)
at nextTickCallbackWith0Args (node.js:420:9)
at process._tickCallback (node.js:349:13)
[13:27:13] E/launcher - Process exited with error code 100
onPrepare is being evaluated in built/util.js in the runFilenameOrFn_ function. The stacktrace unfortunately is not helpful but what this means is that onPrepare has errors. Looking at your onPrepare method, the error is made when assigning the browserName from the browser capabilities. In your code, caps.caps_ is actually undefined. Because caps.caps_ is undefined, caps.caps_.browserName is throwing an error. The capabilities object should be accessed as the following:
browser.getCapabilities().then(capabilities => {
let browserName = capabilities.browserName.replace(/ /g, "_");