how to mock on global object in backstop.js/puppetter - puppeteer

So backstop.js provides ability to run custom script against underlying engine. I use puppeteer as an engine so I try to mock Date.now with 'onReadyScript':
page.evaluate('window.Date.now = () => 0; Date.now = () => 0;');
...
page.addScriptTag({
// btw `console.log` here is not executed, do I use it in wrong way?
content: 'Date.now = () => 0;'
});
...
page.evaluate(() => {
window.Date.now = () => 0;
Date.now = () => 0;
});
Last one, I think, is modifying Date in context of Node, not inside the puppeteer, but anyway tried that as well.
Nothing worked, script under the test still output real Date.now. Also I checked Override the browser date with puppeteer but it did not help me.
Yes, I know I'm able to skip particular selectors, but it does not always make sense(think about clock with arrows).

After trying onBeforeScript with evaluateOnNewDocument() it works for me. Complete script:
module.exports = async function (page, scenario) {
if (!page.dateIsMocked) {
page.dateIsMocked = true
await page.evaluateOnNewDocument(() => {
const referenceTime = '2010-05-05 10:10:10.000';
const oldDate = Date;
Date = function(...args) {
if (args.length) {
return new oldDate(...args);
} else {
return new oldDate(referenceTime);
}
}
Date.now = function() {
return new oldDate(referenceTime).valueOf();
}
Date.prototype = oldDate.prototype;
})
}
};
Reason: onReadyScript is executed when page under testing has already been loaded and executed. So code is bound to original Date by closure, not the mocked version.

Related

Save filereader result to variable for later use

I can't find simple answer, but my code is simple.
I tried something like that, but always when i try to console.log my testResult, then i always recieving null. How to save data from file correctly?
public getFile(
sourceFile: File
): string {
let testResult;
const file = sourceFile[0]
const fileReader = new FileReader();
fileReader.readAsText(file, "UTF-8")
fileReader.onloadend = (e) => {
testResult = fileReader.result.toString()
}
console.log(testResult)
return testResult
}
This problem is related to my other topics, main reason is i can't handle load json file, translate them and upload to user. If i can save this file outside onloadend, then i hope i can handle rest of them (other attempts failed, this one blocking me at beginning)
Your issue is quite classical and is related to the asynchronous operations. Function which you assign to the onloadend request is called only when loadend event fires, but the rest of code will not wait for that to happen and will continue execution. So console.log will be executed immediately and then return will actually return testResult while it is still empty.
Firstly, in order to understand what I just said, put the console.log(testResult) line inside of your onloadend handler:
fileReader.onloadend = (e) => {
testResult = fileReader.result.toString();
console.log(testResult);
}
At this point testResult is not empty and you may continue handling it inside this function. However, if you want your getFile method to be really reusable and want it to return the testResult and process it somewhere else, you need to wrap this method into a Promise, like this:
public getFile(
sourceFile: File
): Promise<string> {
return new Promise((resolve) => {
const file = sourceFile[0]
const fileReader = new FileReader();
fileReader.onloadend = (e) => {
const testResult = fileReader.result.toString();
resolve(testResult);
}
fileReader.readAsText(file, "UTF-8");
});
}
Now whereever you need a file you can use the yourInstance.getFile method as follows:
yourInstance.getFile().then(testResult => {
// do whatever you need here
console.log(testResult);
});
Or in the async/await way:
async function processResult() {
const testResult = await yourInstance.getFile();
// do whatever you need
console.log(testResult);
}
If you are now familiar with promises and/or async/await, please read more about here and here.

ReactJS DraftJS Initialize from Serialized Data

So I am using the DraftJS package with React along with the mentions plugin. When a post is created, I store the raw JS in my PostreSQL JSONField:
convertToRaw(postEditorState.getCurrentContent())
When I edit the post, I set the editor state as follows:
let newEditorState = EditorState.createWithContent(convertFromRaw(post.richtext_content));
setEditorState(newEditorState);
The text gets set correctly, but none of the mentions are highlighted AND I can't add new mentions. Does anyone know how to fix this?
I am using the mention plugin: https://www.draft-js-plugins.com/plugin/mention
to save data
function saveContent() {
const content = editorState.getCurrentContent();
const rawObject = convertToRaw(content);
const draftRaw = JSON.stringify(rawObject); //<- save this to database
}
and retrieval:
setEditorState(()=> EditorState.push(
editorState,
convertFromRaw(JSON.parse(draftRaw)),
"remove-range"
););
it should preserve your data as saved.
the example provided (which works ok) is for inserting a new block with mention, saving the entityMap as well.
mentionData is jus a simple object {id:.., name:.., link:... , avatar:...}
One more thing:
initialize only once:
in other words do not recreate the state.
const [editorState, setEditorState] = useState(() => EditorState.createEmpty() );
und then populate something like:
useEffect(() => {
try {
if (theDraftRaw) {
let mtyState = EditorState.push(
editorState,
convertFromRaw(JSON.parse(theDraftRaw)),
"remove-range"
);
setEditorState(mtyState);
} else editorClear();
} catch (e) {
console.log(e);
// or some fallback to other field like text
}
}, [theDraftRaw]);
const editorClear = () => {
if (!editorState.getCurrentContent().hasText()) return;
let _editorState = EditorState.push(
editorState,
ContentState.createFromText("")
);
setEditorState(_editorState);
};

DialogflowSDK middleware return after resolving a promise

I'm currently playing around with the actions-on-google node sdk and I'm struggling to work out how to wait for a promise to resolve in my middleware before it then executes my intent. I've tried using async/await and returning a promise from my middleware function but neither method appears to work. I know typically you wouldn't override the intent like i'm doing here but this is to test what's going on.
const {dialogflow} = require('actions-on-google');
const functions = require('firebase-functions');
const app = dialogflow({debug: true});
function promiseTest() {
return new Promise((resolve,reject) => {
setTimeout(() => {
resolve('Resolved');
}, 2000)
})
}
app.middleware(async (conv) => {
let r = await promiseTest();
conv.intent = r
})
app.fallback(conv => {
const intent = conv.intent;
conv.ask("hello, you're intent was " + intent );
});
It looks like I should at least be able to return a promise https://actions-on-google.github.io/actions-on-google-nodejs/interfaces/dialogflow.dialogflowmiddleware.html
but I'm not familiar with typescript so I'm not sure if I'm reading these docs correctly.
anyone able to advise how to do this correctly? For instance a real life sample might be I need to make a DB call and wait for that to return in my middleware before proceeding to the next step.
My function is using the NodeJS V8 beta in google cloud functions.
The output of this code is whatever the actual intent was e.g the default welcome intent, rather than "resolved" but there are no errors. So the middleware fires, but then moves onto the fallback intent before the promise resolves. e.g before setting conv.intent = r
Async stuff is really fiddly with the V2 API. And for me only properly worked with NodeJS 8. The reason is that from V2 onwards, unless you return the promise, the action returns empty as it has finished before the rest of the function is evaluated. There is a lot to work through to figure it out, here's some sample boilerplate I have that should get you going:
'use strict';
const functions = require('firebase-functions');
const {WebhookClient} = require('dialogflow-fulfillment');
const {BasicCard, MediaObject, Card, Suggestion, Image, Button} = require('actions-on-google');
var http_request = require('request-promise-native');
process.env.DEBUG = 'dialogflow:debug'; // enables lib debugging statements
exports.dialogflowFirebaseFulfillment = functions.https.onRequest((request, response) => {
const agent = new WebhookClient({ request, response });
console.log('Dialogflow Request headers: ' + JSON.stringify(request.headers));
console.log('Dialogflow Request body: ' + JSON.stringify(request.body));
function welcome(agent) {
agent.add(`Welcome to my agent!`);
}
function fallback(agent) {
agent.add(`I didn't understand`);
agent.add(`I'm sorry, can you try again?`);
}
function handleMyIntent(agent) {
let conv = agent.conv();
let key = request.body.queryResult.parameters['MyParam'];
var myAgent = agent;
return new Promise((resolve, reject) => {
http_request('http://someurl.com').then(async function(apiData) {
if (key === 'Hey') {
conv.close('Howdy');
} else {
conv.close('Bye');
}
myAgent.add(conv);
return resolve();
}).catch(function(err) {
conv.close(' \nUh, oh. There was an error, please try again later');
myAgent.add(conv);
return resolve();
})})
}
let intentMap = new Map();
intentMap.set('Default Welcome Intent', welcome);
intentMap.set('Default Fallback Intent', fallback);
intentMap.set('myCustomIntent', handleMyIntent);
agent.handleRequest(intentMap);
});
A brief overview of what you need:
you have to return the promise resolution.
you have to use the 'request-promise-native' package for HTTP requests
you have to upgrade your plan to allow for outbound HTTP requests (https://firebase.google.com/pricing/)
So it turns out my issue was to do with an outdated version of the actions-on-google sdk. The dialogflow firebase example was using v2.0.0, changing this to 2.2.0 in the package.json resolved the issue

How do I load a WASM module in a Vue component without initializing the module every time?

I have created a Rust library of type cdylib using
cargo web build --target=wasm32-unknown-unknown
I use a modified version of the "rust-wasm-loader" NPM package to build and load the WASM file. rust-wasm-loader uses this as a way to use the Rust code:
const wasm = require('./main.rs')
wasm.initialize().then(module => {
// Use your module here
const doub = module.cwrap('doub', 'number', ['number'])
console.log(doub(21))
})
I do not want to initialize the module every time I want to use the code. How do I load the module and use it like a library?
Since the loading of WebAssembly is asynchronous and may actually take some time for large modules, you need to handle the state when the module is not loaded, and then let the rest of the application know when the WebAssembly module is loaded.
You do not say how you are handling state in your Vue application, but if you are e.g. using Vuex you can do something like this:
const doubPlugin = store => {
wasm.initialize().then(module => {
const doub = module.cwrap('doub', 'number', ['number'])
store.subscribe((mutation, state) => {
if (mutation.type === 'DOUB_REQUEST') {
store.commit('DOUB_RESULT', doub(mutation.payload))
}
})
store.commit('DOUB_READY')
})
}
const store = new Vuex.Store({
state,
mutations,
plugins: [doubPlugin]
})
I've done a similar thing in an Elm/WebAssembly application (relevant JavaScript), so if you want to see how this can be applied in practice you can check that out.
Making a wrapper JS module that performs initialization and re-exports the promise seems like the most straightforward approach.
// main.js
module.exports = require("./main.rs").initialize().then(module => {
return {
doub: module.cwrap('doub', 'number', ['number'])
};
});
Then anything can do
require("./main.js").then(api => {
console.log(api.doub(21));
});
and will always get the same module. Or alternatively you could invert the async part and do
// main.js
const api = require("./main.rs").initialize().then(module => {
return {
doub: module.cwrap('doub', 'number', ['number'])
};
});
exports.doub = async function (val) {
return (await api).doub(val);
};
Then users of your module could do
const api = require("./main.js");
api.doub(21).then(result => {
console.log(result);
});
I created a class to wrap the WebAssembly loading and created a cwrap for every function:
class mkLib {
ready = false
_mod = require("./main.rs").initialize().then(module => {
this._mod = module
this.doub = module.cwrap('doub', 'number', ['number'])
this.ready = true
})
}
export default mkLib
In the Vue component's data there is a variable for the new class and in watch I wait for a change in the ready property:
data () {
return {
mod: new mkLib,
ready: false
}
},
watch: {
'mod.ready': function () {
this.ready = true
// now this.mod.FUNC(PARAMS) can be used
console.log(this.mod.doub(20))
}
}

Possible to run Headless Chrome/Chromium in a Google Cloud Function?

Is there any way to run Headless Chrome/Chromium in a Google Cloud Function? I understand I can include and run statically compiled binaries in GCF. Can I get a statically compiled version of Chrome that would work for this?
The Node.js 8 runtime for Google Cloud Functions now includes all the necessary OS packages to run Headless Chrome.
Here is a code sample of an HTTP function that returns screenshots:
Main index.js file:
const puppeteer = require('puppeteer');
exports.screenshot = async (req, res) => {
const url = req.query.url;
if (!url) {
return res.send('Please provide URL as GET parameter, for example: ?url=https://example.com');
}
const browser = await puppeteer.launch({
args: ['--no-sandbox']
});
const page = await browser.newPage();
await page.goto(url);
const imageBuffer = await page.screenshot();
await browser.close();
res.set('Content-Type', 'image/png');
res.send(imageBuffer);
}
and package.json
{
"name": "screenshot",
"version": "0.0.1",
"dependencies": {
"puppeteer": "^1.6.2"
}
}
I've just deployed a GCF function running headless Chrome. A couple takeways:
you have to statically compile Chromium and NSS on Debian 8
you have to patch environment variables to point to NSS before launching Chromium
performance is much worse than what you'd get on AWS Lambda (3+ seconds)
For 1, you should be able to find plenty of instructions online.
For 2, the code that I'm using is the following:
static executablePath() {
let bin = path.join(__dirname, '..', 'bin', 'chromium');
let nss = path.join(__dirname, '..', 'bin', 'nss', 'Linux3.16_x86_64_cc_glibc_PTH_64_OPT.OBJ');
if (process.env.PATH === undefined) {
process.env.PATH = path.join(nss, 'bin');
} else if (process.env.PATH.indexOf(nss) === -1) {
process.env.PATH = [path.join(nss, 'bin'), process.env.PATH].join(':');
}
if (process.env.LD_LIBRARY_PATH === undefined) {
process.env.LD_LIBRARY_PATH = path.join(nss, 'lib');
} else if (process.env.LD_LIBRARY_PATH.indexOf(nss) === -1) {
process.env.LD_LIBRARY_PATH = [path.join(nss, 'lib'), process.env.LD_LIBRARY_PATH].join(':');
}
if (fs.existsSync('/tmp/chromium') === true) {
return '/tmp/chromium';
}
return new Promise(
(resolve, reject) => {
try {
fs.chmod(bin, '0755', () => {
fs.symlinkSync(bin, '/tmp/chromium'); return resolve('/tmp/chromium');
});
} catch (error) {
return reject(error);
}
}
);
}
You also need to use a few required arguments when starting Chrome, namely:
--disable-dev-shm-usage
--disable-setuid-sandbox
--no-first-run
--no-sandbox
--no-zygote
--single-process
I hope this helps.
As mentioned in the comment, work is being done on a possible solution to running a headless browser in a cloud function. A directly applicable discussion:"headless chrome & aws lambda" can be read on Google Groups.
The question at. had was can you run headless chrome or chromium in Firebase Cloud Functions... the answer is NO! since the node.js project will not have access any chrome/chromium executables and therefore will fail! (TRUST ME - I've Tried!).
A better solutions is to use the Phantom npm package, which uses PhantomJS under the hood:
https://www.npmjs.com/package/phantom
Docs and info can be found here:
http://amirraminfar.com/phantomjs-node/#/
or
https://github.com/amir20/phantomjs-node
The site i was trying to crawl had implemented screen scraping software, the trick is to wait for the page to load by searching for expected string, or regex match, i.e. i do a regex for a , if you need a regex of any complexity made for you - get in touch at https://AppLogics.uk/ - starting at £5 (GPB).
here is a typescript snippet to make the http or https call:
const phantom = require('phantom');
const instance: any = await phantom.create(['--ignore-ssl-errors=yes', '--load-images=no']);
const page: any = await instance.createPage();
const status = await page.open('https://somewebsite.co.uk/');
const content = await page.property('content');
same again in JavaScript:
const phantom = require('phantom');
const instance = yield phantom.create(['--ignore-ssl-errors=yes', '--load-images=no']);
const page = yield instance.createPage();
const status = yield page.open('https://somewebsite.co.uk/');
const content = yield page.property('content');
Thats the easy bit! if its a static page your pretty much done and you can parse the HTML into something like the cheerio npm package: https://github.com/cheeriojs/cheerio - an implementation of core JQuery designed for servers!
However if it is a dynamically loading page, i.e. lazy loading, or even anti-scraping methods, you will need to wait for the page to update by looping and calling the page.property('content') method and running a text search or regex to see if your page has finished loading.
I have created a generic asynchronous function returning the page content (as a string) on success and throws an exception on failure or timeout. It takes as parameters the variables for the page, text (string to search for that indicates success), error (string to indicate failure or null to not check for error), and timeout (number - self explanatory):
TypeScript:
async function waitForPageToLoadStr(page: any, text: string, error: string, timeout: number): Promise<string> {
const maxTime = timeout ? (new Date()).getTime() + timeout : null;
let html: string = '';
html = await page.property('content');
async function loop(): Promise<string>{
async function checkSuccess(): Promise <boolean> {
html = await page.property('content');
if (!isNullOrUndefined(error) && html.includes(error)) {
throw new Error(`Error string found: ${ error }`);
}
if (maxTime && (new Date()).getTime() >= maxTime) {
throw new Error(`Timed out waiting for string: ${ text }`);
}
return html.includes(text)
}
if (await checkSuccess()){
return html;
} else {
return loop();
}
}
return await loop();
}
JavaScript:
function waitForPageToLoadStr(page, text, error, timeout) {
return __awaiter(this, void 0, void 0, function* () {
const maxTime = timeout ? (new Date()).getTime() + timeout : null;
let html = '';
html = yield page.property('content');
function loop() {
return __awaiter(this, void 0, void 0, function* () {
function checkSuccess() {
return __awaiter(this, void 0, void 0, function* () {
html = yield page.property('content');
if (!isNullOrUndefined(error) && html.includes(error)) {
throw new Error(`Error string found: ${error}`);
}
if (maxTime && (new Date()).getTime() >= maxTime) {
throw new Error(`Timed out waiting for string: ${text}`);
}
return html.includes(text);
});
}
if (yield checkSuccess()) {
return html;
}
else {
return loop();
}
});
}
return yield loop();
});
}
I have personally used this function like this:
TypeScript:
try {
const phantom = require('phantom');
const instance: any = await phantom.create(['--ignore-ssl-errors=yes', '--load-images=no']);
const page: any = await instance.createPage();
const status = await page.open('https://somewebsite.co.uk/');
await waitForPageToLoadStr(page, '<div>Welcome to somewebsite</div>', '<h1>Website under maintenance, try again later</h1>', 1000);
} catch (error) {
console.error(error);
}
JavaScript:
try {
const phantom = require('phantom');
const instance = yield phantom.create(['--ignore-ssl-errors=yes', '--load-images=no']);
const page = yield instance.createPage();
yield page.open('https://vehicleenquiry.service.gov.uk/');
yield waitForPageToLoadStr(page, '<div>Welcome to somewebsite</div>', '<h1>Website under maintenance, try again later</h1>', 1000);
} catch (error) {
console.error(error);
}
Happy crawling!