Getting different results when using Puppeteer page.evaluate() - puppeteer

Why is it that my script will produce the correct results when doing this:
let data = await page.evaluate(async () => {
let multipleVideosUnorderedList = await document
.querySelector('article > div')
.querySelector('ul');
let video = [];
if (multipleVideosUnorderedList != null) {
let multipleVideosList = multipleVideosUnorderedList.children;
console.log(multipleVideosList);
for (i = 0; i < multipleVideosList.length; i++) {
let rightBtn = document.querySelector(
'button > div.coreSpriteRightChevron'
);
if (rightBtn) {
await rightBtn.parentNode.click();
}
let videoUrl = multipleVideosList[i].querySelector('video');
if (videoUrl) {
video.push(videoUrl.getAttribute('src'));
}
}
} else {
video.push(document.querySelector('video').getAttribute('src'));
}
return {
video
};
});
console.log(data);
But when it deduce it down to just this:
let er = await page.evaluate(() => {
let multipleVideosUnorderedList = document.querySelector('article > div').querySelector('ul');
return {
multipleVideosUnorderedList
}
});
console.log(er);
the result is undefined. I know there's a lot more code in the former, but I just wanted to see it produce the correct element before I move on to grabbing everything else.
The idea was to take out the document.querySelector in code block and clean it up, to try to use page.$(selector) instead.

Only serializable objects can go into and out of page.evaluate, a NodeList and a Node, which are found with querySelectorAll/querySelector, are not such things.
You probably would like to find an unordered list wich may contain several videos. If this is the case you could rewrite the code in the following way:
let outerVideos = await page.evaluate(() => {
// convert the NodeList to an array
let videos = [...document.querySelectorAll('article > div video')]
// for each member of the array replace the video node with its src value
.map(video => video.getAttribute('src'));
return videos;
});
console.log(outerVideos);

Related

How To Put A Number In Front Of Every Suggestion Corretcly?

Detail Of The Problem
As title, I am using Google App Script and Google Docs API's Batchupdate, trying to put number in front of every suggestion. However, I can place it correctly at the very first one, but it starts to deviate after the first one.
Result I Currently Have
Please refer to the image below.
What I have Tried
Below is the snippet I currently have
function markNumberInFrontOfMark(fileID) {
fileID = "MYFILEID";
let doc = Docs.Documents.get(fileID);
let requests = doc.body.content.flatMap(content => {
if (content.paragraph) {
let elements = content.paragraph.elements;
return elements.flatMap(element => element.textRun.suggestedDeletionIds ? {
insertText: {
text: "(1)",
location: {
index: element.startIndex
}
}
} : []);
}
return [];
});
Docs.Documents.batchUpdate({requests}, fileID);
return true;
}
Result I Want To Have
Please refer to the image below
Post I Refer to
How to change the text based on suggestions through GAS and Google DOC API
Here is an example of how to insert text. In this case I am adding 3 characters "(1)" for example. If the number of additions exceeds 9 you will have to adjust the number of characters added.
function markNumberInFrontOfMark() {
try {
let doc = DocumentApp.getActiveDocument();
let id = doc.getId();
doc = Docs.Documents.get(id);
let contents = doc.body.content;
let requests = [];
let num = 0;
contents.forEach( content => {
if( content.paragraph ) {
let elements = content.paragraph.elements;
elements.forEach( element => {
if( element.textRun.suggestedDeletionIds ) {
num++;
let text = "("+num+")"
let request = { insertText: { text, location: { index: element.startIndex+3*(num-1) } } };
requests.push(request);
}
}
);
}
}
);
if( requests.length > 0 ) {
Docs.Documents.batchUpdate({requests}, id);
}
}
catch(err) {
console.log(err)
}
}
And the resulting updated document.

CSV File object to JSON in Angular

On the front end of my app I wanted to parse some data related to a CSV they upload. Through the file upload tool, I first get a FileList object and then pull the 1 file out of it.
I want to turn it into a json object which I could then iterate. I was thinking to user csv-parser from node, but I dont see a way to leverage a File object stored in memory.
How Can I accomplish this?
At first I was doing:
let f = fileList.item(0);
let decoder = new window.TextDecoder('utf-8');
f.arrayBuffer().then( data => {
let _data = decoder.decode(data)
console.log("Dataset", data, _data)
});
And that was passing the array buffer, and decoding the string. While I Could write a generic tool which process this string data based on \n and ',' I wanted this to be a bit more easier to read.
I wanted to do something like:
let json = csvParser(f)
is there a way to user csv-parser from node, (3.0.0) or is there another tool i should leverage? I was thinking that levering modules based on the browser ( new window.TextDecoder(...) ) is poor form since it has the opportunity to fail.
Is there a tool that does this? im trying to create some sample data and given a File picked from an input type="file" i would want to have this be simple and straight forward.
This example below works, but i feel the window dependancy and a gut feeling makes me think this is naive.
const f : File = fileList.item(0)
console.log("[FOO] File", f)
let decoder = new window.TextDecoder('utf-8');
f.arrayBuffer().then( data => {
let _data = decoder.decode(data)
console.log("Dataset", data, _data)
let lines = _data.split("\n")
let headers = lines[0].split(',')
let results = []
for ( let i = 1; i < lines.length; i++) {
let line = lines[i]
let row = {}
line.split(",").forEach( (item, idx) => {
row[headers[idx]] = item;
})
results.push(row)
}
console.log("JSON ARRAY", results)
})
The issue i run when i stop and do: ng serve is that it does not like using the arrayBuffer function and accessing TextDecoder from window, since that thost functions/classes are not a part of File and window respectively during build.
Any thoughts?
This is what I ended up doing, given the file input being passed into this function:
updateTranscoders(project: Project, fileList: FileList, choice: string = 'replace') {
const f: File = fileList.item(0)
//Reads a File into a string.
function readToString(file) : Promise<any> {
const reader = new FileReader();
const future = new Promise( (resolve,reject) => {
reader.addEventListener("load", () => {
resolve(reader.result);
}, false)
reader.addEventListener("error", (event) => {
console.error("ERROR", event)
reject(event)
}, false)
reader.readAsText(file)
});
return future;
}
readToString(f).then( data => {
let lines = data.split("\n")
let headers = lines[0].split(',')
let results = []
for (let i = 1; i < lines.length; i++) {
let line = lines[i]
let row = {}
line.split(",").forEach((item, idx) => {
row[headers[idx]] = item;
})
results.push(row)
}
if (choice.toLowerCase() === 'replace'){
let rows = project.csvListContents.toJson().rows.filter( row => row.isDeployed)
rows.push( ...results)
project.csvListContents = CsvDataset.fromJson({ rows: rows })
}else if (choice.toLowerCase() === 'append') {
let r = project.csvListContents.toJson();
r.rows.push(...results);
project.csvListContents = CsvDataset.fromJson(r);
}else {
alert("Invalid option for Choice.")
}
this.saveProject(project)
})
}
Now the CHOICE portion of the code is where I have a binary option to do a hard replace on CSV contents or just append to it. I would then save the project accordingly. This is also understanding that the first row contains column headers.

How to access href of a tag in node.js

I have this function
function getxPath(data, path) {
console.log("HEREEEEEEEEEEEEEEEEEEEEEEE");
console.log(path);
//console.log(data);
try {
let root = new dom().parseFromString(data);
console.log(root);
let results = xpath.select(path, root);
console.log(results);
if (results.length > 0) {
let _results = [];
for (let r of results) {
_results.push(r.textContent);
}
return _results;
}
} catch (exc) {
console.log(exc);
}
return null;
}
Path is : //a[#class='classifiedTitle']/#href
I am trying to get links of all the ads on this page:
https://www.sahibinden.com/en/cars/used?date=1day&a5_min=2005&a5_max=2020
But it returns null even though the xpath seems correct.
Edit:
I used full xpath and it gave me the title of a. But when I use even #href it doesnt give
Edit:
I am able to get links using this xpath :
/html/body/div[4]/div[4]/form/div/div[3]/table/tbody/tr//a[1]/#href

Access a nested JSON object property via a single string

This line: let X = this.appGlobal.GetNavigationLanguage().data;
retuns JSON as you can see below.
I want to take NAV.REPORTS.BMAIL.TITLE.
Translate code (NAV.REPORTS.BMAIL.TITLE) is dynamically created.
X.NAV.REPORTS.BMAIL.TITLE => works
X['NAV']['REPORTS']['BMAIL']['TITLE'] => works
But keep in mind I have dynamically created translation code I need something like this:
let transCode = 'NAV.REPORTS.BMAIL.TITLE';
console.log(X[transCode]);
How I can achieve this?
test_data = {
NAV: {
REPORTS: {
BMAIL: {
TITLE: "hello"
}
}
}
}
let transCode = 'NAV.REPORTS.BMAIL.TITLE';
properties = transCode.split('.'); //--> ["NAV","REPORTS","BMAIL","TITLE"]
result = test_data
properties.forEach(function(property) {
result = result[property]
})
console.log(result) // --> hello
The short and evil route would be the following:
console.log(eval(`X.${transCode}`));
The less evil way is to use a recursive function call, this means you only look into the number of items in your string-path (rather than looping the whole collection).
const X = {
NAV: {
REPORTS: {
BMAIL: {
TITLE: 'Test'
}
}
}
}
const transCode = 'NAV.REPORTS.BMAIL.TITLE';
// Evil...
console.log(eval(`X.${transCode}`)); // Test
// Less Evil (but needs exception handling)...
function getData(input: any, splitPath: string[]) {
const level = splitPath.pop();
if (splitPath.length === 0) {
return input[level];
} else {
return getData(input[level], splitPath);
}
}
const result = getData(X, transCode.split('.').reverse());
console.log(result); // Test

How to wait to finish subscribe before moving to next index in for loop in Angular 6

I'm using Angular 6.
I have an array of links and a variable to store fetched information in same order as of array one by one.
Here is what I'm trying to do using for loop.
products: any;
processedItems: Array<any> = [];
private _processItem() {
for (let i = 0; i < this.products.length; i++) {
this.scraperService.scrapSingle(this.products[i].url).subscribe(
res => {
if (res.status.http_code === 200) {
const properties = this.scraperService.processSingleProduct(res.contents);
const p_item = {};
p_item['info'] = this.products[i];
p_item['properties'] = properties;
this.processedItems.push(p_item);
}
console.log(res);
}
);
}
console.log(this.products.length);
}
But how to wait for subscribe before moving to next index in the loop?
Just splice the p_item into your array at the required index given i.
For example instead of doing,
this.processedItems.push(p_item);
do this,
this.processedItems.splice(p_item, 0, i);
That solves your problem :)
Use promises instead of rx.js subscriptions via using toPromise method. You might need to map the res to json. res.map(item => item.json());
products: any;
processedItems: Array < any > =[];
private _processItem() {
this.products.array.forEach(async (element) => {
const res = await this.scraperService.scrapSingle(element.url).toPromise();
if (res.status.http_code === 200) {
const properties = this.scraperService.processSingleProduct(res.contents);
const p_item = {};
p_item['info'] = element
p_item['properties'] = properties;
this.processedItems.push(p_item);
}
console.log(res);
});
console.log(this.products.length);
}