So here's my code, I'm trying to access the anum and articleHTML inside the page.evaulate scope. It says anum and articleHTML is not defined
const num = context.params.event.content.split(' ').slice(1).join(' ')
let an = parseInt(num)
let anum = an - 1
let articleHTML = an + 1
let website = 'https://newkrunktimes.github.io/'
let browser = await puppeteer.launch();
let page = await browser.newPage()
await page.goto(website, {waitUntil: 'networkidle2'})
let data = await page.evaluate(() => {
articleTitle = document.getElementsByClassName("post-title h4 font-weight-bold")[anum].innerText
author = document.getElementsByClassName("post-supertitle")[anum].innerText
aImage = document.getElementsByClassName("img-fluid")[anum].currentSrc
aDesc = document.getElementsByClassName("post-content font-weight-light")[anum].innerText
artLink = document.querySelector(`a[class="ytlink"][target="_blank"][href="articles/article${articleHTML}.html"]`).href
return {
articleTitle,
author,
aImage,
aDesc,
artLink
}
})
the reason behind this is that whatever is inside page.evaluate() run in the browser's context which is not the context of where anum and articleHTML are defined (the node.js context).
I think you'll find the answer here: How can I pass variable into an evaluate function?
Related
I'm having problems showing an image brought from the database.
-> First I'll show you how this image is being stored:
Here I do the import and send it to the backend
import profile_default_image from "../../../other/imgs/profile_default_image.png";
resRegister = await register(user_name, user_email, user_password, profile_default_image);
profile_default_image generates a type string with value =/static/media/profile_default_image.a9136072d073801df253.png
This value is stored in the MYSQL database in a BLOB type field.
Below is the profileImage value of a row
-> Now I'll show you what's happening on the front
const [profileImage, setProfileImage] = useState("test");
useEffect(() => {
async function getUser() {
if (id) {
const res = await apiFindUserById(id);
const base64Flag = "data:image/jpeg;base64,";
const b64Image = await Buffer.from(res.data.profileImage.data).toString("base64");
setProfileImage(base64Flag + b64Image);
}
}
getUser();
}, []);
res.data returns the following:
Below I try to put the image on the screen
<div>
<img style={{ width: 200, height: 200, }}
src={profileImage}
alt="test"
/>
</div>
But the only result I have is this:
I gave a console.log in profileImage and it returned the value of: data:image/jpeg;base64,L3N0YXRpYy9tZWRpYS9wcm9maWxlX2RlZmF1bHRfaW1hZ2UuYTkxMzYwNzJkMDczODAxZGYyNTMucG5n, which basically a blank image.
I tried another way to do(found it here on stackoverflow)
const [profileImage, setProfileImage] = useState("test");
const arrayBufferToBase64 = (buffer: any) => {
let binary = "";
let bytes = new Uint8Array(buffer);
let len = bytes.byteLength;
for (let i = 0; i < len; i++) {
binary += String.fromCharCode(bytes[i]);
}
return window.btoa(binary);
};
useEffect(() => {
async function getUser() {
if (id) {
const res = await apiFindUserById(id);
const imageStr = await arrayBufferToBase64(res.data.profileImage.data);
const base64Flag = "data:image/jpeg;base64,";
setProfileImage(base64Flag + imageStr);
console.log(profileImage);
}
}
getUser();
}, []);
I got the same result.
The original image is a default user avatar.
It could be that the problem is even when I'm storing this image in the database, can anyone help me? I'm very lost.
Thank you very much!!!
On the front end of my app I wanted to parse some data related to a CSV they upload. Through the file upload tool, I first get a FileList object and then pull the 1 file out of it.
I want to turn it into a json object which I could then iterate. I was thinking to user csv-parser from node, but I dont see a way to leverage a File object stored in memory.
How Can I accomplish this?
At first I was doing:
let f = fileList.item(0);
let decoder = new window.TextDecoder('utf-8');
f.arrayBuffer().then( data => {
let _data = decoder.decode(data)
console.log("Dataset", data, _data)
});
And that was passing the array buffer, and decoding the string. While I Could write a generic tool which process this string data based on \n and ',' I wanted this to be a bit more easier to read.
I wanted to do something like:
let json = csvParser(f)
is there a way to user csv-parser from node, (3.0.0) or is there another tool i should leverage? I was thinking that levering modules based on the browser ( new window.TextDecoder(...) ) is poor form since it has the opportunity to fail.
Is there a tool that does this? im trying to create some sample data and given a File picked from an input type="file" i would want to have this be simple and straight forward.
This example below works, but i feel the window dependancy and a gut feeling makes me think this is naive.
const f : File = fileList.item(0)
console.log("[FOO] File", f)
let decoder = new window.TextDecoder('utf-8');
f.arrayBuffer().then( data => {
let _data = decoder.decode(data)
console.log("Dataset", data, _data)
let lines = _data.split("\n")
let headers = lines[0].split(',')
let results = []
for ( let i = 1; i < lines.length; i++) {
let line = lines[i]
let row = {}
line.split(",").forEach( (item, idx) => {
row[headers[idx]] = item;
})
results.push(row)
}
console.log("JSON ARRAY", results)
})
The issue i run when i stop and do: ng serve is that it does not like using the arrayBuffer function and accessing TextDecoder from window, since that thost functions/classes are not a part of File and window respectively during build.
Any thoughts?
This is what I ended up doing, given the file input being passed into this function:
updateTranscoders(project: Project, fileList: FileList, choice: string = 'replace') {
const f: File = fileList.item(0)
//Reads a File into a string.
function readToString(file) : Promise<any> {
const reader = new FileReader();
const future = new Promise( (resolve,reject) => {
reader.addEventListener("load", () => {
resolve(reader.result);
}, false)
reader.addEventListener("error", (event) => {
console.error("ERROR", event)
reject(event)
}, false)
reader.readAsText(file)
});
return future;
}
readToString(f).then( data => {
let lines = data.split("\n")
let headers = lines[0].split(',')
let results = []
for (let i = 1; i < lines.length; i++) {
let line = lines[i]
let row = {}
line.split(",").forEach((item, idx) => {
row[headers[idx]] = item;
})
results.push(row)
}
if (choice.toLowerCase() === 'replace'){
let rows = project.csvListContents.toJson().rows.filter( row => row.isDeployed)
rows.push( ...results)
project.csvListContents = CsvDataset.fromJson({ rows: rows })
}else if (choice.toLowerCase() === 'append') {
let r = project.csvListContents.toJson();
r.rows.push(...results);
project.csvListContents = CsvDataset.fromJson(r);
}else {
alert("Invalid option for Choice.")
}
this.saveProject(project)
})
}
Now the CHOICE portion of the code is where I have a binary option to do a hard replace on CSV contents or just append to it. I would then save the project accordingly. This is also understanding that the first row contains column headers.
I am trying to insert array in my firebase collection from cloud function. I need to have multiple lines in one document so for each line i am inserting an array. Please check my attached screenshot where you can see line0 , same way i need to have Line1,Line2,Line3..,Line n in the same document.
for line0 i am passing array from code like below and its working fine.
admin.firestore().collection("qbContestWinners").add(
{
'cmpientryid': context.params.processId,
'qbid': '',
'qbsyncdate': '',
'qbsyncstatus': 'pending',
'Line0':
{
id: "0",
description: 'PRIZE AMOUNT',
amount: 1000,
accountrefid: contestresultData.qbcontestid,
accountrefname: contestresultData.qbcontestname,
contestresultId: context.params.processId,
},
})
when i am looping through data i am getting from another table , i am not able to generate proper JSON to insert.
below is how i am looping and creating JSON after getting data from another table.
i = 1;
admin.firestore().collection("results").where('cid', '==', 'LKRRk2XXXXXXXX')
.orderBy("rank", "asc").get().then(snapshots =>
{
snapshots.forEach(doc =>
{
const contestresultId = doc.id;
const prizeAmount = doc.data().prizeamt;
const userId = doc.data().userid;
const lineNum = "Line" + i;
console.log("new line numner is: ", lineNum);
console.log(`lineNum? ${lineNum}`);
const linetxt = "Line" + String(i);
const insertData = "{"+linetxt +
":{id:'" + i +
"', description: 'PRIZE AMOUNT'"+
", amount:" + prizeAmount + "," +
"accountrefid:"+ contestresultData.qbcontestid +","+
"accountrefname:'" +contestresultData.qbcontestname +"',"+
"contestresultId:'" + contestresultId +"'," +
"},}"
const finalInsert = JSON.stringify(insertData);
const finalJSON = JSON.parse(finalInsert);
admin.firestore().collection("qbContestWinners").doc(mainID).set(
finalInsert.toJSON(),
{
merge: true
});
i= i+1;
});
});
using this code i am getting error
finalInsert.toJSON is not a function
Actually, the Line0 field is a map and not an Array, see this doc for more details.
So, if you want to create similar fields (Line1, Line2, ...), you simply need to pass a JavaScript Object to the set() method, as follows:
snapshots.forEach(doc => {
const contestresultId = doc.id;
const prizeAmount = doc.data().prizeamt;
const userId = doc.data().userid;
const lineNum = "Line" + i;
console.log("new line numner is: ", lineNum);
console.log(`lineNum? ${lineNum}`);
const lineObj = {
id: i,
description: 'PRIZE AMOUNT',
accountrefid: contestresultData.qbcontestid, //Not sure if you have defined contestresultData somewhere...
//...
}
const dataObj = {};
dataObj["Line" + i] = lineObj // See https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Operators/Property_accessors
admin.firestore().collection("qbContestWinners").doc(mainID).set(dataObj, {merge: true});
i= i+1;
});
HOWEVER, note that you must return a promise that resolves when all the asynchronous work in your Cloud Function is complete (i.e. call to the Firestore set() method).
This is explained in the official Firebase video series, watch in particular the three videos titled "Learn JavaScript Promises".
Since you are calling several times the set() method in a forEach loop, you need to use Promise.all() in order to return a Promise when all these parallel calls to the set() method are completed.
The following should do the trick:
let i = 1;
return admin.firestore().collection("results") // <-- See the return here
.where('cid', '==', 'LKRRk2XXXXXXXX')
.orderBy("rank", "asc").get()
.then(snapshots => {
const promises = [];
snapshots.forEach(doc => {
const contestresultId = doc.id;
const prizeAmount = doc.data().prizeamt;
const userId = doc.data().userid;
const lineNum = "Line" + i;
const lineObj = {
id: i,
description: 'PRIZE AMOUNT',
accountrefid: contestresultData.qbcontestid,
//...
}
const dataObj = {};
dataObj[lineNum] = lineObj;
promises.push(admin.firestore().collection("qbContestWinners").doc(mainID).set(dataObj, {merge: true}));
i= i+1;
});
return Promise.all(promises) // <-- See the return here
});
A last remark: if mainID keeps the same value in the snapshots.forEach loop, you may adopt a totally different approach, consisting in building a JavaScript object with several LineXX properties and call the set() method only once. Since you didn't share the entire code of your Cloud Function it is impossible to say if this approach should be used or not.
first to the error
You stringify and parse a string. The problem here seems to be the order. You have to parse a "String" and to stringify an "Object". The result won't have a toJSON Method as well, but u can just stringify the Object to get a json.
the second thing
Why do you use a string to create your object? You shouldn't. Just use an object.
the third thing
You should not use Objects as Arrays. Not even in firebase.
Just use arrays. Example:
[Line0Object, Line1Object, ...]
Hint: If your array can work as its own collection. Just use a SubCollection. This might fit your needs.
Why is it that my script will produce the correct results when doing this:
let data = await page.evaluate(async () => {
let multipleVideosUnorderedList = await document
.querySelector('article > div')
.querySelector('ul');
let video = [];
if (multipleVideosUnorderedList != null) {
let multipleVideosList = multipleVideosUnorderedList.children;
console.log(multipleVideosList);
for (i = 0; i < multipleVideosList.length; i++) {
let rightBtn = document.querySelector(
'button > div.coreSpriteRightChevron'
);
if (rightBtn) {
await rightBtn.parentNode.click();
}
let videoUrl = multipleVideosList[i].querySelector('video');
if (videoUrl) {
video.push(videoUrl.getAttribute('src'));
}
}
} else {
video.push(document.querySelector('video').getAttribute('src'));
}
return {
video
};
});
console.log(data);
But when it deduce it down to just this:
let er = await page.evaluate(() => {
let multipleVideosUnorderedList = document.querySelector('article > div').querySelector('ul');
return {
multipleVideosUnorderedList
}
});
console.log(er);
the result is undefined. I know there's a lot more code in the former, but I just wanted to see it produce the correct element before I move on to grabbing everything else.
The idea was to take out the document.querySelector in code block and clean it up, to try to use page.$(selector) instead.
Only serializable objects can go into and out of page.evaluate, a NodeList and a Node, which are found with querySelectorAll/querySelector, are not such things.
You probably would like to find an unordered list wich may contain several videos. If this is the case you could rewrite the code in the following way:
let outerVideos = await page.evaluate(() => {
// convert the NodeList to an array
let videos = [...document.querySelectorAll('article > div video')]
// for each member of the array replace the video node with its src value
.map(video => video.getAttribute('src'));
return videos;
});
console.log(outerVideos);
I've just started using feathers to build REST server. I need your help for querying tips. Document says
When used via REST URLs all query values are strings. Depending on the service the values in params.query might have to be converted to the right type in a before hook. (https://docs.feathersjs.com/api/databases/querying.html)
, which puzzles me. find({query: {value: 1} }) does mean value === "1" not value === 1 ? Here is example client side code which puzzles me:
const feathers = require('#feathersjs/feathers')
const fetch = require('node-fetch')
const restCli = require('#feathersjs/rest-client')
const rest = restCli('http://localhost:8888')
const app = feathers().configure(rest.fetch(fetch))
async function main () {
const Items = app.service('myitems')
await Items.create( {name:'one', value:1} )
//works fine. returns [ { name: 'one', value: 1, id: 0 } ]
console.log(await Items.find({query:{ name:"one" }}))
//wow! no data returned. []
console.log(await Items.find({query:{ value:1 }})) // []
}
main()
Server side code is here:
const express = require('#feathersjs/express')
const feathers = require('#feathersjs/feathers')
const memory = require('feathers-memory')
const app = express(feathers())
.configure(express.rest())
.use(express.json())
.use(express.errorHandler())
.use('myitems', memory())
app.listen(8888)
.on('listening',()=>console.log('listen on 8888'))
I've made hooks, which works all fine but it is too tidious and I think I missed something. Any ideas?
Hook code:
app.service('myitems').hooks({
before: { find: async (context) => {
const value = context.params.query.value
if (value) context.params.query.value = parseInt(value)
return context
}
}
})
This behaviour depends on the database and ORM you are using. Some that have a schema (like feathers-mongoose, feathers-sequelize and feathers-knex), will convert values like that automatically.
Feathers itself does not know about your data format and most adapters (like the feathers-memory you are using here) do a strict comparison so they will have to be converted. The usual way to deal with this is to create some reusable hooks (instead of one for each field) like this:
const queryToNumber = (...fields) => {
return context => {
const { params: { query = {} } } = context;
fields.forEach(field => {
const value = query[field];
if(value) {
query[field] = parseInt(value, 10)
}
});
}
}
app.service('myitems').hooks({
before: {
find: [
queryToNumber('age', 'value')
]
}
});
Or using something like JSON schema e.g. through the validateSchema common hook.