How to serialize form data of iframe with Apify - puppeteer

Ok,So I try to serialize a form data, but I get undefined
Here is code:
const frame = page.frames().find(frame => frame.url().includes('reservation'));
const aHandle = await frame.evaluateHandle('document');
const form= await aHandle.$eval('#reservationData', element => element.outerHTML);
var theform = await serialize(form);
console.log(theform);

I don't know how the function serialize looks like. Can you provide code of this function?
But you can do it easily with frame.$eval(selector, pageFunction[, ...args]) and jQuery.
const Apify = require('apify');
...
await Apify.utils.puppeteer.injectJQuery(page);
const frame = page.frames().find(frame => frame.url().includes('reservation'));
const theForm = frame.$eval('#reservationData', (form) => $(form).serialize())
console.log(theform);

Related

How To Return An object representing the queryObjects Handle

Consider the following function
const countObjects = async (page: any) => {
const prototypeHandle = await page.evaluateHandle(() => Object.prototype);
const objectsHandle = await page.queryObjects(prototypeHandle);
const numberOfObjects = await page.evaluate((instances: any) => instances.length, objectsHandle);
await Promise.all([
prototypeHandle.dispose(),
objectsHandle.dispose()
]);
return numberOfObjects;
};
export default countObjects;
That i got from an idea here: https://addyosmani.com/blog/puppeteer-recipes/#measuring-memory-leaks
This works rather well but it only returns numbers, i'd like to make it return the entire page object so i can compare it to the previous only and find a diff.
But the problem im having is
a. i cant debug
const objects = await page.evaluate((instances: any) => {
debugger; // wont stop
return instances.length
}, objectsHandle);
b. the object returns as empty....
const objects = await page.evaluate((instances: any) => instance, objectsHandle);
console.log(object); // empty
Does anyone have any ideas how i could get some info on the diff back i.e. info / names on whats leaking. Also an example usage of the test.
it('should not have memory leaks', async () => {
const numberOfObjects = await countObjects(page);
await page.evaluate(() => {
const onMessage = () => { /* ... */ };
window.addEventListener('message', onMessage);
});
expect(await countObjects(page)).toEqual(numberOfObjects);
});
Also trying to look in chrome dev tools doesnt bring anything obvious up so hoping the name helps from the object.

How to download Csv from google trends using javascript apify puppeteer

i use that code to download csv from google trends using apify js, but it doesn't work, could you help me?
the result is a csv with a wrong content.
i try to get all csv from google trends.
const path = require('path');
const downloadPath1 = path.resolve(__dirname, './downloads');
const fs = require('fs');
try{
const util = require('util');
await util.promisify(fs.mkdir)(downloadPath1);
}
catch( e){
}
await page.setRequestInterception(true);
//await page.click(csvSelector);
await page.waitForSelector(csvSelector)
const hrefElement = await page.$(csvSelector);
await hrefElement.click();
const xRequest = await new Promise(resolve => {
page.on('request', async interceptedRequest => {
const type = interceptedRequest.resourceType();
log.info(type)
if ( type == "xhr" ) {interceptedRequest.abort();
resolve(interceptedRequest);}
else interceptedRequest.continue();
});
});
log.info(xRequest._url);
const request = require('request-promise');
const options = {
encoding: null,
method: xRequest._method,
uri: xRequest._url,
body: xRequest._postData,
headers: xRequest._headers
}
/* add the cookies */
const cookies = await page.cookies();
options.headers.Cookie = cookies.map(ck => ck.name + '=' + ck.value).join(';');
const response = await request(options);
fs.writeFileSync(downloadPath1+'/binary.csv', response);
const fileObjs = fs.readdirSync(downloadPath1, { withFileTypes: true });
console.log("\nCurrent directory files:");
fileObjs.forEach(file => {
console.log(file);
});
// There won't be more files so let's pick the first
const fileData = fs.readFileSync(downloadPath1+`/${fileObjs[0].name}`);
log.info(fileData);
// Now we can use the data or save it into Key-value store.
await Apify.setValue('MY-Csv.csv', fileData, { contentType: 'application/csv'});
i hope that someone can propose a solution for this thanks

Cannot read property 'getAllProducts' of undefined

The problem is in db.js because it's trying to load something from db which does not exist.
In my index.js page:
const dataB = require("./db").getAllProducts;
app.get('/scrapper', async (req, res) => {
const Myobjects = await dataB.getAllProducts();
res.send(Myobjects)
})
And in my db.js page:
async function getAllProducts() {
const connection = await getConnection();
const pageRepo = connection.getRepository(Crawlers);
const pages = await pageRepo.find();
connection.close();
return pages;
}
async function InsertScrappedData(texte, image, price){
const connection = await getConnection();
const page = new Crawler();
page.texte = texte;
page.image = image;
page.price = price;
const crawlertrepo=connection.getRepository(Crawlers);
const res=await crawlertrepo.save(Crawlers);
Console.log('saved',res);
const Allpages = await crawlertrepo.find();
connection.close();
return Allpages;
}
Exporting my functions
module.exports = [
getAllProducts,
InsertScrappedData
]
module.exports is an object type, not an array, so you need to use curly brackets when assigning it.
module.exports = {
getAllProducts,
InsertScrappedData
}
Relevant documentation
This is actually a condensed form of assigning the getAllProducts function to the getAllProducts key of a new object and then assigning that object to module.exports
// Equivalent but unnecessarily verbose
module.exports = {
getAllProducts: getAllProducts,
InsertScrappedData: InsertScrappedData
}

How can I map the complete json data into my code? Right now I can only do it for one entry at a time

export default class FetchData extends React.Component{
state = {
loading: true,
currentPrice: null,
oneDayChange: null,
sevenDayChange: null
};
//these are the three values i need to get from the json result
async componentDidMount(){
const url = 'https://pro-api.coinmarketcap.com/v1/cryptocurrency/listings/latest?CMC_PRO_API_KEY=xxxx'
const response = await fetch(url);
const info = await response.json();
this.setState({currentPrice:info.data[5].quote.USD.price})
this.setState({oneDayChange:info.data[5].quote.USD.percent_change_24h});
this.setState({sevenDayChange:info.data[5].quote.USD.percent_change_7d});
console.log(info.data[5].id);
}
//can only update states by giving array index of 1 entry
This might help
async componentDidMount(){
....
const info = await response.json();
const currentPrice = [];
const oneDayChange = [];
const sevenDayChange = [];
info.data.map(item => {
currentPrice.push(item.quote.USD.price);
oneDayChange.push(item.quote.USD.percent_change_24h);
sevenDayChange.push(item.quote.USD.ercent_change_7d);
});
this.setState({currentPrice, oneDayChange, sevenDayChange});
}
Multiple state changes might not be working as you might be trying to modify state object directly, below solution will replace state with new object copying all the old values and replacing mentioned properties.
This should work
async componentDidMount(){
const url = 'https://pro-api.coinmarketcap.com/v1/cryptocurrency/listings/latest?CMC_PRO_API_KEY=xxxx'
const response = await fetch(url);
const [loading, newArray]
const info = await response.json();
const infoUSD = info.data[5].quote.USD;
this.setState(Object.assign({}, state , {currentPrice: infoUSD.price, oneDayChange: infoUSD.percent_change_24h, sevenDayChange: infoUSD.percent_change_7d});
console.log(info.data[5].id);
}

Can Cypress expose function to global object?

I learned that puppeteer can add a function on the page's window object like this:
const browser = await puppeteer.launch();
const page = await browser.newPage();
await page.exposeFunction('md5', text =>
crypto.createHash('md5').update(text).digest('hex')
);
await page.evaluate(async () => {
// use window.md5 to compute hashes
const myString = 'PUPPETEER';
const myHash = await window.md5(myString);
console.log(`md5 of ${myString} is ${myHash}`);
});
So I'm wondering is there any way that Cypress can attach functions to window function like puppeteer does?