I am trying to write a script to access the specific values that are displayed in the following line graph:
Ideally, I want to get the last value (43) from the line graph. For that I tried to run the following sample code:
function W5(){
var url ='https://trends.google.com/trends/explore?geo=US&q=%2Fm%2F07s_c';
var result = UrlFetchApp.fetch(url);
const $ = Cheerio.load(result.getContentText());
Logger.log($);
return;
}
But it is giving me the following error message, due to which I am unable to parse any data and process it. It's strange because I have not made a single successful request even once but message says too many requests:
Exception: Request failed for https://trends.google.com returned code
429. Truncated server response: Error 429 (Too Many Req...
(use muteHttpExceptions option to examine full response)
I followed this solution by changing the parameters according to my requirement:
function startQuery() {
var sheet = ss.getSheetByName("Sheet2");
// start the query
var result = buildQueryString(sheet.getRange("A2").getValue(),
sheet.getRange("B2").getValue());
// display the resulting link in a cell
sheet.getRange("C2").setValue(result);
var csv_result = generateCsvDownload(sheet.getRange("A2").getValue(),
sheet.getRange("B2").getValue());
sheet.getRange("D2").setValue(csv_result);
}
function buildQueryString(geo,q) {
return "http://www.google.com/trends/explore?" +
"geo=" + encodeURIComponent(geo) +
"&q=" + encodeURIComponent(q)
}
function generateCsvDownload(geo,q) {
return "http://www.google.com/trends/viz?" +
"geo=" + encodeURIComponent(geo) +
"&q=" + encodeURIComponent(q) +
"&graph=all_csv";
}
This script takes parameters from the sheet table (starting from Column A):
geo
q
Query_Result
CSV_Download_Link
csv_result
US
uneployement
But it just creates a link instead of actual CSV data output. Any help with that would be much appreciated. Thank you
Related
I have a script that archives old classrooms, until the end of 2021 it was working fine.
In the lasts months I got an error (the script works ok, but terminate with error) and today I was investigating it, the script runs only once per month.
The error is due to a supposed change in .nextPageToken function.
var parametri = {"courseStates": "ARCHIVED"};
var page = Classroom.Courses.list(parametri);
var listaClassi = page.courses;
var xyz = page.nextPageToken;
if (page.nextPageToken !== '') {
parametri.pageToken = page.nextPageToken;
page = Classroom.Courses.list(parametri);
listaClassi = listaClassi.concat(page.courses);
};
var xyz has been added to better understand what was happening.
So, in this case the list does not have pagination, is only one page. var xyz returns "undefined", and the "if" statement results "true", this makes that variable listaClassi got appended the same content a second time. That generate the error and the abnormal end of the script.
I found an issue reported here https://issuetracker.google.com/issues/225941023?pli=1 that may be related with my problem.
Now I could change .nextPageToken with .getNextPageToken but I found no docs on the second function and many issues reporting that is not working, can anyone help me?
When using the nextPageToken value obtained to the response make sure to enter it as a separate parameter with a slightly different name. You will obtain nextPageToken in the response, the pageToken parameter needs to be entered in the request. It does look like you are doing it right, the way you add the parameter is a bit odd, yet it should be functional.
To discard problems with the Classroom API (that we can certainly take a look at) try with this simple code example in a new Google Apps Script project, remember you will need to add an Advanced service, information about advanced services can be found in this documentation article https://developers.google.com/apps-script/guides/services/advanced. Use listFiles as the main method in your Apps Script project.
function listFiles() {
var totalClasses = 0;
nextPageToken = "";
console.log("Found the following classes:")
do {
var response = loadPage(nextPageToken);
var classes = response.courses;
for (let x in classes){
console.log("Class ID: " + classes[x].id + " named: '" + classes[x].name + "'.");
}
totalClasses += classes.length;
} while (nextPageToken = response.nextPageToken)
console.log("There are " + totalClasses + " classes.")
}
function loadPage(token = ""){
return Classroom.Courses.list({
fields: 'nextPageToken,courses(id,name)',
pageSize: 10,
pageToken: token
});
}
When we first make the API call with Apps Script we don't specify a pageToken, since it is the first run we don't have one. All calls to the List method may return a nextPageToken value if the returned page contains an incomplete response.
while (nextPageToken = response.nextPageToken)
In my code at the line above once response.nextPageToken is empty (not in the response) inside the condition block JavaScript will return false, breaking the loop and allowing the code to finish execution.
To have your incident reviewed by a Google Workspace technician you can also submit a form to open a ticket with the Google Workspace API Support team at https://support.google.com/a/contact/wsdev.
We are running into sporadic errors (more often than not) on a project that generates Google Doc documents based on info entered into Google Sheet spreadsheets.
The Google Apps Script project pulls data (from sheets), caches it in-memory with standard var statements, massages the data a bit (formatting, de-duping, etc), then copies a Doc template and does a bunch of token substitution and also inserts some charts based on the in-memory variables.
We are encountering the following error during a high quantity of executions:
Exception: We're sorry, a server error occurred. Please wait a bit and try again.
at insertChart(Code:5890:10)
at processForm(Code:4540:5)
The main method creates several (between 5 - 20 or more) charts depending on parameters entered at run-time by the user (which happens via a standard Web Form created by GAS:
function doGet() {
return HtmlService
.createTemplateFromFile('index')
.evaluate();
}
The web page returned by the above is a very standard GAS HTMLService webapp. It contains a form that allows a user to select a few standard types of reporting criteria (date range, filter by certain types of data, etc).
We are duplicating a "template" Docs file which has a bunch of common text and some tokens that we use as placeholders to find and replace with data loaded from the Sheets. The Sheets are very common spreadsheets with the kind of data you'd imagine: date ranges, selections from drop-down lists, free-text columns, etc. (The filtering info specified by the user is used to query the Sheets for the data we want to "merge" into the new Doc. It's basically a much more complex version of your standard "mail merge".)
var templateId = 'asdfasdfasdfasdfasdfasdfasdfasdfasdf'; // DEV
var documentId = DriveApp.getFileById(templateId).makeCopy().getId();
DriveApp.getFileById(documentId).setName('Demo Report - Project Report');
DriveApp.getFileById(documentId).setSharing(DriveApp.Access.ANYONE_WITH_LINK, DriveApp.Permission.COMMENT);
Logger.log('templateId: ' + templateId);
Logger.log('documentId: ' + documentId);
var doc = DocumentApp.openById(documentId);
var body = doc.getBody();
We have the following method which takes the Doc body, placeholder, and formatted data from a chart builder. It then identifies the Doc element token to replace (simply a string in the Doc file), and runs a body.insertImage after that token element.
function insertChart(body, placeholder, chart) {
if (placeholder.getParent().getType() === DocumentApp.ElementType.BODY_SECTION && typeof chart !== 'undefined' && chart !== null) {
var offset = body.getChildIndex(placeholder);
//Logger.log(chart);
body.insertImage(offset + 1, chart);
}
}
We have several helper methods like the following buildStackedColumnChart(). It's meant to wrap the Chart API commands to use a dataTable and Chart Builder to return a specific type of chart (bar, stacked bar, line, etc).
Its resulting chart gets passed into the insertChart() method above.
function buildStackedColumnChart(title, header, data) {
Logger.log('buildStackedColumnChart');
Logger.log('title: ' + title);
Logger.log('header: ' + header);
Logger.log('data: ' + data);
try {
var dataTable = Charts.newDataTable();
for (var hr = 0; hr < header.length; hr++) {
var headerRow = header[hr];
dataTable.addColumn(headerRow[0], headerRow[1]);
}
for (var r = 0; r < data.length; r++) {
var row = data[r];
dataTable.addRow(row);
}
dataTable.build();
var chart = Charts.newColumnChart()
.setDataTable(dataTable)
.setDimensions(600, 500)
.setTitle(title)
.setStacked()
.build();
return chart;
} catch (ex) {
Logger.log('ex: ' + ex);
return null;
}
}
In the main processForm() method (called directly by the webapp menioned above when the user selects their criteria and clicks a "Generate Report" button. We have a few calls to a method which finds the token in the template file (just text with {{}} around it as shown below). It iterates through a hard-coded list of values in the in-memory data variables storing accumulations from the Sheets and creates chartHeader and chartData (which contains the values to be charted in a mechanism the helper methods above can translate back into calls that make sense for the Chart API) and uses the insertChart() and `` helper methods to insert the chart after the bookmark token and then remove the bookmark from the document (cleanup so the tokens aren't present in the end report Doc).
var chartTrainingsSummaryBookmark = findPlaceholder(body, '{{CHART_TRAININGS_SUMMARY}}');
Logger.log('chartTrainingsSummaryBookmark: ' + chartTrainingsSummaryBookmark);
var chartCategories = [
'Call',
'Conference Attendee',
'Meeting (External)',
'Partnership',
'Quarterly Board Meeting',
'Alliance or Workgroup',
'Training (Others)',
'Training (Professional Development)',
'Youth Training',
'Webinar or Zoom',
'Other',
'Coalition',
'Email',
'Face-to-face',
'Phone',
'Site Visit',
];
// {Call=10.0, Conference Attendee=10.0, Quarterly Board Meeting=10.0, Alliance or Workgroup=10.0, Coalition=10.0, Meeting (External)=10.0}
var chartHeaderData = [];
var chartData = [];
var monthChartHeader = [[Charts.ColumnType.STRING, 'Month']];
var monthChartData = [monthName];
for (var cci = 0; cci < chartCategories.length; cci++) {
var chartCategory = monthName + ':' + chartCategories[cci];
if (getChartData(chartCategory, trainingsChartData) > 0) {
monthChartHeader.push([Charts.ColumnType.NUMBER, chartCategory.split(':')[1]]);
monthChartData.push(getChartData(chartCategory, trainingsChartData));
}
}
if (monthChartData.length > 1) {
if (chartHeaderData.length < 1) {
chartHeaderData = chartHeaderData.concat(monthChartHeader);
}
chartData.push(monthChartData);
}
Logger.log('----- CHART - TRAININGS: SUMMARY -----');
if (chartData.length > 0 && chartData[0].length > 1) {
insertChart(body, chartTrainingsSummaryBookmark, buildStackedColumnChart('', chartHeaderData, chartData));
}
chartTrainingsSummaryBookmark.removeFromParent();
We are also seeing slight variations that also error out the document generation with a Timeout. (This process can take a while if there is a lot of data that fits within the parameters which the user specified.)
We were wondering why these errors would happen on a seemingly random interval. We can literally just click the button that makes the call again and it might work, or might error out on a different chart, etc.
This script has been working for the most part (we did encounter a few specific errors) in months previous; however, this "Exception: We're sorry, a server error occurred. Please wait a bit and try again." started occuring last weekend and got far worse on Monday / Tuesday (nearly every execution failed, very few succeeded). It's not happening quite as badly today; but we are seeing a few errors.
Also, it's a bit strange, but copying the GAS Project and executing the copy has thrown the error less frequently than the original project in production.
We have some ideas for workarounds; bu, ideally, we would like to identify a root cause so we can correctly fix the issue.
To set the scene, here is the request I'm making via Google Apps Script...
function getHvdcData() {
// call the API
var response = UrlFetchApp.fetch(dDomain + epHvdc, dParams);
var json = JSON.parse(response.getContentText());
Logger.log(json);
// define array location in response
var dataLevel = json['data']['hvdcFlow'];
for (var i = 0; i <= dataLevel.length; i++) {
var input = [];
input.push(dataLevel[i].date);
input.push(dataLevel[i].period);
input.push(dataLevel[i].flow);
input.push(dataLevel[i].direction);
input.push(now = new Date());
shtHvdc.appendRow(input);
}
}
This is an example of what comes back in the response...
{
resultCode:200,
message:"Success",
generated:"2018-11-07T14:00:46",
expires:"2018-11-07T14:05:46",
data:{
hvdcFlow:[
{
date:"2018-11-07T00:00:00",
period:27,
flow:387.446,
direction:"North"
}
]
}
}
I'm hoping someone can help me understand why I'm getting this message
TypeError: Cannot read property "date" from undefined. at getHvdcData(demo_site:79)
Line 79 in my code singles out an attribute of a JSON response, which I'm then adding to an array which gets pushed to a google sheet (input.push(dataLevel[i].date);). I realise that I don't have lines in my code that specifically defined the attribute(s) but thought that was part of the JSON.parse function. The script is still able to read .date and add the associated value to var input = []; and insert it into the google sheet.
If the error is generated because I don't specifically declare what .date is, is there a way that I can turn off validation for this part of the code? Given that the script still inserts the data as expected, I wouldn't normally be concerned however I have also created a time based trigger for this to run. Currently, my executions are sitting at 100% Failed because of this and I'd prefer the Trigger Execution page only showed where my script failed to run.
Any help or advice is greatly appreciated.
Given the following Google Apps Script code snippet:
function myFunction() {
var files = DriveApp.getFolderById('xxx').searchFiles('123')
while (files.hasNext()) {
var file = files.next()
Logger.log(file.getName())
}
}
I'm getting:
Invalid argument: q (line x, file "Code")
from the line with while (files.hasNext()) statement.
The line doesn't contain any error I'm aware of. The funny thing is when I exchange .searchFiles('123') with .getFiles(), the error doesn't appear and the code executes.
Does it mean that there's a problem with the object that .searchFiles('123') returns (it should return FileIterator object)? Unfortunately, Javascript only checks the return data type at runtime so I cannot really see if it's correct until I run it.
Per the function documentation of searchFiles, the expected argument is a query string that conforms to the syntax described in the Google Drive API's "Search for Files " guide
Gets a collection of all files in the user's Drive that match the given search criteria. The search criteria are detailed in the Google Drive SDK documentation. Note that the params argument is a query string that may contain string values, so take care to escape quotation marks correctly (for example "title contains 'Gulliver\\'s Travels'" or 'title contains "Gulliver\'s Travels"').
Note that Drive v2 uses title for the filename, and Drive v3 uses name for the filename. Apps Script's advanced service client library Drive uses v2, so this may be the syntax used by the Drive Service (DriveApp) as well.
The reason the error message does not appear on the line with searchFiles is because the return value is a FileIterator, rather than the actual results. This (and most) iterators are lazily-evaluated, and thus your search query is not actually executed until you call files.hasNext().
To resolve this issue, you must indicate what should be searched and how, i.e. provide the Field and the Operator, not just the Value.
function foo() {
const queries = ["fullText contains '123'",
"mimeType='" + MimeType.GOOGLE_SHEETS + "'"];
const matches = queries.map(function (qs) {
return getMatchingFiles_(qs);
});
// Log the results in the Apps Script logger. (Use console.log for Stackdriver)
matches.forEach(function (queryResult, i) {
Logger.log("Results for query #" + i
+ ", \"" + queries[i] + "\" are:");
queryResult.forEach(function (file) {
Logger.log(file.getId() + ": '" + file.getName() + "'");
});
});
}
function getMatchingFiles_(query, folderId) {
// Default to DriveApp.searchFiles if no folder ID given.
folder = folderId ? DriveApp.getFolderById(folderId) : DriveApp;
const search = folder.searchFiles(query);
const results = [];
while (search.hasNext())
results.push(search.next());
return results;
}
I am converting MS office files to google files (gsheet, gdoc, mostly). I got my script working well enough, but google can't convert some files for whatever reason. When this happens, the convert times out, I get an error message in the script editor window, and the entire script exits (including functions that called this function). Here is the error:
We're sorry, a server error occurred. Please wait a bit and try again.
Try, catch doesn't work here because this kind of error doesn't seem to get caught.
I was thinking of implementing a timer so the function is terminated if it takes longer than 5 seconds or so, but I can't seem to find any way to make that happen.
Here is my code:
function convertFile(oldFile, newMimeType, safeKeepingFolder){
try{
var name = oldFile.getName().split(".")[0];
var ID = oldFile.getId();
var folderID = Drive.Files.get(ID).parents[0].id;
var oldBlob = oldFile.getBlob();
var newFile = {
title: name,
mimeType: newMimeType,
parents: [{id: folderID}],
}
//This is where it times out:
Drive.Files.insert(newFile, oldBlob, {
convert: true});
moveFile(oldFile, safeKeepingFolder);
}
catch(e) {
Utils.errorLog(e, errorSsId, "convertFile");
console.error('convertFile yielded an error: ' + e);
}
}