how to use nextPageToken - google-apps-script

I have a script that archives old classrooms, until the end of 2021 it was working fine.
In the lasts months I got an error (the script works ok, but terminate with error) and today I was investigating it, the script runs only once per month.
The error is due to a supposed change in .nextPageToken function.
var parametri = {"courseStates": "ARCHIVED"};
var page = Classroom.Courses.list(parametri);
var listaClassi = page.courses;
var xyz = page.nextPageToken;
if (page.nextPageToken !== '') {
parametri.pageToken = page.nextPageToken;
page = Classroom.Courses.list(parametri);
listaClassi = listaClassi.concat(page.courses);
};
var xyz has been added to better understand what was happening.
So, in this case the list does not have pagination, is only one page. var xyz returns "undefined", and the "if" statement results "true", this makes that variable listaClassi got appended the same content a second time. That generate the error and the abnormal end of the script.
I found an issue reported here https://issuetracker.google.com/issues/225941023?pli=1 that may be related with my problem.
Now I could change .nextPageToken with .getNextPageToken but I found no docs on the second function and many issues reporting that is not working, can anyone help me?

When using the nextPageToken value obtained to the response make sure to enter it as a separate parameter with a slightly different name. You will obtain nextPageToken in the response, the pageToken parameter needs to be entered in the request. It does look like you are doing it right, the way you add the parameter is a bit odd, yet it should be functional.
To discard problems with the Classroom API (that we can certainly take a look at) try with this simple code example in a new Google Apps Script project, remember you will need to add an Advanced service, information about advanced services can be found in this documentation article https://developers.google.com/apps-script/guides/services/advanced. Use listFiles as the main method in your Apps Script project.
function listFiles() {
var totalClasses = 0;
nextPageToken = "";
console.log("Found the following classes:")
do {
var response = loadPage(nextPageToken);
var classes = response.courses;
for (let x in classes){
console.log("Class ID: " + classes[x].id + " named: '" + classes[x].name + "'.");
}
totalClasses += classes.length;
} while (nextPageToken = response.nextPageToken)
console.log("There are " + totalClasses + " classes.")
}
function loadPage(token = ""){
return Classroom.Courses.list({
fields: 'nextPageToken,courses(id,name)',
pageSize: 10,
pageToken: token
});
}
When we first make the API call with Apps Script we don't specify a pageToken, since it is the first run we don't have one. All calls to the List method may return a nextPageToken value if the returned page contains an incomplete response.
while (nextPageToken = response.nextPageToken)
In my code at the line above once response.nextPageToken is empty (not in the response) inside the condition block JavaScript will return false, breaking the loop and allowing the code to finish execution.
To have your incident reviewed by a Google Workspace technician you can also submit a form to open a ticket with the Google Workspace API Support team at https://support.google.com/a/contact/wsdev.

Related

Google Sheets error - Service invoked too many times for one day: urlfetch

I have problem with my function in Google Sheets. I am getting every day this error: "Exception: Service invoked too many times for one day: urlfetch." I have about 1000 urls in document. I am looked for solution at google. I find some topics where is recommended to add cache to function but I dont know how to do it. Does somebody have any idea? My function:
function ImportCeny(url, HTMLClass) {
var output = '';
var fetchedUrl = UrlFetchApp.fetch(url, {muteHttpExceptions: true});
if (fetchedUrl) {
var html = fetchedUrl.getContentText();
}
// Grace period to avoid call limit
Utilities.sleep(1000);
var priceposition = html.search(HTMLClass);
return html.slice(priceposition,priceposition + 70).match(/(-\d+|\d+)(,\d+)*(\.\d+)*/g);
}
You may try to add a randomly generated number, for example 6 digits and add this number to the URL as a parameter each time before calling "UrlFetchApp"
i.e.;
url = url & "?t=458796"
You can certainly use Utilities.sleep() to force the program to stop for some time before making the next call. However, using the built-in Cache class (you can see the docs here) is much more suitable as it is specially designed for these scenarios.
So, if you wanted to leave one second you could replace:
Utilities.sleep(1000); //In milliseconds
with
var cache = CacheService.getScriptCache(); //Create a cache instance
cache.put("my_content", html, 1); // In seconds

Delay in between two calls (POST and GET) in Google Apps script

My first request on StackOverflow! I really hope you can help me!
I want to create a process to automatize a data report from a system to a sheet; I thought I could use their API, apps script, and export the data on google sheets.
To do so, I need to run two calls on the API:
A POST call, which runs the report within the system (it requires a date range as body).
In return, I will get an ID that is associated with the data generated and it expires after some time.
A GET call, which is a URL that contains the ID generated in the first call and created with a concatenation.
The first call works fine; I get in return the ID successfully.
My problem is when I run the second call, I don’t get any data in return, and I don’t understand what’s the issue, I can see the URL is concatenated correctly because if I copy the URL from the log and I test it on another apps script or on Postman, it works perfectly fine!
Could someone help me in case I am doing something wrong?
Here’s the code:
function callEvents() {
var API_KEY = "xxx";
var data = { 'Start Date': '2021-05-03', 'End Date': '2021-06-03' }
var options = {
'method' : 'post',
'contentType': 'application/json',
'payload' : JSON.stringify(data) };
//This is the first call
var urlEncoded = encodeURI('https://website/api/dataviewresult/
etc/json/?api_key=' + API_KEY);
var url = UrlFetchApp.fetch(urlEncoded, options);
var result = JSON.parse(url.getContentText());
Logger.log(url.getContentText());
//here I retrieve the ID to use in the second call
var ipdataview = (result["contents"]["id"]);
Logger.log(ipdataview);
//here is the concatenation and the second call
var urlEncoded2 = encodeURI('https://website/api/dataviewresult/etc/json/'+ipdataview+'/?api_key=' + API_KEY);
Logger.log(urlEncoded2);
var response = UrlFetchApp.fetch(urlEncoded2);
Logger.log(response.getContentText());
I found out the solution, I was doing the second call (Get) too soon from the first one, so the data was not being picked up.
I used utilities.sleep() for about 2 seconds and it worked perfectly.
Thank you for everyone helping!
S

"We're sorry, a server error occurred. Please wait a bit and try again" while creating charts

We are running into sporadic errors (more often than not) on a project that generates Google Doc documents based on info entered into Google Sheet spreadsheets.
The Google Apps Script project pulls data (from sheets), caches it in-memory with standard var statements, massages the data a bit (formatting, de-duping, etc), then copies a Doc template and does a bunch of token substitution and also inserts some charts based on the in-memory variables.
We are encountering the following error during a high quantity of executions:
Exception: We're sorry, a server error occurred. Please wait a bit and try again.
at insertChart(Code:5890:10)
at processForm(Code:4540:5)
The main method creates several (between 5 - 20 or more) charts depending on parameters entered at run-time by the user (which happens via a standard Web Form created by GAS:
function doGet() {
return HtmlService
.createTemplateFromFile('index')
.evaluate();
}
The web page returned by the above is a very standard GAS HTMLService webapp. It contains a form that allows a user to select a few standard types of reporting criteria (date range, filter by certain types of data, etc).
We are duplicating a "template" Docs file which has a bunch of common text and some tokens that we use as placeholders to find and replace with data loaded from the Sheets. The Sheets are very common spreadsheets with the kind of data you'd imagine: date ranges, selections from drop-down lists, free-text columns, etc. (The filtering info specified by the user is used to query the Sheets for the data we want to "merge" into the new Doc. It's basically a much more complex version of your standard "mail merge".)
var templateId = 'asdfasdfasdfasdfasdfasdfasdfasdfasdf'; // DEV
var documentId = DriveApp.getFileById(templateId).makeCopy().getId();
DriveApp.getFileById(documentId).setName('Demo Report - Project Report');
DriveApp.getFileById(documentId).setSharing(DriveApp.Access.ANYONE_WITH_LINK, DriveApp.Permission.COMMENT);
Logger.log('templateId: ' + templateId);
Logger.log('documentId: ' + documentId);
var doc = DocumentApp.openById(documentId);
var body = doc.getBody();
We have the following method which takes the Doc body, placeholder, and formatted data from a chart builder. It then identifies the Doc element token to replace (simply a string in the Doc file), and runs a body.insertImage after that token element.
function insertChart(body, placeholder, chart) {
if (placeholder.getParent().getType() === DocumentApp.ElementType.BODY_SECTION && typeof chart !== 'undefined' && chart !== null) {
var offset = body.getChildIndex(placeholder);
//Logger.log(chart);
body.insertImage(offset + 1, chart);
}
}
We have several helper methods like the following buildStackedColumnChart(). It's meant to wrap the Chart API commands to use a dataTable and Chart Builder to return a specific type of chart (bar, stacked bar, line, etc).
Its resulting chart gets passed into the insertChart() method above.
function buildStackedColumnChart(title, header, data) {
Logger.log('buildStackedColumnChart');
Logger.log('title: ' + title);
Logger.log('header: ' + header);
Logger.log('data: ' + data);
try {
var dataTable = Charts.newDataTable();
for (var hr = 0; hr < header.length; hr++) {
var headerRow = header[hr];
dataTable.addColumn(headerRow[0], headerRow[1]);
}
for (var r = 0; r < data.length; r++) {
var row = data[r];
dataTable.addRow(row);
}
dataTable.build();
var chart = Charts.newColumnChart()
.setDataTable(dataTable)
.setDimensions(600, 500)
.setTitle(title)
.setStacked()
.build();
return chart;
} catch (ex) {
Logger.log('ex: ' + ex);
return null;
}
}
In the main processForm() method (called directly by the webapp menioned above when the user selects their criteria and clicks a "Generate Report" button. We have a few calls to a method which finds the token in the template file (just text with {{}} around it as shown below). It iterates through a hard-coded list of values in the in-memory data variables storing accumulations from the Sheets and creates chartHeader and chartData (which contains the values to be charted in a mechanism the helper methods above can translate back into calls that make sense for the Chart API) and uses the insertChart() and `` helper methods to insert the chart after the bookmark token and then remove the bookmark from the document (cleanup so the tokens aren't present in the end report Doc).
var chartTrainingsSummaryBookmark = findPlaceholder(body, '{{CHART_TRAININGS_SUMMARY}}');
Logger.log('chartTrainingsSummaryBookmark: ' + chartTrainingsSummaryBookmark);
var chartCategories = [
'Call',
'Conference Attendee',
'Meeting (External)',
'Partnership',
'Quarterly Board Meeting',
'Alliance or Workgroup',
'Training (Others)',
'Training (Professional Development)',
'Youth Training',
'Webinar or Zoom',
'Other',
'Coalition',
'Email',
'Face-to-face',
'Phone',
'Site Visit',
];
// {Call=10.0, Conference Attendee=10.0, Quarterly Board Meeting=10.0, Alliance or Workgroup=10.0, Coalition=10.0, Meeting (External)=10.0}
var chartHeaderData = [];
var chartData = [];
var monthChartHeader = [[Charts.ColumnType.STRING, 'Month']];
var monthChartData = [monthName];
for (var cci = 0; cci < chartCategories.length; cci++) {
var chartCategory = monthName + ':' + chartCategories[cci];
if (getChartData(chartCategory, trainingsChartData) > 0) {
monthChartHeader.push([Charts.ColumnType.NUMBER, chartCategory.split(':')[1]]);
monthChartData.push(getChartData(chartCategory, trainingsChartData));
}
}
if (monthChartData.length > 1) {
if (chartHeaderData.length < 1) {
chartHeaderData = chartHeaderData.concat(monthChartHeader);
}
chartData.push(monthChartData);
}
Logger.log('----- CHART - TRAININGS: SUMMARY -----');
if (chartData.length > 0 && chartData[0].length > 1) {
insertChart(body, chartTrainingsSummaryBookmark, buildStackedColumnChart('', chartHeaderData, chartData));
}
chartTrainingsSummaryBookmark.removeFromParent();
We are also seeing slight variations that also error out the document generation with a Timeout. (This process can take a while if there is a lot of data that fits within the parameters which the user specified.)
We were wondering why these errors would happen on a seemingly random interval. We can literally just click the button that makes the call again and it might work, or might error out on a different chart, etc.
This script has been working for the most part (we did encounter a few specific errors) in months previous; however, this "Exception: We're sorry, a server error occurred. Please wait a bit and try again." started occuring last weekend and got far worse on Monday / Tuesday (nearly every execution failed, very few succeeded). It's not happening quite as badly today; but we are seeing a few errors.
Also, it's a bit strange, but copying the GAS Project and executing the copy has thrown the error less frequently than the original project in production.
We have some ideas for workarounds; bu, ideally, we would like to identify a root cause so we can correctly fix the issue.

Split Gmail thread and label by date in a google script

Hy,
I have a server sending me several log mails by day and I want to automaticly label this mails.
I can't touch the server configuration to adapt the mail subject, so the work must be done by "receiver".
The Subject is still same so gmail merge them in a thread of 100, but I want to split them by date. So One Date, one thread. In addition, I want label them whith a nested label: "Server1" -> "Date"
I've only found a way to add label to the thread in globality and no way to split them.
Is it even possible?
After a new look on my issue, perhaps add the date at the message subject can split threads.
Like:
function AddLogSubjectADate() {
var threads = GmailApp.search('from:sender#server.com has:nouserlabels');
threads.forEach(function(messages){
messages.getMessages().forEach(function(msg){
var date = msg.getDate();
var date_of_mail = Utilities.formatDate(date, "GMT+1", "yyyy/MM/dd")
var subj = msg.getSubject()
var newsubj = subj + date_of_mail
//A way to modify subject
});
});
}
But I didn't find a way to change the subject.
Post Scriptum
I don't think it's relevant, but here is my previous work. but it add label to the thread. Like I said I haven't find a way to split threads.
function AddLogLabelbyDate() {
var today = new Date();
var tomorrow = new Date();
var yesterday = new Date();
tomorrow.setDate(today.getDate()+1);
yesterday.setDate(today.getDate()-1);
var date_today = Utilities.formatDate(today, "GMT+1", "yyyy/MM/dd")
var date_tomorrow = Utilities.formatDate(tomorrow, "GMT+1", "yyyy/MM/dd")
var date_yesterday = Utilities.formatDate(yesterday, "GMT+1", "yyyy/MM/dd")
var threads = GmailApp.search('from:sender#server.com has:nouserlabels before:'+ date_tomorrow +' after:'+ date_yesterday +'');
label.addToThreads(threads);
}
Per the API documentation, Gmail follows some rules about thread grouping:
In order to be part of a thread, a message or draft must meet the following criteria:1. The requested threadId must be specified on the Message or Draft.Message you supply with your request.2. The References and In-Reply-To headers must be set in compliance with the RFC 2822 standard.3. The Subject headers must match.
So, you can prevent the automatic grouping into a given conversation thread by modifying any of those 3 parameters.
Alternately, you can apply per-message conversation labels, though this will not really help you if you use "Conversation View" UI.
Both of these methods require the use of the Gmail REST API, for which Apps Script provides an "advanced service" client library. The native GmailApp does not provide a method for per-message thread alteration, or for manipulating messages in the manner needed.
Thread Separation
If you wanted to disable the conversation grouping, in theory you could do this:
Message#get to obtain a full message representation
Modify one of the properties Gmail uses to perform thread grouping
Message#insert or import to create the new message on the server
Message#delete to remove the original
Message#get to get the inserted message metadata, after Gmail has given it a threadId.
Get the remaining messages that should share that new threadId, modify them appropriately, and insert.
Repeat.
I haven't tested that approach, hence my "in theory" comment.
Per-message labeling
The relevant API methods include Gmail.User.Labels.list, Gmail.User.Messages.list, Gmail.User.Messages.modify, and Gmail.User.Messages.batchModify. You'll probably want to use the list and messages.batchModify methods, since you seem to have a large number of messages for which you'd like to make alterations. Note, there are non-trivial rate limits in place, so working in small batches is liable to be most resource-efficient.
This is likely to be the simplest method to implement, since you don't have to actually create or delete messages - just search for messages that should have a given label, add (or create and add) it to them, and remove any non-desired labels. To start you off, here are some minimal examples that show how to work with the Gmail REST API. I expect you will need to refer to the API documentation when you use this information to construct your actual script.
An example Labels#list:
function getLabelsWithName(labelName) {
const search = Gmail.Users.Labels.list("me");
if (!search.labels || !search.labels.length)
return [];
const matches = search.labels.filter(function (label) {
// Return true to include the label, false to omit it.
return label.name === labelName;
});
return matches;
}
An example Messages#list:
function getPartialMessagesWithLabel(labelResource) {
const options = {
labelIds: [ labelResource.id ],
fields: "nextPageToken,messages(id,threadId,labelIds,internalDate)"
};
const results = [];
// Messages#list is paginated, so we must page through them to obtain all results.
do {
var search = Gmail.Users.Messages.list("me", options);
options.pageToken = search.nextPageToken;
if (search.messages && search.messages.length)
Array.prototype.push.apply(results, search.messages);
} while (options.pageToken);
return results;
}
An example Messages#batchModify:
function batchAddLabels(messageArray, labels) {
if (!messageArray || !messageArray.length || !messageArray[0].id)
throw new Error("Missing array of messages to update");
if (!labels || !labels.length || !labels[0].id)
throw new Error("Missing array of label resources to add to the given messages");
const requestMetaData = {
"addLabelIds": labels.map(function (label) { return label.id; }),
"ids": messageArray.map(function (msg) { return msg.id; }) // max 1000 per request!
};
Gmail.Users.Messages.batchModify(requestMetaData, "me");
}
Additional Resources:
Message Searches
"fields" parameter

Newly created Google Task omits the supplied "TaskLink" property

I am trying to make a small Google Script that would automatically add Google Tasks to the "My List" TaskList after searching my GMail emails.
Everything goes fine except for adding a link to the email from which the Task is generated from. Trying to follow the API documentation doesn't really help.
This is the code for the actual task generator function:
function addTask(taskListId, myTitle, myEmailLink) {
var task = Tasks.newTask(); // effectively same as "= {}".
task.title = myTitle
task.notes = 'blank';
task.links = [{}]
task.links[0].description = 'Link to corresponding email';
task.links[0].type = 'email';
task.links[0].link = 'myEmailLink';
task = Tasks.Tasks.insert(task, taskListId);
}
Any ideas why the task I receive back has no links?
As others have noted, according to the Google Tasks API Documentation the links collection is unfortunately read-only.
As a potential work around, it appears you can add links to the notes section of a task, and the links are then directly clickable from the tasks pane in GMail.
Picture: Task with clickable link
Your function can be modified to put the link in the notes section as follows:
function addTask(taskListId, myTitle, myEmailLink) {
var task = Tasks.newTask(); // effectively same as "= {}".
task.title = myTitle
task.notes = 'link: ' + myEmailLink;
task = Tasks.Tasks.insert(task, taskListId);
}
Combining this with the getPermalink() function on the GmailApp threads object allows for grabbing a deep link to the email you are looking for.
Picture: Task with permalink to email
I'm working on a set of scripts that do some of the things you're talking about in addition to a few other things: https://github.com/tedsteinmann/gmailAutoUpdate
In my solution I have a function that grabs the GMail threads with a particular label (in my case #Task) and then creates a task setting the subject to thread.getFirstMessageSubject() and the notes to thread.getPermalink()
The entire function looks like this:
function processPending_() {
var label_pending = GmailApp.getUserLabelByName(LABEL_PENDING);
var label_done = GmailApp.getUserLabelByName(LABEL_DONE);
// The threads currently assigned to the 'pending' label
var threads = label_pending.getThreads();
// Process each one in turn, assuming there's only a single
// message in each thread
for (var t in threads) {
var thread = threads[t];
// Grab the task data
var taskTitle = thread.getFirstMessageSubject();
var taskNote = 'Email: ' + thread.getPermalink();
// Insert the task
addTask_(taskTitle, taskNote, getTasklistId_(TASKLIST));
// Set to 'done' by exchanging labels
thread.removeLabel(label_pending);
thread.addLabel(label_done);
}
// Increment the processed tasks count
Logger.log('Processed %s tasks', threads.length);
}
Per the Google Tasks API Documentation:
links[] list
Collection of links. This collection is read-only.
You cannot set these links by modifying a Task resource, i.e your code
task.links = [{}]
task.links[0].description = 'Link to corresponding email';
task.links[0].type = 'email';
task.links[0].link = 'myEmailLink';
is simply ignored by the server.
TaskLinks are, to my knowledge, unusable and non-configurable outside of the Googleplex. They may as well not exist to API users.
The only way I've been able to generate a Task that has one is by using the Gmail UI and selecting "Add to Tasks". The resulting task then includes this snippet in the last line of the Task item: