How can I bypass the invoked too many times error? - google-maps

I am trying to use the following script to populate the zip codes for over 3000 cells:
function geo2zip(a) {
var response=Maps.newGeocoder()
.reverseGeocode(lat(a),long(a));
return response.results[0].formatted_address.split(',')
[2].trim().split(' ')[1];
}
function lat(pointa) {
var response = Maps.newGeocoder()
.geocode(pointa);
return response.results[0].geometry.location.lat
}
function long(pointa) {
var response = Maps.newGeocoder()
.geocode(pointa);
return response.results[0].geometry.location.lng
}
Obviously, after about 5 cells I am getting the error that the service has been invoked too many times
Simply, how do I pay (or add to the code) to be able to run the script for the cells I need?

I'm not sure about the delay. You'll have to play with it probably. And I'm also not sure if you'll be allowed to get 3000 operations in one day. But disregarding quotas for the moment, which I wouldn't recommend, you could do something like this.
It looks like this service is on a pay as you go basis so my guess is that it will cost you something. Read the links at the bottom about API use and Billing.
function loopGeocode(start) {
var start=start || 0;
var ss=SpreadsheetApp.getActive();
var sh1=ss.getSheetByName('Points');
var sh2=ss.getSheetByName('Locations');
var pt=sh1.getRange(1, 1,sh.getLastRow(),1).getValues();
for(var i=start;i<pt.length;i++) {
sh2.appendRow([i+1,pt[i][0],geo2zip(pt[i][0])])
Utilities.sleep(8000);//I have no idea what this delay should be
}
return i+1;
}
function geo2zip(a) {
var response=Maps.newGeocoder().reverseGeocode(lat(a),long(a));
return response.results[0].formatted_address.split(',')[2].trim().split(' ')[1];
}
function lat(pointa) {
var response = Maps.newGeocoder().geocode(pointa);
return response.results[0].geometry.location.lat
}
function long(pointa) {
var response = Maps.newGeocoder().geocode(pointa);
return response.results[0].geometry.location.lng
}
I'd run it off of a JavaScript timer and try running once every 6 minutes. I don't know if the withSuccessHandler() will wait that long so you might need to store the return value which tells the loop where to start counting every time. You could probably store in PropertiesService. It will take about 15 hours.
Geocoding API Use and Billing
Geocoding API Getting Started
Javascript Timer Example
After reading some of the links I would believe that 50 queries per second is acceptable so the solution may have nothing to do with adding time between samples. It's probably a matter of setting up a billing account and pay as you go.
Create, Modify, or Close Your Billing Account

Related

Google Sheets error - Service invoked too many times for one day: urlfetch

I have problem with my function in Google Sheets. I am getting every day this error: "Exception: Service invoked too many times for one day: urlfetch." I have about 1000 urls in document. I am looked for solution at google. I find some topics where is recommended to add cache to function but I dont know how to do it. Does somebody have any idea? My function:
function ImportCeny(url, HTMLClass) {
var output = '';
var fetchedUrl = UrlFetchApp.fetch(url, {muteHttpExceptions: true});
if (fetchedUrl) {
var html = fetchedUrl.getContentText();
}
// Grace period to avoid call limit
Utilities.sleep(1000);
var priceposition = html.search(HTMLClass);
return html.slice(priceposition,priceposition + 70).match(/(-\d+|\d+)(,\d+)*(\.\d+)*/g);
}
You may try to add a randomly generated number, for example 6 digits and add this number to the URL as a parameter each time before calling "UrlFetchApp"
i.e.;
url = url & "?t=458796"
You can certainly use Utilities.sleep() to force the program to stop for some time before making the next call. However, using the built-in Cache class (you can see the docs here) is much more suitable as it is specially designed for these scenarios.
So, if you wanted to leave one second you could replace:
Utilities.sleep(1000); //In milliseconds
with
var cache = CacheService.getScriptCache(); //Create a cache instance
cache.put("my_content", html, 1); // In seconds

Duplicate contact creation using appscript despite a functional filter function

Context
A bit of context before we can dive into the code: I am currently working for a non-profit organisation for the protection of cats. I'm not a pro developer, I'm not paid to work on this, but since I'm the only one willing to do it and who knows a bit how to code, I volunteered to write a script for creating and updating adopter and abandoner contacts for our cats.
The other volunteers in the organisation are using Google Sheets to keep track of lots of information about the cats, including their adopters' and abandoners' contact information. On the other hand, the person in charge of the organisation wants to have every adopter or abandoner contact in a specific format in her Google Contacts. Before I wrote the script, volunteers used to enter all the info in the spreadsheet and enter it again in the boss' contacts.
The script is mostly functional and handles the update of contact information as well. However, the script creates duplicate contacts for some people, and I don't really understand why (although I may have a lead). It is a bug which only happens when volunteers use the script, but not when I use it; which makes me think something goes wrong when they call the script...
Code
The script creates an array of Person objects. Every person has a method to return a contactObject version of itself, compatible with Google's People API and I use People API's batchUpdateContacts function to create contacts, by batches of 200 while there are new contacts in the array.
In order to know the contacts already created, I first get the created connections using this function:
/** Helper function to list all connections of the current google user
*
* #returns {PeopleAPI.Person[]} connections - All of the connection objects from Google's People API
*/
/** Helper function to list all connections of the current google user
*
* #returns {PeopleAPI.Person[]} connections - All of the connection objects from Google's People API
*/
function getAllConnections_() {
var connections = [];
var apiResponse;
var nextPageToken;
var firstPass = true;
do {
if (firstPass) {
apiResponse = People.People.Connections.list('people/me', {'personFields': 'memberships,emailAddresses,phoneNumbers,names,addresses,biographies,userDefined'});
firstPass = false;
}
else {
apiResponse = People.People.Connections.list('people/me', {'personFields': 'memberships,emailAddresses,phoneNumbers,names,addresses,biographies,userDefined', 'pageToken': nextPageToken});
}
connections = connections.concat(apiResponse.connections);
nextPageToken = apiResponse.nextPageToken;
} while (nextPageToken);
return connections;
}
Then, I use a filter function to eliminate the already existing contacts based on the contacts email addresses (when a cat is adopted, we always ask for 2 email addresses, so I know there is at least one):
/** Helper function to filter the existing contacts and avoid creating them
*
* #param {Person[]} people - people to filter from
* #param {connections[]} connections - existing contacts in person's address book
* #returns {Person[]} filteredPeople - people who are not in connections
*/
function filterExistingContacts_(people, connections) {
if (!connections[0]) {
return people;
}
return people.filter(function (person) {
for (contact of connections) {
if (!contact.emailAddresses) {continue;}
if (contact.emailAddresses.filter(function (email) {return email.value.toLowerCase().replace(/\s/g, '').includes(person.email)}).length > 0) {return false;}
}
return true;
});
}
In the above code, person.email is lowercased and spaces are replaced by ''. When I run those functions, I can't reproduce the bug, but when the script users do, they get any number from 2 to 74 duplicate contacts created.
Supposition and leads
My supposition is that, maybe, the "getAllConnections_" function gets a bad response from Google's People API, for some reason and thus, gets an incomplete array of existing connections. Then my filter function filters correctly (since I can see no fault in my logic here) the contacts, but some existing contacts are re-created because the script is not aware they already exist.
First idea
If this is so, I think possibly a SQL database could solve the problem (and lower the complexity of the algorithm, which is quite slow with the current ~4000 existing contacts). But I don't really know where I could find a free database (for the organisation would much prefer paying for veterinary care than for this) which would function with Appscript ; plus that would mean a lot of work on the code itself to adapt it... So I would like to know if you guys think it may solve the problem or if I'm completely mistaken before I give it some more hours.
Second idea
Also, I thought about using something like the "ADDED" trick described here: Delete duplicated or multiplied contacts from Google Contacts as a workaround... But the spreadsheet is not structured per contact, but per cat. So it would lead to a problem for a specific situation which is, actually and sadly, quite frequent:
Patrick Shmotherby adopts the cat Smoochie → Smoochie's adopter column is marked as "ADDED" and Patrick's contact is created.
Patrick Shmotherby later abandons Smoochie → Smoochie's abandoner column is marked as "ADDED" and Patrick's contact is updated.
Karen Klupstutsy later adopts Smoochie → Smoochie's adopter column is already marked as "ADDED" so Karen's contact is not created.
A solution could be asking the volunteers to delete the "ADDED" marker manually, yet I think you can understand why this is error-prone when updating lots of contacts on the same day and time-consuming for the volunteers.
Third idea
I thought I might create a function for auto-deleting duplicate contacts from the Google account, but I would prefer not to use this solution as I'm afraid I could delete some important data there, especially as this is the boss' professional, organisational and personal account.
How you could help me
I must say, despite my leads, I'm a bit lost and confused by these duplicates, especially since I can't debug anything because I can't reproduce the bug myself. If you have any fourth lead, I would welcome it warmly.
Also, and because I'm a hobbyist, it's very likely that I didn't do things the correct way, or did not know I could do something else (e.g. I suggested using a SQL database because I know of the existence of relational databases, but maybe there are other common tools I've never heard of). So any suggestion would be good too.
Finally, if you think I'm correct on my own diagnosis, telling me so could help me get the motivation to re-write my code almost entirely if needed. And if you know where I could find a free database usable with Google Appscript (I know quality has a price, so I don't have much hope for this, but we never know) and if it's not "host your own database in you basement", that would be awesome!
Tell me if you need more information, if you want me to put some other piece of code or anything.
Have a nice day/afternoon/evening/night,
Benjamin
Alright so I found where the problem was from, thanks to #OctaviaSima who pointed me to the executions logs.
Apparently, for some reason I don't know, sometimes, my function "getAllConnections_()" which was supposed to get all the contacts in the Google Contacts book failed to get some contacts using this code:
/** Helper function to list all connections of the current google user
*
* #returns {PeopleAPI.Person[]} connections - All of the connection objects from Google's People API
*/
function getAllConnections_() {
var connections = [];
var apiResponse;
var nextPageToken;
var firstPass = true;
do {
if (firstPass) {
apiResponse = People.People.Connections.list('people/me', {'personFields': 'memberships,emailAddresses,phoneNumbers,names,addresses,biographies,userDefined'});
firstPass = false;
}
else {
apiResponse = People.People.Connections.list('people/me', {'personFields': 'memberships,emailAddresses,phoneNumbers,names,addresses,biographies,userDefined', 'pageToken': nextPageToken});
}
connections = connections.concat(apiResponse.connections);
nextPageToken = apiResponse.nextPageToken;
} while (nextPageToken);
return connections;
}
E.g. last execution, the actual contact list was 4061 connections long, however the script only got 4056 connections, which led to 5 duplicate contacts being created.
I added a quick patch by ensuring the connections table was as long as the number of contacts by calling the function recursively if it's not the case.
/** Helper function to list all connections of the current google user
*
* #returns {PeopleAPI.Person[]} connections - All of the connection objects from Google's People API
*/
function getAllConnections_() {
var connections = [];
var apiResponse;
var nextPageToken;
var firstPass = true;
do {
if (firstPass) {
apiResponse = People.People.Connections.list('people/me', {'personFields': 'memberships,emailAddresses,phoneNumbers,names,addresses,biographies,userDefined'});
firstPass = false;
}
else {
apiResponse = People.People.Connections.list('people/me', {'personFields': 'memberships,emailAddresses,phoneNumbers,names,addresses,biographies,userDefined', 'pageToken': nextPageToken});
}
connections = connections.concat(apiResponse.connections);
nextPageToken = apiResponse.nextPageToken;
} while (nextPageToken);
if (connections.length != apiResponse.totalItems) {connections = getAllConnections_();} // Hopefully, the magic lies in this line of code
return connections;
}
Posting this here in case it helps someone else.
Edit: Just corrected the test on the magic line from "==" to "!=".
Here is something I did for use with my email whitelist to ensure I didn't get duplicate emails.
function displayCurrentContacts() {
const ss = SpreadsheetApp.getActive();
const sh = ss.getSheetByName('Contacts');
sh.clearContents();
const vs = [['Name', 'Emails']];
const resp = People.People.Connections.list('people/me', { personFields: "emailAddresses,names,organizations" });
//Logger.log(resp);
const data = JSON.parse(resp);
let m = 0;
let n = 0;
data.connections.forEach((ob1, i) => {
if (ob1.emailAddresses && ob1.emailAddresses.length > 0) {
let emails = [... new Set(ob1.emailAddresses.map(ob2 => ob2.value))];//used set to insure I get unique list
//let emails = ob1.emailAddresses.map(ob2 => ob2.value);
let name;
m += emails.length;
//the following cases are derived from the way that I organize all of my contacts
if (ob1.names && ob1.organizations) {
name = ob1.names[0].displayName + '\n' + ob1.organizations[0].name;
++n;
} else if (ob1.names) {
name = ob1.names[0].displayName;
++n;
} else if (ob1.organizations) {
name = ob1.organizations[0].name;
++n;
}
vs.push([name, emails.sort().join('\n')])
}
});
vs.push([n, m])
sh.getRange(1, 1, vs.length, vs[0].length).setValues(vs)
sh.getRange(2, 1, sh.getLastRow() - 2, sh.getLastColumn()).sort({ column: 1, sortAscending: true });
}
Note this is not meant to be a plugin replacement by any means.

How to reduce the latency between two script calls in Google Apps Script

In a self-developed add-on for Google Sheets, the functionality has been added that a sound file will be played from a JavaScript audio player in the sidebar, depending on the selection in the table. For the code itself see here.
When a line is selected in the table the corresponding sound file is played in the sidebar. Every time the next line is selected it takes around 2 seconds before the script will start to run and load the sound file into the sidebar. As the basic idea of the script is to quickly listen through long lists of sound files, it is crucial to reduce the waiting time as fare as possible.
A reproducible example is accessible here; Add-ons > 'play audio' (Google account necessary). To reproduce the error, the sheet has to be opened two times (e.g. in two browsers).
In order to reduce the latency you might try to reduce interval on your poll function as suggested by Cooper on a comment to the question and to change the getRecord function.
poll
At this time the interval is 2 seconds. Please bear in mind that reducing the interval too much might cause an error and also might have an important impact on the consume of the daily usage quotas. See https://developers.google.com/apps-script/guides/services/quotas
getRecord
Every time it runs it make multiple calls to Google Apps Script which are slow so you should look for a way to reduce the number of Google Apps Script calls. In order to do this you could store the spreadsheet table data in the client side code and only read it again if the data was changed.
NOTE: The Properties Service has a 50,000 daily usage quota for consumer accounts.
One way to quickly implement the above is to limit the getRecord function to read the current cell and add a button to reload the data from the table.
Function taken from the script bounded to the demo spreadsheet linked in the question.
function getRecord() {
var scriptProperties = PropertiesService.getScriptProperties();
var sheet = SpreadsheetApp.getActiveSheet();
var data = sheet.getDataRange().getValues();
var headers = data[0];
var rowNum = sheet.getActiveCell().getRow(); // Get currently selected row
var oldRowNum = scriptProperties.getProperty("selectedRow"); // Get previously selected row
if(rowNum == oldRowNum) { // Check if the was a row selection change
// Function returns the string "unchanged"
return "unchanged";
}
scriptProperties.setProperty("selectedRow", rowNum); // Update row index
if (rowNum > data.length) return [];
var record = [];
for (var col=0;col<headers.length;col++) {
var cellval = data[rowNum-1][col];
if (typeof cellval == "object") {
cellval = Utilities.formatDate(cellval, Session.getScriptTimeZone() , "M/d/yyyy");
}
record.push({ heading: headers[col],cellval:cellval });
}
return record;
}
Related
Problems when using a Google spreadsheet add-on by multiple users

Using Google Apps Script, how do you remove built in Gmail category labels from an email?

I'd like to completely undo any of Gmails built in category labels. This was my attempt.
function removeBuiltInLabels() {
var updatesLabel = GmailApp.getUserLabelByName("updates");
var socialLabel = GmailApp.getUserLabelByName("social");
var forumsLabel = GmailApp.getUserLabelByName("forums");
var promotionsLabel = GmailApp.getUserLabelByName("promotions");
var inboxThreads = GmailApp.search('in:inbox');
for (var i = 0; i < inboxThreads.length; i++) {
updatesLabel.removeFromThreads(inboxThreads[i]);
socialLabel.removeFromThreads(inboxThreads[i]);
forumsLabel.removeFromThreads(inboxThreads[i]);
promotionsLabel.removeFromThreads(inboxThreads[i]);
}
}
However, this throws....
TypeError: Cannot call method "removeFromThreads" of null.
It seems you can't access the built in labels in this way even though you can successfully search for label:updates in the Gmail search box and get the correct results.
The question...
How do you access the built in Gmail Category labels in Google Apps Script and remove them from an email/thread/threads?
Thanks.
'INBOX' and other system labels like 'CATEGORY_SOCIAL' can be removed using Advanced Gmail Service. In the Script Editor, go to Resources -> Advanced Google services and enable the Gmail service.
More details about naming conventions for system labels in Gmail can be found here Gmail API - Managing Labels
Retrieve the threads labeled with 'CATEGORY_SOCIAL' by calling the list() method of the threads collection:
var threads = Gmail.Users.Threads.list("me", {labels: ["CATEGORY_SOCIAL"]});
var threads = threads.threads;
var nextPageToken = threads.nextPageToken;
Note that you are going to need to store the 'nextPageToken' to iterate over the entire collection of threads. See this answer.
When you get all thread ids, you can call the 'modify()' method of the Threads collection on them:
threads.forEach(function(thread){
var resource = {
"addLabelIds": [],
"removeLabelIds":["CATEGORY_SOCIAL"]
};
Gmail.Users.Threads.modify(resource, "me", threadId);
});
If you have lots of threads in your inbox, you may still need to call the 'modify()' method several times and save state between calls.
Anton's answer is great. I marked it as accepted because it lead directly to the version I'm using.
This function lets you define any valid gmail search to isolate messages and enables batch removal labels.
function removeLabelsFromMessages(query, labelsToRemove) {
var foundThreads = Gmail.Users.Threads.list('me', {'q': query}).threads
if (foundThreads) {
foundThreads.forEach(function (thread) {
Gmail.Users.Threads.modify({removeLabelIds: labelsToRemove}, 'me', thread.id);
});
}
}
I call it via the one minute script trigger like this.
function ProcessInbox() {
removeLabelsFromMessages(
'label:updates OR label:social OR label:forums OR label:promotions',
['CATEGORY_UPDATES', 'CATEGORY_SOCIAL', 'CATEGORY_FORUMS', 'CATEGORY_PROMOTIONS']
)
<...other_stuff_to_process...>
}

Need a way to parametize an array sort function, with parameters set in a handler

I'm writing a web app that displays a subset of rows from a spreadsheet worksheet. For the convenience of users, the data is presented in a grid, each row selected from the spreadsheet forms a row in the grid. The number of rows relevant to each user grows over time.
The header of the grid is a set of buttons, which allow the user to control the sort order of the data. So if the user clicks the button in the header of the first column, the array that populates the grid is sorted by the first column and so forth. If you click the same column button twice, the sort order reverses.
I used script properties to communicate the selected sort field and order between the handler responding to the button presses, and the order function called by the sort.
So in doGet():
// Sort Field and Order
ScriptProperties.setProperties({"sortField": "date","sortOrder": "asc"});
And in the button handler:
function sortHandler(event) {
var id = event.parameter.source;
if (id === ScriptProperties.getProperty("sortField")) {
ScriptProperties.setProperty("sortOrder", (ScriptProperties.getProperty("sortOrder") === "asc")?"desc":"asc");
} else {
ScriptProperties.setProperties({"sortField": id, "sortOrder": "asc"});
}
var app = UiApp.getActiveApplication();
app.remove(app.getElementById("ScrollPanel"));
createForm_(app);
return app;
}
And the order function itself (this is invoked by a sort method on the array: array.sort(order); in the code that defines the grid):
function orderFunction(a, b) {
var sortParameter = ScriptProperties.getProperties();
var asc = sortParameter.sortOrder === "asc";
switch(sortParameter.sortField) {
case "date":
var aDate = new Date(a.date);
var bDate = new Date(b.date);
return (asc)?(aDate - bDate):(bDate - aDate);
case "serviceno":
return (asc)?(a.serviceno-b.serviceno):(b.serviceno-a.serviceno);
default: // lexical
var aLex = String(a[sortParameter.sortField]).toLowerCase();
var bLex = String(b[sortParameter.sortField]).toLowerCase();
if (aLex < bLex) {
return (asc)?-1:1;
} else if (aLex > bLex) {
return (asc)?1:-1;
} else {
return 0;
}
}
}
The fly in the ointment with this design is Google. Once the array gets to a certain size, the sort fails with an error that the Properties service is being called too frequently. The message suggests inserting a 1s delay using Utilities.sleep(), but the grid already takes a long time to render already - how is it going to go if the array takes 1s to decide the order of two values?
I tried reimplementing with ScriptDB, but that suffers the same problem, calls to ScriptDB service are made too frequently for Google's liking.
So how else can I implement this, without the orderFunction accessing any App Script services?
If you are looking for temporary storage, I prefer CacheService to ScriptProperties.
Use
CacheService.getPrivateCache().put and
CacheService.getPrivateCache().get
You could also use a hidden widget to pass information between the doGet an handler functions, it will also be much faster than calling the scriptProperties service (have a look at the execution transcript to see how long it takes, you'll be surprised)
On the other hand if you really want to keep using it you can also try to (slightly) reduce the number of calls to the scriptProperties service, for example in your sortHandler(event) function :
function sortHandler(event) {
var id = event.parameter.source;
var sortField = ScriptProperties.getProperty("sortField");
var sortOrder = ScriptProperties.getProperty("sortOrder");
if (id === sortField) {
ScriptProperties.setProperty("sortOrder", (sortOrder === "asc")?"desc":"asc");
} else {
ScriptProperties.setProperties({"sortField": id, "sortOrder": "asc"});
}
var app = UiApp.getActiveApplication();
app.remove(app.getElementById("ScrollPanel"));
createForm_(app);
return app;
}
...how else can I implement this?
Use HTMLService with jQuery DataTable which does all the ordering that you can imagine and a whole lot more without scripting button press logic and the like. There is an example of the spreadsheet to basic table here http://davethinkingaloud.blogspot.co.nz/2013/03/jsonp-and-google-apps-script.html