Context
A bit of context before we can dive into the code: I am currently working for a non-profit organisation for the protection of cats. I'm not a pro developer, I'm not paid to work on this, but since I'm the only one willing to do it and who knows a bit how to code, I volunteered to write a script for creating and updating adopter and abandoner contacts for our cats.
The other volunteers in the organisation are using Google Sheets to keep track of lots of information about the cats, including their adopters' and abandoners' contact information. On the other hand, the person in charge of the organisation wants to have every adopter or abandoner contact in a specific format in her Google Contacts. Before I wrote the script, volunteers used to enter all the info in the spreadsheet and enter it again in the boss' contacts.
The script is mostly functional and handles the update of contact information as well. However, the script creates duplicate contacts for some people, and I don't really understand why (although I may have a lead). It is a bug which only happens when volunteers use the script, but not when I use it; which makes me think something goes wrong when they call the script...
Code
The script creates an array of Person objects. Every person has a method to return a contactObject version of itself, compatible with Google's People API and I use People API's batchUpdateContacts function to create contacts, by batches of 200 while there are new contacts in the array.
In order to know the contacts already created, I first get the created connections using this function:
/** Helper function to list all connections of the current google user
*
* #returns {PeopleAPI.Person[]} connections - All of the connection objects from Google's People API
*/
/** Helper function to list all connections of the current google user
*
* #returns {PeopleAPI.Person[]} connections - All of the connection objects from Google's People API
*/
function getAllConnections_() {
var connections = [];
var apiResponse;
var nextPageToken;
var firstPass = true;
do {
if (firstPass) {
apiResponse = People.People.Connections.list('people/me', {'personFields': 'memberships,emailAddresses,phoneNumbers,names,addresses,biographies,userDefined'});
firstPass = false;
}
else {
apiResponse = People.People.Connections.list('people/me', {'personFields': 'memberships,emailAddresses,phoneNumbers,names,addresses,biographies,userDefined', 'pageToken': nextPageToken});
}
connections = connections.concat(apiResponse.connections);
nextPageToken = apiResponse.nextPageToken;
} while (nextPageToken);
return connections;
}
Then, I use a filter function to eliminate the already existing contacts based on the contacts email addresses (when a cat is adopted, we always ask for 2 email addresses, so I know there is at least one):
/** Helper function to filter the existing contacts and avoid creating them
*
* #param {Person[]} people - people to filter from
* #param {connections[]} connections - existing contacts in person's address book
* #returns {Person[]} filteredPeople - people who are not in connections
*/
function filterExistingContacts_(people, connections) {
if (!connections[0]) {
return people;
}
return people.filter(function (person) {
for (contact of connections) {
if (!contact.emailAddresses) {continue;}
if (contact.emailAddresses.filter(function (email) {return email.value.toLowerCase().replace(/\s/g, '').includes(person.email)}).length > 0) {return false;}
}
return true;
});
}
In the above code, person.email is lowercased and spaces are replaced by ''. When I run those functions, I can't reproduce the bug, but when the script users do, they get any number from 2 to 74 duplicate contacts created.
Supposition and leads
My supposition is that, maybe, the "getAllConnections_" function gets a bad response from Google's People API, for some reason and thus, gets an incomplete array of existing connections. Then my filter function filters correctly (since I can see no fault in my logic here) the contacts, but some existing contacts are re-created because the script is not aware they already exist.
First idea
If this is so, I think possibly a SQL database could solve the problem (and lower the complexity of the algorithm, which is quite slow with the current ~4000 existing contacts). But I don't really know where I could find a free database (for the organisation would much prefer paying for veterinary care than for this) which would function with Appscript ; plus that would mean a lot of work on the code itself to adapt it... So I would like to know if you guys think it may solve the problem or if I'm completely mistaken before I give it some more hours.
Second idea
Also, I thought about using something like the "ADDED" trick described here: Delete duplicated or multiplied contacts from Google Contacts as a workaround... But the spreadsheet is not structured per contact, but per cat. So it would lead to a problem for a specific situation which is, actually and sadly, quite frequent:
Patrick Shmotherby adopts the cat Smoochie → Smoochie's adopter column is marked as "ADDED" and Patrick's contact is created.
Patrick Shmotherby later abandons Smoochie → Smoochie's abandoner column is marked as "ADDED" and Patrick's contact is updated.
Karen Klupstutsy later adopts Smoochie → Smoochie's adopter column is already marked as "ADDED" so Karen's contact is not created.
A solution could be asking the volunteers to delete the "ADDED" marker manually, yet I think you can understand why this is error-prone when updating lots of contacts on the same day and time-consuming for the volunteers.
Third idea
I thought I might create a function for auto-deleting duplicate contacts from the Google account, but I would prefer not to use this solution as I'm afraid I could delete some important data there, especially as this is the boss' professional, organisational and personal account.
How you could help me
I must say, despite my leads, I'm a bit lost and confused by these duplicates, especially since I can't debug anything because I can't reproduce the bug myself. If you have any fourth lead, I would welcome it warmly.
Also, and because I'm a hobbyist, it's very likely that I didn't do things the correct way, or did not know I could do something else (e.g. I suggested using a SQL database because I know of the existence of relational databases, but maybe there are other common tools I've never heard of). So any suggestion would be good too.
Finally, if you think I'm correct on my own diagnosis, telling me so could help me get the motivation to re-write my code almost entirely if needed. And if you know where I could find a free database usable with Google Appscript (I know quality has a price, so I don't have much hope for this, but we never know) and if it's not "host your own database in you basement", that would be awesome!
Tell me if you need more information, if you want me to put some other piece of code or anything.
Have a nice day/afternoon/evening/night,
Benjamin
Alright so I found where the problem was from, thanks to #OctaviaSima who pointed me to the executions logs.
Apparently, for some reason I don't know, sometimes, my function "getAllConnections_()" which was supposed to get all the contacts in the Google Contacts book failed to get some contacts using this code:
/** Helper function to list all connections of the current google user
*
* #returns {PeopleAPI.Person[]} connections - All of the connection objects from Google's People API
*/
function getAllConnections_() {
var connections = [];
var apiResponse;
var nextPageToken;
var firstPass = true;
do {
if (firstPass) {
apiResponse = People.People.Connections.list('people/me', {'personFields': 'memberships,emailAddresses,phoneNumbers,names,addresses,biographies,userDefined'});
firstPass = false;
}
else {
apiResponse = People.People.Connections.list('people/me', {'personFields': 'memberships,emailAddresses,phoneNumbers,names,addresses,biographies,userDefined', 'pageToken': nextPageToken});
}
connections = connections.concat(apiResponse.connections);
nextPageToken = apiResponse.nextPageToken;
} while (nextPageToken);
return connections;
}
E.g. last execution, the actual contact list was 4061 connections long, however the script only got 4056 connections, which led to 5 duplicate contacts being created.
I added a quick patch by ensuring the connections table was as long as the number of contacts by calling the function recursively if it's not the case.
/** Helper function to list all connections of the current google user
*
* #returns {PeopleAPI.Person[]} connections - All of the connection objects from Google's People API
*/
function getAllConnections_() {
var connections = [];
var apiResponse;
var nextPageToken;
var firstPass = true;
do {
if (firstPass) {
apiResponse = People.People.Connections.list('people/me', {'personFields': 'memberships,emailAddresses,phoneNumbers,names,addresses,biographies,userDefined'});
firstPass = false;
}
else {
apiResponse = People.People.Connections.list('people/me', {'personFields': 'memberships,emailAddresses,phoneNumbers,names,addresses,biographies,userDefined', 'pageToken': nextPageToken});
}
connections = connections.concat(apiResponse.connections);
nextPageToken = apiResponse.nextPageToken;
} while (nextPageToken);
if (connections.length != apiResponse.totalItems) {connections = getAllConnections_();} // Hopefully, the magic lies in this line of code
return connections;
}
Posting this here in case it helps someone else.
Edit: Just corrected the test on the magic line from "==" to "!=".
Here is something I did for use with my email whitelist to ensure I didn't get duplicate emails.
function displayCurrentContacts() {
const ss = SpreadsheetApp.getActive();
const sh = ss.getSheetByName('Contacts');
sh.clearContents();
const vs = [['Name', 'Emails']];
const resp = People.People.Connections.list('people/me', { personFields: "emailAddresses,names,organizations" });
//Logger.log(resp);
const data = JSON.parse(resp);
let m = 0;
let n = 0;
data.connections.forEach((ob1, i) => {
if (ob1.emailAddresses && ob1.emailAddresses.length > 0) {
let emails = [... new Set(ob1.emailAddresses.map(ob2 => ob2.value))];//used set to insure I get unique list
//let emails = ob1.emailAddresses.map(ob2 => ob2.value);
let name;
m += emails.length;
//the following cases are derived from the way that I organize all of my contacts
if (ob1.names && ob1.organizations) {
name = ob1.names[0].displayName + '\n' + ob1.organizations[0].name;
++n;
} else if (ob1.names) {
name = ob1.names[0].displayName;
++n;
} else if (ob1.organizations) {
name = ob1.organizations[0].name;
++n;
}
vs.push([name, emails.sort().join('\n')])
}
});
vs.push([n, m])
sh.getRange(1, 1, vs.length, vs[0].length).setValues(vs)
sh.getRange(2, 1, sh.getLastRow() - 2, sh.getLastColumn()).sort({ column: 1, sortAscending: true });
}
Note this is not meant to be a plugin replacement by any means.
Just installed MySQL locally, and created a db and one simple table. I'm now trying to connect to it from Google Sheets, but it fails with err:
{ [Exception: Invalid argument: url] name: 'Exception' } undefined
Much googling later, it seems that because Sheets runs on google servers, it can't resolve localhost as a Host.
Is there a way around this?
var HOST = 'localhost';
var PORT = '3306';
var USERNAME = 'xxxx';
var PASSWORD = 'xxxx';
var DATABASE = 'investment';
var DB_TYPE = 'mysql';
var MAXROWS = 1000;
function testConnection(){
var sql = doc.getRange('query!a2').getDisplayValue();
var options = {};
}
function runSql(query, options) {
try {
var fullConnectionString = 'jdbc:' + DB_TYPE + '://' + HOST + ':' + PORT;
var conn = Jdbc.getConnection(fullConnectionString, USERNAME, PASSWORD);
Logger.log('conn :', conn);
Logger.log('query :', query);
var stmt = conn.createStatement();
stmt.execute('USE ' + DATABASE);
var start = new Date();
var stmt = conn.createStatement();
stmt.setMaxRows(MAXROWS);
var rs = stmt.executeQuery(query);
} catch (e) {
console.log(e, e.lineNumber);
Browser.msgBox(e);
return false;
}
}
Just wondering, if I could find a way of getting my IPv4 address online (since it would be DHCP I think, so changing periodically), could I use Host such as 101.146.72.23/localhost? (not my real IP)
[Edit] I tried adding my IPv4 address as above, and maybe that took me a step closer, as now I get a different error:
{ [Exception: Failed to establish a database connection. Check connection string, username and password.] name: 'Exception' } undefined
While reading up on Google: JDBC, it states "In order to create a database connection using the JDBC service you must allow-list certain IP ranges in your database settings to allow Apps Script to access it. These are the address ranges you'll need to allow-list."
Now searching on how to do this in Windows 10.
[Edit 2] Reporting progress for anyone else who needs to do this. Following #Julian Benavides suggestions, I found a link on how to Add Port Forwarding on Telstra Smart Modem . This post pointed out that my pc would need a Static IP address (on the local network), at How to Setup a Static IP Address on a Computer.
I think I have made a successful connection, although now I have to figure out why I have error:
Exception: Table 'investment.investment_type' doesn't exist
Getting closer, but no cigar yet.
[Edit 3 - Success!] I had the wrong table name, now works!
Did you configure a port forwarding or DMZ access ?, if you are at home configure and do access to router or contact with your ISP, or if you are in the company contact with the network administrator.
With this way your ip could be resolve in internet and be accesed by jdbc connection.
A brief summary of how the issue was resolved - more info in the updates in the original post.
The code I used was closely derived from Ben Collins - How To Connect Google Sheets To A Database, Using Apps Script
#Julian Benavides pointed out that I needed to add Port Forwarding, found how to do that at Setup a Port Forward on the Telstra Smart Modem Gen 2 Router
Had to assign a Static IP on my local network for the machine running MySQL, from How to Setup a Static IP Address
Hope this may be useful to another novice like me. :)
I have a C# Console application that uses EWS (Exchange Web Services) to impersonate a user and I create or update his current Contacts list.
In order to determine if I have to create or update his list, I first need to search his existing Contacts for a particular domain name like so:
private static IEnumerable<Contact> GetExistingContacts(ExchangeService service)
{
var domainToFilterOn = "#contoso.com";
SearchFilter sfSearch = new SearchFilter.ContainsSubstring(ContactSchema.EmailAddress1, domainToFilterOn);
FindItemsResults<Item> contacts = service.FindItems(WellKnownFolderName.Contacts, sfSearch, new ItemView(int.MaxValue));
var results = contacts.Cast<Contact>().ToList();
return results;
}
The problem I’m facing is that the method GetExistingContacts() yields 0 results for the given domain name which is wrong since I know I have a bunch of Contacts holding the #contoso.com domain name inside the EmailAddre1 field.
After a little bit of digging and testing, I finally figure out why the method wasn’t returning any results and the reason was because the email addresses are stored in the Exchange distinguished name instead of the SMTP format.
To further my investigation, I created a few new Contacts with fake/non-existing #contoso.com domain name like: test#contoso.com, gazou#contoso.com, etc.
To my surprise, the GetExistingContacts() method started to return these fake Contacts.
The conclusion is that whenever I create new Contacts that have resolvable email addresses, then these Contacts are stored using the Exchange distinguished name but when I create new Contacts that have non-resolvable email addresses, then these Contacts are stored as SMTP (which are returned by my GetExistingContacts() method).
How do I start fixing this?
Is my search method wrong? Is there another way to search inside the EmailAddress1 field?
Meanwhile, I managed to find a workaround using the .Load() method of the Contact object but this workaround seems ugly and costly in terms of execution time.
I basically get all Contacts, loop and call the Load() method, then add them to a List() and make a Linq query to filter the results. If my user has 800 Contacts, that takes a long time to Load() everyone of them.
Here’s the example:
private static IEnumerable<Contact> GetCurrentContacts(ExchangeService service)
{
var contacts = new List<Contact>();
var data = service.FindItems(WellKnownFolderName.Contacts, new ItemView(int.MaxValue));
foreach (var item in data.Items)
{
if (item is Contact)
{
item.Load();
contacts.Add(item as Contact);
}
}
var result = contacts.Where(x => x.EmailAddresses[EmailAddressKey.EmailAddress1].Address.Contains("#contoso.com")).ToList();
return result;
}
Needless to say, I don't think that is the correct approach although it works.
Another alternative I’ve tried was to force the RoutingType to SMTP thinking it would create the new Contact in the SMTP format as opposed to the Exchange distinguished name but unfortunately, the email address still gets stored in the Exchange distinguished name disregarding the fact that I forced the RoutingType like so:
var email = new EmailAddress();
email.Address = "goodemail#contoso.com";
email.RoutingType = "SMTP";
Contact contact = new Contact(service);
contact.EmailAddresses[EmailAddressKey.EmailAddress1] = email;
...
contact.Save();
If anyone can help me shed some light on this, that would be great!
Thanks in advance
If the SMTP address your trying to use for a Contact is visible (or resolvable) in the Global address list then the X500 address of Directory entry will be used to track the Contact to the Directory entry (this is by design). If you want to return the SMTP address instead of the X500 address when you retrieve the contacts all you need to do is make use you do a GetItem or Load in the Managed API on the contact or contacts in question if you have multiple contact use LoadPropertiesFromItems https://blogs.msdn.microsoft.com/exchangedev/2010/03/16/loading-properties-for-multiple-items-with-one-call-to-exchange-web-services/
You can override this behavior by setting the extended properties on the contacts directly see https://social.technet.microsoft.com/Forums/exchange/en-US/2b375c56-bee1-4d88-b638-f95649ef964a/use-ews-create-a-contact-which-has-a-same-email-address-in-gal-it-will-show-up-with-x500-formatting?forum=exchangesvrdevelopment but I would recommended you stay with the default behavior you can make you code more efficient using batch Loads.
Thank you #Glen Scales, the LoadPropertiesForItems() helped.
This is the final result if anyone cares:
private static IEnumerable<Contact> GetExistingContacts(ExchangeService service)
{
var contacts = new List<Contact>();
var filterContactsOnDomain = "#contoso.com";
var data = service.FindItems(WellKnownFolderName.Contacts, new ItemView(int.MaxValue));
if (data.TotalCount > 0)
{
service.LoadPropertiesForItems(data, new PropertySet(ContactSchema.EmailAddress1));
foreach (var item in data.Items)
{
if (item is Contact)
{
contacts.Add(item as Contact);
}
}
}
var result = contacts.Where(x => x.EmailAddresses[EmailAddressKey.EmailAddress1].Address.ToLower().Contains(filterContactsOnDomain.ToLower())).ToList();
return result;
}
The only thing I dislike is having to get all Contacts (800 of them), load the EmailAddress1 field for all 800 of them, loop and add those 800 Contacts to a list and then, filter on that list...
I guess it would've been nice to have the ability to search (or filter) directly on an X500 email address format thus not having to fetch all 800 Contacts.
Oh well...
Thanks again #Glen Scales
I'm newby at Node.js and Socket.io. I wonder if i accumulate the data which users entered from UI, then, these datas collecting at the server and pushing database at intervals. For example, different users entering messages from their accounts continuously, and these datas collecting at the server. every 1 minutes, collected datas push into database, and the data pool will drain off. Is it possible with node.js and socket.io?
You can achieve such result pretty easily in NodeJS.
Just accumulate your messages in an hashmap (let say a key per user) and trigger a timeout on every minute to flush your hashmap into your database.
var messages = {};
io.sockets.on('connection', function (socket) {
socket.on('message', function (data) {
// data.id is user id, data.content the message
if (!(data.id in messages)) {
messages[data.id] = []
}
messages[data.id].push(data.content);
});
});
var flush = function() {
for (userId in messages) {
// ... write in database
// effectively flush messages
messages[userId] = [];
}
// Don't forget to set agin the next flush
setTimeout(flush, 1000);
}
setTimeout(flush, 1000);
Its a naive solution which will be very easy to setup and test.
Don't use setInterval, which is verry greedy on CPU time.
As this example is single-threaded, your server will be unavailable during the (short) time it will write on the database.
If you need a highly available server, consider use another process for storing and flushing messages, and send it your messages instead of storing them into the main (communication) process.
Specifically users.
Currently I have script like this:
function findUsers(s) {
var users = UserManager.getAllUsers();
var r = new Array;
Logger.log("User Count" + users.length );
for( var i = 0 ; i < users.length ; i++ )
{
var u = users[i];
var n = formatUserName( u );
n = n.toUpperCase();
if( n.indexOf( s.toUpperCase() ) > -1 )
{
r.push(u);
}
}
return r;
}
This works, but is a little slow and I know it will get slower, as more users are migrated because of how UserManager.getAllUsers() currently works
So before I go off and look at ways of 'caching' the domain data myself...I'm wondering if there is a efficient way to search for objects within our domain via APIs or GAScript objects?
When you use the control panel for the domain you can search, fast and efficiently...
For my current task, I may abandon searching the domain myself and require the correct user name.
Here are your choices
1) Make use of the Google Apps Profiles API ( https://developers.google.com/google-apps/profiles/#Retrieving ) from within Google Apps Script. This requires use of oAuth. If you haven't used oAuth, try the oAuth Service Library ( https://developers.google.com/apps-script/notable-script-libraries ) to ease your pain of getting it to work.
2) The second option is what you've already noted. Cache the results in, say, a spreadsheet.
3) Irrespective of 1 or 2, you should open an issue in the Issue tracker if there isn't one already.