Doing some testing on google script's ContactsApp and loading in contacts. It looks like it takes as much time to run ContactsApp.getContacts() (loading all contacts) as it does to run ContactsApp.getContact('email') (specific contact). About 14 seconds on each method for my contacts
My assumption is that both methods are calling all contacts and the 2nd only matches on email. This drags quite a bit.
Has anyone confirmed this and is there anyway to keep the loaded contacts in memory between pages (session variable?).
You've got a few options for storing per-user data:
If it's a small amount of data, you can use User Properties
You can store much more data using ScriptDb, but this will be global, so you'll have to segment off user data yourself
If you only need the data for a short amount of time, say, between function calls, you can use the Cache Services. You'll want to use getPrivateCache()
It sounds like for your use case getPrivateCache() is your best option for user specific session-like data storage.
(Just make sure your intended use fits within the terms of service.)
Related
I have an application I am working on that basically takes the data from Active Collab and creates reports / graphs out of the data. The API itself is insufficient to get the proper data on a per request basis so I resorted to pulling the data down into a separate data set that can be queried more efficiently.
So in order to avoid needing to query the entire API constantly I decided to make use of webhooks in order to make the transformations to the relevant data and lower the need to resync the data.
However I notice not all events are sent, notably the following.
TaskListUpdated
MemberUpdated
TimeRecordUpdated
ProjectUpdated
There is probably more but these are the main ones I noticed so far,
Time reports is probably the most important, in fact it missing from webhooks means that almost any application has a good chance of incorrect data if it needs time record data. Its fairly common to do a typo in a time record and then adjust it later.
So am I missing anything here? Is there some way to see these events reliably?
EDIT:
In order to avoid a long comment to Ilija I am putting the bulk here.
Webhooks apart, what information do you need to pull? API that powers
time tracking reports can do all sorts of cross project filtering, so
your approach to keep a separate database may be an overkill.
Basically we are doing a multi-variable tiered time report. It can be sorted / grouped by any conceivable method you may want to look at.
http://www.appsmagnet.com/product/time-reports-plus/
This is the closest to what we are trying to do, back when we used Active Collab 4 this did the job, but even with it we had to consolidate it in our own spreadsheets.
So the idea of this is to better integrate our Active Collab data into our own workflow.
So the main data we are looking for in this case is
Job Types
Projects
Task Lists
Tasks
Time Records
Categories
Members / Clients
Companies
These items can feed not only our reports, but many other aspects of our company as well. For us Active Collab is the point of truth, so we want the data quickly accessible and fully query-able.
So I have set up a sync system that initially grabs all the data it can from Active Collab and then uses a mix of cron's and webhooks to keep it up to date.
Cron jobs work well for all aspects that do not have "sub items" (projects/tasks/task lists/time records). So those I need to rely on the webhook since syncing them takes to much time to be able to keep it up to date in real time.
For the webhook I noticed the above do not carry through. Time Records I figured out a way around it listed in my answer, and member can be done through the cron. However Task list and project updating are the only 2 of some concern. Project is fairly important as the budget can change and that would be used in reports, task lists has the start / end dates that could be used as well. Since going through every project / task list constantly to see if there is a change is really not a great idea I am looking for a way to reliably see updates for them.
I have based this system on https://developers.activecollab.com/api-documentation/ but I know there are at least a few end points that are not listed.
Cross-project time-record filtering using Active Collab 5 API
This question is actually from another developer on the same system (and also shows a TrackingFilter report not listed in the docs). Due to issues with maintaining an accurate set of data we had to adapt it. I actually notice that you (Ilija) are the person replying and did recommend we move over to this style of system.
This is not a total answer but a way to solve the issue with TimeRecordUpdated not going through the webhook.
There is another API endpoint for /whats-new This endpoint describes changes for the last day or so and it has a category called TrackingObjectUpdatedActivityLog this refers to an updated time record.
So I set up a cron job to check this fairly consistently and manually push the TimeRecordUpdated event through my system to keep it consistent.
For MemberUpdated since the data for a member being updated is unlikely to affect much, having a daily cron for checking the users seems good enough.
ProjectUpdated could technically be considered the same, but with the absence of TaskListUpdated that leads to far to many api calls to sync the data. I have not found a solution for this yet unfortunately.
I have a text box in my application which allows a user to select a location with the help of UI autocomplete. There are around 10,000 valid locations out of which the user must select one. There are two implementations for the autocomplete functionality:
Fetch the list of locations when the page loads for the first time and iterate over the array to find matching items on every keystroke in javascript
Make ajax requests on every keystroke as searching in MySQL(the db being used) is much faster?
Performance wise, which one is better?
An initial test shows that loading the data at once is the better approach from a performance point of view. However, this test was done on a MBP where JavaScipt processing is quite fast. I'm not sure whether this technique is the better one for machines with low processing power like lower end android phones, old systems etc.
Your question revolves around which is quicker, processing over 10,000 rows in the browser, or sending a request to a remote server to return the smaller result set. An interesting problem that depends on context and environment at runtime. Sending to the remote server incurs network delay mostly, with small amounts of server overhead.
So you have two variables in the performance equation, processing speed of the client and network latency. There is also a third variable, volume of data, but this is constant 10k in your question.
If both client browser and network are fast, use whatever you prefer.
If the network is faster, use the remote server approach, although be careful not to overload the server with thousands of little requests.
If the client is faster, probably use the local approach. (see below)
If both are slow, then you probably need to chose either, or spend lots of time and effort optimizing this.
Both clients slow can easily happen, my phone browser on 3G falls into this category, network latency for a random Ajax request is around 200mS, and it performs poorly for some JavaScript too.
As user perceieved performance is all that really matters, you could preload the first N values for each letter as variables in the initial page load, then use these for the first keystroke results, this buys you a few mS.
If you go with the server approach, you can always send requested result AND a few values for each of the next keystroke. This overlaps what users see and makes it appear snappier on slow networks. Eg
Client --> request 'ch'
Server responds with a few result for each potential next letter
'cha' = ...
'chb' = ...
Etc
This of course requires some specialized javascript to alternate between Ajax requests and using cached results from previous requests to prefill the selection.
If you are going with the local client searching through all 10k records, then make sure the server returns the records in sorted order. If your autocomplete scanning is able to use 'starting with' selection rather than 'contains' (eg typing RO will match Rotorua but not Paeroa) then you can greatly reduce processing time by using http://en.wikipedia.org/wiki/Binary_search_algorithm techniques, and I'm sure there are lots of SO answers on this area.
If there is no advantage for querying the backend every time, don't do it.
What could be an advantage of querying the backend all the time? If the amount of returned data for the initial call is to heavy (bandwidth, javascript processing time to prepare it, time at all), the partial request every time could be the smarter option.
I'm developing a web application that will use a google spreadsheet as a database.
This will mean parsing up to 30.000 (guestimated size) rows in regular operations for searching ID's etc...
I'm worried about the response times i will be looking at. Does anybody have experience with this? I don't want to waste my time on something that will hit a deadend at an issue like that.
Thanks in advance
Using spreadsheets as a database for this data set is probably not a good idea. Do you already have this spreadsheet set up?
30K rows will allow you to have only 66 columns, is that enough for you? Check the Google Docs size limits help page for more info.
Anyway, Google Spreadsheets have a "live concurrent editing" nature to it that makes it a much slower database than any other option. You should probably consider something else.
Do you intend to use the spreadsheet to display data or only as a storage place.?
In this second option the relative slowness of the spreadsheet will not be an issue since you'll only have to read its data once to get its data in an array and play with that...
This implies of course that you build every aspect of data reading and writing in a dedicated UI and never show the spreadsheet itself , the speed will be depending only on the JavaScript engine on arrays, the speed of UI and the speed of your internet connection... all 3 factors being not very performing if compared to a 'normal' application but with the advantage of being easily shareable and available anywhere.:-)
That said, I have written such a database app with about 20 columns of data and 1000 rows and it is perfectly useable although having some latency even for simple next / previous requests. On the other hand the app can send mails and create docs.... the advantages of Google service integration :-)
You could have a look at this example to see what I'm talking about
I am trying to execute a mySql database query on my website depending on whether a user has "liked" my Facebook page. I have found a few ways to do this, using the PHP and JS SDK's, namely using the API with /USER_ID/likes/PAGE_ID.
When a user has liked my page, I want to add a value to their data in my database, so I thought of adding a function that is called each time the user visits the site, and if they like it, add value to database and also have a boolean value in there so it doesn't keep adding to the value. However, I guessed this would be waste of calls to the server if this happened every time, so I am not sure how to go about setting this up, any ideas?
Unless you are dealing with huge volumes of users I wouldn't worry about it because a check like that on one row of an indexed mysql table should be very quick (200 milliseconds or less for the entire request over a normal internet connection.). And if the data you need is stored on the server, then how could you possibly avoid the trip to the server? Unless you store the data in a cookie.
I tried searching through on stackoverflow as well as googling around a lot, but am not able to find answers to my problem (I guess I'm searching for the wrong keywords / terms).
We are in the process of building a recommendation engine, and while we are initially logging all user activity in custom logs (we use ruby / rails), we need to do an EOD scanning of that file and arrange according to the user. We also have some other user data coming in from some other places (his fb activity, twitter timeline, etc), and hence by EOD we want all data for a particular user to be saved somewhere and then run our analyzer code on all of the user's data to generate the recommendations.
The problem is that we are generating a lot of data, and while for the time being we are using a mysql table to store all this data, we are not sure till how much time can we continue to do this, as our user-base grows (we are still testing it out internally with about 10 users with a lot of activity). Plus, as eager developers we would like to try out something new that can suffice our needs.
Any pointers in this direction will be very helpful.
Check out Amazon Elastic Map Reduce. It was built for this very type of thing.