Can someone suggest an alternative to ScriptProperties as a means of storing and retrieving short-term (textual) data.
I have a long-standing application built in GAS. Because it kicks off a number of 'threads' I store some information about each thread, ultimately to determine when all the threads have finished and thus when the application can release various 'locks'.
I'm using ScriptProperties to store this information. It used to work perfectly but has been beset by troubles over the last 6 months or so. The problems are mainly:
1. Application load times increase massively as more and more data collects in script properties (it isnt possible to clean down scriptproperty data)
2. ScriptProperties are (now) unreliable under concurrent access
3. ScriptProperties can't be read or written to in rapid bursts
Someone has suggested using ScriptDb. Can anyone confirm this (or any other) viable solution?
Many thanks
Can you have a look at CacheService ? I have used with lot of success.
CacheService is only best-effort and is not intended for guaranteed storage (but it is fast!). If you need to be sure it will be there, use ScriptDb.
Related
We use Google Sheets to store and manipulate event-based data from the users of our web application. Recently, I've been using Google Apps Script to automate this data manipulation process, and now I'm looking for some advice.
My scripts were working well for a while and then late last week, I started receiving the error, 'Exceeded Maximum Execution Time'. I received this error on two of my scripts that worked fine the day before, with no edits in between. I'm working with a lot of data and have even hit the 2,000,000 cell limit on my spreadsheet a couple times.
My goal is to be able to sequence my seven scripts together so that I can fire them all with one trigger. How can I do this without hitting the max execution time error? Sometimes, my spreadsheet will crash when I just try to delete one of the sheets within it. Could the size of my spreadsheet be causing the scripts to fail?
I've done my best to follow best practices outlined in Google's documentation, but can anyone recommend best practices for spreadsheet and script organization?
You might be interested in a tool written by Bruce Mcpherson for massively parallel processing in Google Apps Script.
http://ramblings.mcpher.com/Home/excelquirks/htmlservice/parallel
http://ramblings.mcpher.com/Home/excelquirks/htmlservice/parallel/implement
Here you can find the documentation related to quotas on apps script, as you can see the script runtime is 6 mins after that, the server will stop the script and throw an exception as, as you are experiencing.
If you are already applying Google's best practices, try to find where are your bottle necks.
-Do you need to read the complete range of data or just a part of it?
-Is there any information that you could store (e.g. in cache or properties) so you won't have to calculate or read again?.
-Are you tracking the time for each script?
-Are you reading the whole range of data on every script? or reading it once and passing it as parameter to the other scripts?
hope this helps.
What is the term for a webhook that makes another set of webhooks?
A recent kludge (Unbounce, Zapier, Twilio) only allows me one webhook (i.e. custom one from Zapier)
https://zapier.com/zapbook/twilio/unbounce/
Eventually, however, I will need to generate multiple actions on different (customzed) platforms, so I was hoping to have a hook that goes to one place and then from there, vector off hooks to my other beta site (including Zapier).
Any suggestions would be greatly appreciated.
I note your question was about unbounce, but included Zapier. I'm assuming from this that you were looking to use Zapier as the "one place" (i.e in the middle)"
Now I don't know Zapier, but came across your question looking along the same lines. The terms I'm familiar with on this are "one-to-many", "fan-out" or "publish/subscribe" communication styles - but aware that these may be a bit old school in Web 2.0 world.
In my search I came across a Zapier Support post on Multiple Zaps.
In summary, looks like you may be able to achive this using a "Zapier Delay Trigger zap"
The possible downside (depending on delivery time requirements) is that the delivery of onward webhooks would be complete "...within 5-15 minutes". This doesn't quite fit my requirement, by may fit yours.
Edit: A quick search on "fan-out" term also found this on using Zapier Queue. Again,as it's queuing I suspect there may be a delay in onward processing.
Edit2: Following email to Zapier support - situation is still as per links i.e.time delay, but "...in your zap you can define the length of the delay to minimize the lag. This case you shouldn't have any problem setting a 1 minute delay since there's no need for it to be any longer, which can help you get a lot closer to 'near real time'."
Edit3: Further info from Zapier support - apparently if you configure two Webhook-Webhook Zaps you can fire them both with one URL as "https://zapier.com/hooks/catch/xxhook1,xyhook2" - appears to fire both zaps pretty immediately.
Edit4: One thing to watch with the multi-zap approach (as Edit3) is that if one send to target fails then you can replay manually (via Zapier site and paying customer) - there's no ability (currently) to manage automatic retrying (not sure if there's any ability to monitor this event).
I am parsing a large text file and then calling a rate-limited Google API (mail migration) (The api docs state 1 second per call).
Even when I have Utilities.sleep(1000); inside my loop, I'm still getting this error even though I'm only setting the property one time during the loop:
Service invoked too many times in a short time: properties rateMax. Try Utilities.sleep(1000) between calls. (line 146)
Other than not setting the properties that often, what can I do to alleviate this? Should I instead try to use the CacheService to store my property transiently?
Unfortunately the exact short term rate limits aren't exposed. You may want to try to increase your sleep amount, in the hopes of going above the threshold necessary to avoid the error. Ultimately I think your analysis is correct, that you should look into writing to the user properties less often. I'm not sure the CacheService is the right solution, but it depends on how you are using the information.
It really depends on the design of your app. If you are parsing the information and can aggregate that into a summary it would take less calls. Maybe sending as an email is not optimal. Could the parsed data go somewhere else and then direct the user there instead of sending emails?
properties v CacheService
CacheService produces the same error (Unless I have a bug in my code that I missed). If it helps CacheService lets you store longer strings, so this can reduce the number of writes.
I'm experimenting with P2P on Flash, and I've come across a little hurdle that I'd like to clarify before moving forward. The technology itself (Flash) doesn't matter for this problem, as I think this problem occurs in other languages.
I'm trying to create a document that can be edited "live" by multiple people. Just like Google Docs pretty much. But I'm wondering, how would you suggest synchronizing everyone's text? I mean, should I message everyone with all the text in the text field every time someone makes a change? That seems very inefficient.
I'm thinking there has to be a design pattern that I can learn and implement, but I'm not sure where to start.
Optimally, the application should send the connected clients only the changes that have occurred to the document, and have some sort of buffer or error correction that can be used for retrieving earlier changes that may have been missed. Is there any established design pattern that deals with this type of issue?
Thanks,
Sandro
I think your "Optimally" solution is actually the one you should go for.
each textfield has a model, the model has a history (a FILO storing last, let's say, 10 values).
every time you edit that textfield you push the whole text into the model and send the delta to other connected clients.
as other clients receive the data they just pick the last value from the model and merge it to the received data.
you can refine the mechanism by putting an idle timer in the middle: as a user types something in the textfield you flag that model as "toBeSentThroughTheNet" and you start a timer. as the timer "ticks" (TimerEvent.TIMER) you stop it, collect the flagged data and send it to other clients. just remember to reset the timer everytime the user is actually typing (a semplification coul be keydown = reset, keyup = start).
one more optimization could be send the data packed in a compressed bytearray, but this requires you write your own protocol and may be not so an easy and quick path :)
If the requirement is that everyone can edit the document at the same time and the changes should be propagated to everyone and no changes should be lost, then it is a non-trivial problem. There are few different approaches out there, but one that is quite robust is Operational Transformation. This is the same algorithm that Google Docs uses for collaborative editing.
Understanding and Applying Operational Transformation and the attendant hacker news discussion are probably other good places to start.
The Wave Protocol was released as open source so you can take a look on how it is implemented.
You could of course forgo the tricky synchronization and just allow people to take turns and only one person can edit the document at a time and this person just pushes the changes to the remainder of the group.
I mean, I always was wondered about how the hell somebody can develop algorithms to break/cheat the constraints of legal use in many shareware programs out there.
Just for curiosity.
Apart from being illegal, it's a very complex task.
Speaking just at a teoretical level the common way is to disassemble the program to crack and try to find where the key or the serialcode is checked.
Easier said than done since any serious protection scheme will check values in multiple places and also will derive critical information from the serial key for later use so that when you think you guessed it, the program will crash.
To create a crack you have to identify all the points where a check is done and modify the assembly code appropriately (often inverting a conditional jump or storing costants into memory locations).
To create a keygen you have to understand the algorithm and write a program to re-do the exact same calculation (I remember an old version of MS Office whose serial had a very simple rule, the sum of the digit should have been a multiple of 7, so writing the keygen was rather trivial).
Both activities requires you to follow the execution of the application into a debugger and try to figure out what's happening. And you need to know the low level API of your Operating System.
Some heavily protected application have the code encrypted so that the file can't be disassembled. It is decrypted when loaded into memory but then they refuse to start if they detect that an in-memory debugger has started,
In essence it's something that requires a very deep knowledge, ingenuity and a lot of time! Oh, did I mention that is illegal in most countries?
If you want to know more, Google for the +ORC Cracking Tutorials they are very old and probably useless nowdays but will give you a good idea of what it means.
Anyway, a very good reason to know all this is if you want to write your own protection scheme.
The bad guys search for the key-check code using a disassembler. This is relative easy if you know how to do this.
Afterwards you translate the key-checking code to C or another language (this step is optional). Reversing the process of key-checking gives you a key-generator.
If you know assembler it takes roughly a weekend to learn how to do this. I've done it just some years ago (never released anything though. It was just research for my game-development job. To write a hard to crack key you have to understand how people approach cracking).
Nils's post deals with key generators. For cracks, usually you find a branch point and invert (or remove the condition) the logic. For example, you'll test to see if the software is registered, and the test may return zero if so, and then jump accordingly. You can change the "jump if equals zero (je)" to "jump if not-equals zero (jne)" by modifying a single byte. Or you can write no-operations over various portions of the code that do things that you don't want to do.
Compiled programs can be disassembled and with enough time, determined people can develop binary patches. A crack is simply a binary patch to get the program to behave differently.
First, most copy-protection schemes aren't terribly well advanced, which is why you don't see a lot of people rolling their own these days.
There are a few methods used to do this. You can step through the code in a debugger, which does generally require a decent knowledge of assembly. Using that you can get an idea of where in the program copy protection/keygen methods are called. With that, you can use a disassembler like IDA Pro to analyze the code more closely and try to understand what is going on, and how you can bypass it. I've cracked time-limited Betas before by inserting NOOP instructions over the date-check.
It really just comes down to a good understanding of software and a basic understanding of assembly. Hak5 did a two-part series on the first two episodes this season on kind of the basics of reverse engineering and cracking. It's really basic, but it's probably exactly what you're looking for.
A would-be cracker disassembles the program and looks for the "copy protection" bits, specifically for the algorithm that determines if a serial number is valid. From that code, you can often see what pattern of bits is required to unlock the functionality, and then write a generator to create numbers with those patterns.
Another alternative is to look for functions that return "true" if the serial number is valid and "false" if it's not, then develop a binary patch so that the function always returns "true".
Everything else is largely a variant on those two ideas. Copy protection is always breakable by definition - at some point you have to end up with executable code or the processor couldn't run it.
The serial number you can just extract the algorithm and start throwing "Guesses" at it and look for a positive response. Computers are powerful, usually only takes a little while before it starts spitting out hits.
As for hacking, I used to be able to step through programs at a high level and look for a point where it stopped working. Then you go back to the last "Call" that succeeded and step into it, then repeat. Back then, the copy protection was usually writing to the disk and seeing if a subsequent read succeeded (If so, the copy protection failed because they used to burn part of the floppy with a laser so it couldn't be written to).
Then it was just a matter of finding the right call and hardcoding the correct return value from that call.
I'm sure it's still similar, but they go through a lot of effort to hide the location of the call. Last one I tried I gave up because it kept loading code over the code I was single-stepping through, and I'm sure it's gotten lots more complicated since then.
I wonder why they don't just distribute personalized binaries, where the name of the owner is stored somewhere (encrypted and obfuscated) in the binary or better distributed over the whole binary.. AFAIK Apple is doing this with the Music files from the iTunes store, however there it's far too easy, to remove the name from the files.
I assume each crack is different, but I would guess in most cases somebody spends
a lot of time in the debugger tracing the application in question.
The serial generator takes that one step further by analyzing the algorithm that
checks the serial number for validity and reverse engineers it.