I am parsing a large text file and then calling a rate-limited Google API (mail migration) (The api docs state 1 second per call).
Even when I have Utilities.sleep(1000); inside my loop, I'm still getting this error even though I'm only setting the property one time during the loop:
Service invoked too many times in a short time: properties rateMax. Try Utilities.sleep(1000) between calls. (line 146)
Other than not setting the properties that often, what can I do to alleviate this? Should I instead try to use the CacheService to store my property transiently?
Unfortunately the exact short term rate limits aren't exposed. You may want to try to increase your sleep amount, in the hopes of going above the threshold necessary to avoid the error. Ultimately I think your analysis is correct, that you should look into writing to the user properties less often. I'm not sure the CacheService is the right solution, but it depends on how you are using the information.
It really depends on the design of your app. If you are parsing the information and can aggregate that into a summary it would take less calls. Maybe sending as an email is not optimal. Could the parsed data go somewhere else and then direct the user there instead of sending emails?
properties v CacheService
CacheService produces the same error (Unless I have a bug in my code that I missed). If it helps CacheService lets you store longer strings, so this can reduce the number of writes.
Related
The following api link for the Node JS package shows an endpoint that limits to 60 RPM: forge-api docs
Support has said that 60RPM is only for endpoints where forceget is set to true in the options, and that the implicit value for the forceget option when calling getModelviewProperties is set to false, yet I'm still getting a limited to 60 requests per minute.
Even when setting forceget to false it limits the rpm.
How can I have more RPM for this endpoint?
Unfortunately these endpoints have to be limited for reasons. But I agree 60rpm with forceget=true is a very low limit. However, forceget=false has a 14000rpm limit. Note as well that forceget=true is ignored if the resources are still available in the cache from a previous call.
That means that you first call with forceget=false, and if there is a problem, you call the resource with forceget=true only once, and give it some time for the resources to be loaded in the cache. All following calls should be using forceget=false. That way, you would save your forceget=true quota.
If you still have problems, you can either let me know at cyrille at autodesk.com with an example, and I'll debug the problem with you.
Or an alternative is to parse/expand the db files yourself, which can be a very good option if you need to improve your application performance a lot. I got an old sample doing this here which is more powerful than the Forge API for querying properties. I am currently rewriting it in typescript in a new sample, but this one isn't ready yet - it will be posted here.
I have to implement a REST endpoint that receives start and end dates (among other arguments). It does some computations to generate a result that is a kind of forecast according to the server state at invocation epoch and the input data (imagine a weather forecast for next few days).
Since the endpoint does not alter the system state, I plan to use GET method and return a JSON.
The issue is that the output includes also an image file (a plot). So my idea is to create a unique id for the file and include an URI in the JSON response to be consumed later (I think this is the way suggested by HATEOAS principle).
My question is, since this image file is a resource that is valid only as part of the response to a single invocation to the original endpoint, I would need a way to delete it once it was consumed.
Would it be RESTful to deleting it after serving it via a GET?
or expose it only via a DELETE?
or not delete it on consumption and keep it for some time? (purge should be performed anyway since I can't ensure the client consumes the file).
I would appreciate your ideas.
Would it be RESTful to deleting it after serving it via a GET?
Yes.
or expose it only via a DELETE?
Yes.
or not delete it on consumption and keep it for some time?
Yes.
The last of these options (caching) is a decent fit for REST in HTTP, since we have meta-data that we can use to communicate to general purpose components that a given representation has a finite lifetime.
So this reference of the report (which includes the link to the plot) could be accompanied by an Expires header that informs the client that the representation of the report has an expected shelf life.
You might, therefore, plan to garbage collect the image resource after 10 minutes, and if the client hasn't fetched it before then - poof, gone.
The reason that you might want to keep the image around after you send the response to the GET: the network is unreliable, and the GET message may never reach its destination. Having things in cache saves you the compute of trying to recalculate the image.
If you want confirmation that the client did receive the data, then you must introduce another message to the protocol, for the client to inform you that the image has been downloaded successfully.
It's reasonable to combine these strategies: schedule yourself to evict the image from the cache in some fixed amount of time, but also evict the image immediately if the consumer acknowledges receipt.
But REST doesn't make any promises about liveness - you could send a response with a link to the image, but 404 Not Found every attempt to GET it, and that's fine (not useful, of course, but fine). REST doesn't promise that resources have stable representations, or that the resource is somehow eternal.
REST gives us standards for how we request things, and how responses should be interpreted, but we get a lot of freedom in choosing which response is appropriate for any given request.
You could offer a download link in the JSON response to that binary resource that also contains the parameters that are required to generate that resource. Then you can decide yourself when to clean that file up (managing disk space) or cache it - and you can always regenerate it because you still have the parameters. I assume here that the generation doesn't take significant time.
It's a tricky one. Typically GET requests should be repeatable as an import HTTP feature, in case the original failed. Some people might rely on it.
It could also be construed as a 'non-safe' operation, GET resulting in what is effectively a DELETE.
I would be inclined to expire the image after X seconds/minutes instead, perhaps also supporting DELETE at that endpoint if the client got the result and wants to clean up early.
The problem I'm posed with is backfilling a specialized database, using data from the event log of a given smartcontract on an Ethereum blockchain.
The question is however: how to do so without reaching the limits of eth_getLogs (also without limits: how to have reasonably sized RPC responses)
What I tried so far
I prefer to use Infura, but they limit this call at 100 entries per response. And rightfully so, querying should be done in small chunks for load balancing etc. Is api pagination + eth_getLogs the right way to collect data for backfills?
Idea 1: eth_getLogs on ranges of blocks
I don't know of any way to paginate the eth_getLogs other than querying for ranges of blocks. A block may contain more than 100 events however, which prevents me from reading all of the data when using Infura. Maybe there is a way to paginate on log index? (100 is something I came accross when experimenting, but I can't find documentation on this)
Idea 2: log filters
Using a filter RPC call is another option: i.e. start a "watcher" on a range of old blocks. I tried this, but the Infura websocket RPC I am using doesn't seem to give any response, and neither does Ganache when testing locally. Non-archive (i.e. live watching) logs work, so I known that my code is working as intended at least. (My go-ethereum Watch... generated binding call works, but does not result in responses on the output channel when specifying an old block in bind.WatchOpts.Start)
Does anyone have any suggestions on how to retrieve large amounts of log data? Or a link to other projects that tackled this problem?
I'm using Kafka's high-level consumer. Because I'm using Kafka as a 'queue of transactions' for my application, I need to make absolutely sure I don't miss or re-read any messages. I have 2 questions regarding this:
How do I commit the offset to zookeeper? I will turn off auto-commit and commit offset after every message successfully consumed. I can't seem to find actual code examples of how to do this using high-level consumer. Can anyone help me with this?
On the other hand, I've heard committing to zookeeper might be slow, so another way may be to locally keep track of the offsets? Is this alternative method advisable? If yes, how would you approach it?
You could first disable auto commit: auto.commit.enable=false
Then commit after fetching the message: consumer.commitOffsets(true)
There are two relevant settings from http://kafka.apache.org/documentation.html#consumerconfigs.
auto.commit.enable
and
auto.commit.interval.ms
If you want to set it such that the consumer commits the offset after each message, that will be difficult since the only setting is after a timer interval, not after each message. You will have to do some rate prediction of the incoming messages and accordingly set the time.
In general, it is not recommended to keep this interval too small because it vastly increases the read/write rates in zookeeper and zookeeper gets slowed down because it's strongly consistent across its quorum.
I've solved my problem by using:
consumerConfig.EnableAutoCommit = false;
after
var consumer = consumerBuilder.Consume(cancelToken.Token);
using
consumerBuilder.Commit(consumer);
I'm using:
Confluent.Kafka
for my C# client
Can someone suggest an alternative to ScriptProperties as a means of storing and retrieving short-term (textual) data.
I have a long-standing application built in GAS. Because it kicks off a number of 'threads' I store some information about each thread, ultimately to determine when all the threads have finished and thus when the application can release various 'locks'.
I'm using ScriptProperties to store this information. It used to work perfectly but has been beset by troubles over the last 6 months or so. The problems are mainly:
1. Application load times increase massively as more and more data collects in script properties (it isnt possible to clean down scriptproperty data)
2. ScriptProperties are (now) unreliable under concurrent access
3. ScriptProperties can't be read or written to in rapid bursts
Someone has suggested using ScriptDb. Can anyone confirm this (or any other) viable solution?
Many thanks
Can you have a look at CacheService ? I have used with lot of success.
CacheService is only best-effort and is not intended for guaranteed storage (but it is fast!). If you need to be sure it will be there, use ScriptDb.