Official documentation for distributed map says 'EAGER means load is blocked till all partitions are loaded.' Does it mean any operation with distributed map is blocked until all partitions for that map is loaded?
My project has several pretty big storages and initial loading from DB takes roughly 20 mins. It would be practically unnecessary to start all scheduled tasks to run until all partitions are fully loaded. Is there any wan to get notified when all partitions are up and ready?
Yes, that's what it means. No operation for a partition will be executed if the partition has not been loaded yet. The operations will be parked and retried.
Load for each partition is independent from one another so if load for partitionId == 1 is done, and you do map.get(key), for a key that is handled by partitionId == 1, this operation will succeed, even though partitionId == 2 is still loading the values. If you want to block until all partitions are loaded on a calling party you can just invoke map.size() it needs all partitions to be loaded, so it won't finish till everything has been loaded.
Related
My Rails application takes a JSON blob of ~500 entries (from an API endpoint), throws it into a sidekiq/ redis background queue. The background job parses the blob then loops through the entries to perform a basic Rails Model.find_or_initialize_by_field_and_field() and model.update_attributes().
If this job were in the foreground, it would take a matter of seconds (if that long). I'm seeing these jobs remain in the sidekiq queue for 8 hours. Obviously, something's not right.
I've recently re-tuned the MySQL database to use 75% of available RAM as the buffer_pool_size and divided that amongst 3 buffer pools. I originally thought that might be part of the deadlock but the load avg on the box is still well below any problematic status ( 5 CPU and a load of ~ 2.5 ) At this point, I'm not convinced the DB is the problem though, of course, I can't rule it out.
I'm sure, at this point, I need to scale back the sidekiq worker instances. In anticipation of the added load I increased the concurrency to 300 per worker (I have 2 active workers on different servers.) Under a relatively small amount of load there queues operate as expected; even the problematic jobs are completed in ~1 minute. Though, per the sidekiq documentation >50 concurrent workers is a bad idea. I wasn't having any stability issues at 150 workers per instance. The problem has been this newly introduced job that performs ~500 MySQL finds and updates.
If this were a database timeout issue, the background job should have failed and been moved from the active (busy) queue to the failed queue. That's not the case. They're just getting stuck in the queue.
What other either MySQL or Rails/ sidekiq tuning parameters should I be examining to ensure these jobs succeed, fail, or properly time out?
I'm new to couchbase and was wondering if very frequent updates to a single document (possibly every second) will cause all updates to pass through the disk write queue, or only the last update made to the document?
In other words, does couchbase optimize disk writes by only writing the document to disk once, even if updated multiple time between writes.
Based on the docs, http://docs.couchbase.com/admin/admin/Monitoring/monitor-diskqueue.html, it sounds like all updates are processed. If anyone can confirm this, I'd be grateful.
thanks
Updates are held in a disk queue before being written to disk. If a write to a document occurs and a previous write is still in the disk queue, then the two writes will be coalesced, and only the more recent version will actually be written to disk.
Exactly how fast the disk queue drains will depend on the storage subsystem, so whether writes to the same key get coalesced will depend on how quick the writes come in compared to the storage subsystem speed / node load.
Jako, you should worry more about updates happening in the millisecond time frame or more than one update happening in 1 (one) millisecond. The disc write isn't the problem, Couchbase solves this intelligently itself but the fact that you will concurrency issues when you operate in the milliseconds time frame.
I've run into them fairly easily when I tested my application and first couldn't understand why Node.js (in my case) sometimes would write data to CouchBase and sometimes not. If it didn't write to CouchBase usually for the first record.
More problems raised when I first checked if a document with a specific key existed, upon not existing I would try to write it to CouchBase only to find out that in the meantime an early callback had finished and now there was indeed a key for the same document.
In those case you have to operate with the CAS flag and program it iteratively so that your app is continuously trying to pull the right document for that key and then updates. Keep this in mind especially when running tests and updates to the same document is being done!
I am trying to decide between holding static data (gets updated on a nightly basis, not real-time) in a database or in flat JSON files to supply a Node.js application. In preliminary tests the flat file method is twice as fast. My question is about the issue of memory when using the file method.
If my Node.js app reads the data from the file and then does JSON.parse and passes the object to the template to render... does the in-memory size of that data get duplicated with each user connection?
i.e. if the data file is 1MB and there are 1000 concurrent users, does it consume 1000MB of server memory during that period?
Each connection runs separately so if you have 1000 concurrent users, they aren't really running their request all at the same time because node.js is single threaded. It runs one single connection until it either finishes or until it hits a non-blocking operation such as async I/O. Assuming you are using async file I/O, you could have a few connections in process at the same time, but as soon as one finishes, its memory use will be returned to the system by the garbage collector.
Your operation sounds ideal for an in-memory cache. You can decide what lifetime works best for the cache, but you could load the JSON, store it in memory, set an expiration time 10 minutes from now and as long as the current time is not greater than the expiration time, you just return the result from the cache with no disk access. Thus, you'd only ever retrieve the data from disk once every 10 minutes max and the data would be returned even faster and the average memory used per request would be significantly lower.
The only downside to this cache approach is that when the data is updated real-time, it could take up to 10 minutes (on average 1/2 the cache time or 5 minutes) for the cached data to expire and the new data to be returned. Since this only happens once nightly, it may not be a big deal to you, but there are ways to deal with that issue if you want to. For example, you can check the file date/time of the data file on every request and if it hasn't changed since the last time, then you just keep using your cached version of the data. When it does change, you read it from the disk and replace the cached version. This adds an extra disk I/O operation on each request, but guarantees that the user always gets the latest version while still allowing for the benefits of a cached version that only has to be newly read into memory when the data has actually changed.
One other thing to consider. If the data is 1MB and you're generating a giant HTML file from that, your page rendering may be where the largest memory consumption is since expanding a large data structure into HTML can often make it 10-20x larger and how well your rendering engine does with memory consumption depends entirely on the rendering engine.
If there is no per-user customization in the HTML or anything else in the rendered HTML that varies from one rendering to the next (as long as the JSON hasn't changed), you might want to actually cache the rendered HTML so all you have to do is stream it to each request.
We have a service which sees several hundred simultaneous connections throughout the day, peeking at about 2000, for about 3 million hits a day, and growing. With each request I need to log 4 or 5 pieces of data to MySQL, we originally used the logging that came with the app were using however it was terribly inefficient and would run my db server at >3x the average cpu load, and would eventually bring the server to it knees.
At this point we are going to add our own logging to the application (php), the only option I have for logging data is the MySQL db, as this is the only common resource available to all of the http servers. This data will be mostly writes however everyday we generate reports based on the data, then crunch and archive the old data.
What recommendations can be made to ensure that I don't take down our services with logging data?
The solution we took with this problem was to create an archive table then regularly ( every 15 minutes, on an app server) crunch the data and put it back into the tables that were used to generate reports. The archive table of course did not have any indices, the tables which the reports are generated from have several indices.
Some stats on this approach:
Short Version: >360 times faster
Long Version:
The original code/model did direct inserts into the indexed table, and the average insert took .036 seconds, using the new code/model inserts took less than .0001 seconds (I was not able to get an accurate fix on the insert time I had to measure 100,000 inserts and average for the insert time). The post-processing (crunch) took an average 12 seconds for several tens-of-thousands records. Overall we were greatly pleased with this approach and so far it has worked incredibly well for us.
Based on what you describe, I recommend you try to leverage the fact that you don't need to read this data immediately and pursue a "periodic bulk commit route". That is, buffer the logging data in RAM on the app servers and doing periodic bulk commits. If you have multiple application nodes, some sort of randomized approach would help even more (e.g., commit updated info every 5 +/- 2 minutes).
The main drawback with this approach is that if an app server fails, you lose the buffered data. However, that's only bad if (a) you absolutely need all of the data and (b) your app servers crash regularly. Small chance that both are true, but in the event they are, you can simply persist your buffer to local disk (temporarily) on an app server if that's really a concern.
The main idea is:
buffering the data
periodic bulk commits (leveraging some sort of randomization in a distributed system would help)
Another approach is to stop opening and closing connections if possible (e.g., keep longer lived connections open). While that's likely a good first step, it may require a fair amount of work on your part on a part of the system that you may not have control over. But if you do, it's worth exploring.
is there any software that does "lazy" deletion of the rows from the table. I would like to do maintenance of my tables when my server is idle, and ideally i should be able to define what "idle" is (num of database connections/system load/ requests per second). Is there anything remotely similar to this?
If you are on a linux server, you can make your table cleanup scripts only run based on the output of the command "w" which will show you a system load. If your system load is under say .25 you can run your script. Do this with shell scripting.
To some degree, from an internal perspective InnoDB already does this. Rows are initially marked as deleted, but only made free as part of a background operation.
My advice: You can get in to needlessly complicated problems if you try and first check if the server is idle. i.e.
What if it was idle, but the cleanup takes 2 minutes. During that 2 minutes the server load peaks?
What if the server never becomes idle enough? Now you just have an unlimited backlog.
If you just background the task you might improve performance enough, since now at least no users will be sitting in front of web pages waiting for it to complete. Look at activity graphs as to what is the best time to schedule it (3am, 5am etc).