Couchbase documentation says that "Disk persistence enables you to perform backup and restore operations, and enables you to grow your datasets larger than the built-in caching layer," but I can't seem to get it to work.
I am testing Couchbase 2.5.1 on a three node cluster, with a total of 56.4GB memory configured for the bucket. After ~124,000,000 100-byte objects -- about 12GB of raw data -- it stops accepting additional puts. 1 replica is configured.
Is there a magic "go ahead and spill to disk" switch that I'm missing? There are no suspicious entries in the errors log.
It does support data greater than memory - see Ejection and working set management in the manual.
In your instance, what errors are you getting from your application? When you start to reach the low memory watermark, items need to be ejected from memory to make room for newer items.
Depending on the disk speed / rate of incoming items, this can result in TEMP_OOM errors being sent back to the client - telling it needs to temporary back off before performing the set, but these should generally be rare in most instances. Details on handling these can be found in the Developer Guide.
My guess would be that it's not the raw data that is filling up your memory, but the metadata associated with it. Couchbase 2.5 needs 56 bytes per key, so in your case that would be approximately 7GB of metadata, so much less than your memory quota.
But... metadata can be fragmented on memory. If you batch-inserted all the 124M objects in a very short time, I would assume that you got at least a 90% fragmentation. That means that with only 7GB of useful metadata, space required to hold it has filled up your RAM, with lots of unused parts in each allocated block.
The solution to your problem is to defragment the data. It can either be achieved manually or triggered as needed :
manually :
automatically :
If you need more insights about why compaction is needed, you can read this blog article from Couchbase.
Even if none of your documents is stored in RAM, CouchBase still stores all the documents IDs and metadata in memory(this will change in version 3), and also needs some available memory to run efficiently. The relevant section in the docs:
http://docs.couchbase.com/couchbase-manual-2.5/cb-admin/#memory-quota
Note that when you use a replica you need twice as much RAM. The formula is roughly:
(56 + avg_size_of_your_doc_ID) * nb_docs * 2 (replica) * (1 + headroom) / (high_water_mark)
So depending on your configuration it's quite possible 124,000,000 documents require 56 Gb of memory.
Related
When I create a boot disk with gCloud less than 200GB in size, I see this error:
WARNING: You have selected a disk size of under [200GB]. This may result in poor I/O performance. For more information, see: https://developers.google.com/compute/docs/disks/persistent-disks#pdperformance.
I don't, however, see the details about this 200GB size which it alludes to being at somewhere on the page at that url.
Should I care about this warning at all? I wonder is it is more of a ploy for them to make more money trying to encourage you to lease more space?
Note: I'm using a standard disk, not a solid state. My disk access performance which is of any concern is via MySQL with very small read / writes 99% of the time, and occasionally blobs in the range of say 1 to 100 MBs.
It looks like the documentation has shifted around a little, and the warning is out of date.
There is a section of the Block Storage page that explains the relationship between persistent disk size and performance.
We'll fix the URL in gcloud.
I have a lot of data needed to stored on disk.
Since it is only key-value pairs, I want to use couchbase to do it.
The data is several GB and I only allocate 1 GB RAM to the bucket.
I though RAM to couchbase is only a cache.
But after inserting a lot of data I got:
Hard Out Of Memory Error. Bucket "test2" on node 100.66.32.169 is full. All memory allocated to this bucket is used for metadata.
when I open the couchbase web console.
Can couchbase be a database to store data on disk? Or it is RAM oritented?
Update:
OK, let me make the question more specific:
In couchbase:
If I allocate the RAM of a bucket to be 1 GB, can I store 10 GB data to that bucket?
If I can do 1. , can I consider that 1 GB RAM is a kind of cache of the 10 GB data (just like CPU L2 cache is a cache of RAM) ?
By default, Couchbase stores all keys (and some metadata) in RAM, and fills whatever remains with values. Starting with version 3.0, you can set your bucket to full-eviction mode, which only keeps the keys of cached documents in RAM. This lets you store much more data than you have memory, but at a cost to performance to some read operations, especially trying to retrieve keys that don't exist.
To solve your specific problem, edit the bucket and set it to full metadata eviction. Note that this will restart the bucket.
Couchbase tries to keep as much of the "live dataset" (ie. most used / requested keys) into the node's memory. This is key to the performance of the database, and part of the design, so good memory sizing of your nodes and quotas of your buckets are key.
It does offer persistence, but I'd say this is not a disk-first oriented database.
Persistence to disk is mainly for two things: making data durable and resilient to node shutdown (of course) and off-load data (in priority least used data) from RAM to disk.
I think you're asking a bunch of different questions here.
Specifically about the error message: looks like your bucket is simply too small to hold all the data you're storing in it.
About persisting to disc: you can force couchbase to write to disc (and even configure the number of nodes that docs are replicated into) but as noted above, that would probably hurt your performance a little.
Have a look, for example, at the persist_to flag in the set() api of the python client for couchbase.
couchbase client for python
Okay since i don't have 10 repitation I'm unable to post images, but I will try to explain in text.
I have a 7 node Couchbase (Community) cluster with 4 buckets.
Recently I've been getting spammed(constantly) by Metadata overhead warnings for one of the buckets..
The warning pops up and looks like this:
Metadata overhead warning. Over 62% of RAM allocated to bucket XXXX on node "xxx" is taken up by keys and metadata.
And I've read that it is usually a sign that the bucket needs more ram. But I don't thing that is the issue for me. I simply have a lot of metadata I would guess.
When I look at the Data Buckets tab this bucket has RAM/Quota Usage 64GB/75GB. So for me it looks that there is around 11GB(75-64GB) available.
If i look at the Bucket Analytics VBUCKET RESOURCES metrics I see that there is 59GB user data in RAM and 46GB metadata in RAM. So to my understanding there should be 105GB in RAM on a bucket that has a total of 75GB!?!
But that doesn't add up for me so clearly there is something that I don't understand here.
And yes 46GB of 75GB is around 62%. But what about the 59GB user data that is supposedly in RAM?
EDIT:
A typical document can look like this:
ID=1:CAESEA---rldZ5PhdV4msSdEchI
CONTENT=z2TjZEzkZ84=
And to my question. What do I do? Is the situation acceptable in my circumstances. If so, do I change the threshold for that warning(which I read is not recommended since the warning is set at 50% for a reason).
Or do I assign more RAM? And if so how does that help me if there is already 11GB free?
Please help me clarify these numbers and suggest if I need to take any actions.
First of all, there isn't necessarily a problem with having a high percentage of memory used by metadata - it just means there's less RAM available for caching actual documents. If your application is working well then it may be fine for your use-case. However, having said that let me try and address your questions on it, and what to change if you do want to improve things:
If i look at the Bucket Analytics VBUCKET RESOURCES metrics I see that there is 59GB user data in RAM and 46GB metadata in RAM. So to my understanding there should be 105GB in RAM on a bucket that has a total of 75GB!?!
IIRC "user data in RAM" is inclusive of "metadata in RAM" - so you have a total of 59 GB data used, of which 46 GB is metadata.
And to my question. What do I do? Is the situation acceptable in my circumstances. If so, do I change the threshold for that warning(which I read is not recommended since the warning is set at 50% for a reason).
Or do I assign more RAM? And if so how does that help me if there is already 11GB free?
So basically you are storing lots of very small documents, so the per-document metadata overhead (~48 bytes plus the length of the key) is very high compared to the actual document size.
The 11GB free is mainly made up of the difference between the bucket quota and the high watermark.
Here are a few options to improve this:
Allocate more RAM to the bucket (as you mentioned) - if there's any unallocated in the Server Quota.
Add more memory to the nodes (and allocate to the server quota and bucket).
Reduce the number of replicas (if that's acceptable to you) - at the moment you are essentially storing each object (and it's metadata) three times - once for the active vBuckets and twice for the two replica vBucket sets.
Change your documents to have shorter keys - This will reduce the average metadata per document.
Consolidate multiple documents into one - This will reduce the number of documents, and hence the overall metadata overhead.
I had a cluster which inculdes three nodes. We created a bucket inside and set the number of bucket replicas to be 2. Besides the RAM quota is set to be 10G per node, that is, the total RAM quota is 30G.
I used client-side to save data into this bucket. Hours later, the client-side printed Temporary failure error. and Couchbase web console showed that the bucket RAM reached 29G.Repeated data compression but the RAM didn't reduce anymore.
My questions is organized as follows.
1, I guess the key in bucket can only be saved into the RAM but not in hardware, right or wrong?
2,Wheter the 29G data, which can not be compressed into hardware ,is key or not?
3,Wheter each node that saves others node's replica information is saved in hardware or not? If not, how could it be saved.
4,Every time the client-side saves data, it will make use of hash function to evaluate vbucket in order to judge which nodes that the data will be saved in. Is the process carried on the client-side?
In response to your specific questions:
1, I guess the key in bucket can only be saved into the RAM but not in hardware, right or wrong?
If by hardware you mean disk; then yes, currently Couchbase must hold all document keys (along with some additional metadata) in RAM. This is to ensure that any request for a key can be answered immediately, both in the positive ("yes, this key exists and here's it's value) and the negative ("no, such a key doesn't exist.)"
2,Wheter the 29G data, which can not be compressed into hardware ,is key or not?
Some of this is probably the metadata. If you go to the Bucket tab and display it's statistics by clicking on it's name, you can see the amount of memory used - specifically under the VBucket Resources tab to see how much is used for metadata and user data. See the Couchbase Admin Guide - Viewing Bucket and cluster statistics for more details.
3,Wheter each node that saves others node's replica information is saved in hardware or not? If not, how could it be saved.
The replica metadata is also always kept in RAM, but the replica values (like active values) can be ejected to disk to free up memory.
4,Every time the client-side saves data, it will make use of hash function to evaluate vbucket in order to judge which nodes that the data will be saved in. Is the process carried on the client-side?
Yes the vbucket hashing is done on the client - see the Architecture and Concepts - Vbuckets section in the Admin guide.
In general you may want to review the Sizing chapter in the Admin guide to determine how much of you memory is being used for storing key metadata - specifically the Memory Sizing section. The exact calculation depends on the version of Couchbase (and so I won't duplicate here).
Data to be cached:
100 Gb data
Objects of size 500-5000 bytes
1000 objects updated/inserted in average per minute (peak 5000)
Need suggestion for Coherence topology in production and test (distributed with backup)
number of servers
nodes per server
heap size per node
Questions
How much free available memory is needed per node compared to memory used by cached data (assume 100% usage is not possible)
How much overhead will 1-2 additional indexes per cache element generate?
We do not know how many read operations will be done, the solution will be used by clients where low response times are critical (more than data consistency) and depend on each use-case. The cache will be updated from DB by polling at a fixed frequency and populating the cache (since cache is data master, not the system using the cache).
The rule of thumb for sizing a JVM for Coherence is that the data is 1/3 the heap assuming 1 backup: 1/3 for cache data, 1/3 for backup, and 1/3 for index and overhead.
The biggest difficulty in sizing is that there are no good ways to estimate index sizes. You have to try with real-world data and measure.
A rule of thumb for JDK 1.6 JVMs is start with 4GB heaps, so you would need 75 cache server nodes. Some people have been successful with much larger heaps (16GB), so it is worth experimenting. With large heaps (e.g, 16GB) you should not need as much as 1/3 for overhead and can hold more than 1/3 for data. With heaps greater than 16GB, garbage collector tuning becomes critical.
For maximum performance, you should have 1 core per node.
The number of server machines depends on practical limits of manageability, capacity (cores and memory), and failure. For example, even if you have a server that can handle 32 nodes, what happens to your cluster when a machine fails? The cluster will be machine safe (backups are not on the same machine) but recovery would be very slow given the massive amount of data to be moved to new backups. On the other hand 75 machines is hard to manage.
I've seen Coherence have latencies of 250 micro seconds (not millis) for a 1K object put, including network hops and backup. So, the number of inserts and updates you are looking for should be achievable. Test with multiple threads inserting/updating and make sure your test client is not the bottleneck.
A few more "rules of thumb":
1) For high availability, three nodes is a good minimum.
2) With Java 7, you can use larger heaps (e.g. 27GB) and the G1 garbage collector.
3) For 100GB of data, using David's guidelines, you will want 300GB total of heap. On servers with 128GB of memory, that could be done with 3 physical servers, each running 4 JVMs with 27GB heap each (~324GB total).
4) Index memory usage varies significantly with data type and arity. It is best to test with a representative data set, both with and without indexes, to see what the memory usage difference is.