This document from Couchbase says:
The hashing function used by Couchbase Server to map keys to vBuckets
is configurable – both the hashing algorithm and the output space (the total number of vBuckets output by the function).
However, I did not find any way to configure it. In particular I'm interested in the number of vBuckets, not the hashing algorithm. Could someone please help.
Thank you!
If you just want to make the rebalance to go faster, you don't need to change the number of vBuckets itself, you can just change the number of vBuckets moved concurrently during the rebalance. By default, rebalance moves 1 vBucket at a time.
Here is how you can change this setting: http://docs.couchbase.com/admin/admin/Tasks/rebalance-behind-the-scenes.html
If you have views defined, you can also disable consistent view querying for the duration of the rebalance to speed it up. Remember to turn it back on after the rebalance finishes! http://docs.couchbase.com/admin/admin/REST/rest-cluster-disable-query.html
Related
I want to set up a complete server (apache, mysql 5.7) as a fallback of a productive server.
The synchronization on file level using rsync and cronjob is already done.
The mysql-replication is currently the problem. More precisely: the choice of the right replica method.
Multi primary group replication seemed to be the most suitable method so far.
In case of a longer production downtime, it is possible to switch to the fallback server quickly via DNS change.
Write accesses to the database are possible immediately without adjustments.
So far so good: But, if the fallback-server fails, it is in unreachable status and the production-server switches to read only, since its group no longer has the quota. This is of course a no-go.
I thought it might be possible using different replica variables: If the fallback-server is in unreachable state for a certain time (~5 minutes), the production-server should stop the group_replication and start a new group_replication. This has to happen automatically to keep the read-only time relatively low. When the fallback-server is back online, it should be manually added to the newly started group. But if I read the various forum posts and documentation correctly, it's not possible that way. And running a Group_Replication with only two nodes is the wrong decision anyway.
https://forums.mysql.com/read.php?177,657333,657343#msg-657343
Is the master - slave replication the only one that can be considered for such a fallback system? https://dev.mysql.com/doc/refman/5.7/en/replication-solutions-switch.html
Or does the Group_Replication offer possibilities after all, if you can react suitably to the quota problem? Possibilities that I have overlooked so far.
Many thanks and best regards
Short Answer: You must have [at least] 3 nodes.
Long Answer:
Split brain with only two nodes:
Write only to the surviving node, but only if you can conclude that it is the only surviving node, else...
The network died and both Primaries are accepting writes. This to them disagreeing with each other. You may have no clean way to repair the mess.
Go into readonly mode with surviving node. (The only safe and sane approach.)
The problem is that the automated system cannot tell the difference between a dead Primary and a dead network.
So... You must have 3 nodes to safely avoid "split-brain" and have a good chance of an automated failover. This also implies that no two nodes should be in the same tornado path, flood range, volcano path, earthquake fault, etc.
You picked Group Replication (InnoDB Cluster). That is an excellent offering from MySQL. Galera with MariaDB is an equally good offering -- there are a lot of differences in the details, but it boils down to needing 3, preferably dispersed, nodes.
DNS changes take some time, due to the TTL. A proxy server may help with this.
Galera can run in a "Primary + Replicas" mode, but it also allows you to run with all nodes being read-write. This leads to a slightly different set of steps necessary for a client to take to stop writing to one node and start writing to another. There are "Proxys" to help with such.
FailBack
Are you trying to always use a certain Primary except when it is down? Or can you accept letting any node be the 'current' Primary?
I think of "fallback" as simply a "failover" that goes back to the original Primary. That implies a second outage (possibly briefer). However, I understand geographic considerations. You may want your main Primary to be 'near' most of your customers.
I recommend using the Galera MySQL cluster with HAProxy as a load balancer and automatic failover solution. we have used it in production for a long time now and never had serious problems. The most important thing to consider is monitoring the replication sync status between nodes. also, make sure your storage engine is InnoDB because Galera doesn't work with MyISAM.
check this link on how to setup :
https://medium.com/platformer-blog/highly-available-mysql-with-galera-and-haproxy-e9b55b839fe0
But in these kinds of situations, the main problem is not a failover mechanism because there are many solutions out of the box, but rather you have to check your read/write ratio and transactional services and make sure replication delays won't affect them. some times vertically scalable solutions with master-slave replication are more suitable for transaction-sensitive financial systems and it really depends on the service your providing.
I took over a project some time ago, where file binaries were stored as BLOBs. Those were from sizes of 0.5-50 Mb, therefore that table was touched as less as possible (-> eBeans lazyloading etc). The BLOB stuff worked quite fine as long as the whole system was running on one dedicated server, once we switched to AWS EC2 instances + RDS, things were (obviously) slower.
Therefore I switched the data storage from BLOB to S3 (+ reference to the bucket/key stored in the DB), which is much faster for our backend and our clients.
Now my question is, obviously the programmer before set up the mySQL DB to handle bigger chunks of data (max packagesize etc), and I also stumbled over some discussion about connection pool size.
What are critical parameters to check in the mySQL setup, and what are effective ways of evaluating them?
The most likely answer to your question is "change nothing."
MySQL has many, many, many "tunable" parameters, and there is an absolute wealth of bad advice available online about "optimizing" them. But this is a temptation best avoided.
To the extent that system variables have been changed from defaults, if you ever find yourself in a situation where you believe tweaking the configuration is necessary, your first instinct should be to revert settings to their defaults unless you have a specific and justified reason not to.
Settings like max_allowed_packet if set too small, will break some things (like large blobs) but if set larger than necessary will have little or no impact... the "excess" isn't allocated or otherwise harmful. In the case of max_allowed_packet, this does impose a constraint on memory use by limiting the amount of memory the server would ever need to allocate for a single packet, but since it's a brick wall limit, you don't necessarily want to shrink it. If you aren't sending packets that big, it isn't hurting anything.
It is safe to increase the value of this variable because the extra memory is allocated only when needed. For example, mysqld allocates more memory only when you issue a long query or when mysqld must return a large result row. The small default value of the variable is a precaution to catch incorrect packets between the client and server and also to ensure that you do not run out of memory by using large packets accidentally.
http://dev.mysql.com/doc/refman/5.7/en/packet-too-large.html
Other parameters, though, can have dramatically counter-intuitice negative effects because the range of "valid" values is a superset of the range of "optimal" values. The query cache is a prime example of this. "But it's more cache! How can that be bad?!" Well, a bigger house increases the amount of housework you have to do, and the query cache is a big house with only one tiny broom (a global mutex that each thread contends for when entering and leaving).
Still others, like innodb_buffer_pool_size only really have one relatively small optimal range of values for a given server. Too small will increase disk I/O and impair performance because the pool is smaller than the system could support, too large will increase disk I/O due to the server using swap space or crash it entirely by exhausting the system of every last available kilobyte of free RAM.
Perhaps you get the idea.
Unless you have a specific parameter that you believe may be suboptimally configured, leave a working system working. If you change things, change them one at a time, and prove or disprove that each change was a good idea before proceeding. If you are using a non-default value, consider the default as a potentially good candidate value.
And stay away from "tuning scripts" that make suggestions about parameters you should change. Those are interesting to look at, but their advice is often dangerous. I've often thought about writing my own one of these, but all it would do is check for values not set to the default and tell the user to explain themselves or set it back. :) Maybe that would catch on.
When I query a view from a Couchbase cluster,
will each machine return the same result for the view query?
I would expect the cluster to return the same response regardless of which machine actually responds to the request.
How critically does your application depend upon consistent view results? Remember that Couchbase indices are eventually consistent, meaning that the clusters will not be updated all at the same time, especially when there is a high volume of data changes. So, for data that has been around for a while, you could expect consistent result sets between machines; however, data that has very recently changed may not be reflected in the latest view query. The key is to design your application to deal with this case.
The storage of Couchbase views is spread across the cluster nodes, just like the data is. For example on a three node cluster and with one view, 1/3 of the view would be on each node of the cluster. The part of the view on each cluster is the part that corresponds to the data in the vBuckets on that node. So when you query a Couchbase view, it goes to each node to work on that query. This is all transparent to you as you are using the SDK, but that is what is happening in the background. When a rebalance happens, the views change too since the vBuckets are moving. By default IndexAwareRebalances are set to true.
You also have to realize how often views are updated. By default, it is every 5 seconds AND if there are 5000 data mutations. Not or. These defaults can be tuned, but if for example you only have 1000 mutations, the indexer will not run on its own. Be careful with that stale=false thing too. It can get you into real trouble using it all of the time, especially during rebalances.
Also, know that Couchbase is strongly consistent, but views are eventually consistent.
I had a cluster which inculdes three nodes. We created a bucket inside and set the number of bucket replicas to be 2. Besides the RAM quota is set to be 10G per node, that is, the total RAM quota is 30G.
I used client-side to save data into this bucket. Hours later, the client-side printed Temporary failure error. and Couchbase web console showed that the bucket RAM reached 29G.Repeated data compression but the RAM didn't reduce anymore.
My questions is organized as follows.
1, I guess the key in bucket can only be saved into the RAM but not in hardware, right or wrong?
2,Wheter the 29G data, which can not be compressed into hardware ,is key or not?
3,Wheter each node that saves others node's replica information is saved in hardware or not? If not, how could it be saved.
4,Every time the client-side saves data, it will make use of hash function to evaluate vbucket in order to judge which nodes that the data will be saved in. Is the process carried on the client-side?
In response to your specific questions:
1, I guess the key in bucket can only be saved into the RAM but not in hardware, right or wrong?
If by hardware you mean disk; then yes, currently Couchbase must hold all document keys (along with some additional metadata) in RAM. This is to ensure that any request for a key can be answered immediately, both in the positive ("yes, this key exists and here's it's value) and the negative ("no, such a key doesn't exist.)"
2,Wheter the 29G data, which can not be compressed into hardware ,is key or not?
Some of this is probably the metadata. If you go to the Bucket tab and display it's statistics by clicking on it's name, you can see the amount of memory used - specifically under the VBucket Resources tab to see how much is used for metadata and user data. See the Couchbase Admin Guide - Viewing Bucket and cluster statistics for more details.
3,Wheter each node that saves others node's replica information is saved in hardware or not? If not, how could it be saved.
The replica metadata is also always kept in RAM, but the replica values (like active values) can be ejected to disk to free up memory.
4,Every time the client-side saves data, it will make use of hash function to evaluate vbucket in order to judge which nodes that the data will be saved in. Is the process carried on the client-side?
Yes the vbucket hashing is done on the client - see the Architecture and Concepts - Vbuckets section in the Admin guide.
In general you may want to review the Sizing chapter in the Admin guide to determine how much of you memory is being used for storing key metadata - specifically the Memory Sizing section. The exact calculation depends on the version of Couchbase (and so I won't duplicate here).
I just started using couchbase and hoping to use it as my data store.
One of my requirements in performing a query that will return a certain field about all the documents in the store. This query is done once at the server startup.
For this purpose I need all the documents that exist and can't miss any of them.
I understand that views in couchbase are eventually consistent but I still hope this query can be done (at the cost of performance).
Notes about my configurating:
I have only one couchbase server instance (I dont need sharding or
replication)
I am using the java client (1.4.1)
What I have tried to do is saving my documents this way:
client.set(key, value, PersistTo.ONE).get();
And querying using:
query.setStale(Stale.FALSE);
Adding the PersistTo parameter caused the following exception:
Cause by: net.spy.memcached.internal.CheckedOperationTimeoutException: Timed out waiting for operation - failing node: <unknown>
at net.spy.memcached.internal.OperationFuture.get(OperationFuture.java:167)
at net.spy.memcached.internal.OperationFuture.get(OperationFuture.java:140)
So I guess I am actually asking 3 questions:
Is it possible to get the consistent results I need?
If so, is what I suggested the correct way of doing that?
How can I prevent those exceptions?
The mapping I'm using:
function (doc,meta) {
if (doc.doc_type && doc.doc_type == "MyType" && doc.myField) {
emit(meta.id,null);
}
}
Thank you
Is it possible to get the consistent results I need?
Yes it is possible to set Couchbase views to be consistent by setting the STALE flag to false as you've done. However there are performance impacts with this, so dependent on your data size the query may be slow, if you are only going to be doing it once a day then it should be ok.
Couchbase is designed to be a distributed system comprising of more than node, it's not really suitable for single node deployments. I have read (but can't find the link) that view performances are much better in larger clusters.
You are also forcing more of a sync processing model onto a system that shines with async requests, PersistTo is ok to use for some requests but not system wide on every call (personal opinion), it'll definitely throttle throughput and performance.
If so, is what I suggested the correct way of doing that?
You say the query is done after your application server is running, is this once per day or more? If once a day then your application should work (I'd consider upping the nodes ;)), if you have to do this query a lot and you are 'hammering' the node over and over with sets then I'd expect to see what you are currently experiencing.
How can I prevent those exceptions?
It could be a variety of reasons, what are the specs of your computer, RAM,CPU,DISK? How much ram is allocated to Couchbase, how much to your bucket, what % of the bucket ram is used?
I've personally seen this when I've hammered some lower end AWS instances on some not so amazing networks. What version of Couchbase are you using? It could be a whole variety of factors that and deserves to be a separate question.
Hope that helps!
EDIT regarding more information on the Stale = false parameter (from official docs)
http://docs.couchbase.com/couchbase-manual-2.2/#couchbase-views-writing-stale
The index is updated before the query is executed. This ensures that any documents updated (and persisted to disk) are included in the view. The client will wait until the index has been updated before the query has executed, and therefore the response will be delayed until the updated index is available.