If I store an array-like structure in couchbase like this one:
mykey = 3
key_1 = 47
key_2 = 11
key_3 = 17
and my update procedure is something like this:
a = increment(mykey)
set key_a = 42
will this work on a bucket replicated across multiple datacenters? Is there a better way of doing this?
I'm thinking that two clients on different data centers might call increment at the same time, getting the same value, and then setting the same key to different values.
If you're using XDCR, situations with duplicate keys are possible. And even if you check value before set (using couchbase.add operation) it can also produce two identical keys.
Within a cluster, Couchbase Server provides strong consistency at the
document level. On the other hand, XDCR also provides eventual
consistency across clusters. Built-in conflict resolution will pick
the same “winner” on both the clusters if the same document was
mutated on both the clusters. If a conflict occurs, the document with
the most updates will be considered the “winner.” If the same document
is updated the same number of times on the source and destination,
additional metadata such as numerical sequence, CAS value, document
flags and expiration TTL value are used to pick the “winner.” XDCR
applies the same rule across clusters to make sure document
consistency is maintained.
To avoid this couchbase recommends to store some info about datacenter/cluster or use unique keys like GUIDs. I think that the second way is not preferred, so you can implement the first one by adding datacenter location as key prefix and handle them on application side:
US-east.mykey_1
US-west.mykey_1
Related
RDBMSes have tables; also, similar concepts exist in NoSQL, like Kinds in Google Datastore. But Couchbase puts everything into one big namespace. How do I arrange my data in a table-like way?
I want the performance advantages of table-like namespacing. If I have 1,000,000 rows of one type, and 10 rows of another, I'd rather that the query engine not have to look through 1,000,010 rows to find one of those ten.
Buckets are available, but only up to ten. So, these are not really table-like.
Tables could be implemented on the application layer with a type or kind property in each JsonDocument. But this mixes different abstraction layers: metadata with data.
You can prefix each key with a "Table"-like name. "User:111" instead of 111.
How can I achieve the benefits of Tables/Kinds in Couchbase?
Currently, the correct way to do this is to add an attribute which represents the type of the document, and then create indexes with your "type" attribute in it. So your query will scan directly the index instead of a full table scan. This might sound uncommon at first, but indexes are one of the most powerful features in CB.
You can see if your query is using the index you have created in the "Plan" tab of the web console:
https://blog.couchbase.com/couchbase-5-5-enhanced-query-plan-visualization/
If you are using Spring Data, it is done automatically or you through the attribute "_class" https://blog.couchbase.com/couchbase-spring-boot-spring-data/
Creating multiple buckets for this use case isn't a good strategy, as you will need some extra work whenever you need to make a join.
There are some metadata about the document which you can access via meta() in your query (ex: meta().id, meta().cas) but the type itself has to stay as a top-level attribute of the document.
You can prefix each key with a "Table"-like name. "User:111" instead of 111. -> This is useful when you need to filter wich documents should be replicated via Cross Data Center Replication https://blog.couchbase.com/deep-dive-cross-data-center-replication-xdcr/
Our DB is mostly reads, but we want to add a "View count" and "thumbs up/thumbs down" to our videos.
When we stress tested incrementing views in mysql, our database started deadlocking.
I was thinking about handling this problem by having a redis DB that holds the view count, and only writes to the DB once the key expires. But, I hear the notifications are not consistent, and I don't want to lose the view data.
Is there a better way of going about this? Or is the talk of redis notifications being inconsistent not true.
Thanks,
Sammy
Redis' keyspace notifications are consistent, but delivery isn't guaranteed.
If you don't want to lose data, implement your own background process that manually expires the counters - i.e. copies to MySQL and deleted from Redis.
There are several approaches to implementing this lazy eviction pattern. For example, you can use a Redis Hash with two fields: a value field that you can HINCRBY and a timestamp field for expiry logic purposes. Your background process can then SCAN the keyspace to identify outdated keys.
Another way is to use Sorted Sets to manage the counters. In some cases you can use just one Sorted Set, encoding both TTL and count into each member's score (using the float's integer and fractional parts, respectively), but in most cases it is simpler to use two Sorted Sets - one for TTLs and the other fur values.
I decided to use a MySQL Cluster for a bigger project of mine. Beside storing documents in a simple table scheme with only three indexes, a need to store information in the size of 1MB to 50MB arise. Those informations will be serialized custom tables being aggregats of data feeds.
How will be those information be stored and how many nodes will those information hit? I understand that with a replication factor of three those information will be written three times and I understand that there are coordinator nodes (named differently) so I ask myself what will be the impact storing those information?
Is it right that I understand that for a read a cluster will send those blobs to three servers (one requested the information, one coordinator and one data server) and for a write it is 5 (1+1+3)?
Generally speaking MySQL only supports NoOfReplicas=2 right now, using 3 or 4 is generally not supported and not very well tested, this is noted here:
http://dev.mysql.com/doc/refman/5.6/en/mysql-cluster-ndbd-definition.html#ndbparam-ndbd-noofreplicas
"The maximum possible value is 4; currently, only the values 1 and 2 are actually supported."
As also described in the above URL, the data is stored with the same number of replicas as this setting. So with NoOfReplicas=2, you get 2 copies. These are stored on the ndbd (or ndbmtd) nodes, the management nodes (ndb_mgmd) act as co-ordinators and the source of configuration, they do not store any data and neither does the mysqld node.
If you had 4 data nodes, you would have your entire dataset split in half and then each half is stored on 2 of the 4 data nodes. If you had 8 data nodes, your entire data set would be split into four parts and then each part stored on 2 of the 8 data nodes.
This process is sometimes known as "Partitioning". When a query runs, the data is split up and sent to each partition which processes it locally as much as possible (for example by removing non-matching rows using indexes, this is called engine condition pushdown, see http://dev.mysql.com/doc/refman/5.6/en/condition-pushdown-optimization.html) and then it is aggregated in mysqld for final processing (may include calculations, joins, sorting, etc) and return to the client. The ndb_mgmd nodes do not get involved in the actual data processing in any way.
Data is by default partitioned by the PRIMARY KEY, but you can change this to partition by other columns. Some people use this to ensure that a given query is only processed on a single data node much of the time, for example by partitioning a table to ensure all rows for the same customer are on a single data node rather than spread across them. This may be better, or worse, depending on what you are trying to do.
You can read more about data partitioning and replication here:
http://dev.mysql.com/doc/refman/5.6/en/mysql-cluster-nodes-groups.html
Note that MySQL Cluster is really not ideal for storing such large data, in any case you will likely need to tune some settings and try hard to keep your transactions small. There are some specific extra limitations/implications of using BLOB which you can find discussed here:
http://dev.mysql.com/doc/mysql-cluster-excerpt/5.6/en/mysql-cluster-limitations-transactions.html
I would run comprehensive tests to ensure it is performing well under high load if you go ahead and ensure you setup good monitoring and test your failure scenarios.
Lastly, I would also strongly recommend getting pre-sales support and a support contract from Oracle, as MySQL Cluster is quite a complicated product and needs to be configured and used correctly to get the best out of it. In the interest of disclosure, I work for Oracle in MySQL Support -- so you can take that recommendation as either biased or very well informed.
I'm implementing a sequence generator for my database, via Grails. I've defined my domain class, and I want to specify a sequence. At present, I'm using:
static mapping = {
id generator: 'uuid'
version false
}
But this is generating long, 128 bit IDs, which I'm conscious that the user might have a hard time using. To combat this, I decided that it might be better to use normal, incrementing IDs, so I found this resource, informing me about the various options I had for preconfigured sequence generators.
I had a look at increment, and found this description:
increment -
generates identifiers of type long, short or int that are unique only when no other process is inserting data into the same table. Do not use in a cluster.
I have one Grails application inserting data, but several users may be inputting data at the same time. As I understand it, Grails (like a normal Servlet) will assign a new thread to each request made from the user. Does this mean then, that increment is not a good fit, because even though there is only one application, there will be multiple threads attempting to insert?
If increment is not a good fit, what other options do I have?
If increment is not a good fit, what other options do I have?
As IsidroGHIf stated in the comment if you don't specify id generator GORM will by default choose the native strategy to generate ids, which in case of MySQL are auto-incrementing columns.
They are definitely thread-safe, they also work in a cluster (compared to increment) and they will by default start with 1, incrementing also by 1 so it won't be long 128 bit IDs.
All threads running in a single instance application are executed in the same process so using increment is thread safe.
In this case, the Hibernate session associated to the process (associated to a thread running in the process) manages these IDs and assures uniqueness.
Anyway, the increment mode is normally (at least in my case) used for testing and/or early stage development phases.
If your database supports sequences, I'd probably use the sequence mode.
Here's our mission:
Receive files from clients. Each file contains anywhere from 1 to 1,000,000 records.
Records are loaded to a staging area and business-rule validation is applied.
Valid records are then pumped into an OLTP database in a batch fashion, with the following rules:
If record does not exist (we have a key, so this isn't an issue), create it.
If record exists, optionally update each database field. The decision is made based on one of 3 factors...I don't believe it's important what those factors are.
Our main problem is finding an efficient method of optionally updating the data at a field level. This is applicable across ~12 different database tables, with anywhere from 10 to 150 fields in each table (original DB design leaves much to be desired, but it is what it is).
Our first attempt has been to introduce a table that mirrors the staging environment (1 field in staging for each system field) and contains a masking flag. The value of the masking flag represents the 3 factors.
We've then put an UPDATE similar to...
UPDATE OLTPTable1 SET Field1 = CASE
WHEN Mask.Field1 = 0 THEN Staging.Field1
WHEN Mask.Field1 = 1 THEN COALESCE( Staging.Field1 , OLTPTable1.Field1 )
WHEN Mask.Field1 = 2 THEN COALESCE( OLTPTable1.Field1 , Staging.Field1 )
...
As you can imagine, the performance is rather horrendous.
Has anyone tackled a similar requirement?
We're a MS shop using a Windows Service to launch SSIS packages that handle the data processing. Unfortunately, we're pretty much novices at this stuff.
If you are using SQL Server 2008, look into the MERGE statement, this may be suitable for your Upsert needs here.
Can you use a Conditional Split for the input to send the rows to a different processing stage dependent upon the factor that is matched? Sounds like you may need to do this for each of the 12 tables but potentially you could do some of these in parallel.
I took a look at the merge tool, but I’m not sure it would allow for the flexibility to indicate which data source takes precedence based off of a predefined set of rules.
This function is critical to allow for a system that lets multiple members utilize the process that can have very different needs.
From what I have read the Merge function is more of a sorted union.
We do use an approach similar to what you describe in our product for external system inputs. (we handle a couple of hundred target tables with up to 240 columns) Like you describe, there's anywhere from 1 to a million or more rows.
Generally, we don't try to set up a single mass update, we try to handle one column's values at a time. Given that they're all a single type representing the same data element, the staging UPDATE statements are simple. We generally create scratch tables for mapping values and it's a simple
UPDATE target SET target.column = mapping.resultcolumn WHERE target.sourcecolumn = mapping.sourcecolumn.
Setting up the mappings is a little involved, but we again deal with one column at a time while doing that.
I don't know how you define 'horrendous'. For us, this process is done in batch mode, generally overnight, so absolute performance is almost never an issue.
EDIT:
We also do these in configurable-size batches, so the working sets & COMMITs are never huge. Our default is 1000 rows in a batch, but some specific situations have benefited from up to 40 000 row batches. We also add indexes to the working data for specific tables.