Migrate data model from MySQL to Cassandra - mysql

Question

Structure in MySql (for compactness i am using a simplified notation)
Notation: table name->[column1(key or index), column2, …]
documents->[doc_id(primary key), title, description]
elements->[element_id(primary key), doc_id(index), title, description]
Each document can contain a large number of elements (between 1 and 100k+)
We have two key requirements:
Load all elements for a given doc_id quickly
Update the value of one individual element by his element_id quickly
Structure in Cassandra
1st solution
documents->[doc_id(primary key), title, description, elements] (elements could be a SET or a TEXT, each time new elements are added (they are never removed) we would append it to this column)
elements->[element_id(primary key), title, description]
To load a document we would need:
Load document with given and get all element ids: SELECT * from documents where doc_id=‘id’
Load all elements with the given ids: SELECT * FROM elements where element_id IN (ids loaded from query a)
Updating elements would be done by their primary key.
2nd solution
documents->[doc_id(primary key), title, description]
elements->[element_id(primary key), doc_id(secondary index), title, description]
To load a document we would need:
SELECT * from elements where doc_id=‘id’
Updating elements would be done by their primary key.
Questions regarding our solutions:
1st: Will it be efficient to query 100k+ primary keys in the elements table?
SELECT * FROM elements WHERE element_id IN (element_id1,.... element_id100K+)?
2nd: Will it be efficient to query just by a secondary index?
Could anyone give any advice how would we create a model for our use case?

Answer 1

With cassandra it's all about the access pattern (I hope I understood it correctly, if not please comment)
1st
documents should not use sets because set is limited to 65 535 elements and has to be read, updated in it's entirety every time a change is made. Since you need 100k+ it's not what you want. You could use frozen collections etc, but then again, reading everything in memory every time is bound to be slow.
2nd
secondary indexes, well, small cardinality data might be fine But from I understand you have 100k per document, this might even be fine but then again It's not the best practice. I would simply try it out in your concrete case.
3rd - disk is cheap approach - always write the data the way you are going to read it - cassandra's writes are dirt cheap so prepare the views at write time,
this one satisfies reading of all the elements belonging to doc_id
documents->[doc_id(primary key), title_doc (static), description_doc(static), element_id(clustering key), title, description]
elements remain pretty much as they were:
elements->[element_id(primary key), doc_id, title, description]
When doing updates, you update it in documents and elements (for consistency you can use batch operation - should you need it) If having element_id you can quickly issue another query after you get it's doc Id.
Depending on your updating needs documentId could also be a set. (I might have not gotten this part right because not sure what data is available when updating the element do you have the doc_id also and can one element be in more docs?)
Also since having 100k+ elements in single partition is not the best thing to have because of the retrievals ( all requests will go to one node) I would propose to have composite partitioning key (bucket) I think in your case a simple fixed int would be just fine. So every time you go to retrieve the elements you just issue selects to documentid + (1, 2, 3, 4 ...) and then merge the result at the client - this will be significantly faster.
One tricky part would be that you don't go into every single bucket for elementid that is stored in the document ... when I think about it then it would be better to use a base of two for buckets. In your case 16 would be ideal ... then when you look to update specific element just use some simple hash function known to you and use last 4 bits.
Now when I think about it if the element id + doc id is always known to you you might not even need the elements table at all.
Hope this helps

Answer 2

Based on the suggestion of Marko, our solution is:
CREATE TABLE documents (
doc_id uuid,
description text,
title text,
PRIMARY KEY (doc_id)
);
CREATE TABLE nodes (
doc_id uuid,
element_id uuid,
title text,
PRIMARY KEY (doc_id, element_id)
);
We can retrieve all elements with the following query:
SELECT * FROM elements WHERE doc_id='id'
And update the elements:
UPDATE elements SET title='Hello' WHERE doc_id='id' AND element_id='id';