Which data-structure on CouchDB with three entities (User, Folder, Files)? - json

Question

I'm trying to build a "relationship" in CouchDB for a Dropbox-like scenario with:
Users
Folders
Files
So far I'm struggeling whether to reference or embed the above things and haven't tackled permissions yet. In my scenario I just want to store the path to the files and don't want to work with attachments. Here's what I have:
Option 1 (Separate Documents)
Here I chain just everything together and it (at least for me) seems to be a copy of a RDBMS model which should not be the goal when using NoSQL.
{
"id": "user1",
"type": "user",
"folders": [
"folder1",
"folder2"
]
}
{
"id": "folder1",
"type": "folder",
"path": "\\user1\\pictures",
"files": [
"file1",
"file2"
]
}
{
"id": "file1",
"type": "file",
"name": "myDoc.txt",
}
Option 2 (Separate Documents)
In this option I would leave the users document as it is and put into the folders document the users id for the purpose of referencing.
{
"id": "user1",
"type": "user",
}
{
"id": "folder1",
"type": "folder",
"path": "\\user1\\pictures",
"owner" "user1",
"files": [
"file1",
"file2"
]
}
{
"id": "file1",
"type": "file",
"name": "myDoc.txt",
}
Option 3 (Embedded Documents)
Similar to Option 2 I here would dismiss the the third document type files and embed everything into the folder document. I read that it is only an option if I don't have to many items to store and I don't know how much items a user will store for example.
{
"id": "user1",
"type": "user",
}
{
"id": "folder1",
"type": "folder",
"path": "\\user1\\pictures",
"owner" "user1",
"files": [{
"id": "file1",
"type": "file",
"name": "myDoc1.txt"
}, {
"id": "file2",
"type": "file",
"name": "myDoc2.txt"
}
]
}
Option 4
I could also put everything in just one document but in this scenario this makes no sense. The JSON documents would get to big in time and thats not something which is desirable in regards to performance / load-time.
Conclusion
For me none of the above options seem to fit my scenario and I would appreciate some input from you in how to design a proper database schema in CouchDB. Or maybe one of the above options is already a good start and I just don't see it.

Answer 1

To provide you with a concrete idea, I'd model a Dropbox clone somehow like this:
Shares: The root folder that is shared. There is no need to model subfolders, as they don't have different permissions. Here I can set the physical location of the folder and the users that are allowed to use them. I'd expect that there are only a few shares per user, so you can keep the list of shares in memory.
Files: The actual files in the share. Depending on your use case, there's no need to keep the files in a database, as the filesystem itself is already a great file database by itself! If you need to hash and deduplicate files (such as Dropbox does it), then you might create a cache in CouchDB.
This would be the document structure:
{
"_id": "share.pictures",
"type": "share",
"owner": "Alice",
"writers": ["Bob", "Carl"],
"readers": ["Dorie", "Eve", "Fred"],
"rootPath": "\\user1\pictures"
},
{
"_id": "file.2z32236e2sdwhatever",
"type": "file",
"path": ["vacations", "2017 maui"],
"filename": "DSC1234.jpg",
"size": 12356789,
"hash": "1235a",
"createdAt": "2017-07-29T15:03:20.000Z",
"share": "share.pictures"
},
{
"_id": "file.sdfwhatever",
"type": "file",
"path": ["vacations", "2015 alaska"],
"filename": "DSC12345.jpg",
"size": 11,
"hash": "acd5a",
"createdAt": "2017-07-29T15:03:20.000Z",
"share": "share.pictures"
}
This way you can build a CouchDB view of files by share and path and query it by folder:
function (doc) {
if (doc.type === 'file') emit([doc.share].concat(doc.path), doc.size);
}
If you want, you can add also add a reduce function with just _sum and get a hierarchical size calculator for free (well, almost)!
Assuming you called the database 'dropclone' and added the view to a design document called 'dropclone' with the view name 'files', you would query it like this:
http://localhost:5984/dropclone/_design/dropclone/_view/files?key=["share.pictures","vacations"]
You'd get 123456800 as a result.
For
http://localhost:5984/dropclone/_design/dropclone/_view/files?key=["share.pictures","vacations"]&reduce=false&include_docs=true
You would get both files as a result.
You can also add the whole share name and path into the _id, because then you can directly access each file just by the known path. You can still add the path redundantly or leave it out and just split the _id into its path component dynamically.
Other approaches would be:
Use one CouchDB database per share and use CouchDB's _security mechanism to manage the access.
Split files into chunks, hash them and store the chunk hashes for each file. This way you can virtualize and deduplicate the complete file system. This is what Dropbox does behind the scenes to save storage space.
One thing you shouldn't do is store the files themselves into CouchDB, this will get dirty quite quickly. NPM had to experience that some years ago, and they had to move away from this model in a huge engineering effort.

Answer 2

Data Modeling starts with the queries the application will use.
If your queries will be that a user sees all his/her folders, and opening a folder displays all docs and sub-folders beneath it, the option 1 is a nature fit to the queries.
However, there is one very important question you need to answer first, especially for CouchDB. Which is how large you database will be. If you will need a DB partitioned across multiple nodes, then the performance would suffer, possibly to a point that DB becomes unresponsive. Because opening a folder with many docs would mean searching every partition. This is due to the partitioning is decided by the hashing of the ID which user has no control. The performance will be fine for a small single node (or non partitioned) DB.
Option 2 requires you build index on "owner", which suffers for the same reason as option 1.
Options 3/4 are kind of denormalization, which addressed the above performance issue. If the docs are large and updated often, the overhead of storage and cost of compaction may be significant. You need bench-marking for your specific workloads.
In summary, if your target DB will be big and partitioned, then there is no easy answer. Careful prototype and bench-marking would be needed.