SQL or NoSQL search? - mysql

Question

Let us suppose I have a site with a certain number of users with the following three distinguishing characteristics:
1) The user is part of a network. (The site contains multiple networks.)
2) The user is a 'contact' of a certain number of other site members.
3) Individual documents uploaded by a user may be shared with certain contacts (excluding other contacts).
In this way, a user's document search is unique for each user based upon his or her network, contacts, and additional documents that have been shared with that user. What would be possible ways to address this -- would I need to append a long unique SQL query for each user for each of his or her searches? I am currently using MySQL as a database -- would using this be sufficient, or would I need to move towards a NoSQL option here to maintain the performance of a similar non-filtered search?

Answer 1

A few questions come to mind to help answer this question:
How many documents do you think the average user will have access to? Will many documents in the network be shared for all to see?
How will users be able to find documents and what do the documents look like? Will they only be able to search by the contact that shared it? By a simple title match? Will they be able to run a full text search against the document's contents?
Depending on the answer to those two questions, a relational system could work just fine, which I'm guessing is preferable since you are already using MySql. I think you could locate the documents for an individual user in a relational system with a few very reasonable queries.
Here is a potential bare bones schema
User
--all users in the system
UserId int
NetworkId int (Not sure if this is a 1 to many relationship)
Document
--all documents in the system
DocumentId int
UserId int -- the author
Name varchar
StatusId -- perhaps a flag to indicate whether it is public or not, e.g. shared with everyone in the same network or shared with all contacts
UserDocumentLink
--Linking between a document and the contacts a user has shared the document with
DocumentId
ContactId
UserContact
--A link between a user and all of their contacts
ContactId -- PK identity to represent a link between two users
UserId -- User who owns the contact
ContactUserId --The contact user
Here is a potential "search" query:
--documents owned by me
SELECT DocumentId
from Document where UserId = #userId
UNION
--documents shared with me explicitly
SELECT DocumentId
From UserContact uc
InnerJoin UserDocumentLink ucl on uc.ContactId = ucl.ContactId
Where
uc.ContactUserId = #userId
UNION
--documents shared with me via some public status, using a keyword filter
Select DocumentId
From Document d
inner join User u on d.UserId = u.UserId
where
u.NetworkId = #userNetworkId
and d.status in ()
and d.Name like '%' + #keyword + '%'
I think what might be a more influential requirement for schema design is one that is not mentioned in your question - how will users be able to search through documents? And what kind of documents are we talking about here? MySql is not a good option for full text search.

Answer 2

It rather depends on what you mean by a "certain number" of users. If you mean a few tens of thousands, then almost any solution can be made to perform adequately. If you mean many millions, then a NoSQL solution may scale up more cheaply and easily.
I suspect that a more general SQL query can be used, rather than a unique one for each user, e.g. selecting documents that belong to users that know the current user, that are marked as being shared with the current user, and match the search string.
Denormalisation can probably be used (as is common in NoSQL approaches) to improve performance.
However, a graph database (as Peter Neubauer suggests) possibly in combination with a document store (CouchDB, MongoDB or Cassandra) would work very well for this type of problem and would scale well.

Answer 3

I would take a look at some of the NOSQL solutions, for this interconnected dataset possibly Neo4j, a Graph Database. It's even pretty straightforward to query it through Cypher so that you get tabular results back.

Answer 4

As others have pointed out the number of users and the frequency of requests (traffic volume) must be looked at. Also, how important is redundancy? How likely are people to work on same documents simultaneously? Are most documents created once and distributed for "readonly" purposes?
NoSQL can help you scale and get redundancy in a much easier way compared to rdbms for this particular scenario. I am assuming that at some point you will want tagging etc. to be enabled on the documents.
Now, I am wondering if there is any particular reason why you are not looking at off the shelf document management and CMS system for this? I am sure there is a good reason, but it might be worth looking at all the those options too.
I hope this helps. Good luck!

Answer 5

Denormalization will give you better read-search performance in this
case.
Don't normalize users, keep frequently joined entities like owner and
text, in one table
e.g. keep names of the owners as FK on text table, to keep their
names on the text table and decrease number of joins, then you can
use sql freely.

Answer 6

I've managed this using long unique queries in MySQL as you suggest for a small-scale social networking project. Nowadays I would suggest using solr and keeping permission information as a denormalized array of interchangeable keywords on each document. Say each network has a unique recognizable code (ie 100N-20000N), similar for users and special permission grants. You can store an array of permission keys, like "5515N 43243N 2342N 603U 203PG 44321PG" and treat those as keywords when searching.

Answer 7

I would address it with a simple business process solution, which will lead to a simple data schema, a simple query and so performances and scalabilty:
Each User has a list of documents... Period.
This list is in fact a list of references to documents in a document table (with owner/security informations...)
When sharing a document to another user this document reference is added to the user's document list (Tagged as a shared one if you want), user is added to the document security list (with permission level for example).
sql query to get documents is a simple: select documentid from userdocument where userid=#userid
With a join on document table, proper indexes and sql tuning it will run with all needed informations and it will run fast.
I hope i understood well what you try to do.

Answer 8

-< = one to many
>-< = many to many (will require link table)
Network -< user -< documents >-< contact(user)
v
|
^
contacts(user,user)
This is relational, I don't see a good reason to go NoSQL unless you have a billion users
Network (unless you can belong to more than one) is an attribute of user
contacts will be maitained in the link table user_contact(user,user)
tables
documents(doc_id,user_id)
user(user_id)
contacts(user_id,c_user_id) with foreign keys on users
document_contact(doc_id,c_user_id) where a trigger constrains the c_user_id
then you get a view for all docs owners and subscribers (contacts)
CREATE OR REPLACE VIEW user_docs AS
SELECT d.user_id, d.doc_id, 'owner' AS role
FROM documents d
JOIN users u ON d.user_id = u.user_id
UNION
SELECT c.user_id, d.doc_id, 'subscriber' AS role
FROM documents d
JOIN contacts c ON d.user_id = c.c_user_id;
you can then filter the view against the document contacts,
select * from user_docs ud
where
(ud.role = 'originator'
or
ud.doc_id in (select doc_id from document_contact dc where ud.doc_id = dc.doc_id)
) and ud.user_id = 'me'

Answer 9

I would trade off immediateness with performance when it comes to full text searching.
I would create a hash table of the user combinations with the documents on a separate thread usually triggered by an asynchronous call when user associations change.
I then query the hash value + other search criteria. This will eliminate the need for the long SQL that appears at the end which may cause a lock.