In Mongo my understanding is that you can have databases and collections. I'm working on a social-type app that will have blogs and comments (among other things) and had previously be using MySQL and pretty heavy partitioning in an attempt to limit possible concurrency issues.
With MySQL I've stuffed all my user data into a _user database with several tables to further partition the data (blogs, pages, etc).
My immediate reaction with Mongo would be to create a 'users' database with one collection per user. In this way user 'zach' blog entries would go into the 'zach' collection with associated comments and such becoming sub-objects in the same collection. Basically like dynamically creating one table per user in MySQL, but apparently without the complexity and limitations that might impose.
Of course since I haven't really used Mongo before I'm having trouble gauging the (ahem..) quality of this idea and the potential problems it might cause down the road.
I'd like user data to be treated a lot like a users directory in a *nix environment where user created/non-shared (mostly) gets put into one place (currently with MySQL that would be the appname_users as mentioned above).
Most of the users data will be specific to the users page(s). Some of the user data which is queried across all site users (searchable user profiles) is currently kept in a separate database/table and I expect things like this could be put into a appname_system database and be broken up into collections and/or application specific databases (appname_profiles).
Anyway, since the available documentation on this is currently a little thin and my experience is extremely limited I thought I might find a little guidance from someone with a better working understanding of the system.
On the plus side I'd really already been attempting to treat MySQL as a schema-less document-store and doing this with Mongo seems much more intuitive/sane/rational so I'm really looking forward to getting started.
Thanks,
Zach
I have the same kind of application.
Some things to consider: you can cross query between collection bu not between databases.
So It's probably better to have a database with all you data and then a collection for each Object.
Then each document can contain any kind and number of fields.
I tried to avoid embedding arrays b/c I had trouble query properly my object (it was working fine, but the architecture of my system was designed for this use)
And a database can be shared between several sever automatically so space is not an issue (if you have more than 1 server)
Related
I've researched a lot for the best multi-tenant setup of an e-commerce project but could not find a fitting answer. I am leaning to use a shared database separate schema setup with either MySQL or PostgreSQL. The structure of the tables is the same for all the tenants. I really like that with that setup the application code doesn't need to provide an extra WHERE clause for every query, so it is very developer friendly!
Now you also have the shared schema approach and that is what I am currently using, but I feel this is bad for security and isolation purposes. I would like to move to a different solution.
The app will be used by +- 100 webshops (a tenant) this year and I expect it to grow to into the thousands. Webshops ranges from small to large, so it is important that I can later pick out a specific shops data and put it in it's own database server.
Since I don't have any experience with a separate schema setup I would like to know if this would benefit me. What issues might I have when walking that path? Especially with changes in the structure of the tables, this is what bothers me the most. What is the limit of schema's to use in a separate schema approach using PostgreSQL (I will have 100-1000 schemas) before it would be a pain to manage?
The following is my take
Blockquote
When using the SharedDatabase with same table for all tenants, isolation happening with tenantid column is easy because you always add a filter that says tenantid = LoggedInTenantId [LoggedInTenantId => set during login]. When you have a base method in your ORM like EF [I'm from .Net], this would auto append to any query that goes out of the code.
When you opt for shared Schema, if there is a use-case like sharing data between tenants [webshops], it is not feasible. Else, if you have an accountant that wants to audit a collection of tenant's and wants a dashboard to view the a/c statistics etc, it becomes impossible
With the scaling point of view, you can better go for a separate db per tenant if a single tenant or a collection of them wants to scale out. This will be better than managing schemas.
Consider the use-cases that you may have for your product and share here so that we can take this discussion forward.
HTH
Little question, I'm developing a saas software (erp).
I designed it with 1 database per account for these reasons :
I make a lot of personalisation, and need to add specific table columns for each account.
Easier to manage db backup (and reload data !)
Less risky : sometimes I need to run SQL queries on a table, in case of an error with bad query (update / delete...), only one customer is affected instead of all of them.
Bas point : I'm turning to have hundreds of databases...
I'm hiring a company to manage my servers, and they said that it's better to have only one database, with a few tables, and put all data in the same tables with column as id_account. I'm very very surprised by these words, so I'm wondering... what are your ideas ?
Thanks !
Frederic
The current environment I am working in, we handle millions of records from numerous clients. Our solution is to use Schema to segregate each individual client. A schema allows you to partition your clients into separate virtual databases while inside a single db. Each schema will have an exact copy of the tables from your application.
The upside:
Segregated client data
data from a single client can be easily backed up, exported or deleted
Programming is still the same, but you have to select the schema before db calls
Moving clients to another db or standalone server is a lot easier
adding specific tables per client is easier (see below)
single instance of the database running
tuning the db affects all tenants
The downside:
Unless you manage your shared schema properly, you may duplicate data
Migrations are repeated for every schema
You have to remember to select the schema before db calls
hard pressed to add many negatives... I guess I may be biased.
Adding Specific Tables: Why would you add client specific tables if this is SAAS and not custom software? Better to use a Postgres DB with a Hstore field and store as much searchable data as you like.
Schemas are ideal for multi-tenant databases Link Link
A lot of what I am telling you depends on your software stack, the capabilities of your developers and the backend db you selected (all of which you neglected to mention)
Your hardware guys should not decide your software architecture. If they do, you are likely shooting yourself in the leg before you even get out of the gate. Get a good senior software architect, the grief they will save you, will likely save your business.
I hope this helps...
Bonne Chance
I am building a web application that requires to be scalable. In a nutshell:
We got users, users have friends, so they got a friendlist. Users can create messages, and messages from your friends are displayed on the homepage, each message is linked to a location and these messages can be filtered by date, for example I want to display all the messages from my friends that where posted yesterday, or display me all messages from location X.
I am now building the application fully in MongoDb, however I am heading into trouble atm. For example:
On the mainpage, we got the message list of the friends of the users, no problem we use:
$db->messages->find(array('users._id' => array('$in' => $userFriendListGoesHere)));
So then we got our messages, however after that, each message has a location, so I have to make a loop through all messages, and get the location from another collection, and also multiple users can be bound to a single message, so we also have to get all the user data from another collection, in MySql simply a join query, in MongoDb 2 loops, and this is my first question: is this a problem? Does this require alot of resources, the looping?
So my idea is to split up with MySql and MongoDb, I use MongoDb to store all the locations (since it are over 350.000+ locations and use lat long calculations) and MySql for the message, users and friends of the users, so second question, can you help me with my decision, should I keep using MongoDb with the loops? Or use a combination?
Thanks for reading and your time.
.. in MySql simply a join query, in MongoDb 2 loops, and this is my first question: is this a problem?
This is par for the course with MongoDB, in fact, it's a core MongoDB trade-off.
MongoDB is based on the precept that joins do not scale. So it has no joins and leaves you to "roll your own". Some libraries like Morphia (for Java) provide built-in logic for loading references.
PHP has the Doctrine project, which should help with some of this.
Does this require alot of resources, the looping?
Kind of? This will really depend on implementation.
It's obviously going to involve a bunch of back and forth with the DB, but it may be less network traffic than the SQL version. You will need memory space for all of the data coming back. But again, that's not terribly different from SQL.
Really, it's up to you to make all of the trade-offs about how this is implemented and who is keeping what in memory.
should I keep using MongoDb with the loops
MongoDB is a great idea when your data is not inherently relational.
In the example you provided, it kinda seems like your data is relational. MySQL and other relational DBs (such as Postgres) are better data stores than MongoDB for relational data. This blog post covers this topic in more detail.
In summary, I'd recommend the following:
Please spend some time analyzing whether your data is inherently relational or not.
If it is not, then MongoDB can give you benefits over using MySQL.
If it is relational, then MySQL is the better solution.
Using both is, of course, possible - but it will create additional work & complexity for you. In the long term - is that worth the effort? Only you will know the answer.
Best of luck with your web app!
I'm currently building a CRM for a niche industry that can be tailored for the end user. We currently are using a base table for leads and then basically a hacked together combination of tables as a key/value store for any custom fields that might be added.
Each lead in the CRM also has tables for various owners, permissions, access logs, scheduled tasks, and generated contracts/documents.
Currently, everything is stored in a MySQL database but it just seems really messy. I'm thinking of using a NoSQL solution like MongoDB and then using Redis to create any relationships between tables.
Example: I'm thinking of storing the leads and users in a MongoDB database and then using Redis for storing what users a certain lead belongs to etc.
Just looking for some general advice on what the best solution is for this situation as I haven't dealt with NoSQL before.
mysql is a great tool
storages is not some stuff that works automagically. For being working well - the great developer needs to implement great schema and great code.
What I wanted to say: if you have some issues now - it doesn't mean that they will disappear once you moved to another database (paradigm, ideology, etc). Even if they moved away - the new ones will come. Nothing happens without hard working.
And especially to big volumes of data and high loaded projects - there is no general advice and silver bullet.
ps:
I'm thinking of using a NoSQL solution like MongoDB and then using Redis to create any relationships between tables.
MongoDB can maintain relations. This is one more argument to not move to the databases you are not well experienced in.
I'm working on an application that's similar to Wufoo in that it allows our users to create their own databases and collect/present records with auto generated forms and views.
Since every user is creating a different schema (one user might have a database of their baseball card collection, another might have a database of their recipes) our current approach is using MySQL to create separate databases for every user with its own tables. So in other words, the databases our MySQL server contains look like:
main-web-app-db (our web app containing tables for users account info, billing, etc)
user_1_db (baseball_cards_table)
user_2_db (recipes_table)
....
And so on. If a user wants to set up a new database to keep track of their DVD collection, we'd do a "create database ..." with "create table ...". If they enter some data in and then decide they want to change a column we'd do an "alter table ....".
Now, the further along I get with building this out the more it seems like MySQL is poorly suited to handling this.
1) My first concern is that switching databases every request, first to our main app's database for authentication etc, and then to the user's personal database, is going to be inefficient.
2) The second concern I have is that there's going to be a limit to the number of databases a single MySQL server can host. Pretending for a moment this application had 500,000 user databases, is MySQL designed to operate this way? What if it were a million, or more?
3) Lastly, is this method going to be a nightmare to support and scale? I've never heard of MySQL being used in this way so I do worry about how this affects things like replication and other methods of scaling.
To me, it seems like MySQL wasn't built to be used in this way but what do I know. I've been looking at document-based databases like MongoDB, CouchDB, and Redis as alternatives because it seems like a schema-less approach to this particular problem makes a lot of sense.
Can anyone offer some advice on this?
Since you are leaving the schema up to your users to decide, it doesn't make sense using a relational database that forces you to define a schema.
Use a NoSQL database. Do some more reading on stack overflow.
What is NoSQL, how does it work, and what benefits does it provide?
Pros/Cons of document based database vs relational database
What is the best Document-oriented database?
Creating tables on the fly like you describe is a very bad idea. Supporting schema changes would be a nightmare. Each time someone added or removed a field you would have to run an ALTER TABLE ... command, and if there's data i the table, that's not a quick operation since it basically creates a new table with the new scehma and moves all the data over to the new one. Don't go down that route.
You could implement some kind of key/value-store on top of MySQL without too much work, or use something like Friendly, but going for a proper document database is probably a much simpler way.
MongoDB would be my choice, but there's a lot of things to consider, and others may say that Cassandra would be better. It's very easy to get going with MongoDB, and using it feels quite familiar to using a SQL database. It does indexing more or less identically, and querying is not too different either. The best thing though, is probably that you don't need an ORM, your objects are stored more or less as-is in the database. Reading and writing can be done very close to the metal without requiring a lot of mapping to and from objects.