How to handle transaction issues in an environment where mongodb and mysql databases work together?
I want to use mongodb for scalability and mysql for transactions. (transactions are used in the inventory management system but product information is stored in a mysql database)
There is good news: As of Mongo version 4.2, multiple document ACID transactions are now fully supported:
For situations that require atomicity of reads and writes to multiple documents (in a single or multiple collections), MongoDB supports multi-document transactions.
As a general comment to your question, there is nothing wrong with having more than one data store in your architecture. However, keep in mind that business operations which would require both Mongo and MySQL for a single logical transaction/unit, there would probably be no way to make this atomic. If you are falling into this category, then you need to rethink your database design, and just stick with one database for each business operation.
Related
I am quite new to the programming environment so any help in which direction to head would be greatly appreciated. What should i use? NoSQL or Sql? What should be my deciding factors?
Should i go with a noSQL db like sembast since my server side application uses mongodb for storage and since i have to mainly deal with JSON data , or should I go with SQL db like sqflite? What should be my considerations when deciding between the two? Are there some other options that i should be aware of too?
SQL databases are called relational databases. They are based on tables, which consists of a number of rows of data. They use SQL (Structured Query Language) for defining and manipulating the data. Thanks to this, SQL database are a good fit for complex querying. The transaction mechanism makes SQL databases a good choice for heavy duty transactional type applications.
NoSQL databases are called as non-relational or distributed database. They are based on key-value pairs document and don't have a standard schema definitions. Queries are focused on collection of documents so they cannot be very complicated. Although NoSQL provides transaction mechanism it's not stable enough for a complex transactional applications. NoSQL databases fit best for the hierarchical data storage (similar to JSON data) so they are highly preferred for large data set of data (big data).
I have a application where I need to maintain the audit log operation performed on the collection. I am currently using the MongoDB for storage purpose which work well so far.
Now for audit log I am thinking to use the MySQL database where reasons are-
1. Using the mongo implicit audit filter degrade the performance.
2. Storage will be huge if I store the logs also in the mongoDB which will impact in replication of nodes in cluster.
Conditions to see the logs are not very often in application, so thinking to store logs out of main storage. I am confused to use mongoDB with MySQL, also is this a right choice for future perspective.
Also Is MySQL a good choice to store the audit log, or any other database can help me in storage and conditional query later.
Performance is not guaranteed to go to a completely different database system only for this purpose.
My first attempt for separation would be creating a new database in your current database system and forward to there or even using a normal text file.
Give your feedbacks.
We have a classified website. I read few things but still not got a solid reason to choose between these.
My requirement is to just create user, listing and manage these. These are paid listings so monetary transactions would be there.
As per my research (https://www.mongodb.com/compare/mongodb-mysql) I found that mongo db has edge over other options except for complex transactions. What are complex transaction in this context? Any specific examples?
What factors should I consider before choosing one. I have experience with My SQL and Entity framework so its more appealing to me now but are these really bad for scaling when compared with mongo?
Feel free to suggest any other good performance option apart from these choices.
There is a vast difference between mongoDB and MySQL or SQL Server.
MongoDB is a NoSQL database meaning it is schemaless and doesnot follow CODD Rules of RDBMS whereas MySQL/SQL Server is relational Database.
Each are meant for different use cases. As per mongoDB doc
Why use MongoDB instead of MySQL?
Organizations of all sizes are adopting MongoDB because it enables them to build applications faster, handle highly diverse data types, and manage applications more efficiently at scale.
Development is simplified as MongoDB documents map naturally to modern, object-oriented programming languages. Using MongoDB removes the complex object-relational mapping (ORM) layer that translates objects in code to relational tables.
MongoDB’s flexible data model also means that your database schema can evolve with business requirements. For example, the ALTER TABLE command required to add a single, new field to Craiglist’s MySQL database would take months to execute. The Craigslist team migrated to MongoDB because it can accommodate changes to the data model without such costly schema migrations.
MongoDB can also be scaled within and across multiple distributed data centers, providing new levels of availability and scalability previously unachievable with relational databases like MySQL. As your deployments grow in terms of data volume and throughput, MongoDB scales easily with no downtime, and without changing your application. In contrast, to achieve scale with MySQL often requires significant, custom engineering work.
What are common use cases for MongoDB?
MongoDB is a general purpose database that is used for a variety of use cases. The most common use cases for MongoDB include Single View, Internet of Things, Mobile, Real-Time Analytics, Personalization, Catalog, and Content Management.
When would MySQL be a better fit?
While most modern applications require a flexible, scalable system like MongoDB, there are use cases for which a relational database like MySQL would be better suited. Applications that require complex, multi-row transactions (e.g., a double-entry bookkeeping system) would be good examples. MongoDB is not a drop-in replacement for legacy applications built around the relational data model and SQL.
A concrete example would be the booking engine behind a travel reservation system, which also typically involves complex transactions. While the core booking engine might run on MySQL, those parts of the app that engage with users – serving up content, integrating with social networks, managing sessions – would be better placed in MongoDB
As there is no concept of join in MongoDB so if you want to store data, either you have to create a hack to store data in two table(although it is expensive cal to join in MongoDB) or you have to store data as a single document,, making your document too complex.In these cases MySQL has upperhand.
We are building a large scale e-comm web site to service over 100,000 users, but we expect the number of users to grow rapidly over the first year. In general, the site functions very much like ebay where users can create, update, and remove listings. User can also search listings and purchase an item of interest. Basically, the system has transactional and non-transactional requirements:
**Transactional**
Create a listing (multi-record update)
Remove a listing
Update a listing
Purchase a listing (multi-record update)
**Non-Transactional**
Search listings
View a listing
We want to leverage the power of scalable, document-based NoSQL data stores such as Couch or MongoDB, but at the same time we need a relational store to support our ACID transactional requirements. So we have come up with a hybrid solution which uses both technologies.
Since the site is "read mostly", and, to meet the scalablity needs, we set up a MongoDB data store. For the transactional needs, we set up a MySQL Cluster. As the middleware component we use JBoss App server cluster.
When a "search" request comes in, JBoss directs the request to Mongo to handle the search which should produce very quick results while not burdening MySQL. When a listing is created, updated, removed, or purchased, JBoss services the transactions against MySQL. To keep MongoDB and MySQL synchronized, all transactional requests handled by JBoss against MySQL would include a final step in the business logic that updates the corresponding document in MongoDB via the listing id; we plan to use the MongoDB Java API to facilitate this integration of updating the document.
So, in essence, since the site is read mostly, the architecture allows us to scale out MongoDB horizontally to accommodate more users. Using MySQL allows us to leverage the ACID properties of relational databases while keeping our MongoDB store updated through the JBoss middleware.
Is there anything wrong with this architecture? No platform can offer consistency, availability, and partition-tolerance at the same time -- NoSQL systems usually give up consistency -- but at least with this hybrid approach we can realize all three at the cost of additional complexity in the system, and we are ok with that since all of our requirements are being met.
There is nothing wrong with this approach.
Infact Currently am also working on the application (E-Commerce) which leverages both SQL & NonSQL. Ours is a rails application and 90% of the data is stored in mongo and only transactional & inventory items stored in mysql. All the transactions are handled in Mysql, and everything else goes to mongo.
If you have already built it, there isn't too much wrong with the architecture aside from being a little too enterprisey. Starting from scratch on a system like this though, I'd probably leave out SQL and the middleware.
The loss of consistency in NoSQL data stores isn't as complete as you suggest. Aside from the fact that many of them do support transactions and can be set up for immediate consistency on particular queries, I suspect some of your requirements are simply an artefact of designing things relationally. Your concern seems to be around operations that require updates to multiple records - Is a listing really multiple records, or just set up that way because SQL records have to have a flat structure?
Also, if search and view are handled outside of MySQL, you have effectively set up an eventual consistency system anyway.
I have been working to learn Mongodb effectively for one week in order to use for my project. In my project, I will store a huge geolocation data and I think Mongodb is the most appropriate to store this information. In addition, speed very important for me and Mongodb responds faster than Mysql.
However, I will use some joins for some parts of the project, and I'm not sure whether I store user's information in Mongodb or not. I heard some issues can occur in mongodb during writing process. should I use only mongodb with collections (instead of join) or both of them?
In most situations I would recommend choosing one db for a project, if the project is not huge. On really big projects (or enterprises in general), I think long term organizations will use a combination of
RDBMS for highly transactional OLTP
NoSQL
a datawarehousing/BI project
But for things of more reasonable scope, just pick the one that does the core of the use case, and use it for everything.
IMO storing user data in mongodb is fine -- you can do atomic operations on single BSON documents so operations like "allocate me this username atomically" are doable. With redo logs (--journal) (v1.8+), replication, slavedelayed replication, it is possible to have a pretty high degree of data safety -- as high as other db products on paper. The main argument against safety would be the product is new and old software is always safer.
If you need to do very complex ACID transactions -- such as accounting -- use an RDBMS.
Also if you need to do a lot of reporting, mysql may be better at the moment, especially if the data set fits on one server. The SQL GROUP BY statement is quite powerful.
You won't be JOINing between MongoDB and MySQL.
I'm not sure I agree with all of your statements. Relative speed is something that's best benchmarked with your use case.
What you really need to understand is what the relative strengths and weaknesses of the two databases are:
MySQL supports the relational model, sets, and ACID; MongoDB does not.
MongoDB is better suited for document-based problems that can afford to forego ACID and transactions.
Those should be the basis for your choice.
MongoDB has some nice features in to support geo-location work. It is not however necessarily faster out of the box than MySQL. There have been numerous benchmarks run that indicate that MySQL in many instances outperforms MongoDB (e.g. http://mysqlha.blogspot.com/2010/09/mysql-versus-mongodb-yet-another-silly.html).
Having said that, I've yet to have a problem with MongoDB losing information during writing. I would suggest that if you want to use MongoDB, you use if for the users as well, which will avoid having to do cross database 'associations', and then only migrate the users to MySQL away if it becomes necessary.