There are plenty of resources out there for me to find out how tables are stored as B-Trees in a relational database. I however cannot seem to find resources on how you would store JSON documents in a NoSQL database (I am thinking document oriented). Do any of you know how JSON files are stored as B-Trees in a document oriented NoSQL database?
Related
I am quite new to the programming environment so any help in which direction to head would be greatly appreciated. What should i use? NoSQL or Sql? What should be my deciding factors?
Should i go with a noSQL db like sembast since my server side application uses mongodb for storage and since i have to mainly deal with JSON data , or should I go with SQL db like sqflite? What should be my considerations when deciding between the two? Are there some other options that i should be aware of too?
SQL databases are called relational databases. They are based on tables, which consists of a number of rows of data. They use SQL (Structured Query Language) for defining and manipulating the data. Thanks to this, SQL database are a good fit for complex querying. The transaction mechanism makes SQL databases a good choice for heavy duty transactional type applications.
NoSQL databases are called as non-relational or distributed database. They are based on key-value pairs document and don't have a standard schema definitions. Queries are focused on collection of documents so they cannot be very complicated. Although NoSQL provides transaction mechanism it's not stable enough for a complex transactional applications. NoSQL databases fit best for the hierarchical data storage (similar to JSON data) so they are highly preferred for large data set of data (big data).
I'm currently using a MySQL where do I store JSON information and I'd fetch them from MySQL and parse them on my application. I would like to get rid of MySQL, but first I would like to know is that wise?
Is that efficient if I move to way that I store the data into data folder that contains .json files and these contains the data I need? There will be my app's coordinate data per user who wants to track themselves on map. Will that cause any issues? I don't need "query", but what about big data like 50K lines in example? Same amount will be in MySQL too. Amount doesn't change, but will there be any problems that appears when moving from "reading from sql to reading from json files"
It's difficult to answer all these questions in one, but I'll address some of them:
There are dedicated NoSQL databases that are very good at the type of data storage you're talking about: MongoDB, CouchDB etc. It might be worth checking these out. They are very good at dealing with JSON data. Querying and parsing are very simple in Node.js.
You can store JSON in MySQL (or other RDMS systems), I've done it in several projects with good results. As of MySQL 5.7.8 there is a dedicated JSON type. Queries can actually work surprisingly well, I know I've queried tables with tens of millions of JSON entries pretty quickly.
Make sure you consider backup and restore scenarios, what happens in the event of a data loss. Using MySQL or a NoSQL database will simplify this for you. Either way make sure you have this covered!
I wouldn't call 50K lines big data! I dealt with databases with tens of millions of rows.. this still wouldn't be called big data.
I would probably not recommend storing your data in files. I've worked in telematics before, we stored millions of JSON blobs in relational databases with very little problems. Later on we planned to move to a NoSQL database for these, but the relational database worked surprising well, especially because you can adopt a hybrid approach of using relational queries and including JSON data in the results (to be parsed by clients).
You might not need the ability to query, but it's very useful to get for example "Give me all JSON for user id 100". An RDBMS or NoSQL system would make this very easy.
I'm doing a project where I have to store data for a NodeJS express server. It's not a LOT of data, but i have to save it somewhere.
I always hear that a database is good for that kinda stuff, but I thought about just saving all the data in objects in NodeJS and back them up as JSON to disk every minute (or 5 minutes). Would that be a good idea?
What im thinking here is that the response time from objects like that are way faster than from a database, and saving them is easy. But then I heared that there are in-memory databases aswell, so my question is:
Are in-memory databases faster than javascript objects? Are JSON-based data backups a good idea in this aspect? Or should I simply go with a normal database because the performance doesn't really matter in this case?
Thanks!
If this is nothing but a school assignment or toy project with very simple models and access patterns, then sure rolling your own data persistence might make sense.
However, I'd advocate for using a database if:
you have a lot of objects or different types of objects
you need to query or filter objects by various criteria
you need more reliable data persistence
you need multiple services to access the same data
you need access controls
you need any other database feature
Since you ask about speed, for trivial stuff, in-memory objects will likely be faster to access. But, for more complicated stuff (lots of data, object relations, pagination, etc.), a database could start being faster.
You mention in-memory databases but those would only be used if you want the database features without the persistence and would be closer to your in-memory objects but without the file writing. So it just depends on if you care about keeping the data or not.
Also if you haven't ever worked with any kind of database, now's a perfect time to learn :).
What I'm thinking here is that the response time from objects like that is way faster than from a database, and saving them is easy.
That's not true. Databases are the persistence storage, there will always be I/O latency. I would recommend using Mysql for sql database and MongoDB or Cassandra for nosql.
An in-memory database is definitely faster but again you need persistence storage for those data. redis is a very popular in-memory database.
MongoDB store data in BSON (a superset of JSON) like formate, so it will be a good choice in your case.
I have a production database server running on MYSQL 5.1, now we need to build a app for reporting which will fetch the data from the production database server, since reporting queries through entire database may slow down, hence planning to switch to nosql. The whole system is running aws stack planning to use DynamoDb. Kindly suggest me the ways to sync data from the production nosql server to nosql database server.
Just remember the simple fact that any NoSQL database is essentially a document database; it's really difficult to automatically convert a typical relational database in MySQL to a good document design.
In NoSQL you have a single collection of documents, and each document will probably contain data that would be in related rows in multiple tables. The advantage of a NoSQL redesign is that most data access is simpler and faster without requiring you to write complex join statements.
If you automatically convert each MySQL table to a corresponding NoSQL collection, you really won't be taking advantage of a NoSQL DB. This is because you'll end up loading many more documents, and thus make many more calls to the database than needed and thus loosing simplicity and speediness of NoSQL DB.
Perhaps a better approach is to look at how your applications use the MySQL database and go from there. You might then consider writing a simple utility script knowing fully well your MySQL database design.
As the data from a NoSQL database like MongoDB, RIAK or CouchDB has a very different structure than a relational database like MySQL the only way to migrate/synchronise the data would be to actually write a job which would write the data from MySQL to the NoSQL database using SELECT queries as stated on the MongoDB website:
Migrate the data from the database to MongoDB, probably simply by writing a bunch of SELECT * FROM statements against the database and then loading the data into your MongoDB model using the language of your choice.
Depending of the quantity of your data this could take awhile to process.
If you have any other questions don't hesitateo to ask.
Anybody use NoSQL / InnoDB with Memcached?
How stable is it? I have set it up yesterday and going to test today, but maybe you can share some knowledge also?
Not sure what you mean by NoSQL/InnoDB - Innodb is a storage engine used in mysql table schemas and isn't really related to NoSQL key/value stores like Mongo, Redis or CouchDB. If you mean a comparison between the two, here is a basic benchmark on an update statement between mongo, a major NoSQL platform, and mysql tables using the InnoDB engine.
http://mysqlha.blogspot.com/2010/09/mysql-versus-mongodb-update-performance.html
That said, most of the NoSQL alternatives have at this point fairly stable libraries. An application my team worked on utilized memcached alongside mongo utilizing their Python APIs in a search app to store query data to train the search results on later. Basically memcached hashes were stored alongside query data and then called after a result set was picked by the user in order to refine the results for those works. Haven't had any problems with utilizing the two together and implementation was a snap.
Most NoSQL engines now use some serialized key-value data, commonly some variant on the JSON spec. This actually makes things generally even easier than the old RDBMS approach of constructing your objects from across multiple tables and running numerous updates for your persistence tier. In the case of Mongo, we handed the whole serialized BSON doc returned from Mongo to memcached for the temp storage and there were no chokes at all.
This NoSQL thing is pretty cool for those already working with the object paradigm.