I have an app and it needs to store a few simple objects that change, and it needs to be able to persist them. Should I save the data to a db like mongo and read it in on startup or should I just save to a json file?
Thanks for any tips.
Either way can work. 'It depends' which is best in your situation.
Things to consider:
Flat files are not (or not easily) queriable.
You will have to manage flat files (e.g. folders, file names, etc)
Is it possible that there will be a collision? Are you going to read & write often?
If you are only persisting a few objects, is a database like mongo overkill?
Is something like redis a better solution?
I'm currently working on a project that involves several data stores. MySQL, flat-files, mongo, et cetera.
The flat files work really well for storing data that will be pulled up later. It's suitable for data that isn't read or written often.
Mongo has modules like Mongoose that take away a lot of complexities. But the documentation for both Mongoose and Mongo can be a bit dry and confusing.
Think about how the data will be used, how it will evolve, and if you need to query it. If it just needs to be pulled up at the app's startup, then a flat file can work. If it is going to grow into larger objects then flat files can still be the solution. But if there's a chance for additional objects in the future, then flat-files will become harder to manage, and it will be a pain to add them into a working system.
And overall, if you want any type of querying (other than manually opening files and inspecting them) then go with Mongoose or a similar database.
It depends mongo will be easier if u want to alter the data. It is basically a storage engine for json lol. Depends on your use case. If you think this will grow, you might as well use mongo. If your storage needs will never change use a json file.
Related
I want to create a website that will have an ajax search. It will fetch the data or from a JSON file or from a database.I do not know which technology to use to store the data. JSON file or MySQL. Based on some quick research it is gonna be about 60000 entries. So the file size if i use JSON will be around 30- 50 MB and if use MySQL will have 60000 rows. What are the limitations of each technique and what are the benefits?
Thank you
I can't seem to comment since I need 50 rep. for commenting, so I will give it as an answer:
MySQL will be preferable for many reasons, not the least of which being you do not want your web server process to have write access to the filesystem (except for possibly logging) because that is an easy way to get exploited.
Also, the MySQL team has put a lot of engineering effort into things such as replication, concurrent access to data, ACID compliance, and data integrity.
Imagine if, for instance, you add a new field that is required in whatever data structure you are storing. If you store in JSON files, you will have to have some process that opens each file, adds the field, then saves it. Compare this to the difficulty of using ALTER TABLE with a DEFAULT value for the field. (A bit of a contrived example, but how many hacks do you want to leave in your codebase for dealing with old data?) so to be really blunt about, MySQL is a database while JSON is not, so the correct answer is MySQL, without hesitation. JSON is just a language, and barely even that. JSON was never designed to handle anything like concurrent connections or any sort of data manipulation, since its own function is to represent data, not to manage it.
So go with MySQL for storing the data. Then you should use some programming language to read that database, and send that information as JSON, rather than actually storing anything in JSON.
If you store the data in files, whether in JSON format or anything else, you will have all sorts of problems that people have stopped worrying about since databases started being used for the same thing. Size limitations, locks, name it. It's good enough when you have one user, but the moment you add more of them, you'll start solving so many problems that you would probably end up by writing an entire database engine just to handle the files for you, while all along you could have simply used an actual database. Do note! Don't take my word for granted, I am not an expert on this field, so let others post their answer and then judge by that. I think enough people here on stackoverflow have more experience then I do haha. These are NOT entirely my words, but I have taken out the parts that were true from what I knew and know and added some of my own knowledge :) Have a great time making your website
For MySQl :you can select specific rows,or specific column using queries ,filter data based on a key,order alphabetically
downside:need a REST API to fetch data because it can't be accessed directly,you have to use php or python or whatever programming language for backend code.
for json file :benefits :no backend code directly accessed using GET http request.
downside:no filtering ,ordering or any queries,you have to do it manually.
For my simple App, i have a ftp server where i can store file (json or xml) or DB. Multiple clients could access that file or DB to read or write (DB or file would have only up to 100 entries).
From one point of view, DB is more suited for having big amounts of entries, due to indexing. But from other point of view, i am not sure if there would be issue with xml or json file if multiple clients try to read or write at the same time from the same file? So i am thinking to use DB just to avoid that issue.
I'd suggest using a database for a few reasons:
Databases are designed for exactly this scenario.
If you ever need to work on a larger scale you won't need to change your code.
You'll get to practice writing db code that will be useful in future, larger scale projects.
There are some really good database technologies that will work well with what you need, for example MongoDB, MySQL, SQL Server. They all have great support, code examples and you'll be able to use Stack Overflow to ask questions about them.
After googling, it seems that SQLite is the best choice. It is good approach for small DB, it is self-contained, allowing safe access from multiple processes or threads. Exactly what i needed.
I am in the process of building my first live node.js web app. It contains a form that accepts data regarding my clients current stock. When submitted, an object is made and saved to an array of current stock. This stock is then permanently displayed on their website until the entry is modified or deleted.
It is unlikely that there will ever be more than 20 objects stored at any time and these will only be updated perhaps once a week. I am not sure if it is necessary to use MongoDB to store these, or whether there could be a simpler more appropriate alternative. Perhaps the objects could be stored to a JSON file instead? Or would this have too big an implication on page load times?
You could potentially store in a JSON file or even in a cache of sorts such as Redis but I still think MongoDB would be your best bet for a live site.
Storing something in a JSON file is not scalable so if you end up storing a lot more data than originally planned (this often happens) you may find you run out of storage on your server hard drive. Also if you end up scaling and putting your app behind a load balancer, then you will need to make sure there are matching copy's of that JSON file on each server. Further more, it is easy to run into race conditions when updating a JSON file. If two processes are trying to update the file at the same time, you are going to potentially lose data. Technically speaking, JSON file would work but it's not recommended.
Storing in memory (i.e.) Redis has similar implications that the data is only available on that one server. Also the data is not persistent, so if your server restarted for whatever reason, you'd lose what was stored in memory.
For all intents and purposes, MongoDB is your best bet.
The only way to know for sure is test it with a load test. But as you probably read html and js files from the file system when serving web pages anyway, the extra load of reading a few json files shouldn't be a problem.
If you want to go with simpler way i.e JSON file use nedb API which is plenty fast as well.
I'm working on a Wordpress theme and it needs to have some settings. Until now I have them in the database but I also want them to be portable. For that reason I also saved the data as an array into a settings.php-file. Now I'm considering to don't use the database at all to avoid storing things twice.
Some questions about this
Are there any bad thing about storing data in an array within a included PHP-file, compared to MySQL database? (It's just settings, nothing needs to be sorted, no relations needed)
Which is fastest? Include a php-file with an array, or load data from the database?
Other thougts about this?
I give a check-vote to the most complete answer to my questions. Short answers might get a vote up.
I'm a pretty hardcore database guy and I would say they do not need to be in a database.
The clues are in your statements "It's just settings, nothing needs to be sorted, no relations needed" and "I also want them to be portable"
My main argument is simplicity. PHP is extremely good with arrays, it likes them, it understands them, can easily load them from files and save them to files. So even if you do change them from time to time from the app, updating and saving an array is no big deal. So, if you use the array, you use a native feature of PHP and that creates architectural simplicity for this feature.
So for portability, the most portable database is the one you do not use. When you have the simplicity of using a native PHP data format, you don't need the database (at least not for this)
For speed, on Linux anyway PHP can open a file and read it faster than it can make a roundtrip to the database for anything.
The only remaining argument against an array solution would be interaction with other data, but you have said there is none.
So, as a hardcore database guy, I would say do not use the database just because it is there. Databases are incredible for structured data, if this is just a flat list of settings it is not structured data. The db can do, the filesystem can do it, pick what is simpler.
Don't create a file based storage for your Wordpress-Theme-Settings if you want it to be portable. Some sites might have the themes folder readonly.
For the initial setup it's ok to read your settings from almost everything filebased. Later on its best stored (and backuped) in the database.
If you store things in a file use the ini-style based files as PHP gives you an API for free. Things usually only tend to be better stored in an array or serialized if your options are not restricted.
Don't care about performance too much, simply use the wordpress options api as a best practice.
If you're worried about portability, you might be interested in using ODBC or PHP Data Objects
As for which is fastest, I'm no expert, but the settings file only involves reading a text file and parsing it. The database option usually will result in TCP connections (unless you use mysqlite, which I would recommend if you are going to store more than just file paths and database names.
Regards,
The main problem of having them in a file occurs if you want the settings to be configurable via the website itself. If not, then having them in the code is no real cause for concern.
Most projects now need some form of a database. When someone says database, I usually think relational databases, but I still hear about flat file XML databases.
What parameters do you take into consideration when deciding between a "real" database and a flat-file XML database. When should one be used over the other, and under what circumstances should I never consider using a flat file (or vice versa a relational) database?
There is no such thing as a xml flat file database. Flat xml files are non-databases in that they have no higher functions like indices - have fun with larger datasets and searches or analytical queries without any index.
XML databases are another topic and may have their needs (content management, document storage in general - complicated schemata you dont care too much from the database point of view).
Flat files are fine for things like settings 8smaller files), but a real database is a real database. ACID conditions are hard to guarantee for flat files.
To add to Rachel's answer.
concurrency
read vs. write
If you have something simple that's going to be read often and is not going to change much it might be more optimal to use a flat file and save the overhead.
On the other hand, if you have to support multiple connections that are going to be adding and updating the data you'll want to use a database.
Few Parameters to consider is
Amount of Data
Complexity of Data
Relationship between Data
If we have less amount of Data with low Complexity and no inter-relationship than people would go for Flat file but in real application this is rarely the case and so you will always find Relational Databases used very often.
XML file is not a database. Read Joel's "Back to basics" article to see the difference.