Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
We don’t allow questions seeking recommendations for books, tools, software libraries, and more. You can edit the question so it can be answered with facts and citations.
Closed 6 years ago.
Improve this question
I have to handle 12000 objects in Node.js from https://data.detroitmi.gov/Public-Safety/DPD-All-Crime-Incidents-2009-Present-Provisional-/b4hw-v6w2/data for map data. (JSON example : https://data.detroitmi.gov/resource/i9ph-uyrp.json)
I heard that Redis helps to deal with a lot of data. but my friend said that Those are not that big enough to use Redis so It would not worth to use Redis. I've only used mongo before but after I deal with those a bunch of data, I felt I can't handle with normal database. So decided to use something like Redis but not really sure. I would like to here some ideas about it!
That basically depends on what you mean by "handle".
Redis is in-memory database, so despite it has (a kind of) persistence layer implemented (regular snapshots on disk), some redundancy support (master-slave with replication), and even features for scalability (autosharding through Redis Cluster) - first and foremost, it's in-memory DB.
Which means: it's excellent in serving reads/writes for relatively small dataset (yours fits perfectly, btw) blazingly fast, putting consistency & scalability way behind (setting aside more tricky & flexible features).
If that is your purpose - go for it, but the good practice here (widespread, at least) is having a persistence layer served by another database, whether would it be more traditional ACID-type one (like, relational) or trendy distributed stuff.
PS In general, I personally better stick to first option (RDBMS) for persistence layer - they are way more mature and still outperform document-oriented even being used as advanced key-value store.
Although in real life there are no "general" cases, each is specific, so you're going to evaluate it in details.
PPS Another personal opinion: no Mongo, it's a total misadventure by itself.
Related
Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
We don’t allow questions seeking recommendations for books, tools, software libraries, and more. You can edit the question so it can be answered with facts and citations.
Closed 7 years ago.
Improve this question
Are there any self-hosted equivalents to Sendgrid, Mandrill, and other like services? With Mandrill, for instance, there are a number of features like bounce and open tracking, all exposed over a nice user interface. I'm looking for a similar technology, but one we could host in-house (I'm aware about the need to warm up IPs, etc). I found one solution called MessageGears but it looks like they're maybe more geared towards large enterprise implementations.
The problem with such in-house systems is that they either concentrate on the laborious task of sending out emails efficiently (and thus, lack a friendly interface) - or they do the opposite which is they have a very nice interface, but not as robust in the meat and potato department of sending messages out.
Ideally, you would want a combination of something like Sendy + a very fast email relay. I can recommend socketlabs for the second part.
They have two main products - an assembly server (think of it like a very robust templating engine), and a MTA. Combine the two for your own hosted campaign management server. Their MTA server is especially robust and full featured; and they have a comprehensive api as well.
I will warn you though - these products are designed as fast delivery engines, so unless you have a large volume of emails they may not be the best solution. The second thing which I alluded to a bit is their interface; its not the best but it will get the job done; have a look at the screenshots and judge for yourself.
You'll have to build something on top of this yourself. Sendy is a good candidate to customize for this.
I haven't tested it, but I've heard of this solution a couple weeks ago:
http://sendy.co/
Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 8 years ago.
Improve this question
Please read before you answer: I don't need any opinion-based answers or "nosql vs sql which is better" debates on the subject, just facts.
I want to slowly convert a php+mysql website I wrote with Symfony2 into a real time application using backbone with nodeJs + websockets.
I want to make a slow transition by changing single features, since I don't want to break a fully functional site.
I have been educating myself about NodeJs by reading books and watching Tutorials and there is one thing I noticed, I own more than 5 nodeJs books and none of them use MySql although its fully supported by node.
They all use MongoDB.
Here is my situation
1. My Website is already integrated into MySql(Doctrine)
2. My MySQL setup is fully functional and needs no improvements so far
I'm really frustated and I have a few questions:
Why is MySql not prefered although its a more mature piece of technology?
What are the advantages of moving to MongoDB over MySQL for the purpose of having a real-time application??
I've seen people choose Node/Mongo development because of the simplicity of the all-JavaScript stack, I've seen people choose Mongo because it's the New Hotness, and I've seen people choose Mongo because it's actually the right tool for the job: they have a large amount of document-like, unstructured data and/or they want to take advantage of Mongo's support for horizontal scaling, among other differences between MySQL and Mongo.
I'm not sure it's possible to answer this question in a non-opinion-based manner and without touching on Sql vs NoSql. Mongo is simply a tool, and it happens to be free and commonly used in the field with Node. If I were writing a Node tutorial, I'd probably choose Mongo too, because it's common and it's cool.
If Mongo is the right tool for your site's use cases, then the transition is probably worth it. If MySQL is the right tool for your site, then congratulations! You've just saved a bunch of time rewriting your DB in Mongo.
As an aside- if your question uses the word "preferred", I can't really think of a way for it to not be opinion-based, by definition.
Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
Questions asking us to recommend or find a tool, library or favorite off-site resource are off-topic for Stack Overflow as they tend to attract opinionated answers and spam. Instead, describe the problem and what has been done so far to solve it.
Closed 9 years ago.
Improve this question
I have a relatively large MySQL database (over 300 tables) which I desperately need to convert to PostgreSQL and synchronise data between the two databases if not real time then something close to it. Ideally I need a bi-directional data sync, or at least one directional - MySQL to Postgres sync.
I have managed to convert the database and import the data, but synchronisation seems to be a real problem.
This solution from DBConvert should supposedly do exactly that. After many days of trying to make it work I gave up. They don't even have a linux client which is strange considering that absolute majority of MySQL and Postgres database would run on linux servers.
Is there an alternative to DBConvert's solution that would do the same?
Check out Pentaho ETL tools Kettle and its client interface, Spoon. http://kettle.pentaho.com
boy, you have a job ahead of you in terms of bidirectional synchronization. This is hard on the best of days, and it poses a lot of problems.
The tool I would look at first, actually, would be RubyRep. This gives you a basic framework for replication between your databases, and it supports a number of RDBMS's.
The second thing you have to think about is what you are actually doing and why this is a really bad idea. Your biggest issue is conflict resolution and managing what happens if two different people update the same record on the different db's. This is not a trivial problem and it requires thinking through the actual workflows and scenarios carefully.
Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 3 years ago.
Improve this question
I know google uses its own Big Tables (is that right?) and Facebook and Twitter use Cassandra but when does your everyday project outgrow mysql (if ever?)
If you were starting out on a potentially massive scale web application, would you use mysql as an engine or start with an alternative from the start?
I think the only way you can know when MySQL isn't good enough is when you start to see performance issues or you feel like your fighting to keep it going. If you are aware that your application is potentially huge then you should be implementing the right tools from the start otherwise it's a huge headache transferring at a later date.
There is no simple metric which will tell you the answer - it depends not only on the amount of data, number of transactions but also the nature of the replication - number of replicated sites, required speed of replication etc.
Yes, a large scale noSQL clsuter can out-perform a a MySQL cluster built for the same budget for OLTP, however its called noSQL for a reason - when you need to start doing somethng useful with the data, the relational model and SQL language make slicing and dicing the data much easier. OTOH, at some point OLAP then overtakes the relational model in terms of performance - but I think it would be rather difficult to use a datawarehouse for transaction processing.
So its quite possible that the functional requirements of an application will outgrow the capabilities of a noSQL database much faster than the perofrmance requriements would outgrow a relational database.
I'd start with an alternative (PostgreSQL), but not because of scaling issues, but because MySQL's support for transactions and referential integrity is worthless.
Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
We don’t allow questions seeking recommendations for books, tools, software libraries, and more. You can edit the question so it can be answered with facts and citations.
Closed 7 years ago.
Improve this question
I know implementing database is a huge topic, but I want to have a basic understanding of how database systems work (e.g. memory management, binary tree, transaction, sql parsing, multi-threading, partitions, etc) by investigating the source code of the database.
Since there are a few already proven very robust open source databases like mysql, sqlite and so on. However, the code are very complicated and I have no clue where to start. Also I find that the old school database textbooks are only explaining the theory, not the implementation details.
Can anyone suggest how I should get started and if there are any books that emphasis on the technology and techniques of building dbms used in modern database industry?
I hate to sound like an grumpy, old academic, but the theory really is what you need to study, if you are determined to build your own RDBMS. The implementation details are really just, erh, implementation details. Apart from textbooks, you might also want to study research papers, which tend to cover the subject in higher detail.
When you start implementing your database engine, you could look into existing open-source implementations, but do expect the learning curve to be steep. As you have already discovered, these projects tend to be quite complex. When you have concrete questions to those projects, try posting them on the relevant mailinglists. When you have concrete questions about your own implementation, post them here :)
http://en.wikibooks.org/wiki/Design_of_Main_Memory_Database_System
Refer, Database system implementation book by Ullman. Start with jdbc or odbc driver and move down to storage layer if you want to read from code.