Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 6 years ago.
Improve this question
I have a table in MySQL that has about 50 million records (continuing growing), and it is about subscription consumptions.
So, everyday I have to select these records and make calculations on it in order to target different kind of consumptions/clients, for example if a client is active/inactive, how long has been active, if it had changed product, and so on.
At the moment, I have different queries to select the different business cases and then I load data to the staging area and data warehouse. Although, some of these queries are very low and they are overloading productive environment.
I would like to know if there is a known solution(s) or technology to this kind of daily tasks.
I am open to continue with MySQl or try a new big data technology. For example, selecting everyday the millions of raw records to a staging area/ODS and then work on them with some technology.
Does anybody know good solutions for these kind of tasks?
Thank you.
One option might be replication - http://dev.mysql.com/doc/refman/8.0/en/replication.html
That way you can run whatever queries you want on the replicated DB without impacting the live DB.
Related
Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 2 years ago.
Improve this question
Currently, I'm developing one project and there are lots of MySQL query operations with billions of records and also mathematic operation included and it takes more time to perform the query.
so I need your help in choosing technology for big data and DB operation
Currently, I'm using nodejs and MySQL DB
Thanks for giving me the right way to develop this.
It depends on your data. If your data is homogeneous (most of the rows has the same number of columns) and you need to perform complex queries with tons of joins, using a relational database as MySQL is a good option. You can also try other relational databases like Oracle DB, MariaDB and others. It shouldn't be difficult to export your current database and try if the performance improves.
On the other way, if your data is heterogenous and you don't need to perform complex join queries, a NoSQL database can be your option. There are a lot of them but one of the most famous ones is MongoDB. Moreover, Mongo has very good integration with NodeJS. Your main problem would be to convert your actual relational database to a non-relational database.
Closed. This question needs details or clarity. It is not currently accepting answers.
Want to improve this question? Add details and clarify the problem by editing this post.
Closed 8 years ago.
Improve this question
I will be writing a program in Delphi that will be reading and writing to a MySQL database tables on a regular basis like every 5 seconds. Is this going to be CPU intensive? or get to a point where computer will freeze completely? I know reading and writing to and from a hardrive nonstop can freeze everything on your computer. I am not really sure about MySQL database.
Databases are designed to handle many transactions frequently, but it really depends on what the queries you are using. A simple SELECT on a couple rows is unlikely to cause an issue, but large scale updates targeting many tables or multiple joins can slow performance. It all depends on what your queries are.
This all depends on the computer and the complexity of the query.
As David has said, it really does depend on the hardware and queries you are processing.
I would suggest measuring the processing time of each query to determine whether the writing processes will be stacking over the other 5 second interval queries.
You can find information on how to measure your MySQL processes here.
Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 8 years ago.
Improve this question
I am thinking of storing persons contact data centrally. So there will be so many persons and each will have their contact list. There will be more number of updates and selects on database as user will be searching their contacts or searching for a person not in his/her contact list. Person may be updating their contact details. But inserts in database will be limited because only one time enrollment will be there. I am confused in using databases MySQL or Neo4j. Because when I think of searching person from database neo4j seems better. But when I think of handling millions of records MySQL seems better. So can anyone suggest which database suits best? MySQL/Neo4j/ both MySQL and Neo4j or some other database?
Neo4j allows you to store the connections between the people via their contacts, so if you want to leverage the network effect in your application it makes sense to look into that.
It all depends on how you want people to search and interact with your app. If treat people as individual records with no connections then MySQL is good enough. Otherwise Neo4j would probably work better.
IF you have the time to a tiny PoC with some realistic data with both and then decide for yourself.
you can use MySQL latest version it is quite simple and relevant to your need , you need to just use locking system on your database or you can lock your table when inserting or updating.
Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 9 years ago.
Improve this question
I am developing a Real Time messaging application (such as WhatsApp and co) and I am facing a big question.
The application itself is not as complicated as what exists on the market. However, I am no sure what storage system I should use. I have several ideas but I don't know which one is better that the others:
A simple mysql database with relations between messages/conversations/conversations
A mongodb with replicate of each conversations for all users in the conversations
A redis store with replicate conversations for all users in the conversations.
I don't know which one is better for what I want to do. If you have some advise so I can choose the right solution. (or if there is a solution I haven't listed which is even better :) )
Note : My API is developped in Ruby On Rails (if this can help make a decision)
Data volume and number of read/writes should be the key factor leading you to the decision. If the data volume and number of read/write is not going to be huge you can do with mysql. I believe few TB of data with few hundreds of read/writes per minute is SQL database territory. Beyond that it is NoSQL world. However, you should be ready to deal with increased complexity of non-SQL data store design, query implementation, and achieving eventual consistency if you choose NoSQL solution. All the best!
Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 9 years ago.
Improve this question
I am in the middle of attempting to help my company digitize their history. One project is taking maintenance records for equipment and putting them into a mysql database. The idea is to be able to pull up a history at any time without flipping through piles and piles of paper.
My experience is limited to using phpMyAdmin to create tables and fumbling through php to output data how I want it. I've never used a relational setup.
The data fields would always be the same, the database would be populated via copy/paste from Excel (until such time comma delimited importing can be figured out), and this data would not need to be edited by endusers. It is strictly for viewing/printing purposes only.
Example fields:
id, unit number, unit_type, date, maintenance_performed
My question is, would putting all this into one table be an acceptable way to accomplish this task? Or would a relational setup be better due to the different types of units? Why?
I would focus on getting the data into the database and not on its storage. You are going to have enough problems copy-and-pasting the data in. For instance, how will you ensure that the dates are always in a consistent format?
After the data is loaded into tables, then you can worry about how to optimize it for querying purposes. How will new records continue to be uploaded? That will be a very important part of the process (I would recommend having field a creation date in the database, in addition to other information in the record).
After the data is loaded, you can worry about the best structure for organizing it. This is analogous to a real archivist, who tends to start by gathering lots and lots of data, and then figuring out the best way to organize it.