Closed. This question is off-topic. It is not currently accepting answers.
Want to improve this question? Update the question so it's on-topic for Stack Overflow.
Closed 11 years ago.
Improve this question
I have a copy of the PAF (UK Postcode) database it is currently stored in a MySql Database, and i use it on my site to pre-fill address details, however the Database huge 28,000,000+ records and it is very slow to search.
Any ideas how I could slit the DB to improve performance?
Thanks for the help guys!
that is not a large database, not even a large table. you must set appropiate indexes over the table and you will get good performance
There could be several ideas:
create indexes, meaningful ofcourse
review your schema. Avoid using huge datatypes like INT, BIGINT, TEXT etc unless absolutely required
optimize your queries so they use indexes, EXPLAIN statement might help
split your table into multiple smaller tables, say for example based on zones - north, east, west, south etc.
If your table doesn't require many INSERTs or UPDATEs, which I assume it might not being a postcode table, query cache can be a big help for faster queries
You will need to play around and see what option works best for you. But I think the first 2 should just be enough.
Hope it helps!
Related
Closed. This question needs details or clarity. It is not currently accepting answers.
Want to improve this question? Add details and clarify the problem by editing this post.
Closed 6 years ago.
Improve this question
Say there are 1,000 tables in a database, and each table has 1,000 rows. When I search for single table from these 1,000 tables, is the search time same as that required to search for data within one of the tables?
In other words, does SQL use the same search algorithm to find a table out of 1,000 tables as it does to get data from a table with 1,000 rows?
No, MySQL doesn't use the same search algorithm to find a table.
MySQL maintains an in-memory "data dictionary" so when you run a query that names a specific table, it looks up that table very quickly. It's much faster for MySQL to identify a table than to search for data within a table. For example, the database servers I maintain at my job have over 150,000 tables, and this isn't a problem.
Does this mean you should split up your data over many tables to make it run faster? No -- that's not usually a good tradeoff. It makes your application code more complex, since your code needs to pick which table to query. You may also find cases where you wish the data were in one table, if you have to search for results across many of your tables.
Here are a couple of principles to follow:
"Everything should be made as simple as possible, but not simpler." (attributed to Albert Einstein)
"First make it work, then make it right, and, finally, make it fast." (Stephen C. Johnson and Brian W. Kernighan, 1983)
Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 7 years ago.
Improve this question
These days I've faced performance issues when binding data with Java object to database. Especially when paring the data from database to java code when a lot of FK-PK relationship involved. I realized the issue and solved the performance slowdown by creating database views and create POJOs to map with the view.
I did some research online but couldn't find a good answer for this: How does database(I am using mysql) keeps the fast data querying speed in views?
For example, if I create a view among 10 tables, with FK-PK relationship, the view is still pretty fast to query and display the result pretty fast. How exactly happened behind the scenes for the database engine?
Indexes.
MySQL implicitly creates a foreign key index (i.e. an index on columns that compose the foreign key), unless one already exists. Not all database engines do so.
A view is little more than an aliased query. As such, any view, as trivial as it may seem, could kill the server if written poorly. Execution time is not proportional with the number of joined tables, but with the quality of indexes*.
Side effect: the default index might not be the most efficient one.
*tables sizes also start to matter when the tables grow large, as in millions of records
Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 8 years ago.
Improve this question
Currently I am working on a database that requires me to take raw data from a 3rd party and store it into a database. The problem is that the raw data is obviously not optimized, and the people who I'm building the database for, don't want any data entry involved when uploading the raw data into the database, they pretty much just want to upload the data and be done with it. Some of the raw data files have empty cells all over the place and many instances of duplicate names/numbers/entries. Is there a way to still optimize the data quickly and efficiently without too much data entry or reworking each time data is uploaded or is this an instant where optimization is impossible due to constrants? Does this happen a lot, or do I need to tell them their dreams of just uploading are not possible for long team success?
There are many ways to optimize data and one way to optimize data in one use case may be horrible in another use case. There are tools that will tell you there are multiple values in columns that need to be optimized but there is no single advice which works in all cases.
without specific details, this is always good:
With regards to empty entries, that should not be an issue
With regards to duplicate data, it may be worth considering adding a one to many relationship
One thing need to make sure is to put a key in any field you are going to search for, this will speed up a lot your queries no matter the dataset
as far as changing the database schema... rare are the schemas that do not change over time.
My advice is think through your schema but do not try to over optimize things because you can not plan in advance what the exact usage will be. As long as it is working and there is no bottleneck, focus on other areas. If there is a bottleneck, then by all means, rewrite the affected part, making sure indices are present (consider composite indices in some cases). Consider avoiding unions when possible. and remember the KISS principle (Keep It Simple and Sweet).
Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 9 years ago.
Improve this question
I need to know if it is more or less efficient to have multiple databases with an index of databases relative to each dataset.
I do not know to what extent multicache can adversely affect performance.
Suppose 10 bases in 2GB data each rather than a single 20GB.
For example: the data of userid 293484 are in third database.
Thanks.
Yes, this is a common technique known as sharding.
http://en.wikipedia.org/wiki/Shard_%28database_architecture%29
Altimately the code you will have to write to maintain such a structure will kill you.
Keep it simple, keep it in one database, and use proper design patterns and indexing.
Database engines are design to deal with large amounts of data, so if your hadrware is sufficient, your queries well structured and the design good, you should not have to many performance problems.
Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 9 years ago.
Improve this question
For a website having 2000000 users, out of which each user shares thousands of pictures and on each picture there will be thousands of comments, in such scenario there will be more than 2000 million comments, so how can I manage this much of big data using MySql. How can following methods improve performance of my database server
Use of table partationing
Use MySQL clusters
Use MySQL with memcached
Please explain other methods and best practices to handle such big database tables
On top of the mentioned optimization, choosing the right indexes on the right fields is crucial for your query performance, make sure your tables are indexed on everything you group, order or search based on.
Also make sure to check out Chapter 8 of the MySql reference which discusses optimization
What you really should be focusing on is optimizing the structure, queries and indexes before getting into memcached and MySql clusters.
As your database grows you monitor the performance and optimize accordingly.
In this case i dont thinl traditional RDBMS is what you need :) , more like NoSQl is what would serve you best