In my DB project for website which will be created using PHP I have round 60 tables with 2 to 6 columns in each. Some of them can have thousands records, some hundreds and some only a few (1-10). These tables where are only a few records are the one that describe user type or city size from where later in other table is only reference to this. The bigger tables are for example the user contact data info.
What I guess is also important in this architecture is that the tables are created the way that only inserts are complex because for example user insert (creation) is operation that is inserting values to 6 tables. But all updates or queries are done generally on max 3 tables.
With such big database I am wondering is it better to stay with smaller tables and during query make it more complex call, or create fewer tables with bigger number of columns?
If staying with smaller tables is better then should I use JOIN, create some VIEW, or maybe something else?
I hope I've explained completely my concerns and how database looks like.
Thank you for answers in advance.
In my view you should stay with smaller tables. because it is easy to maintain data in them. Also Normalise your database. save reference in your master Tables. Use foreign keys to make relationships between tables.
Than use joins to fetch data you want its very easy. I always follow this approach. and one thing Only You Know What is best suitable for you .
Related
So I started working for a company where they had 3 to 5 different tables that were often queried in either a complex join or through a double, triple query (I'm probably the 4th person to start working here, it's very messy).
Anyhow, I created a table that when querying the other 3 or 5 tables at the same time inserts that data into my table along with whatever information normally got inserted there. It has drastically sped up the page speeds for many applications and I'm wondering if I made a mistake here.
I'm hoping that in the future to remove inserting into those other tables and simply inserting all that information into the table that I've started and to switch the applications to that one table. It's just a lot faster.
Could someone tell me why it's much faster to group all the information into one massive table and if there is any downside to doing it this way?
If the joins are slow, it may be because the tables did not have FOREIGN KEY relationships and indexes properly defined. If the tables had been properly normalized before, it is probably not a good idea to denormalize them into a single table unless they were not performant with proper indexing. FOREIGN KEY constraints require indexing on both the PK table and the related FK column, so simply defining those constraints if they don't already exist may go a long way toward improving performance.
The first course of action is to make sure the table relationships are defined correctly and the tables are indexed, before you begin denormalizing it.
There is a concept called materialized views, which serve as a sort of cache for views or queries whose result sets are deterministic, by storing the results of a view's query into a temporary table. MySQL does not support materialized views directly, but you can implement them by occasionally selecting all rows from a multi-table query and storing the output into a table. When the data in that table is stale, you overwrite it with a new rowset. For simple SELECT queries which are used to display data that doesn't change often, you may be able to speed up your pageloads using this method. It is not advisable to use it for data which is constantly changing though.
A good use for materialized views might be constructing rows to populate your site's dropdown lists or to store the result of complicated reports which are only run once a week. A bad use for them would be to store customer order information, which requires timely access.
Without seeing the table structures, etc it would be guesswork. But it sounds like possibly the database was over-normalized.
It is hard to say exactly what the issue is without seeing it. But you might want to look at adding indexes, and foreign keys to the tables.
If you are adding a table with all of the data in it, you might be denormalizing the database.
There are some cases where de-normalizing your tables has its advantages, but I would be more interested in finding out if the problem really lies with the table schema or with how the queries are being written. You need to know if the queries utilize indexes (or whether indexes need to be added to the table), whether the original query writer did things like using subselects when they could have been using joins to make a query more efficient, etc.
I would not just denormalize because it makes things faster unless there is a good reason for it.
Having a separate copy of the data in your newly defined table is a valid performance enchancing practice, but on the other hand it might become a total mess when it comes to keeping the data in your table and the other ones same. You are essentially having two truths, without good idea how to invalidate this "cache" when it comes to updates/deletes.
Read more about "normalization" and read more about "EXPLAIN" in MySQL - it will tell you why the other queries are slow and you might get away with few proper indexes and foreign keys instead of copying the data.
I have a huge Database table containing around 5 Million rows. Now retrieving records make the server slow in some cases. How can i manage the table now as it growing over the days.
I was thinking to make some archiving technique on yearly basis for example breakdown the complete tables into many small tables on yearly basis, but that cost me a lot of changes in coding. I have to change the complete structure of querying from database. So, most probably changes on most of the places in project.
What else i can do to reduce fetching time of records from database tables? Is there any other technique that i can adopt to avoid many changes in my code?
Thanks in advance
You can easily PARITION the table horizontally using PARITION BY RANGE in MySQL.
Also if you have many columns in table then you can break that table into two or more tables by Vertical partitioning method.
Also add proper indexes preferably clustered or covering indexes on tables and test queries for performance by using EXPLAIN.
May be table partitioning will help in that case. The following provide the info :
http://dev.mysql.com/doc/refman/5.1/en/partitioning-overview.html
https://dba.stackexchange.com/questions/19313/partitioning-a-table-will-boost-the-performance
I already saw a few forums with this question but they do not answer one thing I want to know. I'll explain first my topic:
I have a system where each log of multiple users are entered to the database (ex. User1 logged in, User2 logged in, User1 entered User management, User2 changed password, etc). So I would be expecting 100 to 200 entries per user per day. Right now, I'm doing it in a single table and to view it, I just have to filter out using UserID.
My question is, which is more efficient? Should I use one single table or create a table per user?
I am worried that if I use a single table, the system might have some difficulty filtering thousands of entries. I've read some pros and cons using multiple tables and a single table especially concerning updating the table(s).
I also want to know which one saves more space? multiple table or single table?
As long as you use indexes on the fields you're selecting from, you shouldn't have any speed problems (although indexes slow writes, so too many are a bad thing). A table with a few thousand entries is nothing to mySQL (or any other database engine).
The overhead of creating thousands of tables is much worse -- say you want to make a change to the fields in your user table -- now you'd have to change thousands of tables.
A table we regularly search against for a single record # work has about 150,000 rows, and because the field we search for is indexed, the search time is in very small fractions of a second.
If you're selecting those records without using the primary key, create an index on the field you use to select like this:
CREATE INDEX my_column_name ON my_table(my_column_name);
Thats the most basic form. To learn more about it, check here
I would go with a single table. With an index on userId, you should be able to scale easily to millions of rows with little issue.
A table per user might be more efficient, but it's generally poor design. The problem with a table per user is it makes it difficult to answer other kinds of questions like "who was in user management yesterday?" or "how many people have changed their passwords?"
As for storage space used - I would say a table per user would probably use a little more space, but the difference between the two options should be quite small.
I would go with just 1 table. I certainly wouldn't want to create a new table every time a user is added to the system. The number of entries you mention for each day really is really not that much data.
Also, create an index on the user column of your table to improve query times.
Definitely a single table. Having tables created dynamically for entities that are created by the application does not scale. Also, you would need to create your queries with variable tables names, something which makes things difficult to debug and maintain.
If you have an index on the user id you use for filtering it's not a big deal for a db to work through millions of lines.
Any database worth its salt will handle a single table containing all that user information without breaking a sweat. A single table is definitely the right way to do it.
If you used multiple tables, you'd need to create a new table every time a new user registered. You'd need to create a new statement object for each user you queried. It would be a complete mess.
I would go for the single table as well. You might want to go for multiple tables, when you want to server multiple customers with different set of users (multi tenancy).
Otherwise if you go for multiple tables, take a look at this refactoring tool: http://www.liquibase.org/. You can do schema modifications on the fly.
I guess, if you are using i.e. proper indexing, then the single table solution can perform well enough (and the maintenance will be much more simple).
Single table brings efficiency in $_POST and $_GET prepared statements of PHP. I think, for small to medium platforms, single table will be fine. Summary, few tables to many tables will be ideal.
However, multiple tables will not cause any much havoc as well. But, the best is on a single table.
How many fields are OK to have in a MySQL table?
I have a form to get information from users, it is divided into 6 or 7 unites, but all related to each other. They are about 100 fields.
Do you recommend to keep all of them in one table?
Thank you all for your responses,well my table's data are about some personal information of users,all the work that is needed to be done is to get them from the user,save and show them and have the edit option.
should i use the normalization? doesn't it increase the process time in my situation?
Providing you are following database normalization, you generally should be ok - although you may find some performance issues down the road.
To me, it seems like perhaps there could be some normalization?
Also, you should consider how many of these columns will just have null values, and what naming conventions you are using (not just name, name2 etc)
In case you want to read into it more.:
Data normalization basics
MySql :: An introduction to database normalization
MySQL supports up to 4096 columns per table. Dont use a field in main table if it often takes empty values.
if a column takes NULL no problem but if one stored as " " its memory wastage.
Too often null values means optional field data then why should one store it in main table.
Thats what i meant in the statement.
For data that does not need to be indexed or searched I create a single column in my database like col_meta which holds a serialized PHP array of values, it saves space and can expand or contract as needed. Just a thought.
There's no general rule to that. However, 100 columns definitely hints that your DB design is plain wrong. You should read about normalization and database design before continuing. I have a DB design with ~150 columns split up into nearly 40 tables, just to give you an idea.
If they all relate and can not be normalised (not to mention nothing > 1 is serialised) then I guess it will be OK.
Are you sure some things can't be split into multiple tables though? Can you provide a sample of the information you are gathering?
Could you split things into such as
user_details
user_car_details
...
I would typically start splitting them into separate tables after around 20-30 columns. However, it is really driven by how you are going to use the data.
If you always need all the columns, then splitting them out into separate tables will add unnecessary complication and overhead. However, if there are groups of columns that tend to be accessed together, then splitting along those lines with a 1-1 relationship between the tables might be useful.
This technique can be useful if performance is a concern, especially if you have large text columns in your table.
Lastly, I would echo the other posters comments about normalisation. It is relatively rare to have 100s of columns in one table and it may be that you need to normalise the table.
For example columns, SETTING_1, SETTING_2, SETTING_3, SETTING_4 might be better in a separate SETTINGS table, which has a 1-many relationship to the original table.
Thank you all,so i think it's better for me to merge some of my data into one column,which save me more space and decrease overhead
I've noticed a performance loss around 65 Columns in a DB with around 10 000 Rows. But could be lack of serverstructure or something else.
Assuming I have an application where I expect no more than 50,000 users.
I have 7 tables in my DB schema. Is it a good idea to replicate all the
tables for every user? So, in my case, number of tables will roughly be
50,000 * 7 = 350,000.
Is it in anyway better than 7 monolithic tables?
NO, I would not recomend creating a table schema per user in the same database.
mysql should handle the load perfectly well, given the correct indexes on tables.
What you're proposing is horizontal partitoning. http://en.wikipedia.org/wiki/Partition_%28database%29
i.e. you take all the rows in what would (logically) be one table, and you spread those rows across multiple tables, according to some partitioning scheme (in this case, one table per user).
In general, for starter applications of this nature, only consider partitioning (along with a host of other options) when performance starts to suffer.
No, it's definitely not. For two reasons:
The database design should not change just because you store more data in it.
Accessing different tables is more expensive and more complicated than accessing specific data in a single table.
Just image if you wanted statistics for a value from each user. You would have to gather data from 50 000 tables using 50 000 separate queries instead of running a single query against one table.
A table can easily contain millons of records, but i'm not sure that the database can even contain 350 000 tables.
Certainly not. A DB is typically optimized for having multiple rows in a given table... Try to rethink your DB schema to have a field in each of your tables that holds the user's ID or another associative table that holds the user id and key for the particular entry in the data table.
There's a decent intro to DB design here.