i have a huge data.what is the best way for store this data in database.store this data to one table or store in several table?
if i want to save data to several table i must create table for every user.
I think a table per users is not a good, please read above link it will hwlp you to design a better and more efficient database
MySQL :: An Introduction to Database Normalization
http://ftp.nchu.edu.tw/MySQL/tech-resources/articles/intro-to-normalization.html
In a relational dbms, data table size doesn't dictate the database design, but only the relations of entities.
So even with a table of billions of records holding large multi media data, you would not make this several tables, just because the table gets so big.
Maybe using an RDBMS is a wrong approach for your task. Maybe a NoSQL dbms would be better. Even a simple file system could be the way to go.
Maybe however, an RDBMS is the right approach. We don't know, because you have told us almost nothing about your database. And what seems huge to you may be considered small by your dbms.
Are you saying that you would have a huge table holding data for all users, but every user will only be interested in their own data? Then simply partition the table by user. The database design will remain the same, only the underlying storage and internal data access will be different. As long as your queries always select one user's data, you will stay within one partition and data access will be fast.
Here is how to partition a table:
alter table user_movies
add primary key (user_id, movie_id)
partition by hash(user_id) partitions 100;
So many factors you should be taken into consideration before making decisions. But how about this one, try "no-sql" type db, each user will be treated as key, the info about this user will be treated as value.
Related
I have an online service (online vocabulary trainer). Each user has its vocabulary.
Now, I'm not sure, how I should structure my Mysql-DB.
As far as I know, I have different possibilities:
everything in one table (MyISAM): I store all the vocabulary in one large MyISAM-table and add a column "userid" to identify each user's vocabulary
every user has its own table (MyISAM): Every time, when a user is created, the programm adds a table named like "vocabulary_{userid}" where {userid} is to connect the table to a user.
everything in one table (InnoDB): Like point one, but with InnoDB instead of MyISAM.
The problem is, that one large vocabulary table can reach up to 100 millions rows. With MyISAM, the problem is, that every query locks the whole table. So I imagine, if there are many users online (and send many queries), the table might be locked a lot. And with InnoDB, I'm simply not sure, wheather this is a good solution as I'm having quite some SELECT-, UPDATE-, and INSERT- commands.
I hope anyone can help me. Thank you in advance.
It is almost always better to go with InnoDB. InnoDB can handle 100 milllions rows, the max size is 64tb.
It doesn't sound like you have a relational dataset, but more of a key/value store. Maybe Riak is a better solution.
It depends
If you start having one table per user (aka sharding) you will have some troubles at the beginning.
if you don't have the need of scale right now. go for 1 table with good indexes. I wouldn't use MyISAM but InnoDB instead otherwise you can get hit by the bigests issue of MyISAM (locks...)
The normal relational design for this would, I think, use three tables:
Users — user ID, and other attributes: name, email, etc
Vocabulary — least clear from the question, but presumably words with attributes such as part of speech and maybe meaning, probably including a word ID (because some word spellings have multiple meanings).
User_Vocabulary — a table with a User ID, Word ID, and maybe attributes such as 'date learned'.
If MyISAM locks the table while a query is going on, then you can't afford to use MyISAM if you need concurrent updates to the User_Vocabulary table. So, go with InnoDB for all the tables.
So I started working for a company where they had 3 to 5 different tables that were often queried in either a complex join or through a double, triple query (I'm probably the 4th person to start working here, it's very messy).
Anyhow, I created a table that when querying the other 3 or 5 tables at the same time inserts that data into my table along with whatever information normally got inserted there. It has drastically sped up the page speeds for many applications and I'm wondering if I made a mistake here.
I'm hoping that in the future to remove inserting into those other tables and simply inserting all that information into the table that I've started and to switch the applications to that one table. It's just a lot faster.
Could someone tell me why it's much faster to group all the information into one massive table and if there is any downside to doing it this way?
If the joins are slow, it may be because the tables did not have FOREIGN KEY relationships and indexes properly defined. If the tables had been properly normalized before, it is probably not a good idea to denormalize them into a single table unless they were not performant with proper indexing. FOREIGN KEY constraints require indexing on both the PK table and the related FK column, so simply defining those constraints if they don't already exist may go a long way toward improving performance.
The first course of action is to make sure the table relationships are defined correctly and the tables are indexed, before you begin denormalizing it.
There is a concept called materialized views, which serve as a sort of cache for views or queries whose result sets are deterministic, by storing the results of a view's query into a temporary table. MySQL does not support materialized views directly, but you can implement them by occasionally selecting all rows from a multi-table query and storing the output into a table. When the data in that table is stale, you overwrite it with a new rowset. For simple SELECT queries which are used to display data that doesn't change often, you may be able to speed up your pageloads using this method. It is not advisable to use it for data which is constantly changing though.
A good use for materialized views might be constructing rows to populate your site's dropdown lists or to store the result of complicated reports which are only run once a week. A bad use for them would be to store customer order information, which requires timely access.
Without seeing the table structures, etc it would be guesswork. But it sounds like possibly the database was over-normalized.
It is hard to say exactly what the issue is without seeing it. But you might want to look at adding indexes, and foreign keys to the tables.
If you are adding a table with all of the data in it, you might be denormalizing the database.
There are some cases where de-normalizing your tables has its advantages, but I would be more interested in finding out if the problem really lies with the table schema or with how the queries are being written. You need to know if the queries utilize indexes (or whether indexes need to be added to the table), whether the original query writer did things like using subselects when they could have been using joins to make a query more efficient, etc.
I would not just denormalize because it makes things faster unless there is a good reason for it.
Having a separate copy of the data in your newly defined table is a valid performance enchancing practice, but on the other hand it might become a total mess when it comes to keeping the data in your table and the other ones same. You are essentially having two truths, without good idea how to invalidate this "cache" when it comes to updates/deletes.
Read more about "normalization" and read more about "EXPLAIN" in MySQL - it will tell you why the other queries are slow and you might get away with few proper indexes and foreign keys instead of copying the data.
What is big table(i.e Google DataBase design), I have such type of requirement, but I don't know how to design it.
In Big Table, how to maintain relations among them?
create all table with innodb storage engine which maintain relationships
Choose table fields according to and limited to requirement
The paper of Big table, published by google, may be hard to read. Hope my answer can help you to start understanding.
In old days, RDBMS stores data according rows, one record one row, 1,2,3,4,5.....
Then if you want to find record 5, it's ok, database will seek in a B+ tree(or something similar) to get the address of record 5, load it for you.
But the nightmare is when you want to get records that have column "user=Michael", the database has no way but seek every record to check out if the user is "Michael".
Big Table has a different way to store data. It stores all the columns by an inverted table. When we want to find out all the records that satisfy "user=Michael", it seeks this as a key via a B+ tree or hash table, and gets the address of inverted table where stores the list of all records satisfying.
Maybe a good starting point is Lucene, an open source full text search engine, a fully implementation of big table principles.
Be noticed, inverted table is not a column-based storage in RDBMS. They are different, please must remember this.
I already saw a few forums with this question but they do not answer one thing I want to know. I'll explain first my topic:
I have a system where each log of multiple users are entered to the database (ex. User1 logged in, User2 logged in, User1 entered User management, User2 changed password, etc). So I would be expecting 100 to 200 entries per user per day. Right now, I'm doing it in a single table and to view it, I just have to filter out using UserID.
My question is, which is more efficient? Should I use one single table or create a table per user?
I am worried that if I use a single table, the system might have some difficulty filtering thousands of entries. I've read some pros and cons using multiple tables and a single table especially concerning updating the table(s).
I also want to know which one saves more space? multiple table or single table?
As long as you use indexes on the fields you're selecting from, you shouldn't have any speed problems (although indexes slow writes, so too many are a bad thing). A table with a few thousand entries is nothing to mySQL (or any other database engine).
The overhead of creating thousands of tables is much worse -- say you want to make a change to the fields in your user table -- now you'd have to change thousands of tables.
A table we regularly search against for a single record # work has about 150,000 rows, and because the field we search for is indexed, the search time is in very small fractions of a second.
If you're selecting those records without using the primary key, create an index on the field you use to select like this:
CREATE INDEX my_column_name ON my_table(my_column_name);
Thats the most basic form. To learn more about it, check here
I would go with a single table. With an index on userId, you should be able to scale easily to millions of rows with little issue.
A table per user might be more efficient, but it's generally poor design. The problem with a table per user is it makes it difficult to answer other kinds of questions like "who was in user management yesterday?" or "how many people have changed their passwords?"
As for storage space used - I would say a table per user would probably use a little more space, but the difference between the two options should be quite small.
I would go with just 1 table. I certainly wouldn't want to create a new table every time a user is added to the system. The number of entries you mention for each day really is really not that much data.
Also, create an index on the user column of your table to improve query times.
Definitely a single table. Having tables created dynamically for entities that are created by the application does not scale. Also, you would need to create your queries with variable tables names, something which makes things difficult to debug and maintain.
If you have an index on the user id you use for filtering it's not a big deal for a db to work through millions of lines.
Any database worth its salt will handle a single table containing all that user information without breaking a sweat. A single table is definitely the right way to do it.
If you used multiple tables, you'd need to create a new table every time a new user registered. You'd need to create a new statement object for each user you queried. It would be a complete mess.
I would go for the single table as well. You might want to go for multiple tables, when you want to server multiple customers with different set of users (multi tenancy).
Otherwise if you go for multiple tables, take a look at this refactoring tool: http://www.liquibase.org/. You can do schema modifications on the fly.
I guess, if you are using i.e. proper indexing, then the single table solution can perform well enough (and the maintenance will be much more simple).
Single table brings efficiency in $_POST and $_GET prepared statements of PHP. I think, for small to medium platforms, single table will be fine. Summary, few tables to many tables will be ideal.
However, multiple tables will not cause any much havoc as well. But, the best is on a single table.
I want to crate new table for each new user on the web site and I assume that there will be many users, I am sure that search performance will be good, but what is with maintenance??
It is MySQL which has no limit in number of tables.
Thanks a lot.
Actually tables are stored in a table too. So in this case you would move searching in a table of users to searching in the system tables for a table.
Performance AND maintainibility will suffer badly.
This is not a good idea:
The maximum number of tables is unlimited, but the table cache is finite in size, opening tables is expensive. In MyISAM, closing a table throws its keycache away. Performance will suck.
When you need to change the schema, you will need to do one ALTER TABLE per user, which will be an unnecessary pain
Searching for things for no particular user will involve a horrible UNION query between all or many users' tables
It will be difficult to construct foreign key constraints correctly, as you won't have a single table with all the user ids in any more
Why are you sure that performance will be good? Have you tested it?
Why would you possibly want to do this? Just have one table for each thing that needs a table, and add a "user" column. Having a bunch of tables vs a bunch of rows isn't going to make your performance better.
To give you a direct answer to your question: maintenance will lower your enthousiasm at the same rate that new users sign up for your site.
Not sure what language / framework you are using for your web site, but in this stage it is best to look up some small examples in that. Our guess is that in every example that you'll find, every new user gets one record in a table, not a table in the database.
I would go with option 1 (a table called tasks with a user_id foreign key) in the short run, assuming that a task can't have more than one user? If so then you'll need a JOIN table. Check into setting up an actual foreign key as well, this promotes referential integrity in the data itself.