I have a table with 1 million records. As my query will maily based on one column(32 constants). I am trying to add 32 partitions use type list.
My application can't stop, there will be insert some record in the meanwhile? Can I add partition to the table? Does it impace my application. Such as lock some rows duirng the partition.
I search the internet, but didn't find too much material abou the story of Add partition to existing table?
THank you.
A common way to do a table migration such as this without any impact to the application is to follow these steps:
Create a duplicated table that contains the revisions you require (in your case the partition)
Setup a trigger on the original table to insert into the duplicated table (this will act as a form of replication for a short period of time)
Start a migration from the original table to the new table, at a rate that will not hinder your application (say 1000 rows at a time)
When the migration completes, you'll have your tables in perfect sync, this is the time for you to modify your application to start reading and writing using the new table.
Once you're happy that your app is functional using the new table, drop the old table.
Migration complete, have a beer.
Related
Since partition also splits the table into subtables, I wanted to know if there is any way to index the partitioned table one by one based on the partition name or id. I am asking this because, my table can have 1 Billion+ rows and add index query takes long hours/day, so wanted to check if I can start adding index based on the partition that I think is more important first or vice versa.
No, MySQL has no syntax to support creating indexes on a partitioned table one partition at a time. The index will be added to all partitions in one ALTER TABLE or CREATE INDEX statement.
At my company, we execute schema changes using pt-online-schema-change, a script that allows clients to continue reading and writing the table while the alter is running. It might even take longer to run the schema change, but since it doesn't block clients, this doesn't cause a problem.
The script is part of the Percona Toolkit, which is a free, open-source collection of tools written in Perl and Bash.
I was working on table which has near about 50 million data(2GB-size). I had requirement to optimize the performance. So when I add index on column through phpmyadmin panel, table got lock and result in holding up all queries in queue on that table and ultimately results in restart/kill all queries. (And yeah, I forgot to mention I was doing this on production. My bad!)
When I did some research I found out some solution like creating duplicate table but any alternative method ?
You may follow this steps,
Create a temp table
Creates triggers on the first table (for
inserts, updates, deletes) so that they are replicated to the temp
table
In small batches, migrate data When done, rename table to new
table, and drop the other table
But as you said you are doing it in production then you need to consider live traffic while dropping a table and creating another one
I have a table which stores data out of a complex query. This table is truncated and new populated once per hour. As you might assume this is for performance reason so the application accesses this table and not the query.
Is truncate and insert the only way to resolve this task cheap, or are there other possibilities in respect of the transaction log?
If I am assuming right, you are using this table as a temp table to store some records and want to remove all records from this table every one hour, right?
Truncate is always minimally logged. So yes, truncate and then insert will work. Another option is to create a new table with same structure. Drop old table and then rename new table to the old table name.
If you want to avoid the above, you can explore the "simple" recovery model (this has implications on point of time recovery - so be very careful with this if you have other tables in this same database). Or you can create a new database which will just have this one table, set recovery for this DB to "simple". Simple recovery model will help you keep your t-log small.
Lastly, if you have to have full recovery and also cannot use "truncate" or "drop" options from above, you should at the very least backup your t-log at very regular intervals (depending on how big its growing and how much space you have).
We have a very large (10million+) row table stored in MySql using the InnoDB engine. Column 'x' is defined as 'smallint(5) unsigned not null'.
Requirements have now changed since the original design a few years ago, and column 'x' needs to store a minimum datatype size of 'int unsigned not null'.
We are allowed a "short" downtime to deploy the application (less than 5 mins) so any database change would need to fit within this time window if it requires the database table to be made temporarily unavailable in any way (e.g. full table lock). If the change can be done "online" then we can probably accept degraded performance for a longer period.
Has anyone here had experience of altering column type in MySql/InnoDB on a very large table? If so, did it go OK and roughly how long did it take (I realise this is hardware dependent, I am just trying to understand if what we are asking to do in the time window is even vaguely possible)?
Thanks in advance,
heres a recipe i've used
where you have old column, a
create a new column b
create a trigger to update b on update/insert on a
update b = a
drop all fkey relations referencing a
create fkey relations for b
update code to use column b instead of a (deploy)
drop triggers on column a
drop column a
repeat all steps if you must change the column name back.
You could do it online by creating a new table as a clone of the original's layout.
CREATE TABLE newtable LIKE oldtable;
Next, alter your new table's columns to meet the new spec
ALTER TABLE newtable ... ;
Once that's done, copy the data.
INSERT INTO newtable SELECT * FROM oldtable;
Finally, use your few minutes downtime to rename your old table to something else, then give your new table the old table's name.
Of course, you need to have enough storage for two copies of your table available. Also, you might want to disable the indexes on the new table when doing the copy to reduce stress on the server during the copy operation.
Once you're confident that all the data in the old table has been copied to the new without any lossage you can finally delete the old table entirely.
EDIT:
Actually, a better approach might be to just add a new column to your existing table that meets the new requirements for the column you want to update, copy the values of the old column to the new column, drop the old column and rename the new column to the old column's name. It would certainly put lower demands on your storage.
Also, if you have any FK constraints that depend on the column in question, you will need to remove them before starting and reinstate them on the new column once you've done.
What kind of high-availability solution do you currently have in place?
I ask because, 5 minutes is not enough downtime to allow a reboot for normal tasks such as OS updates.
You must therefore, have some kind of high-availability solution in place to allow these regular (I assume your operations team continues to apply patches from time to time) updates.
It should be possible to use such a system to do this.
However, ALTER TABLE allows the table to remain available for read operations throughout its run, and only blocks reads for a short time at the end (much less than five minutes on most systems).
So ask, what time can you reasonably block writes for?
Also, 10M rows is not a large table, by any stretch of the imagination. It probably fits in ram hence will be very quick to alter.
This approach is pretty quick
Query OK, 10000000 rows affected (6 min 0.14 sec)
mysql slow query
I have a table with 8 millions records in mysql.
I want to keep last one week data and delete the rest, i can take a dump and recreate the table in another schema.
I am struggling to get the queries right, please share your views and best approaches to do this.Best way to delete so that it will not affect other tables in the production.
Thanks.
MySQL offers you a feature called partitioning. You can do a horizontal partition and split your tables by rows. 8 Million isn't that much, how is the insertion rate per week?
CREATE TABLE MyVeryLargeTable (
id SERIAL PRIMARY KEY,
my_date DATE
-- your other columns
) PARTITION BY HASH (YEARWEEK(my_date)) PARTITIONS 4;
You can read more about it here: http://dev.mysql.com/doc/refman/5.1/en/partitioning.html
Edit: This one creates 4 partitions, so this will last for 4 weeks - therefore I suggest changing to partitions based on months / year. Partition limit is quite high but this is really a question how the insertion rate per week/month/year looks like.
Edit 2
MySQL5.0 comes with an Archive Engine, you should use this for your Archive table ( http://dev.mysql.com/tech-resources/articles/storage-engine.html ). Now how to get your data into the archive table? It seems like you have to write a cron-job that runs on the beginning of every week, moving all records to the archive table and deleting them from the original one. You could write a stored procedure for this but the cron-job needs to run on the shell. Keep in mind this could affect your data integrity in some way. What about upgrading to MySQL 5.1?