I need to drop a column from a big table (about 20 GB) in a production MySQL database,
but I don't want the MySQL server to hang or make a risk on production database.
This table is using the InnoDB engine and it contains around 10,000,000 records.
The best possible way as far as I know, is to use a MASTER-MASTER setup in mysql
You can modify MASTER1 first, and just use MASTER2 in production. Then you switch over and do the same.
Related
If we want to create mysql databases for cadence. Assuming we want 10 shards for cadence, we should create a set of mysql cadences tables for each shard? If we want 5 machine to create mysql database for 10 shards, how should we do?
Assuming we want 10 shards for cadence, we should create a set of mysql cadences tables for each shard
No. You will only need one set of tables for the whole Cadence cluster.
The sharding mechanism is implemented within Cadence server. Unless you have a sharded MySQL solution, you don't need to worry about anything about sharding when setting up database schema.
If you do have a sharded MySQL, just make sure to use shardID as partition(sharding) key for the table.
Sharding in Cadence is only needed for History service(that's why it's called numHistoryShards in the config).
More reading about the sharding:
https://cadenceworkflow.io/docs/operation-guide/setup/#static-configuration
Typically you will need 2K shards in production if MySQL is the database.
here is some reference that might help you for mysql , postgresql
I need a fresh opinion on the case. Any thoughts are appreciated.
Input: we have a huge percona mysql (5.5) database that takes a couple of Tb (terabytes). Tables on innodb engine.
More than a half (2/3) of that size should be deleted as quick as possible.
Also we have master-slave configuration.
As the quickest way to achieve that I am considering the following solution:
Execute for each table on the slave server (to avoid production downtime) :
Stop replication
Select the rows NOT to be deleted into an empty new table that has the same structure as the original table
Rename original table to "table_old", new table - to correct name
Drop the original table "table_old"
Start replication
The problem is that we have a lot of FK constraints. Also I am afraid to break the replication during this process.
Questions:
1) What the potential problems can be with FK constraints in this solution?
2) How do not break replication?
3) Opinions? Alternative solutions?
Thank you in advance.
if you can put db offline (aka no one is accessing the db except you) for a while, you can go with your solution but you need to drop the FK involved before and to recreate them after. You should also check for AUTO_INCREMENT columns that will change number with copy operation.
the FK are needed if you want the db online, I had a similar problem with some huge log tables, any try to delete all the records at a time will probably lock the database or corrupt the table.
so I went for a slow approach, I made a procedure that will delete batches of rows from the tables using clustered primary key, and then I scheduled it to run every n seconds.
Our server database is in mysql 5.1
we have 754 tables in our db.We create a table for each project. Hence the large no of tables.
From past one week i have noticed a very long delay in inserts and updates to any table.If i create a new table and insert into it,It takes one min to insert around 300 recs.
Where as our test database in the same server has 597 tables Same insertion is very fast in test db.
Default engine is MYISAM. But we have few tables in INNODB .
There were a few triggers running. After i deleted triggers it has become some what faster. But it is not fast enough.
USE DESCRIBE to know your query execution plans.
Look more at http://dev.mysql.com/doc/refman/5.1/en/explain.html for its usage.
As #swapnesh mentions, the DESCRIBE command is very usefull for performance debugging.
You can also check your installation for issues using:
https://raw.github.com/rackerhacker/MySQLTuner-perl/master/mysqltuner.pl
You use it like this:
wget https://raw.github.com/rackerhacker/MySQLTuner-perl/master/mysqltuner.pl
chmod +x mysqltuner.pl
./mysqltuner.pl
Of course, here I am assuming that you run some kind of a Unix based system.
You can use OPTIMIZE. According to Manual it does the following:
Reorganizes the physical storage of table data and associated index
data, to reduce storage space and improve I/O efficiency when
accessing the table. The exact changes made to each table depend on
the storage engine used by that table
The syntax is:
OPTIMIZE TABLE tablename
Inserts are typically faster when made in bulk rather than one by one. Try inserting 10, 30, or 100 records per statement.
If you use jdbc you may be able to achieve the same effect with batching, without changing the SQL.
I have about 100 databases (all the same structure, just on different servers) with approx a dozen tables each. Most tables are small (lets say 100MB or less). There are occasional edge-cases where a table may be large (lets say 4GB+).
I need to run a series of ALTER TABLE commands on just about every table in each database. Mainly adding some rows to the structure, but a few changes like change a row from a varchar to tinytext (or vice versa). Also adding a few new indexes (but indexing new rows, not existing ones, so assuming that isn't a big deal).
I am wondering how safe this is to do, and if there are any best practices to this process.
First, is there any chance I may corrupt or delete data in the tables. I suspect no, but need to be certain.
Second, I presume for the larger tables (4GB+), this may be a several-minutes to several-hours process?
Anything and everything I should know about performing ALTER TABLE commands on a production database I am interested in learning.
If its of any value knowing, I am planning on issuing commands via PHPMYADMIN for the most part.
Thanks -
First off before applying any changers, make backups. Two ways you can do it: mysqldump everything or you can copy your mysql data folder.
Secondly, you may want to use mysql from the command line. PHPMyAdmin will probably time out. Most PHP server has timeout less than 10 minutes. Or you accidently close the browser.
Here is my suggestion.
You can do fail-over the apps.(make sure no connections on all dbs) .
You can create indexes by using "create index statements" .don't use alter table add index statements.
Do these all using script like(keep all these statements in a file and run from source).
Looks like table sizes are very small so it wont create any headache.
I've a database with 3 of the tables having rows in excess of 20 million each. I've used GUIDs as primary keys (unfortunately). Now our database is about 20GB and growing 5GB per month.
It takes about 2 hrs to take full backup of the database, and 30hrs to restore on a box with 4GB RAM.
We once have all the tables from database disappeared. other mysql databases in same server were alright except one - for which only data was disappeared leaving empty tables.
A select query (among many slow queries) - which get max of a date column in one of 20m table takes about 5 mins to return result. This query used pretty frequently.
What I'm looking answers for
recommended db design changes
ways to improved select query performance - max date column on 20m records
other queries' performance
how to go about handling future db growth
Thanks all for your attention.
I've seen setups of larger size (with InnoDB as storage engine and a GUID as a primary key), and there were no such problems.
We once have all the tables from database disappeared. other mysql databases in same server were alright except one - for which only data was disappeared leaving empty tables.
The tables may seem empty if the system LSN has gone below the each page's LSN. This may happen if the InnoDB logfiles are corrupt. InnoDB, however, will issue a warning in this case.
A select query (among many slow queries) - which get max of a date column in one of 20m table takes about 5 mins to return result. This query used pretty frequently.
Create an index on this column, the query will be instant.
Please post the exact query and I'll tell you how to create the best index.
I see no problem in the DB design as such, most probably it's something with your server.
Is it possible to reproduce this behavior on another server with a clean vanilla MySQL installation?
You may also want to try to split data between the tables. Set innodb_file_per_table and restore from the backup.
A free alternaive to innodb hot backup is Percona XtraBackup Tool.
For backup, you could use the innodb hot backup tool. This not only lets you do consistent backups while your database is up, but the restore is much faster than the one you're doing (I'm assuming mysqldump?). It does cost money.
You might also try Mydumper: http://www.mydumper.org/
It is a great tool and is free and open source