I have a wamp local server with only MyISAM tables, Total size is 350 MB and increases by about 10 MB per day. Would like to have a hourly backup solution. As recently WAMP server crashed and lost data.Should I create a trigger or schedule a task or use some utility it would be great if someone can guide. So that database backup is takeon either on local network or on different partition of server HDD. Have mirror RAID enabled on the server.
Since InnoDB does not lose data in a crash, and since you are talking about backing up a lot of data, and you are in the 'early stages', the only real answer to your question is to switch to InnoDB.
When the server crashes, MyISAM almost always leaves some indexes in need of REPAIR TABLE. This problem vanishes with InnoDB.
If the server crashes in the middle of a multi-row INSERT, UPDATE, or DELETE on a MyISAM table, some rows will be done and other rows won't be. And you have no way to know how much was done. With InnoDB, the entire query is "rolled back", making it as if it had not occurred. Very predictable.
See my tips
on converting.
For better recovery from a crash where the server cannot be restarted at all, consider Replication.
Related
I have MySQL database which is in size of 4 TB and when I'm dumping it using mysqldump then it is taking around 2 days to dump that database in the .sql format
Can anyone help to faster this process?
OS ubuntu 14
MySQL 5.6
The single database of size 4 TB
hundreds of table average tables size is around 100 to 200 GB
Please help if anyone have any solution to this
I would:
stop the database,
copy the files in a new database
restart the database
process the data from the new place (maybe in an other machine).
If you are replicating, just stop replication, process, start replication.
These methods should improve speed, because of lack of concurrent processes that access the database (and all lock logic).
On such large databases, I would try not to have to make dumps. Just use mysql table files if possible.
In any case 2 days seems a lot, also for a old machine. Check that you are not swapping, and try to check your mysql configuration for possible problems. In general, try to get a better machine. Computer are cheaper than time to optimize.
I have an AWS RDS MySQL 5.7 database with MyISAM tables that I would like to migrate to another RDS in a custom VPC, and once migrated, convert those MyISAM tables to InnoDB.
If I undertood correctly, the only way to create a correct automatic backup is using the following procedure explained here: "Automated Backups with Unsupported MySQL Storage Engines"
https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_WorkingWithAutomatedBackups.html#Overview.BackupDeviceRestrictions
Stop all activity to your MyISAM tables (that is, close all sessions).
You can close all sessions by calling the mysql.rds_kill command for each process that is returned from the SHOW FULL PROCESSLIST command.
Lock and flush each of your MyISAM tables
Create a snapshot of your DB instance. When the snapshot has completed, release the locks and resume activity on the MyISAM tables
Has someone done this procedure before?
How is that the snapshots are being created successfully every night from the current RDS DBInstance, even though it contains MyISAM tables?
Thanks!
The problem isn't with snapshot creation. It's what can go wrong when you actually try to use one of the snapshots.
RDS snapshots work by capturing a snapshot if your RDS instance's underlying EBS volume (you can't see this volume, but it's there -- RDS runs on EC2, with "hidden" instances and volumes).
EBS snapshots capture the entire contents of the hard drive exactly as they happened to exist at the moment in time when the snapshot process starts.
What ends up on the snapshot is essentially the same thing that you would have on a MySQL Server if you executed sudo killall -9 mysqld -- it is as if the server had halted everything, immediately, without doing any of the things it normally does to clean up for a graceful shutdown. With RDS, things are not quite that dramatic, because RDS does take some precautions, but fundamentally, this is the nature of what is happening.
When you create an RDS instance from a snapshot, the first thing that happens when the instance starts up is the same thing your hypothetical server would do when you restarted the killed MySQL Server daemon: InnoDB Crash Recovery.
InnoDB Crash Recovery
To recover from a MySQL server crash, the only requirement is to restart the MySQL server. InnoDB automatically checks the logs and performs a roll-forward of the database to the present. InnoDB automatically rolls back uncommitted transactions that were present at the time of the crash.
https://dev.mysql.com/doc/refman/5.7/en/innodb-recovery.html#innodb-crash-recovery
Crash recovery is InnoDB's mechanism for bringing everything back into harmony in it internal data structures and ensure that all data is intact, exactly as your application left it. It's possible because InnoDB is a transactional storage engine. That means a lot of different things, but what it specifically means in this case is that InnoDB doesn't just change table data when you change a table. It goes through a process that can be simplified something like this:
store the proposed changes to disk¹
actually make the changes
mark the changes as complete
What this means is that until the changes are finalized, InnoDB can be interrupted and will subsequently be able to pick up where it left off, without corrupting or losing data.
MyISAM has no such mechanisms. It just writes to the data files, directly. Even if a MyISAM table isn't actively being used, it may still need to be repaired when the server comes up, to clean up its structures. In some circumstances, repairing the table can be impossible, and all or part of the data in the table will be lost.
If your MyISAM tables are flushed and locked when the snapshot occurs, they are in a quiescent state on the disk, as though the server had actually been gracefully shut down before the snapshot had occurred, so they will be stable on the snapshot.
But the snapshot process will always appear to succeed, because the snapshot is just duplicating whatever is on the disk, as it appears at the moment in time when the snapshot gets underway.
The problem is that what the snapshot captured may not be usable, and you have no way of knowing whether the snapshot will be fully viable.
¹ Note that the first step, "store the proposed changes to disk" is related to the system variable innodb_flush_log_at_trx_commit which makes the system slower if set to 1 but also is the safest setting, because your query doesn't actually succeed until that first step is done. A setting of 2 is still reasonably safe, because it still writes the changes but continues without requiring that the operating system confirm that they have actually been written to the hard drive before your query returns success... but in a crash, a transaction your application thinks was committed may or may not have survived.
Inside my cronjobs I make a full mysqldump every night.
My database has total 1.5GB data inside 20 tables.
Nearly every table has indexes.
I make backup like this:
mysqldump --user=user --password=pass --default-character-set=utf8 database
--single-transaction| gzip > "mybackupfile"
I make this for 2 months. This process takes nearly 1,5 minutes for 2 months.
Last week my hosting company changed my server. Just after the server change, this process started to long for 5 minutes. I told this to server company and they increased my CPU from 4GHz to 6 GHz so mysqldump process became 3,5 minutes. Then they increased to 12 GHz. But this didn't change the performance.
I checked my shared SSD disk performance with hdparm. It was 70 MB/sec. So I complain again. So they changed my hard disk to another one. Hard disk read speed became 170 MB/sec. So mysqldump process became 3 minutes.
But the duration is far from the previous value. What would be the cause for this performance degradation ? How can I isolate the problem ?
(Server is Centos 6.4, 12 GHz CPU, 8 GB RAM)
Edit: My company changed server again and I still have same problem. Old server has 3,5 minutes backup time now new server has 5 minutes time. Resultant file is 820 MB when zipped, 2.9 GB when unzipped.
I'm trying to find out what makes this dump slow.
Dump process started at 11:24:32 and stopped at 11:29:40. You can check it from screenshots' timestamps.
Screenshots:
General
Consumed memory
Memory and CPU of gzip
Memory and CPU of mysqldump
Disk operations
hdparm results:
/dev/sda2:
Timing cached reads: 3608 MB in 1.99 seconds = 1809.19 MB/sec
Timing buffered disk reads: 284 MB in 3.00 seconds = 94.53 MB/sec
/dev/sda2:
Timing cached reads: 2120 MB in 2.00 seconds = 1058.70 MB/sec
Timing buffered disk reads: 330 MB in 3.01 seconds = 109.53 MB/sec
Obvious thing I'd look at is whether the amount of data increased in recent months. Most database-driven applications collect more data over time, so the database grows. If you still have copies of your nightly backups, I'd look at the file sizes to see if they have been increasing steadily.
Another possibility is that you have other processes doing read queries while you are backing up. Mysqldump by default creates a read lock on the database to ensure a consistent snapshot of data. But that doesn't block read queries. If there are still queries running, this could compete for CPU and disk resources, and naturally slow things down.
Or there could be other processes besides MySQL on the same server competing for resources.
And finally, as #andrew commented above, there could be other virtual machines on the same physical server, competing for the physical resources. This is beyond your control and not even something you can test for within the virtual machine. It's up to the hosting company to manage a balanced allocation of virtual machines.
The fact that the start of the problems coincides with your hosting company moving your server to another host makes a pretty good case that they moved you onto a busier host. Maybe they're trying to crowd more VM's onto a single host to save rackspace or something. This isn't something StackOverflow can answer for you -- you should talk to the hosting company.
The number or size of indexes doesn't matter during the backup, since mysqldump just does a SELECT * from each table, and that's a table-scan. No secondary indexes are used for those queries.
If you want a faster backup solution, here are a few solutions:
If all your tables are InnoDB, you can use the --single-transaction option, which uses transaction isolation instead of locking to ensure a consistent backup. Then the difference between 3 and 6 minutes isn't so important, because other clients can continue to read and write to the database. (P.S.: You should be using InnoDB anyway.)
Mysqldump with the --tab option. This dumps data into tab-delimited files, one file per table. It's a little bit faster to dump, but much faster to restore.
Mydumper, an open source alternative to mysqldump. This has the option to run in a multi-threaded fashion, backing up tables in parallel. See http://2bits.com/backup/fast-parallel-mysql-backups-and-imports-mydumper.html for a good intro.
Percona XtraBackup performs a physical backup, instead of a logical backup like mysqldump or mydumper. But it's often faster than doing a logical backup. It also avoids locking InnoDB tables, so you can run a backup while clients are reading and writing. Percona XtraBackup is free and open-source, and it works with plain MySQL Community Edition, as well as all variants like Percona Server, MariaDB, and even Drizzle. Percona XtraBackup is optimized for InnoDB, but it also works with MyISAM and any other storage engines (it has to do locking while backing up non-InnoDB tables though).
My question is: Do you really need a dump or just a copy?
There is a great way that is far away from mysql dump, it uses Linux LVM "LVM Snapshot":
http://www.lenzg.net/mylvmbackup/
The idea is to hold the database for a milli second, then LVM will make a hot copy (which takes another milli second) and then the database can continue to write data. The LVM copy is now ready for every action you want: copy the table files or open it as new mysql instance for a dump (which is not on the production database!).
This needs some modifications to your system. Maybe those mylvmbackup scripts are not completely finished jet. But if you have time yourself you can build on them and do your own work.
Btw: if you really go this way, I'm very interested in the scripts as I am also need a fast solution to clone a database from production environment to a test system for developers. We decided to use this LVM snapshot feature but - as always - had no time to start with it.
Currently my database is almost 20 GB big and still growing.
I'm taking a daily backup with mysqldump and it's getting really slow.
So slow that in the meanwhile new connections stack up and eventually cause this error:
SQLSTATE[HY000] [1040] Too many connections
(I could improve the amount of connections that's accepted but that won't do anything because the connections are still just frozen, waiting for the backup to complete, which will lead to timeout)
I've been reading up on some options to improve the speed and this is what I've found:
option --quick (Will probably help)
option --single-transaction (Will prevent tables from being locked, but may cause database to become incorrect)
Master-Slave replication (Probably the best thing I could do, one problem, I have only one server available)
The master-slave replication really sounds like it's the best option since I can stop the slave from updating, take the backup, and let it resume syncing. The problem is I only have one fysical machine to work with.
I know that I can set up multiple mysql instances on this one server. The question is: Is it wise to do so?
The slave is really only used to generate that backup file (which will be copied to a different disk on the network) so that the master can stay live.
if you use just innodb - try xtrabackup.
if you use both myisam and innodb - flush + lvm snapshot + file-level copy might work for you.
indeed replication slave for backups is good idea as well. just remember to periodically check data consistency between the master and the slave.
I have several slave dbs replicated from the same master db, however, for one of the slaves, i would like to keep it as a backup db, which will never have rows updated or deleted.
basically the purpose is to have a backup db with all rows stored by using the replication(mysqldump is waaay slow to do the backup), no update/delete query get replicated, insert query only. i know there will be some conflicts going on no doubt, but still wonder if any filtering options on statement/query on the slave end or any other solutions.
You should never run a production database without a working backup scheme in place - at least as long as you value your data. If you fear that a wrong sql instruction can ruin your database, then you may try point in time recovery.
If you already use replication your master server will log all write/update operations to its binlog - which it will send to the slave servers for replication. You can do for example nightly backups of you complete database. If you destroy your database in the morning, you can import the backup from the night and reapply the instructions from the binlog from after the backup till before the instruction that killed your database.
You could then skip this instruction and apply the instructions that came afterwards. This can also cause consistency issues, as the instruction after the skipped instruction see different data in the database as they did when they were originally executed.
I have similar problem. I know it's old thread but it can help others:
link: mysql replication works only if I choose database by USE database