We are using Percona MySQL 5.6 on Debian 8 for an ecommerce aggregator. For this there is a master backend server doing the whole ETL (processing product feeds from partners) and a slave MySQL server used by the frontend web servers. It's a single product database with about 600GB data.
Both machines have RAID10 with datacenter series SSDs. Master MySQL is a dual Xeon E5 with 128GB RAM and slave is a single Xeon E5 with 64GB RAM. Our problem is, that the ETL is so I/O heavy (with thousands of IOPS), that the master is able to handle the I/O load but the slave server cannot keep up with replication. Most work is done in a ramdisk and only real neccessary data is written to database, already.
The slave has more reads than writes, but replication cannot catch up. What options are there for scaling the replication writes (i.e. the I/O load) on the slave?
Edit 2016-11-18: With "options" I'm not only asking for optimizing MySQL, but for completely other techniques or software which could handle this situation better.
Upgrading to 5.7 should hopefully solve your immediate problem: http://mysqlhighavailability.com/multi-threaded-replication-performance-in-mysql-5-7/.
This is usually a precursor to the master running out of write capacity. You should start thinking about sharding soon. You can check out YouTube's vitess.io project, which I'm part of.
Related
I'm planning to create a system which tracks visitors clicks into the database. I'm expecting around 1M inserts/day into the Database.
On the backend, I'll have an analytics system which will analyze all the data that's been collected over the days/weeks/months/years.
My question is: is it a practical approach to have 2 different MySQL Servers + 1 Web server? MySQL Server A would insert the clicks into it's DB and it would be connected to MySQL Server B by group replication, so whenever I create reports, etc on MySQL Server B, it doesn't load Server A heavily.
These 2 Database servers would then be connected to the Web Server which would handle all the click requests and displaying the backend reports also.
Is it a practical solution, or is it better to have one bigger server to handle all the MySQL data? Or have multiple MySQL servers that are load balancing each other? Anything else perhaps?
1M inserts/day is not a high load by modern standards. That's less than 12 per second on average.
On sufficiently powerful servers with fast storage and proper tuning of MySQL options, you can expect to support at least 100x that load with a single MySQL server.
A better reason to use multiple MySQL servers is redundancy. Inevitably, any MySQL server needs to be upgraded, or you might have hardware failures and need to replace a disk, or other components. To avoid downtime, you should have a standby database server, which stays in sync with the primary server, either using MySQL replication or by disk-level replication like DRBD.
Inside my cronjobs I make a full mysqldump every night.
My database has total 1.5GB data inside 20 tables.
Nearly every table has indexes.
I make backup like this:
mysqldump --user=user --password=pass --default-character-set=utf8 database
--single-transaction| gzip > "mybackupfile"
I make this for 2 months. This process takes nearly 1,5 minutes for 2 months.
Last week my hosting company changed my server. Just after the server change, this process started to long for 5 minutes. I told this to server company and they increased my CPU from 4GHz to 6 GHz so mysqldump process became 3,5 minutes. Then they increased to 12 GHz. But this didn't change the performance.
I checked my shared SSD disk performance with hdparm. It was 70 MB/sec. So I complain again. So they changed my hard disk to another one. Hard disk read speed became 170 MB/sec. So mysqldump process became 3 minutes.
But the duration is far from the previous value. What would be the cause for this performance degradation ? How can I isolate the problem ?
(Server is Centos 6.4, 12 GHz CPU, 8 GB RAM)
Edit: My company changed server again and I still have same problem. Old server has 3,5 minutes backup time now new server has 5 minutes time. Resultant file is 820 MB when zipped, 2.9 GB when unzipped.
I'm trying to find out what makes this dump slow.
Dump process started at 11:24:32 and stopped at 11:29:40. You can check it from screenshots' timestamps.
Screenshots:
General
Consumed memory
Memory and CPU of gzip
Memory and CPU of mysqldump
Disk operations
hdparm results:
/dev/sda2:
Timing cached reads: 3608 MB in 1.99 seconds = 1809.19 MB/sec
Timing buffered disk reads: 284 MB in 3.00 seconds = 94.53 MB/sec
/dev/sda2:
Timing cached reads: 2120 MB in 2.00 seconds = 1058.70 MB/sec
Timing buffered disk reads: 330 MB in 3.01 seconds = 109.53 MB/sec
Obvious thing I'd look at is whether the amount of data increased in recent months. Most database-driven applications collect more data over time, so the database grows. If you still have copies of your nightly backups, I'd look at the file sizes to see if they have been increasing steadily.
Another possibility is that you have other processes doing read queries while you are backing up. Mysqldump by default creates a read lock on the database to ensure a consistent snapshot of data. But that doesn't block read queries. If there are still queries running, this could compete for CPU and disk resources, and naturally slow things down.
Or there could be other processes besides MySQL on the same server competing for resources.
And finally, as #andrew commented above, there could be other virtual machines on the same physical server, competing for the physical resources. This is beyond your control and not even something you can test for within the virtual machine. It's up to the hosting company to manage a balanced allocation of virtual machines.
The fact that the start of the problems coincides with your hosting company moving your server to another host makes a pretty good case that they moved you onto a busier host. Maybe they're trying to crowd more VM's onto a single host to save rackspace or something. This isn't something StackOverflow can answer for you -- you should talk to the hosting company.
The number or size of indexes doesn't matter during the backup, since mysqldump just does a SELECT * from each table, and that's a table-scan. No secondary indexes are used for those queries.
If you want a faster backup solution, here are a few solutions:
If all your tables are InnoDB, you can use the --single-transaction option, which uses transaction isolation instead of locking to ensure a consistent backup. Then the difference between 3 and 6 minutes isn't so important, because other clients can continue to read and write to the database. (P.S.: You should be using InnoDB anyway.)
Mysqldump with the --tab option. This dumps data into tab-delimited files, one file per table. It's a little bit faster to dump, but much faster to restore.
Mydumper, an open source alternative to mysqldump. This has the option to run in a multi-threaded fashion, backing up tables in parallel. See http://2bits.com/backup/fast-parallel-mysql-backups-and-imports-mydumper.html for a good intro.
Percona XtraBackup performs a physical backup, instead of a logical backup like mysqldump or mydumper. But it's often faster than doing a logical backup. It also avoids locking InnoDB tables, so you can run a backup while clients are reading and writing. Percona XtraBackup is free and open-source, and it works with plain MySQL Community Edition, as well as all variants like Percona Server, MariaDB, and even Drizzle. Percona XtraBackup is optimized for InnoDB, but it also works with MyISAM and any other storage engines (it has to do locking while backing up non-InnoDB tables though).
My question is: Do you really need a dump or just a copy?
There is a great way that is far away from mysql dump, it uses Linux LVM "LVM Snapshot":
http://www.lenzg.net/mylvmbackup/
The idea is to hold the database for a milli second, then LVM will make a hot copy (which takes another milli second) and then the database can continue to write data. The LVM copy is now ready for every action you want: copy the table files or open it as new mysql instance for a dump (which is not on the production database!).
This needs some modifications to your system. Maybe those mylvmbackup scripts are not completely finished jet. But if you have time yourself you can build on them and do your own work.
Btw: if you really go this way, I'm very interested in the scripts as I am also need a fast solution to clone a database from production environment to a test system for developers. We decided to use this LVM snapshot feature but - as always - had no time to start with it.
I have the following situation:
I have to set up a high-performance server-cluster with maximum availability with nginx and MySQL. The cluster consists of four web servers which are load ballanced with nginx+gluster which works just fine.
In addition there's another server with 2 SSDs in RAID1. On that server I intend to install 2 VMs each with 12GB of RAM where I set up the MySQL cluster with Master-Master replication.
But that only prevents the system to break down if the MySQL service breaks down on one of the VMs, not if the host system is offline.
To counter that I thought of adding 2 more nodes on other machines to the MySQL cluster as failover. Unfortunately I don't have more machines with SSDs.
Now my question: Would I have to expect performance issues because of the much slower hard drives on the failover machines? And if so, would these issues occur only when inserting data or also when calling pure select queries?
Of course I'd set the loadballancer to prioritize the faster nodes.
I have an application that runs on server A and the database is on the same server
there is a backup server B which I use in case the server A is down
the application will remain unchanged but the data in the DB is changing constantly
Is there a way to synchronize those 2 databases real-time automatically?
currently I wait till all the users are gone so I can manually backup and restore in the backup server.
Edit: When I said real-time I didn't mean it literally, I can handle up to one hour delay but the faster sync the better.
My databases are located on 2 servers on the same local network.
2 of them are SQL Server 2008, the main DB is on windows server 2008
the backup is on windows server 2003
A web application (intranet) is using the DB
I can use sql agent (if that can help)
I don't know what kind of details could be useful to solve this, kindly tell me what can help. Thanks.
Edit: I need to sync all the tables and table only.
the second database is writable not read-only
I think what you want is Peer to Peer Transactional Replication.
From the link:
Peer-to-peer replication provides a scale-out and high-availability
solution by maintaining copies of data across multiple server
instances, also referred to as nodes. Built on the foundation of
transactional replication, peer-to-peer replication propagates
transactionally consistent changes in near real-time. This enables
applications that require scale-out of read operations to distribute
the reads from clients across multiple nodes. Because data is
maintained across the nodes in near real-time, peer-to-peer
replication provides data redundancy, which increases the availability
of data.
mysql-cluster-expert-5.1 document mentioned that:
"MySQL Cluster tables in MySQL 5.1 are normally stored completely in memory rather than on disk (this is why we refer to MySQL cluster as an in-memory database)"
which means mysql cluster is a distributed memory database, so has anybody ever done a comparison of mysql cluster & redis on speed?
I don't think redis does anything remotely similar to MySQL cluster, so you can't compare them.
MySQL cluster is a high-availability, fuly durable SQL cluster with fully synchronous replication. Redis is not. As far as I understand, redis supports neither synchronous replication, nor SQL.
MySQL cluster means that when you can COMMIT TRANSACTION**, if you lose any cluster member permanently 1ms later, your data are still safe. MySQL cluster query nodes (which are the clients) automatically fail-over in a very short time (typically < 5 seconds).
Redis does absolutely none of this. It is a non-SQL based data store which has master-slave replication (failover? You'd better implement it yourself)
You may as well ask if a motorbike is faster than an ocean liner.
** I don't think redis supports transactions, so that notion is rubbish as well.