Consider a situation, we've scheduled a job which backups(Full) database at a interval of 2 hrs. Similarly two more jobs are scheduled, one for the differential and another for log backup scheduled at interval 30 mins and 5 mins each . We've started job ,After two hours all the three jobs will be fired(Full,Differential,Log) therefore which will be done first as they all are scheduled to same time.
1) If Full BackUp is performed, then diff(backup) will be of Zero changes?
2) Similarly what will be case of transaction, and how can we know which fired first rather than using LSN (EASY METHOD to know).
There are 10 InnoDB partitioned tables. MySQL is configured with option innodb-file-per-table=1 (innodb-file per table/partition - for some reasons). Tables size is abount 40GB each. They contains statictics data.
During normal operation, the system can handle the load. The accumulated data is processed every N minutes. However, if for some reason, there was no treatment for more than 30 minutes (eg, maintenance of the system - it is rare, but once a year is necessary to make changes), begin to lock timeout.
I will not tell you how to come to such an architecture, but it is the best solution - way was long.
Đ•ach time, making changes requires more and more time. Today, for example, a simple ALTER TABLE took 2:45 hours. This is unacceptable.
So, as I said, processing the accumulated data requires a lot of resources and SELECT statements are beginning to return lock timeout errors. Of course, the tables in the query are not involved, and the work goes to the results of queries to them. Total size of these 10 tables is about 400GB, and a few dozen small tables, the total size of which is comparable to (and maybe not yet) to the size of an big table. Problems with small tables there.
My question is: how can I solve the problem with a lock timeout error? A server is not bad - 8 core xeon, 64 RAM. And this is only the database server. Of course, the entire system is not located on the same machine.
There is an only reason why I get this errors: on data transfrom process from big tables to small.
Any ideas?
I have a 6 nodes couchbase cluster with about 200 million documents in one bucket. I need to delete about 100 million documents within that bucket. I'm planning to have a view that gives me an index of the documents I need to delete and then do a touch operation to set the expiry of those documents to the next day.
I understand that couchbase will run a background expiry pager operation on regular intervals to delete the documents. Will this expiry pager process on 100 million documents have an impact on couchbase cluster performance?
If you set them to expire all around the same time, maybe. It depends on your cluster's sizing if it will effect performance. If it were me, unless you have some compelling reason to get rid of them all right this moment, I would play it safe just set a random TTL for a time between now and a few days from now. Then the server will take care of it for you and you do not have to worry about this.
Document expiration in Couchbase is in seconds or UNIX epoc time. If over 30 days, it has to be UNIX epoc time.
We have nightly load jobs that writes several hundred thousand records to an Mysql reporting database running in Amazon RDS.
The load jobs are taking several hours to complete, but I am having a hard time figuring out where the bottleneck is.
The instance is currently running with General Purpose (SSD) storage. By looking at the cloudwatch metrics, it appears I am averaging less than 50 IOPS for the last week. However, Network Receive Throughput is less than 0.2 MB/sec.
Is there anyway to tell from this data if I am being bottlenecked by network latency (we are currently loading the data from a remote server...this will change eventually) or by Write IOPS?
If IOPS is the bottleneck, I can easily upgrade to Provisioned IOPS. But if network latency is the issue, I will need to redesign our load jobs to load raw data from EC2 instances instead of our remote servers, which will take some time to implement.
Any advice is appreciated.
UPDATE:
More info about my instance. I am using an m3.xlarge instance. It is provisioned for 500GB in size. The load jobs are done with the ETL tool from pentaho. They pull from multiple (remote) source databases and insert into the RDS instance using multiple threads.
You aren't using up much CPU. Your memory is very low. An instance with more memory should be a good win.
You're only doing 50-150 iops. That's low, you should get 3000 in a burst on standard SSD-level storage. However, if your database is small, it is probably hurting you (since you get 3 iops per GB- so if you are on a 50gb or smaller database, consider paying for provisioned iops).
You might also try Aurora; it speaks mysql, and supposedly has great performance.
If you can spread out your writes, the spikes will be smaller.
A very quick test is to buy provisioned IOPS, but be careful as you may get fewer than you do currently during a burst.
Another quick means to determine your bottleneck is to profile your job execution application with a profiler that understands your database driver. If you're using Java, JProfiler will show the characteristics of your job and it's use of the database.
A third is to configure your database driver to print statistics about the database workload. This might inform you that you are issuing far more queries than you would expect.
Your most likely culprit accessing the database remotely is actually round-trip latency. The impact is easy to overlook or underestimate.
If the remote database has, for example, a 75 millisecond round-trip time, you can't possibly execute more than 1000 (milliseconds/sec) / 75 (milliseconds/round trip) = 13.3 queries per second if you're using a single connection. There's no getting around the laws of physics.
The spikes suggest inefficiency in the loading process, where it gathers for a while, then loads for a while, then gathers for a while, then loads for a while.
Separate but related, if you don't have the MySQL client/server compression protocol enabled on the client side... find out how to enable it. (The server always supports compression but the client has to request it during the initial connection handshake), This won't fix the core problem but should improve the situation somewhat, since less data to physically transfer could mean less time wasted in transit.
I'm not an RDS expert and I don't know if my own particular case can shed you some light. Anyway, hope this give you any kind of insight.
I have a db.t1.micro with 200GB provisioned (that gives be 600 IOPS baseline performance), on a General Purpose SSD storage.
The heaviest workload is when I aggregate thousands of records from a pool of around 2.5 million rows from a 10 million rows table and another one of 8 million rows. I do this every day. This is what I average (it is steady performance, unlike yours where I see a pattern of spikes):
Write/ReadIOPS: +600 IOPS
NetworkTrafficReceived/Transmit throughput: < 3,000 Bytes/sec (my queries are relatively short)
Database connections: 15 (workers aggregating on parallel)
Queue depth: 7.5 counts
Read/Write Throughput: 10MB per second
The whole aggregation task takes around 3 hours.
Also check 10 tips to improve The Performance of your app in AWS slideshare from AWS Summit 2014.
I don't know what else to say since I'm not an expert! Good luck!
In my case it was the amount of records. I was writing only 30 records per minute and had an Write IOPS of round about the same 20 to 30. But this was eating at the CPU, which reduced the CPU credit quite steeply. So I took all the data in that table and moved it to another "historic" table. And cleared all data in that table.
CPU dropped back down to normal measures, but Write IOPS stayed about the same, this was fine though. The problem: Indexes, I think because so many records needed to indexed when inserting it took way more CPU to do this indexing with that amount of rows. Even though the only index I had was a Primary Key.
Moral of my story, the problem is not always where you think it lies, although I had increased Write IOPS it was not the root cause of the problem, but rather the CPU that was being used to do index stuff when inserting which caused the CPU credit to fall.
Not even X-RAY on the Lambda could catch an increased query time. That is when I started to look at the DB directly.
Your Queue depth graph shows > 2 , which clearly indicate that the IOPS is under provisioned. (if Queue depth < 2 then IOPS is over provisioned)
I think you have used the default AUTOCOMMIT = 1 (autocommit mode). It performs a log flush to disk for every insert and exhausted the IOPS.
So,It is better to use (for performance tuning) AUTOCOMMIT = 0 before bulk inserts of data in MySQL, if the insert query looks like
set AUTOCOMMIT = 0;
START TRANSACTION;
-- first 10000 recs
INSERT INTO SomeTable (column1, column2) VALUES (vala1,valb1),(vala2,valb2) ... (val10000,val10000);
COMMIT;
--- next 10000 recs
START TRANSACTION;
INSERT INTO SomeTable (column1, column2) VALUES (vala10001,valb10001),(vala10001,valb10001) ... (val20000,val20000);
COMMIT;
--- next 10000 recs
.
.
.
set AUTOCOMMIT = 1
Using the above approach in t2.micro and inserted 300000 in 15 minutes using PHP.
I have a MySql Db with innoDB tables.
I need to alter a couple of big tables (~50M records), since altering locks the tables I want to make the process as fast as possible.
What is best in term of speed:
1. altering one table at a time
2. alter both tables on the same time (simultaneously)
any ideas?
I did a test.
I created a table with 4 million rows. Very simple table, one column and all values are "dude" for all rows. I then duplicated that table into big_2 containing the exact same data.
My computer is a macbook pro 13.3" from mid 2010 so everything is related to that.
I then did three things.
I ran an alter on both tables in serial, it took 34 and 33 seconds to add the column (67s total).
I ran alter on both tables in parallell, it took 1.1 min before they returned (basically at the same time) (61s total)
I redid the first test and this time it took 35 + 35 seconds (70 in total)
This confirms my suspicion that it won't be any faster in parallel. The most likely reason is that this is almost entirely an operation on disk, and that cannot be paralleled at all.
Really depends on how much memory you have in your server.
When you're doing ALTER TABLE, you really want the table and its largest secondary index (remember innodb clusters the primary key, so PK is stored with the rows) to fit into memory. If it doesn't, it's going to be slow (NB: This discussion assumes the table is not partitioned).
As your table has a tiny 50M rows, the chances are it fits in RAM trivially (you have 32G+ on your server, right?) with all its secondary indexes.
If it all fits in the innodb buffer pool, do them in parallel. If it doesn't do them in series.
Try it on your development server which has the same spec as production (obviously configure them with the same size innodb_buffer_pool)
Doing it simultaneously won't give you much gain. It still has to wait until the first is finished to do the second one.
You may prefer to run the queries with a short delay between them so other queries that have been waiting for the lock since the beginning of the first update don't have to wait for the second as well. It your database is serving a website for example, two 15 seconds hangs is better than a single 30 seconds one.