I have a production DB (running on SQL Server 2008) with some ever-growing tables (orders etc). These tables are large and keep growing, so I want to make a cutoff at some point, but naturally, I do not want to lose the history entirely. So, I thought along the lines of:
One time: Backup the entire DB to another server
Periodically:
Back up differentially / synchronize from Production DB to Backup DB
In Production DB, delete all rows older the cutoff period
This would not, of course, replace the regular backup plan of the production server, but rather would allow shrinking its size while keeping the historical data available off-site, where I can use it for statistics and whatnot.
Does this make sense? And if it does, could you point me towards some solution / tool which allow this, other than manually writing code for EACH of the ever-growing tables.
Any advice will be appreciated.
Micky
May be partioning will help you.
It helps you to split table on different datafiles and filegroups. You can backup and restore each partition independenly.
Related
I have a few very large MySql tables on an Amazon Std EBS 1TB Volume (the file-per-table flag is ON and each ibd file is about 150 GB). I need to move all these tables from database db1 to database db2. Alongwith this, I would also like to move the tables out to a different Amazon Volume (which I think is considered a different Partition/File-System, even if the FileSystem type is the same). The reason I am moving to another volume is so I can get another 1TB space.
Things I have tried:
RENAME TABLE db1.tbl1 TO db2.tbl1 does not help because I cannot move it out to a different volume. I cannot mount a Volume at db2 because then it is considered a different file-system and MYSQL fails with an error:
"Invalid cross-device link" error 18
Created a stub db2.tbl1, stopped mysql, deleted db2's tbl1 and copied over db1's tbl.ibd. Doesn't work (the db information is buried in the ibd?)
I do not want to try the obvious mysqldump-import OR selectinto-loadfile because each table takes a day and a half to move even with most optimizations (foreign-key checks off etc). If I take indexes out before the import , re-indexing takes long and the overall time taken is still too long.
Any suggestions would be much appreciated.
Usually what I would suggest in this case, is to create an ec2 snapshot of the volume and write that snapshot into your larger volume.
You'll need to resize the partition afterwards.
As a sidenote, if your database is that large, EBS might be a major bottleneck. You're better off getting locally attached storage, but unfortunately the process is a bit different.
You might want to use Percona xtrabackup for this:
https://www.percona.com/doc/percona-xtrabackup/LATEST/index.html
I am using Microsoft SQL Server 2008 R2.I have copied database A(myproduction database) to database B(Myreportin database) by creating SSIS package.Both databases are in same server.I want to run a job so that If any change(data modifications like inserting new rows or updating values of any row in any table) take place in database A that will also take place in my B database and sql job will run and acomplish the changing automatically.I don't want that in database B table will be dropped and recreated (as its not our business rule )instead only the change will take place.
Can any one help me please.Thanks in Advance.
I would suggest that you investigate using replication. Specifically, transactional replication if you need constant updates. Here's a bit from MSDN:
Transactional replication typically starts with a snapshot of the publication database objects and data. As soon as the initial snapshot is taken, subsequent data changes and schema modifications made at the Publisher are usually delivered to the Subscriber as they occur (in near real time). The data changes are applied to the Subscriber in the same order and within the same transaction boundaries as they occurred at the Publisher; therefore, within a publication, transactional consistency is guaranteed.
If you don't need constant updating (that comes at a price in performance, of course), you can consider the alternatives of merge replication or snapshot replication. Here's a page to start examining those alternatives.
I would like to ask for help on how it would be best to replicate 4 tables from our OLTP production database into another database for reporting and keep the data there forever.
Our OLTP database cleans up data older than 3 months and now we have a requirement that 4 of the tables in that OLTP database need to be replicated to another database for reporting data should never be removed from those tables?
The structure of the tables is not optimal for reporting so once we have replicated/copied the tables over to the reporting database we would select from those tables into new tables with slightly fewer columns and slightly different data types. (e.g. they are using money data type for date for few columns).
It is enough if the data is replicated/copied on nightly basis but can be more frequently.
I know this is not detailed information I provide here but this is a rough description of what I have at the moment. Hopefully this enough so that someone could provide me with some suggestions/ideas for me.
Any suggestions for a good solution that would put the least amount of load to the OLTP database is highly appreciated?
Thanks in advance!
Have staging tables where you load new data (e.g. each night you can send data over for the previous day), then you can insert with transformations into the main history table on the reporting server (then truncate the staging table). To limit impact on the OLTP server, you can use backup / restore or log shipping and pull the data from a copy of the production database. This will also have the added benefit of thoroughly testing your backup/restore process.
Some others might suggest SSIS. I think it is overkill for a lot of these scenarios, but YMMV.
We have a moderately large production MySQL database. Periodically, we will run commands, usually via a rails migration, that while running, bog down the database. As a specific example, we might add an index to a large table.
Is there any method that can reduce the priority MySQL gives to a specific task. A sort of "nice" within MySQL itself? I found this, which is what inspired the question:
PostgreSQL tips and tricks
Since adding an index causes the work to be done within the DB and MySQL process, lowering the priority of the Rails migration process seems like it won't help. Are there other ways we can lower the priority?
We use multiple, replicated database servers to make changes like this.
In our case, db1 is the master, replicated to db2. (db1->db2).
Start by making the change to db2. If things lock, replication will stall, but that's OK.
Move your traffic to db2. Any remnant traffic going to db1 will replicate over, so you won't lose anything.
Once there's no traffic on db1, rebuild it as a slave of db2 (db2->db1).
That's the general idea and you get very little downtime and you don't have to pull an all-nighter! We actually have three servers, so it's a little more complicated, but not much.
Good luck.
Unfortunately, there is no simple way to do this: commands that alter the database structure don't have a priority option.
If your tables are MyISAM, you can try this:
mysqlhotcopy to make a backup of the table
import that backup it into a different database server (one that's not under load)
make the changes there
make a mysqlhotcopy backup of the altered table
import it into the live server
Note that this may or may not be faster than adding the index on the live server, depending on the time it takes you to transfer the table back and forth.
We have a live MySQL database that is 99% INSERTs, around 100 per second. We want to archive the data each day so that we can run queries on it without affecting the main, live database. In addition, once the archive is completed, we want to clear the live database.
What is the best way to do this without (if possible) locking INSERTs? We use INSERT DELAYED for the queries.
http://www.maatkit.org/ has mk-archiver
archives or purges rows from a table to another table and/or a file. It is designed to efficiently “nibble” data in very small chunks without interfering with critical online transaction processing (OLTP) queries. It accomplishes this with a non-backtracking query plan that keeps its place in the table from query to query, so each subsequent query does very little work to find more archivable rows.
Another alternative is to simply create a new database table each day. MyIsam does have some advantages for this, since INSERTs to the end of the table don't generally block anyway, and there is a merge table type to being them all back together. A number of websites log the httpd traffic to tables like that.
With Mysql 5.1, there are also partition tables that can do much the same.
I use mysql partition tables and I've achieve wonderful results in all aspects.
Sounds like replication is the best solution for this. After the initial sync the slave gets updates via the Binary Log, thus not affecting the master DB at all.
More on replication.
MK-ARCHIVER is a elegant tool to archive MYSQL data.
http://www.maatkit.org/doc/mk-archiver.html
MySQL replication would work perfectly for this.
Master -> the live server.
Slave -> a different server on the same network.
Could you keep two mirrored databases around? Write to one, keep the second as an archive. Switch every, say, 24 hours (or however long you deem appropriate). Into the database that was the archive, insert all of todays activity. Then the two databases should match. Use this as the new live db. Take the archived database and do whatever you want to it. You can backup/extract/read all you want now that its not being actively written to.
Its kind of like having mirrored raid where you can take one drive offline for backup, resync it, then take the other drive out for backup.