I'm quite new to MySQL and I was wondering: when dumping a mysql database it takes only a few seconds, but when loading it sometimes it takes a few minutes! Is there a reverse of mysqldump to load the database in a few seconds?
Some easy tuning here might help.
Also, there are techniques that can help in specific situations, such as using --disable-keys.
In addition, there is an older post. Be careful of the chosen answer though, the comment said it is dangerous, which is correct, and this tool is now officially deprecated.
In mysql, for storage engines that use file-based storage, you can backup and restore using the files. See this relevant page:
http://dev.mysql.com/doc/refman/5.1/en/backup-methods.html
It's slower when you load it because it has to recreate the indexes. So the short answer is "no". However you can improve it by using --opt option when you dump. This adds some SQL to the dump file that does various things such as disabling the keys until all the data is loaded so it rebuilds indexes all at once.
This offers a nice improvement.
No - writes always take longer than reads - and with a relational database it has to rebuild the indexes too. There are some things you can do to make it go faster though (e.g. use extended inserts, defer index rebuilds)
there is an interesting to restore from mysql dump files (created using mysqldump) ... the technique is explained in detail at
http://www.geeksww.com/tutorials/database_management_systems/mysql/tips_and_tricks/fast_parallel_restore_from_sql_dumps_mysqldump_for_mysql.php
it uses different user accounts to run backups in parallel
Related
My site runs on a VDS-server. I've just found out that my MySQL server doesn't support InnoDB engine, therefore I can't use database transactions in my application.
It makes me think, that some people might never use transactions. Is this the case? If so, how does one manage to coordinate related operations on different tables in MyISAM?
Otherwise, is there a way to install InnoDB on a MySQL server which is run on a VDS?
Thanks!
If you need transactions, then you need transactions and MyISAM isn't going to cut the mustard.
Some applications won't need transactions. For example; an application that never runs more than one related SQL statement at a time and has no need to rollback multiple SQL statements. Another example is an application that uses MySQL as a simple Key-Value Store. There are many use cases that don't require database level transaction support.
It's hard to answer the second part of your question without knowing more details about your VDS. Who is you hosting provider? Do you have shell access and permissions to change my.cnf? If not, then you probably won't be able to enable InnoDB. If you do, then here is a another SO answer that details how to enable InnoDB on MySQL: How to enable INNODB in mysql
You can often either enable the engine, install the InnoDB components manually, or simply re-install a version of MySQL that includes that engine by default. MyISAM is the crazypants database, stupidly fast but also unreliable and prone to complete destruction if your system isn't shut down properly.
Running a mission critical application on MyISAM is an extremely bad idea. Where you need MyISAM tables for performance reasons they should always be disposable, easily re-built from another more reliable source of data.
I read somewhere that it is, so I was wondering.
Don't do that, the .frm/.myd/.myi may be much larger than the actual data and may cause crash (data not consistence) or very hard to transfrom/recover.
Use mysqldump to transfer MySQL database.
It might or might not work, depending on the state of the data.
it should work if you stop the database completely and restart it again after the backup, but there you have a downtime.
A slightly more detailed response at askers request.
Some details of the dangers of a straight file copy:
If the database is live the database might change some of the files
before others so when you copy them you copy them you may get copies
in that are inconsistent with each other. If the database is offline
this method is probably reliable.
Advantages of using the documented methods:
Should work on future version of DBMS
Should work consistently across underlying engines
Always a consistent snapshot like copy
I am working on a development copy of a MySQL / InnoDB database that is about 5GB.
I need to restore it pretty frequently for testing alter scripts, and using the mysqldump file is taking quite a while. The file itself is about 900MB and takes around an hour to load in. I've removed data inserts for unimportant tables, and done extended inserts, etc., but it's still pretty slow.
Is there a faster way to do this? I am thinking to just make a copy of the database files from .../mysql/database-name, id_logfile#, and ibdata1, and copy them back in when I need to 'reset' the db, but is this viable with InnoDB? Is the ibdata file for one database? I only see one, even though I have multiple InnoBD db on this box.
Thanks!
I believe you can just copy the file named for the database (with the server daemon down) and all should be well. It seems like something that a little testing on a sample db should answer quickly, no?
I have no idea if it would be faster -- it might be slower, in fact, depending on how extensively your tests alter the data --, but what if you put all your tests in a transaction, and then rolled it back?
Use an LVM snapshot of the disk. Load your data (lengthy process) and take a snapshot (seconds). Then, whenever you need to go back to the snapshot, play LVM games -- again only seconds to re-load and re-clone any sized disk.
You might be able to do something with http://dev.mysql.com/doc/refman/5.1/en/multiple-tablespaces.html
See the ALTER TABLE xxx IMPORT TABLESPACE option especially.
For innodb backup and restore speed you might want to look at mydumper/myloader - much faster than mysqldump due to multi-threaded nature http://vbtechsupport.com/1716/
What's the fastest way to export/import a mysql database using innodb tables?
I have a production database which I periodically need to download to my development machine to debug customer issues. The way we currently do this is to download our regular database backups, which are generated using "mysql -B dbname" and then gzipped. We then import them using "gunzip -c backup.gz | mysql -u root".
From what I can tell from reading "mysqldump --help", mysqldump runs wtih --opt by default, which looks like it turns on a bunch of the things that I can think of that would make imports faster, such as turning off indexes and importing tables as one massive import statement.
Are there better ways to do this, or further optimizations we should be doing?
Note: I mostly want to optimize the time it takes to load the database onto my development machine (a relatively recent macbook pro, with lots of ram). Backup time and network transfer time currently aren't big issues.
Update:
To answer some questions posed in the answers:
The production database schema changes up to a couple times a week. We're running rails, so it's relatively easy to run the migrate scripts on stale production data.
We need to put production data into a development environment potentially on a daily or hourly basis. This entirely depends on what a developer is working on. We often have specific customer issues that are the result of some data spread across a number of tables in the db, which needs to be debugged in a development environment.
I honestly don't know how long mysqldump takes. Less than 2 hours, since we currently run it every 2 hours. However, that's not what we're trying to optimize, we want to optimize the import onto the developer workstation.
We don't need the full production database, but it's not totally trivial to separate what we do and don't need (there are a lot of tables with foreign key relationships). This is probably where we'll have to go eventually, but we'd like to avoid it for a bit longer if we can.
It depends on how you define "fastest".
As Joel says, developer time is expensive. Mysqldump works and handles a lot of cases you'd otherwise have to handle yourself or spend time evaluating other products to see if they handle them.
The pertinent questions are:
How often does your production database schema change?
Note: I'm referring to adding, removing or renaming tables, columns, views and the like ie things that will break actual code.
How often do you need to put production data into a development environment?
In my experience, not very often at all. I've generally found that once a month is more than sufficient.
How long does mysqldump take?
If it's less than 8 hours it can be done overnight as a cron job. Problem solved.
Do you need all the data?
Another way to optimize this is to simply get a relevant subset of data. Of course this requires a custom script to be written to get a subset of entities and all relevant related entities but will yield the quickest end result. The script will also need to be maintained through schema changes so this is a time-consuming approach that should be used as an absolute last resort. Production samples should be large enough to include a sufficiently broad sample of data and identify any potential performance problems.
Conclusion
Basically, just use mysqldump until you absolutely can't. Spending time on another solution is time not spent developing.
Consider using replication. That would allow you to update your copy in real time, and MySQL replication allows for catching up even if you have to shut down the slave. You could also use a parallell MySQL instance on your normal server that replicates the data to a MyISAM table, which supports online backup. MySQL allows for this as long as the tables have the same definition.
Another option that might be worth looking into is XtraBackup from renowned MySQL performance specialists Percona. It's an online backup solution for InnoDB. Haven't looked at it myself, though, so I won't vouch for it's stability or that it's even a workable solution for your problem.
I'm rewriting a PHP+MySQL site that averages 40-50 hits a day using Django.
Is SQLite a suitable database to use here? Are there any advantages/disadvantages between them?
I'm just using the db to store a blog and the users who can edit it. I am using fulltext search for the blog search, but no complex joins anywhere.
40-50 hits per day is very small and SQLLite can be used without any problem.
MySql might be better once you will get more hit because it handles in a better way multiple connexion (lock isn't the same with MySql and SqlLite).
The major problem with sqlite is concurrency. If you expect 40-50 hits a day, that's probably a non-issue. However, if that load increases you should be ready to migrate to a database daemon such as MySQL - better abstract your database specific code to make such a switch as painless as possible.
The performance section of the SQLite wiki might be of use to you.
Since you're already using an adequate database, I don't see a reason to migrate to a smaller one.
While sqlite might be perfectly adequate, too - changing to a less-capable platform from a more-capable one doesn't seem the best choice :)
SQLite will work just fine for you. It sounds as though you're largely using the database as read-only (with occasional writes to update the content). SQLite excels at this kind of access pattern. The only place where SQLite chokes is when you have a lot of writes to a database, because once a process attempts to write the file is locked until the write is complete. Also, if you do lots of writes (like updating rows in a loop) you should look into putting all those writes into a transaction - while the file is locked once the transaction hits a write query, the updates themselves take much less time because they're written to the file at once and not individually.
SQLite would be fine for this level of traffic. It actually performs quite well, the only thing that it is lacking is caching of data and queries because it needs to be spun up every time your page is accessed. That said, it is still very quick and it shouldn't be too hard to migrate to MySQL later if need be.