Without stopping mysql I am trying to copy it to disk and copy that directory over to a new server to use. Apparently FLUSH and mysqldump don't work, since mysqldump points to a new file, and FLUSH only updates the logfile files.
Is there any way to dump what is in the mysql database to the filesystem?
From Restoring MySQL database from physical files:
You should be able to restore by copying them in your database folder (In linux, the default location is /var/lib/mysql/)
If the server is running the database is stored in memory. It feels deficient if mysql is inable to flush the memory to disk. Please let me know if this is possible while the mysql server is running.
EDIT: Any type of snapshot, AWS, DigitalOcean, Azure, is worthless without this figured out.
Related
Percona XtraBackup gets a lot of praise, from what I can see, but I find it incredibly frustrating. I'm using:
root#GR-00258:~# xtrabackup --version
xtrabackup version 2.4.9 based on MySQL server 5.7.13 Linux (x86_64) (revision id: a467167cdd4)
I can create backups of one or more single databases without problems, but there doesn't appear to be any way to restore them. The only way I have found is to restore them as a full backup into an empty /var/lib/mysql, which means mysql won't start up, of course. It seems like a remarkably poor tool for restore - what is the purpose of being able to make a backup of individual databases, if they cannot be restored?
Enough ranting - is there a way to get this to work, or am I just wasting my time? I know, I can use mysqldump into a csv file, but that is not an attractive option, when the databases are ~500GB - ~1TB.
Percona XtraBackup allows you to make a physical backup of a MySQL datadir without blocking access for clients. It's significantly faster than mysqldump to make a backup.
Restoring is also very fast. All you have to do is copy the backup files to a new datadir (after doing the prepare step, which you could do at the time you create the backup).
When restoring a full backup, you can't do that when the destination MySQL Server is running. You must shut down mysqld then copy the files into place, make sure the files have the right ownership and permissions, then start mysqld.
It's not easy to import selected tables or schemas without overwriting. But it is possible:
You must use the --export option when you prepare the backup. Then you can import individual tablespaces to an existing MySQL datadir. But you must do this one tablespace at a time, unfortunately. There's no way to do it in one step for all tables in a schema. You should be able to write a script to do that.
See a complete example of importing tablespaces from a backup here: https://www.percona.com/doc/percona-xtrabackup/8.0/xtrabackup_bin/restoring_individual_tables.html
I'm trying to create a DataBase for a very little program that just I and 2 more friends use, so the database will be little.
What I want to do is to share this database with these 2 friends, so I thought about storing the data in a OneDrive shared folder.
At the moment, I'm using a txt file as a "database". Which is placed in a shared OneDrive folder, so when my friends execute the program, it can read the data from there and make it sort of real time "online".
The thing is that I can't find the my.ini file in my
C:\Program Files\MySQL\MySQL Server 8.0\
directory, so I can't change the data folder.
Another trouble I have is that I also don't have the Data folder in that directory, instead I've found it in this one:
C:\Program Files\MySQL\MySQL Workbench 8.0 CE
Is it possible to do what I want to do? And how should I proceed?
Do you think I should be using MySQL 5.x version?
Thanks
If you want to share a database with a couple of other users, putting the datadir in OneDrive isn't going to work.
MySQL stores data in the files under the datadir, that's true. They're stored in files called tablespaces which have the .ibd extension.
But as you do INSERT/UPDATE/DELETE operations, data is temporarily stored in memory and in the transaction logs (ib_logfile*. MySQL has to do a delicate coordination between these, to save data in a durable way, while ensuring good performance. It works well, but only if the MySQL Server is the only process writing to the files.
OneDrive is not coordinating with MySQL at all. It will periodically check for files that have changed since it last did a sync. The interval OneDrive checks files is about every 10 minutes, and this is not configurable.
OneDrive may choose to sync the files at a moment after you've done some INSERT/UPDATE/DELETE operations, and the data is modified in RAM, but hasn't yet been updated in the tablespace files on disk.
Once you've committed a change, it must also be safely stored in the transaction log, even if it isn't updated in the tablespace. But if your friends receive the files in this state (transaction log contains changes which aren't present in the tablespace), they can reconstruct the data from your RAM that they didn't get. This is called InnoDB crash recovery. MySQL Server does if automatically if it starts up and finds the transaction log contains changes that aren't in the tablespace. It assumes you had a sudden reboot and lost what was in RAM.
If your friends try to just keep their MySQL Server running continuously, reading a datadir that is simultaneously being updated by their one OneDrive, it'll basically overwrite their files and MySQL will become confused. It only checks to see if it should do crash recovery as MySQL Server starts. So if the files change in unexpected ways while MySQL Server is already running, it'll just conclude that your hard drive has gotten corrupted. It'll probably report a fatal error and shut down MySQL.
Also if your friends try to make changes of their own to the database, their changes would conflict with the updates from OneDrive. Then their attempts to overwrite files would eventually be synchronized in the opposite direction via OneDrive, and would eventually corrupt your database too. This would happen without warning at 10 minute intervals, whenever OneDrive chose to do its file sync.
So I'm afraid OneDrive is not the solution to share your database.
Alternatives that do have a chance of working include:
Hosting a single instance of MySQL Server on a website that you all share, and giving each of you a client that can use the database. A popular free client is phpMyAdmin. That way there'd be only one instance of MySQL Server, one datadir, even though you each would be concurrent clients reading and writing data. This is the simplest solution, most likely to work.
Exporting the data from your MySQL Server instance using mysqldump periodically, and putting the export file on OneDrive, or sending it to your friends through email or any other means. Then they would have to import that data to their MySQL Server manually. This would overwrite any changes they had done to their database, but it wouldn't appear as corruption. If they want to send changes back to you, they could do a similar operation: export their database, put the dump file on OneDrive, then you'd get their dump file and import it to your MySQL Server instance, overwriting any changes you had done locally since the last time you sent your export to your friends.
Use a MySQL add-on for multi-server synchronous replication, such as InnoDB Group Replication or Percona XtraDB Cluster. But this is probably too complex for you to set up if you're a newbie with MySQL.
I am using mysqldump to move my database to another sever. But database has tables with million of rows and mysql restore takes too long(4 houres).
is there any way I do this faster?
Here's the way I have done this in the past using mysql replication
Dump SQL on source machine with binary logging turned on (use the --master-data option) this will give you data at that point in time and allow you to import the data on your new server while new data is being populated on the old server.
after the import (4 hours you said?) then you can START SLAVE on the new server and the new server will replay the binary logs and catch up to the old server and keep in sync until the actual switchover happens.
How to setup mysql replication
Yes, you can kill the mysqld on the source server, once it is down you can copy the entire datadir to the new server and start both servers once copy is done.
Objective: to be able to synchronize 2 linux server realtime.
My concern is after using Rsync to mirror the mysql server. The only thing it wasnt able to synchronize is the entries (ie. inserting data to the database using the insert query). How will I be able to solve this?
Things I've done:
scp the keys of the 2 server so that password wont be asked for each transaction
I used
rsync -avc /var/lib/mysql/ root#10.1.99.XXX:/var/lib/mysql/
to sync the database/tables, but wasn't able to sync the entries.
Isubaki,
It's not quite as simple as just using rsync, as mysql may have the files open at the time you are pushing them across. Linux will do the file copy ok, but using this technique, the table is locked in memory until the database is restarted.
I do have a script that will do the sync part, but it does require a database restart, which may not be what you want (you mention realtime sync)
The databases are prohibitively large (> 400MB), so dump > SCP > source is proving to be hours and hours work.
Is there an easier way? Can I connect to the DB directly and import from the new server?
You can simply copy the whole /data folder.
Have a look at High Performance MySQL - transferring large files
Use can use ssh to directly pipe your data over the Internet. First set up SSH keys for password-less login. Next, try something like this:
$ mysqldump -u db_user -p some_database | gzip | ssh someuser#newserver 'gzip -d | mysql -u db_user --password=db_pass some_database'
Notes:
The basic idea is that you are just dumping standard output straight into a command on the other side, which SSH is perfect for.
If you don't need encryption then you can use netcat but it's probably not worth it
The SQL text data goes over the wire compressed!
Obviously, change db_user to user user and some_database to your database. someuser is the (Linux) system user, not the MySQL user.
You will also have to use --password the long way because having mysql prompt you will be a lot of headache.
You could setup a MySQL slave replication and let MySQL copy the data, and then make the slave the new master
400M is really not a large database; transferring it to another machine will only take a few minutes over a 100Mbit network. If you do not have 100M networks between your machines, you are in a big trouble!
If they are running the exact same version of MySQL and have identical (or similar ENOUGH) my.cnf and you just want a copy of the entire data, it is safe to copy the server's entire data directory across (while both instances are stopped, obviously). You'll need to delete the data directory of the target machine first of course, but you probably don't care about that.
Backup/restore is usually slowed down by the restoration having to rebuild the table structure, rather than the file copy. By copying the data files directly, you avoid this (subject to the limitations stated above).
If you are migrating a server:
The dump files can be very large so it is better to compress it before sending or use the -C flag of scp. Our methodology of transfering files is to create a full dump, in which the incremental logs are flushed (use --master-data=2 --flush logs, please check you don't mess any slave hosts if you have them). Then we copy the dump and play it. Afterwards we flush the logs again (mysqladmin flush-logs), take the recent incremental log (which shouldn't be very large) and play only it. Keep doing it until the last incremental log is very small so that you can stop the database on the original machine, copy the last incremental log and then play it - it should take only a few minutes.
If you just want to copy data from one server to another:
mysqldump -C --host=oldhost --user=xxx --database=yyy -p | mysql -C --host=newhost --user=aaa -p
You will need to set the db users correctly and provide access to external hosts.
try importing the dump on the new server using mysql console, not an auxiliar software
I have no experience with doing this with mysql, but to me it seems the bottleneck is transferring the actual data?
4oo MB isnt that much. But if dump -> SCP is slow, i dont think connecting to the db server from the remove box would be any faster?
I'd suggest dumping, compressing, then copying over network or burning to disk and manually transfering the data.
Compressing such a dump will most likely give you quite good compression rate since, most likely , theres a lot of repeptetive data.
If you are only copying all the databases of the server, copy the entire /data directory.
If you are just copying one or more databases and adding them to an existing mysql server:
create the empty database in the new server, set up the permissions for users etc.
copy the folder for the database in /data/databasename to the new server /data/databasename
I like to use BigDump: Staggered Mysql Dump Importer after Exporting my database from the old server.
http://www.ozerov.de/bigdump/
One thing to note though, if you don't set the export options (namely the maximum length of created queries) respective to the load your new server can handle, it'll just fail and you will have to try again with different parameters. Personally, I set mine to about 25,000, but that's just me. Test it out a bit and you'll get the hang of it.