I'm running an web application in google compute engine and have scheduled a snapshot for the VM [Ubuntu 16.04].
I tried restoring the VM from the last available snapshot. I'm able to bring up the web application from the restored VM. But the problem is there are no any data in the database [mongodb]. All the collections created by application and default data [data seeded during deployment] are present in the mongodb in restored VM, but other than that, there is no data.
Is this how Google snapshots work? Isn't the new restored VM supposed to have all the data till the time of snapshot creation?
Creating snapshot while all the apps are running may not prove 100% accurate because some data are in buffers / caches etc.
Your missing data might have been not yet written to the disk when the snapshot was being created.
Google documentation about creating snapshots is quite clear about it:
You can create a snapshot of a persistent disk even while your apps
write data to the disk. However, you can improve snapshot consistency
if you flush the disk buffers and sync your file system before you
create a snapshot.
Pause apps or operating system processes that write data to that
persistent disk. Then flush the disk buffers before you create the
snapshot.
Try following instructions and test the results.
If for some reason you can't completely stop the database try to just flush the buffer to the disk, freeze the file system (if possible) and then create a snapshot.
You can freeze file system by logging into the instance and typing sudo fsfreeze -f [example-disk_location] and unfreeze sudo fsfreeze -u [example-disk_location].
The perfect way (with guaranteed data integrity) is to either stop the VM or unmount the disk.
Related
I have a GCE ubuntu 18.04 system that overflowed the / partition. Consequently, I can't start a remote shell to fix it. I created a new system from a snapshot, and stopped the old system.
I want to attach the old system disk to the new system, mount it, and delete a bunch of stuff so I can restart it.
However, when I edit the new system disk in the cloud console, the old system disk does not show up as a possible disk to attach. What do I need to do to the old system disk to both preserve it as a system disk, and allow attaching it to the new system?
What do I need to do to the old system disk to both preserve it as a
system disk, and allow attaching it to the new system?
Either the disk is still attached or the new instance is located in a different zone. Follow the steps below.
STEP 1:
Shut down your instance with the disk space problem. Login into the Google Cloud Console. Go to Compute Engine -> VM instances. Click on your instance and make note of the “Boot disk” name. This will be the first disk under “Boot disk and local disks”.
STEP 2:
Create a snapshot of the boot disk before doing anything further. While still in Compute Engine -> Disk. Click on your boot disk. Click on “CREATE SNAPSHOT”.
STEP 3:
Create a new instance in the same zone. A micro instance will work.
STEP 4:
Open a Cloud Shell prompt (this also works from your desktop if gcloud is set up). Execute this command. Replace NAME with your instance name (broken system) and DISK with the boot disk name and ZONE with the zone that the system is in:
gcloud compute instances detach-disk NAME --disk=DISK --zone=ZONE
Make sure that the command did not report an error.
STEP 5:
Now we will attach this disk to the new instance that you created.
Make sure that the repair instance is running before attaching the second disk. Sometimes an instance can get confused on which disk to boot from if more than one disk is bootable.
Go to Compute Engine -> VM instances. Click on your instance. Click Edit. Under “Additional disks” click “Add item”. For name enter/select the disk that you detached from your broken instance. Click Save.
STEP 6:
SSH into your new instance with both disks attached.
STEP 7:
Mount the second disk to the root file system.
Become superuser. Execute sudo -s
Execute df. Make sure that /dev/sdb1 is not mounted.
Create a directory for the mount point: mkdir /mnt/repair
Mount the second disk: mount /dev/sdb1 /mnt/repair
I'm looking for some insight whether or not rsyncing a copy of the data folder from MariaDB well running in docker will provide a usable backup. I'm deploying several containers with mapped folders in a production environment using docker.
I'm thinking of using rsnapshot for nightly backups as it uses hardlinks incrementally and I can specify the number of weekly / daily / monthly copies to keep.For the code and actual files I suspect this will work wonderfully.
For MariaDB I could run mysqldump every night but this would essentially use a new copy of the database each time instead of an incremental one. If I could rsync the data folder and be 100% sure the backup would be fully intact it would be advantageous I presume. Is there any chance this backup method would fail if data was written during the rsync? Would all the files inside of MariaDB change with daily usage (it wouldn't be advantageous if so)?
This is probably a frequent question, but I can't find a really exact match right now.
The answer is NO — you can't use filesystem-level copy tools to back up a MySQL database unless the mysqld process is stopped. In a Docker environment, I would expect the container to stop if the mysqld process stops.
Even if there are no queries running, the InnoDB engine is probably doing writes in the background to flush pages from memory into the tablespace, clean up rolled-back transactions, or finish some deferred index merges.
If you try to use rsync or cp or any other filesystem-level tools to copy InnoDB files, you will only get corrupted files that can't be restored.
Some people use LVM snapshots to get an atomic snapshot of the whole filesystem as of a single instant, and this can be used to get quick backups.
Another useful tool is Percona XtraBackup, which copies the InnoDB tablespace files while it is also copying the InnoDB transaction log continually. Only with both of these in sync can the backup be restored. Read the documentation here: https://www.percona.com/doc/percona-xtrabackup/LATEST/index.html
At my current job, we use Percona XtraBackup to make nightly backups for thousands of MySQL instances. We run Percona Server (not MariaDB) in Docker pods, and Percona XtraBackup runs as another container in the pod. It works very well, and it's free, open-source software.
A MySQL database is used for test purpose, so I want to run backend API tests on this (multiple transactions), simulating a real database in production.
When tests execution are finished, I want to restore the database in the point before that execution, to be able to run more tests with that database snapshot.
I don't have any MySQL version restriction, because it'll be a fresh database server.
What's the most performant way to restore this database?
I'd use Percona XtraBackup to create a physical backup. This can be restored a lot faster than a dump created by mysqldump.
If the database is large enough (over 100GB), even a physical backup won't be fast enough. I'd use LVM snapshots of the filesystem. See https://www.lullabot.com/articles/mysql-backups-using-lvm-snapshots for example.
You have a lot of options:
Create a snapshot with a tool like mysqldump and restore from this snapshot later to reset the database.
Create a volume-level snapshot on a filesystem that supports it like ZFS and restore it later.
Create a virtual machine and snapshot that, later restoring it to the snapshot. Most VM environments like VirtualBox support this feature.
Use a Docker container with an attached volume you can snapshot and restore.
The hard drive on my VM filled up (it was under a SSH login attack I believe) and now the VM won't boot.
How can I access the boot hard drive so I can clear out space and get it booting again?
You'll need to delete the instance without deleting the boot disk, attach the disk on a temporary instance and then mount it there. The instructions to do this can found here: https://cloud.google.com/compute/docs/troubleshooting#ssherrors (Mount your disk on a temporary instance)
You'll then be able to access the disk through the /mnt/myinstance directory, and delete the files. Once the disk is no longer full, you can detach the disk from the temporary instance, and then use it to recreate your original instance.
I have created a Google Compute Engine instance with CentOS and added some stuff there, such as Apache, Webmin, ActiveCollab, Gitolite etc.. etc.
The problem is that the VM is always running out of memory because the RAM is too low.
How do I change the assigned RAM in Google Compute Engine?
Should I have to copy the VM to another with bigger RAM? If so will it copy all the contents from my CentOS installation?
Can anyone give me some advises on how to get more RAM without having to reinstall everything.
Thanks
The recommended approach for manually managed instances is to boot from a Persistent root Disk. When your instance has been booted from Persistent Disk, you can delete the instance and immediately create a new instance from the same disk with a larger machine type. This is similar to shutting down a physical machine, installing faster processors and more RAM, and starting it back up again. This doesn't work with scratch disks because they come and go with the instance.
Using Persistent Disks also enables snapshots, which allow you to take a point-in-time snapshot of the exact state of the disk and create new disks from it. You can use them as backups. Snapshots are also global resources, so you can use them to create Persistent Disks in any zone. This makes it easy to migrate your instance between zones (to prepare for a maintenance window in your current zone, for example).
Never store state on scratch disks. If the instance stops for any reason, you've lost that data. For manually configured instances, boot them from a Persistent Disk. For application data, store it on Persistent Disk, or consider using a managed service for state, like Google Cloud SQL or Google Cloud Datastore.