I'm a bit stuck, I switched recently to Google Cloud MySQL and I would like to clone one of my database (not instance) for an external development environment for freelancers.
The idea is to clone/duplicate existing live database, then scrap sensitive datas (emails, etc...).
I know I need to use "gcloud" command line function but I don't really know to do it.
Can someone help me ?
The easiest way to do this would be to restore a backup made on the first instance to a new instance. I recommend you review the Cloud SQL documentation around backups
Example steps:
Create an on demand backup
gcloud sql backups create --async --instance [SOURCE_INSTANCE_NAME]
You can see a list of backup ids for the source instance with this:
gcloud sql backups list --instance [SOURCE_INSTANCE_NAME]
Restore to the new instance. After preparing the new instance (creating, ensuring it has no replicas, etc).
gcloud sql backups restore [BACKUP_ID] --restore-instance=[TARGET_INSTANCE_NAME] \
--backup-instance=[SOURCE_INSTANCE_NAME]
You can also do all of the above through the console.
Once the restore is complete, you can remove the backup. The easiest way to do this is through the console, but it can be done via the REST API if necessary.
Of course, there isn't a gcloud command to do the data cleanup you describe, you would need to do that yourself, based on your own data and anonymization requirements. Doing good anonymization can be tricky unless you have a very limited amount of sensitive data.
If instead you just want to export a single database, then you can use the export and import functionality. This is subject to some limitations, for example, triggers, stored procedures, and possibly views, etc, will need to be manually recreated.
Full instructions for export, but here's a quick summary.
You will need a cloud storage bucket to hold the output, and the service account for the database will need to be a writer on that bucket. Once that is in place:
gcloud sql export sql [INSTANCE_NAME] gs://[BUCKET_NAME]/[DUMP_FILE_NAME] \
--database=[DATABASE_NAME]
You can then either download the file and use it on a local database, or import it into a new instance, as so:
gcloud sql import sql [INSTANCE_NAME] gs://[BUCKET_NAME]/[DUMP_FILE_NAME] \
--database=[DATABASE_NAME]
Obviously, sanitizing the data is still up to you.
Related
I need to clone a MySQL database in one GCP to another GCP account.
The most obvious way I can think of is exporting MySQL and then importing it another account.
What are other alternatives?
Go to Cloud SQL page in the console and chose migrate data. Here you have several cases of migration, and among them this one which match your requirement:
Google Cloud project to Google Cloud project
Move an instance from another Google Cloud project into this one
You can choose to set this read replica as master (and thus to finish the migration), or you can keep the state of read replica and your clone will be always the image of your original project. Here the described steps:
The Cloud SQL Migration Assistant will guide you through the following steps:
Providing details on your data source
Creating a Cloud SQL read replica
Synchronising the read replica with the source
Promoting the read replica to your primary instance (optional)
1. Export data to Cloud Storage in the source account
Choose Cloud Storage export location, Format
The SQL export process may take a long time (possibly an hour or more
for large instances). You will not be able to perform operations on
your instance for the entire duration of the export. Once begun, this
process cannot be canceled.
2. Copy the exported dump file to destination account
a. Create a bucket
b. Edit bucket permissions
c. Add member
d. Enter the mail source account
e. Select Role
Copy the file from soure account to destination account :
gsutil mv gs://source/export gs://destination/export
If the dump file is to big use: Cloud Data Transfer
3. Select Cloud SQL Migrate data
Begin Migration
a. Choose Data Source Details: Name of data source, Public IP address of source, Port number of source, MySQL replication username, Password,
b. Create and configure a new Cloud SQL read replica of the external primary instance. Choose Read replica instance ID, Location, Region, Machine type, Storage type, Storage capacity, Import SQL dump from Google Cloud Storage
c. Data synchronization
d. Read replica promotion (optional)
I have a SQL script which selects data from DB and stores it to files. I am unable to create a directory to store these files.
I have shell script that loads the SQL file. Shell and the SQL are on separate server than MySQL db. I would prefer to create this directory using SQL as I want to avoid ssh.
Any suggestions? Surprisingly I couldn't find anything on Google.
I will assume that you're using mysql, according to your tags. You could do it with a Microsoft SQL Server or Oracle database but unfortunately, at the moment, there is no solution to create a directory from MySQL.
Some will guide you with a workaround based on the creation of a data directory, I wouldn't recommand this, as it could lead to performances issues in the future, or worst.
The best solution would be to use a script (java, vbscript, SSH, batch, ...). Again, you won't be able to start this script within your SQL query easily. I know that's no good news, but it is important not to lead you on the wrong direction.
I would suggest to reverse your thinking, and start your SQL query from a script (again, any language you're used to).
I couldn't find any other way other than opening ssh session to the target box.
Open ssh session
Create directory
close ssh session
Load sql file using shell
The sql adds the generated files to the directory created in step 2.
ssh -t $USER#$HOST <<-SSH-END;
mkdir -p "dir/path";
exit;
SSH-END
Sharing just in case someone else needs to do the same.
This may sound stupid and it is. But i'm wondering if it is possible to create a GCE instance that its sole purpose is to copy data from another GCE's MySQL database and copies all data to a Google Cloud SQL instance every few minutes and essentially updates the GCloud SQL.
Essentially i'm trying to get around how GAE can't connect to a GCE MySQL database but you can connect to a Google cloud SQL database.
I have tried "FEDERATED Tables" however Google Cloud SQL doesn't support that. So this is my last resort.
Thanks
Why do you need the GCE database at all? That is, why can't you just use a Cloud SQL database for all of your database needs?
You could try manually replaying the binary log to your Cloud SQL instance, ie:
Enable binary logging on your GCE MySQL instance.
Use mysqlbinlog to periodically extract the latest contents of the log as SQL statements. Use the positioning functions to make sure each run starts where the last finished.
Send the SQL outputted by mysqlbinlog to your Cloud SQL instance.
Our Java server application logs data to a SQL database, which may or may not be on the same machine. Currently we use MS SQL Server, and we're now porting to MySQL. A user configures database backup parameters on our app server, e.g. time of day to run a backup, and the app server executes SQL Server's BACKUP DATABASE command at the appropriate time, via a sproc. It does incremental backups daily and full backups weekly.
MySQL lacks an equivalent feature to tell the database from a client connection to back itself up. Options we're considering are:
Create a UDF to shell out to mysqldump (or copy database files), which can be called from our app server via a sproc. Essentially we'd be implementing a version of BACKUP DATABASE for MySQL.
Create a service to run on the MySQL box that can get the backup settings from the app server and run mysqldump (or file copy) locally.
Create a backup sproc to mimic mysqldump, e.g. SHOW CREATE TABLES and SELECT INTO OUTFILE for each table.
Setting up a cron job, Perl script, third-party app or other tricks that'd work great in a data center aren't preferred; this is a shrink-wrap package that needs to be pretty robust and hands off.
Database sizes can range from roughly 10MB to 10GB.
I'm aware of the binary logs for the incremental piece. I figure the general solution will probably apply to them as well, if we decide to use them.
This is all on Windows 2003 32-bit or 2008R2 64-bit, MySQL 5.1.
The UDF option seems the best to me. The UDF Repository (http://www.mysqludf.org/) has mysqludf_sys, which may be all we need, but I thought I'd ask for opinions since after extensive googling it doesn't seem like others have reached the same conclusion, or maybe our needs are just out of the ordinary. Our app is the only thing in MySQL, so I'm not worried about other users having access to our UDF.
Any solutions I'm overlooking? Any experience with using UDFs in such a way?
Thanks,
Eric
For this an other reasons we decided to collocate our application with the database, so this problem became moot.
The title is self explanatory. Is there a way of directly doing such kind of importing?
The .BAK files from SQL server are in Microsoft Tape Format (MTF) ref: http://www.fpns.net/willy/msbackup.htm
The bak file will probably contain the LDF and MDF files that SQL server uses to store the database.
You will need to use SQL server to extract these. SQL Server Express is free and will do the job.
So, install SQL Server Express edition, and open the SQL Server Powershell. There execute sqlcmd -S <COMPUTERNAME>\SQLExpress (whilst logged in as administrator)
then issue the following command.
restore filelistonly from disk='c:\temp\mydbName-2009-09-29-v10.bak';
GO
This will list the contents of the backup - what you need is the first fields that tell you the logical names - one will be the actual database and the other the log file.
RESTORE DATABASE mydbName FROM disk='c:\temp\mydbName-2009-09-29-v10.bak'
WITH
MOVE 'mydbName' TO 'c:\temp\mydbName_data.mdf',
MOVE 'mydbName_log' TO 'c:\temp\mydbName_data.ldf';
GO
At this point you have extracted the database - then install Microsoft's "Sql Web Data Administrator". together with this export tool and you will have an SQL script that contains the database.
MySql have an application to import db from microsoft sql.
Steps:
Open MySql Workbench
Click on "Database Migration" (if it do not appear you have to install it from MySql update)
Follow the Migration Task List using the simple Wizard.
I did not manage to find a way to do it directly.
Instead I imported the bak file into SQL Server 2008 Express, and then used MySQL Migration Toolkit.
Worked like a charm!
In this problem, the answer is not updated in a timely. So it's happy to say that in 2020 Migrating to MsSQL into MySQL is that much easy. An online converter like RebaseData will do your job with one click. You can just upload your .bak file which is from MsSQL and convert it into .sql format which is readable to MySQL.
Additional note: This can not only convert your .bak files but also this site is for all types of Database migrations that you want.
Although my MySQL background is limited, I don't think you have much luck doing that. However, you should be able to migrate over all of your data by restoring the db to a MSSQL server, then creating a SSIS or DTS package to send your tables and data to the MySQL server.
hope this helps
I highly doubt it. You might want to use DTS/SSIS to do this as Levi says. One think that you might want to do is start the process without actually importing the data. Just do enough to get the basic table structures together. Then you are going to want to change around the resulting table structure, because whatever structure tat will likely be created will be shaky at best.
You might also have to take this a step further and create a staging area that takes in all the data first n a string (varchar) form. Then you can create a script that does validation and conversion to get it into the "real" database, because the two databases don't always work well together, especially when dealing with dates.
The method I used included part of Richard Harrison's method:
So, install SQL Server 2008 Express
edition,
This requires the download of the Web Platform Installer "wpilauncher_n.exe"
Once you have this installed click on the database selection ( you are also required to download Frameworks and Runtimes)
After instalation go to the windows command prompt and:
use sqlcmd -S \SQLExpress (whilst
logged in as administrator)
then issue the following command.
restore filelistonly from
disk='c:\temp\mydbName-2009-09-29-v10.bak';
GO This will list the contents of the
backup - what you need is the first
fields that tell you the logical names
- one will be the actual database and the other the log file.
RESTORE DATABASE mydbName FROM
disk='c:\temp\mydbName-2009-09-29-v10.bak' WITH MOVE 'mydbName' TO
'c:\temp\mydbName_data.mdf', MOVE
'mydbName_log' TO
'c:\temp\mydbName_data.ldf'; GO
I fired up Web Platform Installer and from the what's new tab I installed SQL Server Management Studio and browsed the db to make sure the data was there...
At that point i tried the tool included with MSSQL "SQL Import and Export Wizard" but the result of the csv dump only included the column names...
So instead I just exported results of queries like "select * from users" from the SQL Server Management Studio
SQL Server databases are very Microsoft proprietary. Two options I can think of are:
Dump the database in CSV, XML or similar format that you'd then load into MySQL.
Setup ODBC connection to MySQL and then using DTS transport the data. As Charles Graham has suggested, you may need to build the tables before doing this. But that's as easy as a cut and paste from SQL Enterprise Manager windows to the corresponding MySQL window.
For those attempting Richard's solution above, here are some additional information that might help navigate common errors:
1) When running restore filelistonly you may get Operating system error 5(Access is denied). If that's the case, open SQL Server Configuration Manager and change the login for SQLEXPRESS to a user that has local write privileges.
2) #"This will list the contents of the backup - what you need is the first fields that tell you the logical names" - if your file lists more than two headers you will need to also account for what to do with those files in the RESTORE DATABASE command. If you don't indicate what to do with files beyond the database and the log, the system will apparently try to use the attributes listed in the .bak file. Restoring a file from someone else's environment will produce a 'The path has invalid attributes. It needs to be a directory' (as the path in question doesn't exist on your machine).
Simply providing a MOVE statement resolves this problem.
In my case there was a third FTData type file. The MOVE command I added:
MOVE 'mydbName_log' TO 'c:\temp\mydbName_data.ldf',
MOVE 'sysft_...' TO 'c:\temp\other';
in my case I actually had to make a new directory for the third file. Initially I tried to send it to the same folder as the .mdf file but that produced a 'failed to initialize correctly' error on the third FTData file when I executed the restore.
The .bak file from SQL Server is specific to that database dialect, and not compatible with MySQL.
Try using etlalchemy to migrate your SQL Server database into MySQL. It is an open-sourced tool that I created to facilitate easy migrations between different RDBMS's.
Quick installation and examples are provided here on the github page, and a more detailed explanation of the project's origins can be found here.