I'm looking for some advice over the best way to transfer data from a local MSSQL DB to a remote MYSQL database every 5 mins?
At the moment I am using a tool called SQLYOG which allows me to create a connection to my local MSSQL database and to my remote MYSQL server and set up some queries and mapping.
It also allows me to create a windows scheduled task which runs every 5 minutes.
This solution has been working ok but is only about 95% reliable and seems to struggle sometimes.
This software runs in GUI mode and via cmd prompt so is reasonably light weight when running a scheduled task.
Does anyone know of any other light weight tools or methods I can use to get the same outcome?
You can use SymmetricDS. its a good & efficient open source tool for this purpose.
you can use SSIS packages to do that.
SQL Server Integration Services (SSIS) is a component of the Microsoft SQL Server database software that can be used to perform a broad range of data migration tasks.
Couple of options:
1. create linked server to mysql then copy rows via linked server in sp, set sql job to execute.
2. build ssis package and run it in sql job.
What I feel is the best soultion for this is to design an ETL job using either SSIS or Talen Open Studio.
Related
I want to establish 2 way sync between MS SQL Database with WordPress Mysql Database. I even tried to use Workbench or Microsoft SQL Server Migration Assistant for MySQL. But nothing worked. Every time i get some "Connection to MySQL Failed" Error.
I have even tried all option i have seen online but noting worked.
You have to tell us more about what you are trying to do.
As far as I can tell yo have 2 options:
Use PolyBase: Which is very impractical, close to retirement, lacking in documentation, and personally I wasn't able to install it not even once.
Use Linked Servers: As I posted in the reply you can follow the step by step guide and create a linked server to MySQL. From there you can query the linked server and eventually create a stored procedure that import/export data but forget about an on-time sync
Azure Data Factory: If you are on Azure SQL Database you can setup Azure Data Factory and create a pipeline for sync data. Or eventually use SSIS if you are on prem.
sorry if title is not so clear, probably I am not finding what I need due I do not know how to search
I have few MySQL servers is separated online servers (from different wordpress) and I want to load some of the data on those databases/tablets into a SQL database located on Azure.
inside Azure portal itself I do not see where to establish external connections, neither at server level, neither at database level
I download and install Microsoft SQL Server Management Studio, connect to the server, I can see my databse and the master one, Security with logins, and Integration Service Catalog, nothing else.
I was looking for something like:
https://www.jetbrains.com/help/go/db-tutorial-connecting-to-ms-sql-server.html#step-3-connect-to-microsoft-sql-server-with-datagrip
but nowhere ...
maybe something like this:
https://www.devart.com/odbc/mysql/docs/microsoft_sql_server_manager_s.htm
but no Servers objects option available on my SSMS
Can be this done?
Note: Azure database is a basic wfor now, if that is a limitation
Some choices.
In your SQL Server Management Studio create a linked server pointing to each MySQL instance. You found the instructions for that. https://www.devart.com/odbc/mysql/docs/microsoft_sql_server_manager_s.htm But it probably will not work in Azure SQL Server; you don't have access to the underlying Windows OS to install stuff like MySQL ODBC drivers, which you need. (You could ask Azure techsupport if they can help.)
In each MySQL instance, try creating a federated table connection to appropriate table in SQL Server. That cross-vendor federation stuff only works in MariaDB, however; MySQL's federation only goes MySQL <--> MySQL.
Write yourself a purpose-built extract / transform / load (ETL) program, and arrange to run it every so often. Program it to connect to all the servers involved, retrieve the data needing to be transferred from your MySQL servers, and update / insert that data on the SQL server.
(edit) You may be able to use command-line SQL client programs. mysqldump, with its --compatible option, may generate usable INSERT statements in a file. You then may be able to use sqlcmd to run those INSERTs on your Azure server. It's going to take some hacking, and may take using sed(1) or awk(1) to make the MySQL output compatible with SQL Server.
I believe the third option is the most robust one for production use.
I have a somewhat small database in SQL Server Express 2005 that I really need to migrate over to a MySQL install on my hosting service (Dreamhost). After reading for a couple days, everything pointed to the MySQL Migration Toolkit, which is unfortunately EOL. I was able to find an archive and install it on my server running Sql Server. I set the source database, and set my Dreamhost MySQL as the destination. For whatever reason I get tons of permission errors trying to migrate although the user I'm connecting to MySQL as full permissions (working with dreamhost on this).
Is there a better way to do this? I've heard that I should use some third party tools, (like dbtools) and then I heard NOT to use third party tools.
Like I said the database is small, with a few views, a few functions, and a few stored procs, which I can manually move over if needed.
What are my options? Thank you!
Export your SQL Server database to a downloadable package (SSIS?)
Install SQL Server Express locally.
Install MySQL locally.
Run the Migration Toolkit locally.
Dump the MySQL database
Upload and run the Dump file # DreamHost (via phpMyAdmin if possible).
For such a small database you may spend more time trying to get a 3rd party tool to work for your situation than it would take you to just move the stuff manually. If you used standard SQL and little to nothing proprietary to SQL Server, creating the objects manually in MySQL should be easy enough... you just have to be aware of the slight syntax differences between the two platforms. Once the structures are created, generating insert statements to populate the data should also be trivial.
Our Java server application logs data to a SQL database, which may or may not be on the same machine. Currently we use MS SQL Server, and we're now porting to MySQL. A user configures database backup parameters on our app server, e.g. time of day to run a backup, and the app server executes SQL Server's BACKUP DATABASE command at the appropriate time, via a sproc. It does incremental backups daily and full backups weekly.
MySQL lacks an equivalent feature to tell the database from a client connection to back itself up. Options we're considering are:
Create a UDF to shell out to mysqldump (or copy database files), which can be called from our app server via a sproc. Essentially we'd be implementing a version of BACKUP DATABASE for MySQL.
Create a service to run on the MySQL box that can get the backup settings from the app server and run mysqldump (or file copy) locally.
Create a backup sproc to mimic mysqldump, e.g. SHOW CREATE TABLES and SELECT INTO OUTFILE for each table.
Setting up a cron job, Perl script, third-party app or other tricks that'd work great in a data center aren't preferred; this is a shrink-wrap package that needs to be pretty robust and hands off.
Database sizes can range from roughly 10MB to 10GB.
I'm aware of the binary logs for the incremental piece. I figure the general solution will probably apply to them as well, if we decide to use them.
This is all on Windows 2003 32-bit or 2008R2 64-bit, MySQL 5.1.
The UDF option seems the best to me. The UDF Repository (http://www.mysqludf.org/) has mysqludf_sys, which may be all we need, but I thought I'd ask for opinions since after extensive googling it doesn't seem like others have reached the same conclusion, or maybe our needs are just out of the ordinary. Our app is the only thing in MySQL, so I'm not worried about other users having access to our UDF.
Any solutions I'm overlooking? Any experience with using UDFs in such a way?
Thanks,
Eric
For this an other reasons we decided to collocate our application with the database, so this problem became moot.
Often I need to pull the production database of some project to my local sql server to add features, test out stuff etc.
Today my procedure is to create a backup on the production server, somehow get that to my local machine - and then create a new database locally and restore the backup on top of that.
It is a pain, and takes more time than I like - and I would like to think, there must be a better way.
I have access via SQL Server Management Studio to the production database - isn't there an easier way, that requires fewer manual steps?
How do you do it?
Can't think of a quicker way using SQL Sever Management Studio. I'd recommend SQL Compare from Red Gate for synchronising the schema, SQL Data Compare can sync the data, but it's not quick for large databases over the internet.
You can use SSIS and copy objects between these environments, assuming you have a direct connection to the production server.
Another way, it's a must to make regular backups of the production databases, a simple maintenance plan can make a full backup at night, by example. If this is the case here, just request an early backup to the administrators and then mount it into your environment.
One more way, because production data disclosure can involve legal issues, you can just extract the database schema and then use testing data to make any development. This is also the fastest way to get a database.
A new option is to clone the database. Red Gate SQL Clone is one solution, and Windocks provides SQL Server containers with built-in DB cloning support. Full disclosure, I work for Windocks.
Backup and Restore is slow for my databases, so what I do is:
1. Detach production database
2. Copy files to my dev machine
3. Attach database to dev server.
but no body should work on the production database.
My backup procedure is similar - I am detaching the database and archive the files, it is faster than backing up.