I want to ask about how to solve communication_link_failure error when I migrate mysql data to bigquery by using embulk. the table which I want to migrate is more than 1GB. I think that size is reason for error return.
How can I set my embulk config.yml file? Or how to migrate big table by using embulk.
communication_link_failure probably, Embulk can't connect MySQL server. I recommend you check the network setting.
If you can't solve the issue, Could you tell me the following?
Embulk version:
Java version: java -version
OS version
Embulk config (remove sensitive information)
Error message (Embulk run result)
Related
I'm trying to connect to a Google Cloud SQL second generation in Python from AppEngine standard (Python 2.7).
Until now, I was using MySQLDB driver directly and it was fine.
I've tried to switch to SQLAlchemy, but now I'm always having this error when the code is deployed (it seems to work fine in local) resulting in a error 500 (It's not just some connections which are lost, it constantly fails) :
OperationalError: (_mysql_exceptions.OperationalError) (2013, "Lost connection to MySQL server at 'reading initial communication packet', system error: 38") (Background on this error at: http://sqlalche.me/e/e3q8)
I don't understand because the setup doesn't differ from before, so it must be related to the way I use SQLAlchemy.
I use something like this :
create_engine("mysql+mysqldb://appuser:password#x.x.x.x/db_name?unix_socket=/cloudsql/gcpProject:europe-west1:instanceName")
I've tried different values (with, without the ip, ...). But it is still the same. Is is a version compatibility problem ?
I use
MySQL-python in the app.yaml and SQLAlchemy 1.2.4 :
app.yaml :
- name: MySQLdb
version: "latest"
requirements.txt :
SQLAlchemy==1.2.4
It was a problem in the url. I was adding in a specific part of the code "/dbname" at the end of the connection string, resulting in something like this :
mysql+mysqldb://appuser:password#/db_name?unix_socket=/cloudsql/gcpProject:europe-west1:instanceName/dbname
So in the end, the meaning of this error can also be that the unix socket is wrong.
There are a number of causes for connection loss to Google CloudSQL server but quite rightly, you have to ensure that your setup is appropriate first. I don't think this issue is about version compatibility.
According to the documentation, for your application to be able to connect to your Cloud SQL instance when the app is deployed, you require to add the user, password, database, and instance connection name variables from Cloud SQL to the related environment variables in the app.yaml file(Your displayed app.yaml does not seem to contain these environment variables).
I recommend you review the details in the link for details on how to set up your CloudSQL instance and connecting to the instance.
I'm trying to convert my MySQL database to PostgreSQL using EnterpriseDB Migration Tool. My setup is as follows:
Windows 10 x64
MySQL 5.5.52 Community Edition GPL
EDB PostgreSQL 9.6.2-2
EnterpriseDB Migration Toolkit 50.0.1
The issue is - it connects successfully to to source MySQL and target PostgreSQL but then fails with the error "unrecognized configuration parameter db_dialect":
C:\Program Files (x86)\edb\mtk\bin>runMTK -sourcedbtype mysql s2c_data
Running EnterpriseDB Migration Toolkit (Build 50.0.1) ...
Source database connectivity info...
conn=jdbc:mysql://localhost:3306/s2c_data?zeroDateTimeBehavior=convertToNull
user =s2c_data_user
password=******
Target database connectivity info...
conn =jdbc:edb://localhost:5432/s2c_data
user =s2c_data_user
password=******
Connecting with source MySQL database server...
Connected to MySQL, version '5.5.52-log'
Connecting with target EDB Postgres database server...
Connected to PostgreSQL, version '9.6.2'
Importing mysql schema s2c_data...
MTK-17003:Error in copy tables: java.sql.SQLException: com.edb.util.PSQLException: ERROR: unrecognized configuration parameter: "db_dialect"
MTK-03000: General Error DB-null: java.sql.SQLException: java.sql.SQLException: java.sql.SQLException: com.edb.util.PSQLException: ERROR: unrecognized configuration parameter: "db_dialect"
One or more schema objects could not be imported during the migration process. Please review the migration output for more details.
Migration logs have been saved to C:\Users\Alex_R\.enterprisedb\migration-toolkit\logs
******************** Migration Summary ********************
Total objects: 0 Successful count: 0 Failed count: 0 Invalid count: 0
*************************************************************
C:\Program Files (x86)\edb\mtk\bin>
Anybody faced the same issue? Seems like a bug in EDB MTK but may be I'm missing something. Would be grateful for any help.
According to EDB's documentation, this is a parameter of their Postgres PLUS fork of PostgreSQL.
So it looks like this migration tool only works when you migrate to their proprietary version of PostgreSQL, not the standard version.
I think you should refer to
this tool's documentation.
According to this document, maybe you can try to add one more command line option:
C:\Program Files (x86)\edb\mtk\bin>runMTK -sourcedbtype mysql -targetdbtype postgresql s2c_data
I found the solution - DBTransfer. The original version was pretty old and buggy so I had to fix few things. The source is here. Feel free to use it for your projects.
I have a database in mysql that I would like to migrate to mssql since I would like to use that database for an asp.net application.I have everything setup but anytime I try to convert the schema using SSMA(SQL Server Migration Assistant) I keep getting this error
An unexpected error occured.Please send the log file to the product support."For more information see getting SSMA assistant"
in the product documentation.
Error Message: Object reference not set to an instance of an object.
How do I get that out of the way and then migrate the data to MSSQL.
I also had the same issue. Once I installed the latest version of ssma for mysql, it resolves the issue you have mentioned above.
https://www.microsoft.com/en-us/download/details.aspx?id=51218
Long time lurker, first time poster; hoping anyone can help me out.
Im using MySQL Workbench 5.2.41 to migrate a database from SQL Server to MySQL 5.0.8
The entire process goes smoothly: both SQL and MySQL connection Tests are good, the skemea and table create as expected, everything checks out until the 'Bulk Data Transfer' step. At that point I receive this error:
...
Migrating data...
wbcopytables.exe --odbc-source=DSN=SQL Server 11;DATABASE=;UID=sa --target=root#127.0.0.1:3306 --progress --passwords-from-stdin --thread-count=1 --table [GSAClosers_v2] [dbo].[AccountBase] `dbo` `AccountBase`
`dbo`.`AccountBase`:Copying 84 columns of 169530 rows from table [GSAClosers_v2].[dbo].[AccountBase]
ERROR: `dbo`.`AccountBase`:SQLGetData: HY009:10:[Microsoft][ODBC Driver Manager] Invalid argument value `dbo`.`AccountBase`:
Finished copying 0 rows in 0m00s
Copy helper has finished
...
For connectors I'm using Microsoft SQL Server / ODBC Data Source / DSN: SQL Server and for MySQL the IP and port(3306).
SQL Server 2012 Management Studio connects and all operations work as expected.
O, side not: both are on the same localhost machine.
If anyone can shed some light on this I would be forever indebted. Thanks in advance
From here:
[Microsoft][ODBC Driver Manager] Invalid argument value.
Regarding to the error message and code you provided, which seems is
database is invalid or cannot be accessed. That means either the
database does not exist or the user does not have permission to access
the database.
It turns out you need specify the instance after the server name, so
in the server name in the dialog box for creating the ODBC Data
Source, you must enter it as either: MyServer\SQLEXPRESS or
.\SQLEXPRESS
I think you may have a problem connecting to your database from wbcopytables.exe. Keep in mind that this is a separate tool so the fact that you can connect from the rest of the Migration Wizard doesn't imply that you will connect in wbcopytables.exe.
The thing that worries me the most is that your DSN has whitespace characters. This might be interpreted by the Windows terminal as independent command line parameters. One thing you can do is to edit your DSN name removing the whitespaces and try again.
You may also want to connect without a DSN by putting all your connection parameters explicitely as explained in my blog post: How-To: Guide to Database Migration from Microsoft SQL Server using MySQL Workbench.
And, by the way, since MySQL Workbench 5.2.42 is out you should go and get it. The Migration Wizard is pretty new so important bug fixes are likely launched in each recent Workbench release.
In either case I think you should file a bug with your issues with a detailed explanation about how to reproduce it, possible solutions, etc.
Hope this helps.
Ok, got it figured out: un-install java. install java 1.6, use MySQL Migration Tool. change all data types to varchar, ints and bits. run migrations....eat data.
I am trying to run a JSP file deployed to Amazon's Beanstalk, which uses DataNucleus to connect to RDS, and am getting the following error:
javax.jdo.JDOFatalDataStoreException: No suitable driver found for jdbc:mysql://mydbblahblah.rds.amazonaws.com:3306/mydb?autoReconnect=true
root cause
java.sql.SQLException: No suitable driver found for jdbc:mysql://bmydbblahblah.rds.amazonaws.com:3306/mydb?autoReconnect=true
java.sql.DriverManager.getConnection(DriverManager.java:640)
java.sql.DriverManager.getConnection(DriverManager.java:200)
Now I have the MySQL connector files in the classpath (it is at WebContent/WEB-INF/lib/mysql-connector-java-5.1.15-bin.jar, so this should work.
I have also confirmed that if I remove the library from there, I get the "com.mysql.jdbc.Driver not found on CLASSPATH" error.
Here is what my datanucleus.properties file looks like this:
javax.jdo.PersistenceManagerFactoryClass=org.datanucleus.api.jdo.JDOPersistenceManagerFactory
javax.jdo.option.ConnectionDriverName=com.mysql.jdbc.Driver
javax.jdo.option.ConnectionURL=jdbc:mysql://bmydbblahblah.rds.amazonaws.com:3306/mydb
#javax.jdo.option.ConnectionURL=jdbc:mysql://localhost:3306/mydb?autoReconnect=true
javax.jdo.option.ConnectionUserName=user
javax.jdo.option.ConnectionPassword=pwnothere
All of this works on my local Tomcat connecting to RDS. It only fails when running on Beanstalk.
Any ideas what might be causing this issue?
Thanks
I [almost] resolved this issue using the following procedure:
http://www.cubehouse.org/blog/2011/03/04/using-rds-aws-mysql-on-an-elastic-beanstalk-java-tomcat-server-with-ec2-security-permissions/
With the addition that your EC2 Security Group needs a rule set for port 3306.
Sadly, this code can read but not write.
So we downgraded to DataNucleus 2.0 without any other changes, and the issue disappeared. Sad, but it worked. I cannot recommend newer versions of DataNucleus with MySQL.