Airflow MySQL connection parameters with URI - mysql

I'm trying to use the MySqlHook.bulk_load() method docs in my Airflow task, and if my understanding is correct, I have to enable "local_infile" and "loose-local-infile" when I create the connection to my MySQL database in order for this to work.
Otherwise, it results in the following error:
"The used command is not allowed with this MySQL version."
If I create my connection from the Connections page on the webserver UI, and pass the following parameters as "extras", the method is successful and the DAG works as intended:
{"local_infile":"1","loose-local-infile":"1"}
image of connection settings on Airflow webserver UI
However, I am looking create my database connection by passing it as an environmental variable instead. To do so, requires me to pass my connection settings as a URI. see "Connections" in docs
ie: scheme://[user[:[password]]#]target[:port][/schema][?attribute1=value1&attribute2=value2...
docs
I have tried the following formats, but none have worked successfully for me:
mysql://user:password#hostname:port/schema?local_infile=1&loose-local-infile=1
mysql://user:password#hostname:port/schema?local_infile=1;loose-local-infile=1
I have also tried percent encoding my values like so:
mysql://user:password#hostname:port/schema?local_infile%3D1%26loose-local-infile%3D1
Any help would be GREATLY appreciated, thank you!

Related

How do I configure IntelliJ IDEA to use clear-text password authentication for a MySQL data source?

I've tried a few different ways of defining my MySQL database connection in IntelliJ, but can't seem to connect.
Requirements:
Uses an SSH proxy server
Needs the password sent in clear text.
My understanding is that the MySQL driver should have a built in plugin for this, called mysql_clear_password, and that I should be able to enable it on the Advanced tab of the database configuration by declaring its classname (com.mysql.jdbc.authentication.MysqlClearPasswordPlugin) and/or setting the LIBMYSQL_ENABLE_CLEARTEXT_PLUGIN variable to 1, Y or y.
I can't seem to get it to work. Can someone provide guidance on the correct way to do this in the IDE?
One needs to specify additional JDBC parameter authenticationPlugins with value mysql_clear_password.
Also, see https://dev.mysql.com/doc/connector-j/5.1/en/connector-j-reference-configuration-properties.html

SQLAlchemy AppEngine standard - Lost connection to MySQL server

I'm trying to connect to a Google Cloud SQL second generation in Python from AppEngine standard (Python 2.7).
Until now, I was using MySQLDB driver directly and it was fine.
I've tried to switch to SQLAlchemy, but now I'm always having this error when the code is deployed (it seems to work fine in local) resulting in a error 500 (It's not just some connections which are lost, it constantly fails) :
OperationalError: (_mysql_exceptions.OperationalError) (2013, "Lost connection to MySQL server at 'reading initial communication packet', system error: 38") (Background on this error at: http://sqlalche.me/e/e3q8)
I don't understand because the setup doesn't differ from before, so it must be related to the way I use SQLAlchemy.
I use something like this :
create_engine("mysql+mysqldb://appuser:password#x.x.x.x/db_name?unix_socket=/cloudsql/gcpProject:europe-west1:instanceName")
I've tried different values (with, without the ip, ...). But it is still the same. Is is a version compatibility problem ?
I use
MySQL-python in the app.yaml and SQLAlchemy 1.2.4 :
app.yaml :
- name: MySQLdb
version: "latest"
requirements.txt :
SQLAlchemy==1.2.4
It was a problem in the url. I was adding in a specific part of the code "/dbname" at the end of the connection string, resulting in something like this :
mysql+mysqldb://appuser:password#/db_name?unix_socket=/cloudsql/gcpProject:europe-west1:instanceName/dbname
So in the end, the meaning of this error can also be that the unix socket is wrong.
There are a number of causes for connection loss to Google CloudSQL server but quite rightly, you have to ensure that your setup is appropriate first. I don't think this issue is about version compatibility.
According to the documentation, for your application to be able to connect to your Cloud SQL instance when the app is deployed, you require to add the user, password, database, and instance connection name variables from Cloud SQL to the related environment variables in the app.yaml file(Your displayed app.yaml does not seem to contain these environment variables).
I recommend you review the details in the link for details on how to set up your CloudSQL instance and connecting to the instance.

DB2 Connect issue using Native OLE DB\MS OLEDB Provider for DB2

I downloaded and installed the driver setup file, DB2OLEDB.exe, from here:
http://download.microsoft.com/mwg-internal/de5fs23hu73ds/progress?id=HYLbKUfGNl
Using the connection string that worked on another PC, I tried to create a Connection Object in an SSIS package. When I tested the connection I got this error:
Test connection failed because of an error in initializing provider. A TCPIP socket error has occurred (10057): A request to send or receive data was disallowed because the socket is not connected and (when sending on a datagram socket using a sendto call) no address was supplied.
Any suggestions on what the cause of this error is and how I might resolve this issue?
By the way, when I use the DB2 Configuration set up utility and test a connection from within that, I am able to successfully connect.
What other info can I provide to help you answer this question?
Thank you
Could this be related to a blocked port?
If you follow all the steps illustrated here: http://www.bidn.com/blogs/PatrickLeBlanc/ssis/700/connecting-to-db2-using-ssis do you still get the same result?
Maybe a silly question, did you restart the computer after the installation?
Are you an admin user on one machine and not on the other?
You could try to verify the port connectivity with a quick telnet command:
telnet your-db-host your-db-listening-port
If it connects, that one is off the list.
Doing some research I've found two possible fixes.
The first link suggests calling BeginReceive after the EndAccept logic is complete. Are you using script code, or just using the GUI without any scripting?
TCP async sockets throwing 10057
The second link points to drivers / software on the PC. It could be that you are missing a windows update or have faulty hardware / drivers.
I think this is less likely the case since you could connect to a different machine with the same connection string(?). Can you verify this is a valid statement?
http://social.msdn.microsoft.com/Forums/en-US/1bc3df95-c86d-4d25-aa20-30f61ed00c63/odd-socket-errors
If you could show the connection strings used for both the working and non working, and give a little more detail about The "Other PC" in comparison to the non-working PC... that would be helpful =]
If neither of the posts I've linked are the solution, this specific Google search has proven to yield some seemingly helpful results
"socket" "10057" "no address was supplied."

Cakephp is not establishing a database link

I have a cake installation on a webserver and a database on a separate server.
I am able to connect to the database remotely via shell, but my cake gives
Error: Mysql requires a database connection
Error: Confirm you have created the file : app/Config/database.php.
Notice: If you want to customize this error message, create app/View/Errors/missing_connection.ctp.
I checked and PDO is setup, mod_rewrite is enabled and I have a similar setup on development server running properly. I checked core.php and it echoes proper base site url, and database.php echoes proper database selection.
Any ideas what may be causing it?
Trying to cover all of the possibilities...
With regard to the database.php file, make sure that the database credentials are being set to the $default variable and not $test, unless of course you're trying to run database unit tests of course.
The file that handles all of the initializing is the webroot's index.php file. You'll want to verify that all paths within that file are using the proper paths. If all you did was extract the CakePHP framework without any folder rearrangements, it should be all correct.
You mention that you checked mod_rewrite is enabled - did you do this with a phpinfo() call to a file located on the same (sub)domain just to verify the settings in the same location?
Although not related to the errors you're experiencing, you'll also want to verify that the external DB allows connection from your webserver's IP.

CloverETL mySQL Database SSL secure connection & Parameters

I am trying to make the connection to a remote mysql database in CloverETL Designer (Community Version)
Documentation with examples on this subject seem fragmented so I have tried to piece the bits together but I'm not use if they are correct. I would be grateful for any suggestions or further clarification on this subject.
Here is what I have so far:
The SSL Certificate
As an alternative VPN connection the server guys passed me the raw CA certificate, which I saved as a .crt file in the conn folder of the CloverETL workspace.
CloverETL Connections
I created a new DB connection and tested it was working without SSL
In the graph window I created a simple graph
Switching to Source View I added the parameter attribute to the connection code like this:
<Connection database="MYSQL" dbURL="jdbc:mysql://hostname/database_name" id="JDBC0" jdbcSpecific="MYSQL" name="CPM" parameters="ssl=${WORKSPACE}/conn/mysql_cpm_cert.crt" password="password" type="JDBC" user="username"/>
Then I tested the connection to the database by creating new metadata from a DB table
The problem is that I don't know if the connect is now secure or not.
Here's the list of resources that I used to piece together the information:
components:bulkloaders:mysql_data_writer [CloverETL wiki]
MySQL :: MySQL 5.0 Reference Manual :: 6.3.6.3 SSL Command Options
Thanks
CPM
Your setup almost certainly does not use ssl. Both links you posted are not usable - first describes bulk loader and second connection via command line client. But what you need is to configure JDBC. So I would use http://dev.mysql.com/doc/refman/5.0/en/connector-j-reference-configuration-properties.html section "Security."
I would set:
useSSL=true
requireSSL=true
You will do that by adding items in table (Custom JDBC properties) on second tab (Advanced) of Connection Wizard. In source it should appear like "jdbc.useSSL=true jdbc.requireSSL=true" not "parameters=".
Then connection should fail, because in your default java key store is missing your certificate. It can be imported via http://docs.oracle.com/javase/1.4.2/docs/tooldocs/windows/keytool.html section "Importing Certificates"
I hope this helps.