SSL integration with Cloud SQL instance based MySQL - mysql

I enabled SSL in a MySQL Cloud SQL instance. In order to connect to the instance , I downloaded the necessary certficates and can connect fine using mysql command. The CloudSQL instance is running with Private IP on a sharedVPC network .
$ mysql -h 192.168.0.3 --ssl-ca=server-ca.pem --ssl-cert=client-cert.pem --ssl-key=client-key.pem -u testuser -p
Enter password:
Now to test connectivity from a code to connect to SQL instance I deployed the following in Cloud Functions
import pymysql
from sqlalchemy import create_engine
def sql_connect(request):
engine = create_engine('mysql+pymysql://testuser:<password>#192.168.0.3/mysql',echo=True)
tab = engine.execute('show databases;')
return str([t[0] for t in tab])
It shows "Access Denied" error as shown below
Error: function terminated. Recommended action: inspect logs for termination reason. Additional troubleshooting documentation can be found at https://cloud.google.com/functions/docs/troubleshooting#logging Details:
(pymysql.err.OperationalError) (1045, "Access denied for testuser'#'192.168.60.4' (using password: YES)")
When I disable SSL it works fine as shown below
['information_schema', 'mysql', 'performance_schema', 'sys', 'testdb']
A) To enable SSL in code I did the following
ssl_args = {'sslrootcert':'server-ca.pem','sslcert':'client-cert.pem','sslkey':'client-key.pem'}
engine = create_engine('mysql+pymysql://testuser:<password>#192.168.0.3/mysql',echo=True,connect_args=ssl_args)
but it is failing with below error
__init__() got an unexpected keyword argument 'sslrootcert'
B) Also tried disabling ssl=False in code but it is failing with below error
Invalid argument(s) 'ssl' sent to create_engine(), using configuration MySQLDialect_pymysql/QueuePool/Engine
UPDATE:
Changed the code for SSL as follows:
ssl_args = {'ssl': {'ca':'./server-ca.pem', 'cert':'./client-cert.pem', 'key':'./client-key.pem'}}
Uploaded the certs to cloud function source
Added 0.0.0.0/0 as authorized networks in CloudSQL to allow connecting from Cloud functions
Now seeing the following error
"Can't connect to MySQL server on 'X.X.X.181' ([SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: IP address mismatch, certificate is not valid for 'X.X.X.181'. (_ssl.c:1091))") . However can connect using the same certificates using `mysql` command
Need help to resolve both A) fixing the error as observed so that the code is integrated with SSL and B) Modify code so that it does not uses SSL

Use ssl_ca for the root, ssl_cert for the cert and ssl_key for the key
ssl_args = {'ssl_ca':'server-ca.pem','ssl_cert':'client-cert.pem','ssl_key':'client-key.pem'}
engine = create_engine('mysql+pymysql://testuser:<password>#192.168.0.3/mysql',echo=True,connect_args=ssl_args)

Use SSL parameter in the form ssl = {"ssl":{"ca":"server-ca.pem"}} within the connect function
from pymysql import connect
from sqlalchemy import create_engine
import os
SQLALCHEMY_DATABASE_URI='mysql+pymysql://testuser:<password>#192.168.0.3/mysql?ssl_ca=server-ca.pem'
engine = create_engine(SQLALCHEMY_DATABASE_URI)
args, kwargs = engine.dialect.create_connect_args(engine.url)
# Create connection to the DB
db_conn = connect(kwargs["host"], kwargs["user"], kwargs["passwd"], kwargs["db"], ssl = {"ssl":{"ca":kwargs["ssl"]["ca"]}})
cursor = db_conn.cursor()
# Execute query
cursor.execute("show tables")
cursor.fetchall()

Related

MySQLdb._exceptions.OperationalError: (2002, "Can't connect to server on '<servername>.database.windows.net') | Django+Azure+MySql

I'm having issues connecting my Django app in my local machine to MySql Database in Azure? I added my IP in the Rules and am connecting with this:
'default': {
'ENGINE': 'django.db.backends.mysql',
'HOST': '<servername>.database.windows.net',
'PORT': '3306',
'NAME': '<database_name>',
'USER': '<admin>#<servername>',
'PASSWORD': '<cecret>',
'OPTIONS': {'ssl': {'pem': 'tls.pem'} }
},
I can connect using AzureDataStudio, but not with this configuration in django. I Nmaped my host, found a bunch of open ports but 3306 and 1433 are bound to Sql servers.
Django's runserver shows MySQLdb._exceptions.OperationalError: (2002, "Can't connect to server on '<servername>.database.windows.net' (115)") with this configuration even if I have that server and database within it running.
One example php query string in Azure portal has:
$conn = new PDO("sqlsrv:server = tcp:<server_name>.database.windows.net,1433; Database = <database_name>", "<admin>", "{your_password_here}");
So, I'm assuming I should connect to 1433 but only 3306 works from DataStudio. From python manage.py runserver it shows django.db.utils.OperationalError: (2013, "Lost connection to server at 'handshake: reading initial communication packet', system error: 104") if I try port 1433. I'm at the limit of my knowledge regarding this.
Correction-1: 3306 doesn't seem to work with Azure DataStudio. But using 1433 in Django settings won't even initialize connection.
You need to install the MySQL connector in your local machine to connect the Python with Azure Database for MySQL using below command.
pip install mysql-connector-python
Later, on server's Overview page, make a note of the Server name and Server admin login name.
Add the code example to the file. In the code, replace the <mydemoserver>, <myadmin>, <mypassword>, and <mydatabase> placeholders with the values for your MySQL server and database.
Note: SSL is enabled by default on Azure Database for MySQL servers. You may need to download the DigiCertGlobalRootG2 SSL certificate to connect from your local environment. Replace the ssl_ca value in the code with path to this file on your computer.
import mysql.connector
from mysql.connector import errorcode
# Obtain connection string information from the portal
config = {
'host':'<mydemoserver>.mysql.database.azure.com',
'user':'<myadmin>#<mydemoserver>',
'password':'<mypassword>',
'database':'<mydatabase>',
'client_flags': [mysql.connector.ClientFlag.SSL],
'ssl_ca': '<path-to-SSL-cert>/DigiCertGlobalRootG2.crt.pem'
}
# Construct connection string
try:
conn = mysql.connector.connect(**config)
print("Connection established")
except mysql.connector.Error as err:
if err.errno == errorcode.ER_ACCESS_DENIED_ERROR:
print("Something is wrong with the user name or password")
elif err.errno == errorcode.ER_BAD_DB_ERROR:
print("Database does not exist")
else:
print(err)
else:
cursor = conn.cursor()
# Drop previous table of same name if one exists
cursor.execute("DROP TABLE IF EXISTS inventory;")
print("Finished dropping table (if existed).")
# Create table
cursor.execute("CREATE TABLE inventory (id serial PRIMARY KEY, name VARCHAR(50), quantity INTEGER);")
print("Finished creating table.")
# Insert some data into table
cursor.execute("INSERT INTO inventory (name, quantity) VALUES (%s, %s);", ("banana", 150))
print("Inserted",cursor.rowcount,"row(s) of data.")
cursor.execute("INSERT INTO inventory (name, quantity) VALUES (%s, %s);", ("orange", 154))
print("Inserted",cursor.rowcount,"row(s) of data.")
cursor.execute("INSERT INTO inventory (name, quantity) VALUES (%s, %s);", ("apple", 100))
print("Inserted",cursor.rowcount,"row(s) of data.")
# Cleanup
conn.commit()
cursor.close()
conn.close()
print("Done.")
Refer the steps given in this official tutorial to deploy the same.
Apparently I was using connection string and backend that didn't support the "MySql database" I was using. I'm still vague on how it worked but here it goes.
I created a new Azure Database for MySql servers and created a new Database there. Then used connection strings as provided by #UtkarshPal-MT in my original Django's DATABASE={} entry. Didn't have to do anything else. It just connected.
Note: You do compulsorily require to pass that certificate.
Edit1: If you're using this, as I did.
Then you need to use this mssql-django external backend with proper drivers. You can find details on using this backend in that PyPi page.

How to select MySQL version when installing RMySQL in linux

I am running an R application in a Docker container and would like to connect to a MySQL 8 database using the RMySQL library. I can connect to the database from the container using the mysql terminal command without any problems, but when I call dbConnect() in my R script I get an unknown SSL connection error (see below).
As I am using the same SSL certificates with the terminal command as well as in my R script, it must have something to do with the MySQL version used by RMySQL. I need to have MySQL 5.7 installed in the container, too, as some other R packages cannot be built otherwise. However, according to the thread below you can specify the directory of the MySQL version to be used when installing RMySQL:
adding RMySQL package to R fails (on Windows)?
So I set $MYSQL_HOME in /etc/R/Renviron.site and called install.packages('RMySQL',type='source').
Apparently you also need to copy some .lib and .dll files to make it work in Windows, but where do I find the equivalent files in Linux?
Database handler script:
#' #import DBI
#' #import RMySQL
connectToDatabase <- function(myproject.db_config) {
dbConfig <- read.properties(myproject.db_config)
print(dbConfig)
dbHandle <- dbConnect(
MySQL(),
dbname = dbConfig$databaseName,
host = dbConfig$host,
port = as.integer(dbConfig$port),
user = dbConfig$user,
password = dbConfig$password
)
return(dbHandle)
}
Config file passed to the read.properties() command:
host=domain.subdomain.de
port=3306
user=john.doe
password=mypassword
databaseName=my_db
ssl-ca=/staging/mysql-ssl/ca.pem
ssl-cert=/staging/mysql-ssl/client-cert.pem
ssl-key=/staging/mysql-ssl/client-key.pem
Error message in R:
Error in connection_create(host, username, password, dbname, as.integer(port), :
Failed to connect: SSL connection error: unknown error number
I found the solution by myself. The SSL certificates were not applied in the dbConnect() call. They need to be defined in a mysql.cnf file which is passed to dbConnect() in the default.file parameter:
dbHandle <- dbConnect(
MySQL(),
dbname = dbConfig$databaseName,
host = dbConfig$host,
port = as.integer(dbConfig$port),
user = dbConfig$user,
password = dbConfig$password,
default.file = dbConfig$configFile
)
Config file passed to the read.properties() command:
host=domain.subdomain.de
port=3306
user=john.doe
password=mypassword
databaseName=my_db
configFile=/staging/config/mysql.cnf
mysql.cnf:
[mysqld]
ssl-ca=/staging/mysql-ssl/ca.pem
ssl-cert=/staging/mysql-ssl/client-cert.pem
ssl-key=/staging/mysql-ssl/client-key.pem

How do I connect to an AWS mysql database in python3?

I can connect to mysql using cli like this:
mysql -u cwaugh -p******** -h example.com mydb
However, when I try to use python3 with the same parameters, I get an error: mysql.connector.errors.ProgrammingError: 1045 (28000): Access denied for user 'cwaugh'#'xxx.xxx.xxx.xxx' (with my ip where the x should be)
My code for python looks like this:
import mysql.connector
import datetime
thinknode = mysql.connector.connect(
host = "example.com",
user = "cwaugh",
passwd = "********",
db = "mydb")
The credentials also work in javascript, but only if I used ssl: "Amazon RDS" when I created the connection (the database is on AWS RDS).
What do I need to do to connect from python? I can connect from other applications on this same computer, so I can rule out AWS Security Groups and MySQL host limits. This seems to only happen with python. Does this have something to do with ssl like it did with javascript?
You need to add the ssl_ca argument to the mysql.connector.connect function. Eg.
thinknode = mysql.connector.connect(
host = "example.com",
user = "cwaugh",
passwd = "********",
db = "mydb",
ssl_ca = "./rds-combined-ca-bundle.pem")
I'm not sure why javascript wass able to use "Amazon RDS", and it would be a lot easier if python did too, but it doesn't.

Cloud SQL Access denied

import webapp2
import MySQLdb
import os
class MainPage(webapp2.RequestHandler):
def get(self):
if (os.getenv('SERVER_SOFTWARE') and
os.getenv('SERVER_SOFTWARE').startswith('Google App Engine/')):
db = MySQLdb.connect(unix_socket='/cloudsql/fluent-outlet-604:test-db' , db='guestbook', user='root',passwd='root')
# connect to the cloud SQL
else:
db = MySQLdb.connect(host='173.194.248.221', port=3306, db='guestbook', user='root',passwd='root')
cursor = db.cursor()
cursor.execute('SELECT guestName, content, entryID FROM entries')
data = cursor.fetchall()
db.close()
self.response.write(data)
application = webapp2.WSGIApplication([
('/',MainPage),
],debug=True)
when i deploy this app to the app engine i gain error says
"(1045, "Access denied for user 'root'#'localhost' (using password: YES)")
I had the same problem, and solved it.
This problem is on Google Cloud SQL.
In the prompt console, re-start Cloud SQL like below,
gcloud sql instances --project [app engine project name] restart [sql instance name]
You should not specify the root password (even if you have set), when connecting from AppEngine.
Remove the passwd param from line 9 of your code so that it looks like:
db = MySQLdb.connect(unix_socket='/cloudsql/fluent-outlet-604:test-db' , db='guestbook', user='root')
The example code in this article also shows this.
I saw this problem when I didn't specify the hostname in the database URI:
SQLALCHEMY_DATABASE_URI = (
'mysql+pymysql://<user-name>:<password>#/<database-name>'
'?unix_socket=/cloudsql/<connection_name>'
Changing it to the following fixed it:
SQLALCHEMY_DATABASE_URI = (
'mysql+pymysql://<user-name>:<password>#<host-name>/<database-name>'
'?unix_socket=/cloudsql/{connection_name}'

Ruby SSH MySQL Sequel (or DataMapper) remote connection with keys using Net::SSH gem

How would I connect to my VPS based MySQL database remotely (from a cloud based app) using the Ruby Net::SSH or Net::SSH::Gateway gems and key, not password, authentication?
And then connect to the database with Sequel or DataMapper. I'm assuming that after I manage to get the SSH connection working, I would just setup a Sequel/DM connection to 'sql_user#localhost:3306/database'.
I did locate a couple of similar question here, but they all use password authentication, not keys, and only demonstrate executing raw commands to query the database.
UPDATE: I just cannot seem to get this (Net::SSH with key manager) to work.
UPDATE2: Alright I have managed to get authorization when logging in from a computer that has authorized keys stored in the users local .ssh folder, with the following (port is my custom SQL port on the VPS):
sql_gate = Net::SSH::Gateway.new('192.xxx.xxx.xx','sqluser', port: 26000)
However, I will not be able to create a .ssh folder in the app's VM, so I need to somehow pass the path and filename (I will be creating a public key just for SQL access for specified user) as an option ... but haven't been able to figure out how.
UPDATE: Just need to figure out DataMapper access now. Current code being tested (remote_user_sql is my Ubuntu user, sql_user is the MySQL database user with localhost/127.0.0.1 privileges):
require 'net/ssh/gateway'
require 'data_mapper'
require 'dm-mysql-adapter'
class User
include DataMapp......
.
.
end
ssh_gate = Net::SSH::Gateway.new('192.n.n.n','remote_user_sql', {port: 25000, keys: ["sql_rsa"], keys_only: true})
port = ssh_gate.open('localhost',3306,3307)
child = fork do
DataMapper.setup(:default, {
adapter: 'mysql',
database: 'sql_test',
username: 'sql_user',
password: 'passwd',
host: 'localhost',
port: port})
DataMapper.auto_upgrade!
exit
end
puts "child: #{child}"
Process.wait
ssh_gate.close(port)
My solution, in two parts:
Well I have figured how to make the Net::SSH::Gateway gem using a specified keyfile, and then connect to the VPS through ssh via a port other than 22:
Part 1: Net::SSH::Gateway key authentication
First you must generate the keyfiles you want to use, copy the .pub to the remove server and append it to the ~/.ssh/authorized_keys file (cat sql_rsa.pub >> authorized_keys), and then make sure user_sql (the user I created on the VPS to be used only for this purpose) has been added to AllowUsers list in sshd_config. Make note of port used for ssh (25000 for this example) and use the following code to establish the connection:
ssh_gate = Net::SSH::Gateway.new('192.n.n.n','user_sql', {port: 25000, keys: ["sql_rsa"], keys_only: true})
That will read the keyfile sql_rsa in the same directory as script file, then create a new ssh gateway for 'user_sql'#'192.n.n.n' on port 25000.
I can successfully execute raw shell commands on the remove VPS with:
ssh_gate.exec("ls -la")
To close:
ssh_gate.shutdown!
Unfortunately I am still having problems using DataMapper (do-mysql-adapter) to use the gateway. I will update this answer if I figure that part out, but at least the first half of the problem has been solved.
These are the errors that DataMapper::Logger has reported:
When 127.0.0.1 was used:
Can't connect to local MySQL server through socket '/tmp/mysql.sock' (2) (code: 2002, sql state: HY000, query: , uri: )
When localhost was used:
Access denied for user 'user_sql'#'localhost' (using password: YES) (code: 1045, sql state: 28000, query: , uri: )
When the VPS hostname was used:
Unknown MySQL server host 'hostname' (25) (code: 2005, sql state: HY000, query: , uri: )
UPDATE (No success yet): So far the only way I can access the remote MySQL database is by using Net::SSH::Gateway to establish a gateway, and then use the .sshmethod to open a new Net::SSH connection over that gateway, like so:
ssh_gate.ssh('192.n.n.n','user_sql',{port: 25000, keys: ["sql_rsa"], keys_only: true}) do |ssh|
ssh.exec("mysql -u sql_user -p'passwd' -h localhost -P 3306 -e 'SELECT DATABASE();'")
end
In other words, I can only execute SQL commands using the mysql command line. I cannot figure out how to get Sequel or DataMapper to use the gateway to connect.
Part 2: DataMapper/Sequel/mysql2 connection through Net::SSH::Gateway
Make sure your MySQL server is bound to 127.0.0.1 in /etc/mysql/my.cnf, setup your connection - DataMapper example:
DataMapper.setup(:default, {
adapter: 'mysql',
database: 'DATABASE',
username: 'username',
password: 'passwd',
host: '127.0.0.1',
port: 3307}) # local port being forwarded via Net::SSH:Gateway
Followed by any class table definitions and DataMapper.finalize if required. Note that DataMapper doesn't actually connect to the remote MySQL server until either an auto_upgrade!, auto_migrate!, or query is executed, so no need to create the forwarded port yet.
Then create a new Net::SSH::Gateway, and then whenever you need DataMapper/Sequel to access the remote database, just open a port for the process, like so:
port = ssh_gate.open('127.0.0.1',3306,3307)
child = fork do
DataMapper.auto_upgrade! # DM call that accesses MySQL server
exit
end
Process.wait
ssh_gate.close(port)
You may want to put the Net::SSH::Gateway/.open code in a begin..ensure..end block, ensure'ing the port closure and gateway shutdown.
I had to use a fork and Process.wait to establish the connection, without it the method just hangs.