I am trying to connect to a database using an SSH connection. However, when I try to add a connection, I do not see fields that allow me to set up an SSH connection. I am aware that one can pass extra parameters, but am not sure what key values does Airflow accept for my use case.
Airflow currently (itself) does not allow SSH connection. With airflow's Extra params you can only encrypt your connection (which btw. is a good idea).
So if you're interested in encrypted connection, in Extra fields you need to specify parameters like that:
"cert":"/path/to/certs/client-cert.pem", "key":"/path/to/certs/client-key.pem", "ca":"/path/to/certs/server-ca.pem"}
In details, if you "dig" MySqlOperator you will see following chain:
MySqlOperator uses MySqlHook
MySqlHook uses python's mysql library MySQLdb
MySQLdb is an interface to C version which has function mysql_ssl_set() described here in details:
https://dev.mysql.com/doc/refman/8.0/en/mysql-ssl-set.html
This link nicely explains what cert, key and ca are and what should be put in there.
Now, if that's still not what you want then you need to "step out" of the Airflow. Assuming that you can already ssh to your server (ssh username#your-host) then, in separate terminal window (or background) you should launch forwarding using command: ssh -L <bind_address>:127.0.0.1:<host_port> username#your-host where:
<bind_address> is port on which you will locally connect your Airflow
<host_port> is port on which your remote host's database listens on (so for mysql it's 3306)
When your forwarding is in place, you can use Airflow's Connections settings and specify connection that will use localhost as your Host and <bind_address> as Port.
Example: Let's say your remote database listens on port 3306 and working ssh connection is ssh me#my-host. Your forwarding command should be e.g. ssh -L 9876:127.0.0.1:3306 and Airflow's settings:
Host: localhost
Port: 9876
all other connection settings are similar to connecting to mysql database on your localhost
Keep in mind that your forwarding session has to be "ON" all the time, otherwise the connection will break. Just put it in background.
Related
I want to use the RMySQL package to query a database. I usually type ssh website.com in my mac terminal before using dbConnect() and dbGetQuery() in a R script.
My database is behind a firewall and only accessible locally and to the best of my knowledge needs to be ssh'd into.
Is it possible to do the entire process in R?
I've tried system('ssh website.com')) without success
Thanks
You need to establish a port forward between your system and the ssh server.
Add the following to your ~/.ssh/config file:
Host mysql-tunnel-website.com
LocalForward 3306 localhost:3306
I highly recommend SSH key usage for this. Github's SSH key guide is pretty good. I'm not an R coder, but R might dislike the need to enter a password interactively in a system() call and SSH keys (when passwordless or when added to an ssh-agent) remove that need.
Now you should be able to start up the tunnel in R with:
system('ssh -f mysql-tunnel-website.com')
This will map website.com's localhost port 3306 to your ssh client's localhost on port 3306, allowing you to run the same code on your ssh client system as you would have on the remote website.com system.
Your R code needs to point to host=localhost and port=3306 (which should be the default).
If the remote SQL server isn't served by website.com's localhost with port 3306, simply change the localhost:3306 to the appropriate server:port combination relative to what is accessible from website.com. If you're unable to use port 3306 on your ssh client system (perhaps you're running MySQL locally?), you can forward to a different port by changing that first 3306 to any other port (I tend to prefix a digit like 13306) and then be sure to specify that alternate port (e.g. port=13306) in your R code.
I am trying Peewee to connect and retrieve data from a MySQL remote database, but I get the following error:
InternalError: (1130, "Host 'x.x.x.x' is not allowed to connect to this MariaDB server")
Could you help me?
"retrieve data from a MySQL remote database"
"Host is not allowed to connect to this MariaDB server"
Seem to point on a simple problem:
You're not allowed to connect on the DB from "outside".
By default, MySql / MariaDB are only listening on the "inside" of the server, from MariaDb doc :
MariaDB packages bind MariaDB to 127.0.0.1 (the loopback IP address) by default as a security measure using the bind-address configuration directive.
This mean apart for an application that run on the same machine (accessing 127.0.0.1 or localhost), you'll not be able to connect.
Solutions:
SSH tunnelling
This is probably the safest way to allow a connexion on a remote DB.
SSH is a protocol that allow you to connect to a server. It's mainly used on unix server to manage them, but can do a lot more.
How to use it in your case?
if you can connect with SSH to your DB server, then running this simple command on your notebook the will do the trick:
ssh -L 3306:localhost:3306 user#x.x.x.x
Lets explain a bit: first, your run SSH, then, you tell him to enable a port forwarding from your 3306 port to the localhost:3306 port of the server you connect through user#IP.
With this command running, every query from your local machine:3306 will by send to your MariaDB:3306 server, allowing you to use it as if you where on the server.
Allowing a remote-access user
This one is way more dangerous than the previous one. You'll need to take your time and think about every outcome it mean.
As already said, you're not allowed to connect from outside, ssh let you be "inside", but if you know what you do, you can just remove the security.
The point is:
to make an account that'll be able to login from a remote IP,
allow MariaDB to listen on external requests,
and at least, secure other account to disable remote connection.
[I'm not putting the how-to now, if you really need it, I'll update this answer]
I'm trying to connect to a remote DB with MySQL Workbench over SSH. The remote DB I'm trying to connect to has firewall which is private key protected (I have coverted this to OPEN SHH). I keep getting an error when I try to connect, I'm just a little confused on what is the information of the DB I should be using. I consistently see people use localhost and the DB changed intermittadely at the same point in different questions. My set up is currently that I have to connect to the server of the firewall(check) now there are multiple VMs running I wish to connect to e.g. 192.1.91.0. I have also a port set up to this on putty( I don't know if this is relevant) on localhost 5000. When I try to connect using 192.1.91.0 port 3306 or localhost 5000. I get the same error. Can anyone help. Attached is an image of the information I put into workbench
I think there are a few misunderstandings here. A firewall is not protected by a keypair. An SSH connection is. The firewall only filters network traffic using specific rules (e.g. only let it pass for enabled network ports). When you use an SSH connection you have to use the MySQL address as seen from the remote SSH connection end. That means if the MySQL server runs on the same machine as the SSH server (which is what you connect to when you use an SSH tunnel) then the address is localhost (or the IPv4/IPv6 loopback address). See my video about connection creation and troubleshooting on Youtube for more details.
To connect workbench with a private database you will need a 'jump host' also called 'bastion host' which can be any EC2 instance in a public subne in same VPC as database.
Follow Below Steps:
Open the security group attached to the database, and add new rule as below:-
Type:MYSQL/Aurora, Protocol:TCP, PortRange:3306,
Source:securitygroupofEC2 (you can all security group by entering
'sg-')
Open the security group attached to the EC2, and make port 22 is open. If not, add a new rule as below:-
Type:SSH, Protocol:TCP, PortRange:22, Source:MY IP
Open Workbench, Click New connection
- Standard TCP/IP over SSH
- SSH Hostname : < your EC2 Public IP > #34.3.3.1
- SSH Username : < your username > #common ones are : ubuntu, ec2-user, admin
- SSH KeyFile: < attach your EC2 .pem file>
- MYSQL Hostname: <database endpoint name> #mydb.tbgvsblc6.eu-west-1.rds.amazonaws.com
- MYSQL Port: 3306
- Username : <database username>
- Password: <database password>
Click 'test connection' and boom done!!
If the MySQL Server is configured to accept remote connections, you can use the servers IP address, but then you don't need to connect over ssh. Once you choose connect over ssh, the workbench creates a portforwarding using the ssh credentials to the specified server. So you have to use localhost as MySQL hostname. Due to the portforwarding, the request will be forwarded to your remote machine. In short: As far as I understand your question, you have to use localhost.
I want to connect to my database with the tool SquirreL in ssh.
Is it possible?
Sadly I have no other options for this server, so I ll have to change of tool if it is not working.
There's no way to do it directly through Squirrel, but it's ridiculously easy (when you know how) to set up the ssh tunnel that Squirrel can use.
I'd forgotten how, and came here looking for something to refresh my memory... but I had no luck on StackOverflow, so I did the research and I'm leaving the solution here for future-me... and hopefully it'll help someone else, too.
Create the ssh tunnel:
$ ssh -v -N -L3307:localhost:3306 remotehostname
I chose to use port 3307, since I also have MySQL running locally on port 3306.
Details (or you can just skip to step 2):
3307 is the port you want to use locally to refer to the remote system.
I believe that localhost here is referring to the remote system, from its own point of view, so localhost:3306 references the the standard MySQL port on the remote system.
-v is optional; it just makes ssh's output more verbose, which can help with troubleshooting. You're not going to be using the window running ssh for anything else, so might as well let it be chatty.
-N says you're not interested in actually opening a shell on the remote host. We're here for a database connection only.
If you have the verbose option turned on, you should see a message like this:
debug1: Local connections to LOCALHOST:3307 forwarded to remote address localhost:3306
If you don't have verbose output turned on, you'll only see something like Authenticated to 10.0.1.234 (via proxy).
Set up an "alias" in Squirrel that looks something like this:
Click Test and then Connect to test your connection.
If you ran ssh with the -v option, you can watch the window while you attempt to connect via Squirrel. You should see something like this for a successful connection:
debug1: Connection to port 3307 forwarding to localhost port 3306 requested.
debug1: channel 2: new [direct-tcpip]
debug1: channel 2: free: direct-tcpip: listening port 3307 for localhost port 3306, connect from 127.0.0.1 port 54536 to 127.0.0.1 port 3307, nchannels 3
When you're done with the database, just hit CtrlC in the ssh window to kill the tunnel (if you left off the -N option, I guess you'd have to hit CtrlD to close the shell).
Sadly squirrelSQL does not support ssh tunneling. Fortunately, MySQL Workbench does support ssh tunneling so I would recommend using that instead if you are connecting to a MySQL database.
An alternative to using a different tool is to use ssh port forwarding. Essentially you can forward port 3306 (or whatever port your MySQL DB uses) from the remote box to your local box. There are a number of guides on the web on how to do this. Once the tunnel with port forwarding is established you can use a local connection string like 'localhost:3306'. to connect to the remote db. It's good idea to put the ssh connection string in a .sh or .bat file because you'll have to run when you need to connect to the DB for the first time.
I have created a VM with all the specs from this post to a T:
http://azure.microsoft.com/blog/2014/09/02/create-your-own-dedicated-mysql-server-for-your-azure-websites/
The one item that I have not completed "sudo ssh -fNg -L 3307:127.0.0.1:3306 azurevmuser#servername" I am not sure what to put for "azurevmuser#servername" Should this be a user on my server or a mysql?
I also I would like to run it at 3306 I believe I would just make it "3306:127.0.0.1:3306"
Also I have created the endpoints in Azure so 3306 is listed as the mysql post on private and public.
My current error in workbench is cannot not connect. I have added my user with a wild card "%" I believe I am just fuzzy on the port binding and is that necessary every time i need to connect a port to the outside world.
I had the same issue once and got around it by using the Standard TCP/IP over the SSH option in connection mode.
Simply provide your SSH username and password or specify a SSH key file, if you are using one instead of a password, along with your MySQL credentials.