I want to use the RMySQL package to query a database. I usually type ssh website.com in my mac terminal before using dbConnect() and dbGetQuery() in a R script.
My database is behind a firewall and only accessible locally and to the best of my knowledge needs to be ssh'd into.
Is it possible to do the entire process in R?
I've tried system('ssh website.com')) without success
Thanks
You need to establish a port forward between your system and the ssh server.
Add the following to your ~/.ssh/config file:
Host mysql-tunnel-website.com
LocalForward 3306 localhost:3306
I highly recommend SSH key usage for this. Github's SSH key guide is pretty good. I'm not an R coder, but R might dislike the need to enter a password interactively in a system() call and SSH keys (when passwordless or when added to an ssh-agent) remove that need.
Now you should be able to start up the tunnel in R with:
system('ssh -f mysql-tunnel-website.com')
This will map website.com's localhost port 3306 to your ssh client's localhost on port 3306, allowing you to run the same code on your ssh client system as you would have on the remote website.com system.
Your R code needs to point to host=localhost and port=3306 (which should be the default).
If the remote SQL server isn't served by website.com's localhost with port 3306, simply change the localhost:3306 to the appropriate server:port combination relative to what is accessible from website.com. If you're unable to use port 3306 on your ssh client system (perhaps you're running MySQL locally?), you can forward to a different port by changing that first 3306 to any other port (I tend to prefix a digit like 13306) and then be sure to specify that alternate port (e.g. port=13306) in your R code.
Related
I've installed MySQL on a Google Cloud Virtual Machine (debian buster). What I want to do is to make this accessible publicly (using username / password obviously).
As far as I can tell the server is visible from the outside world, as I can ping the IP and I get results, and I think I've set up a user correctly and given the appropriate permissions so I can log in.
For info, my firewall settings on GCP look like this (this is just the egress, there is one exactly the same for ingress):
Which I'm assuming is correct and leaves the correct port open?
The issue I have when I use MySQL Workbench is that when I try to create a new connection, it gives me the following error:
Your connection attempt failed for user 'username' to the MySQL server at [my ip address]:3306:
Unable to connect to localhost
Please:
1 Check that MySQL is running on address [my ip address]
2 Check that MySQL is reachable on port 3306 (note: 3306 is the default, but this can be changed)
3 Check the user username has rights to connect to [my ip address] from your address (MySQL rights define what clients can connect to the server and from which machines)
4 Make sure you are both providing a password if needed and using the correct password for [my ip address] connecting from the host address you're connecting from**
Any pointers would be gratefully received.
Update: What is really confusing me is the 'Unable to connect to localhost' error. I'm not trying to connect to localhost...?
Update 2: As per comments, results of the following commands:
Note I am trying to connect using the matprichardson username. The svc2toria user is pointing to my own IP address.
Mat, If you want to use your Google Cloud Instance Database using your MySQL workbench. I suggest you connect to it through an SSH tunnel. So, this problem won't happen. I also ran into this problem several times. Connecting through SSH made the job done.
But if your need is something else, this would not help you at all. If your only purpose is managing your database from your local machine using the MySQL workbench. This will work nicely. Create a USER in your Debian VM. and open port 22 to the public. Also, make sure to have strong credentials or a better key file when connecting through SSH. This method is working for every cloud VM database. I'm using this method for G-Cloud, Azure, and AWS. After all of your work is done. Close port 22 (SSH).
My best guess will be because of number 2. "Check that MySQL is reachable on port 3306 (note: 3306 is the default, but this can be changed)".
Your Virtual Machine will have network security controls / firewall which will be blocking port 3306 by default.
I don't use Google cloud but I believe you are looking for "network details" -> "Firewall rules".
did you change your mysqld.cnf already?
bind-address = 0.0.0.0
As none of the suggestions posted worked I went for the rather more nuclear option of deleting and rebuilding my VM and setting MySQL up again from scratch. I must have done something wrong in my initial setup, as things worked without any issues at all once I’d done this.
The location of the MySQL configuration file differs depending on the distribution.
In Ubuntu and Debian the file is located at /etc/mysql/mysql.conf.d/mysqld.cnf
while in Red Hat based distributions such as CentOS, the file is located at /etc/my.cnf
Open the file with your text editor :
sudo nano /etc/mysql/mysql.conf.d/mysqld.cnf
Search for a line that begins with bind-address and set its value to the IP address on which a MySQL server should listen.
By default, the value is set to 127.0.0.1 (listens only in localhost).
In this example, we’ll set the MySQL server to listen on all IPv4 interfaces by changing the value to 0.0.0.0
bind-address = 0.0.0.0
# skip-networking
If there is a line containing skip-networking, delete it or comment it out by adding # at the beginning of the line.
In MySQL 8.0 and higher, the bind-address directive may not be present. In this case, add it under the [mysqld] section.
Once done, restart the MySQL service for changes to take effect. Only root or users with sudo privileges can restart services.
To restart the MySQL service on Debian or Ubuntu, type:
sudo systemctl restart mysql
On RedHat based distributions like CentOS to restart the service run:
sudo systemctl restart mysqld
For more Detail Read Here
I am trying to connect to a database using an SSH connection. However, when I try to add a connection, I do not see fields that allow me to set up an SSH connection. I am aware that one can pass extra parameters, but am not sure what key values does Airflow accept for my use case.
Airflow currently (itself) does not allow SSH connection. With airflow's Extra params you can only encrypt your connection (which btw. is a good idea).
So if you're interested in encrypted connection, in Extra fields you need to specify parameters like that:
"cert":"/path/to/certs/client-cert.pem", "key":"/path/to/certs/client-key.pem", "ca":"/path/to/certs/server-ca.pem"}
In details, if you "dig" MySqlOperator you will see following chain:
MySqlOperator uses MySqlHook
MySqlHook uses python's mysql library MySQLdb
MySQLdb is an interface to C version which has function mysql_ssl_set() described here in details:
https://dev.mysql.com/doc/refman/8.0/en/mysql-ssl-set.html
This link nicely explains what cert, key and ca are and what should be put in there.
Now, if that's still not what you want then you need to "step out" of the Airflow. Assuming that you can already ssh to your server (ssh username#your-host) then, in separate terminal window (or background) you should launch forwarding using command: ssh -L <bind_address>:127.0.0.1:<host_port> username#your-host where:
<bind_address> is port on which you will locally connect your Airflow
<host_port> is port on which your remote host's database listens on (so for mysql it's 3306)
When your forwarding is in place, you can use Airflow's Connections settings and specify connection that will use localhost as your Host and <bind_address> as Port.
Example: Let's say your remote database listens on port 3306 and working ssh connection is ssh me#my-host. Your forwarding command should be e.g. ssh -L 9876:127.0.0.1:3306 and Airflow's settings:
Host: localhost
Port: 9876
all other connection settings are similar to connecting to mysql database on your localhost
Keep in mind that your forwarding session has to be "ON" all the time, otherwise the connection will break. Just put it in background.
I am trying Peewee to connect and retrieve data from a MySQL remote database, but I get the following error:
InternalError: (1130, "Host 'x.x.x.x' is not allowed to connect to this MariaDB server")
Could you help me?
"retrieve data from a MySQL remote database"
"Host is not allowed to connect to this MariaDB server"
Seem to point on a simple problem:
You're not allowed to connect on the DB from "outside".
By default, MySql / MariaDB are only listening on the "inside" of the server, from MariaDb doc :
MariaDB packages bind MariaDB to 127.0.0.1 (the loopback IP address) by default as a security measure using the bind-address configuration directive.
This mean apart for an application that run on the same machine (accessing 127.0.0.1 or localhost), you'll not be able to connect.
Solutions:
SSH tunnelling
This is probably the safest way to allow a connexion on a remote DB.
SSH is a protocol that allow you to connect to a server. It's mainly used on unix server to manage them, but can do a lot more.
How to use it in your case?
if you can connect with SSH to your DB server, then running this simple command on your notebook the will do the trick:
ssh -L 3306:localhost:3306 user#x.x.x.x
Lets explain a bit: first, your run SSH, then, you tell him to enable a port forwarding from your 3306 port to the localhost:3306 port of the server you connect through user#IP.
With this command running, every query from your local machine:3306 will by send to your MariaDB:3306 server, allowing you to use it as if you where on the server.
Allowing a remote-access user
This one is way more dangerous than the previous one. You'll need to take your time and think about every outcome it mean.
As already said, you're not allowed to connect from outside, ssh let you be "inside", but if you know what you do, you can just remove the security.
The point is:
to make an account that'll be able to login from a remote IP,
allow MariaDB to listen on external requests,
and at least, secure other account to disable remote connection.
[I'm not putting the how-to now, if you really need it, I'll update this answer]
I want to connect to my database with the tool SquirreL in ssh.
Is it possible?
Sadly I have no other options for this server, so I ll have to change of tool if it is not working.
There's no way to do it directly through Squirrel, but it's ridiculously easy (when you know how) to set up the ssh tunnel that Squirrel can use.
I'd forgotten how, and came here looking for something to refresh my memory... but I had no luck on StackOverflow, so I did the research and I'm leaving the solution here for future-me... and hopefully it'll help someone else, too.
Create the ssh tunnel:
$ ssh -v -N -L3307:localhost:3306 remotehostname
I chose to use port 3307, since I also have MySQL running locally on port 3306.
Details (or you can just skip to step 2):
3307 is the port you want to use locally to refer to the remote system.
I believe that localhost here is referring to the remote system, from its own point of view, so localhost:3306 references the the standard MySQL port on the remote system.
-v is optional; it just makes ssh's output more verbose, which can help with troubleshooting. You're not going to be using the window running ssh for anything else, so might as well let it be chatty.
-N says you're not interested in actually opening a shell on the remote host. We're here for a database connection only.
If you have the verbose option turned on, you should see a message like this:
debug1: Local connections to LOCALHOST:3307 forwarded to remote address localhost:3306
If you don't have verbose output turned on, you'll only see something like Authenticated to 10.0.1.234 (via proxy).
Set up an "alias" in Squirrel that looks something like this:
Click Test and then Connect to test your connection.
If you ran ssh with the -v option, you can watch the window while you attempt to connect via Squirrel. You should see something like this for a successful connection:
debug1: Connection to port 3307 forwarding to localhost port 3306 requested.
debug1: channel 2: new [direct-tcpip]
debug1: channel 2: free: direct-tcpip: listening port 3307 for localhost port 3306, connect from 127.0.0.1 port 54536 to 127.0.0.1 port 3307, nchannels 3
When you're done with the database, just hit CtrlC in the ssh window to kill the tunnel (if you left off the -N option, I guess you'd have to hit CtrlD to close the shell).
Sadly squirrelSQL does not support ssh tunneling. Fortunately, MySQL Workbench does support ssh tunneling so I would recommend using that instead if you are connecting to a MySQL database.
An alternative to using a different tool is to use ssh port forwarding. Essentially you can forward port 3306 (or whatever port your MySQL DB uses) from the remote box to your local box. There are a number of guides on the web on how to do this. Once the tunnel with port forwarding is established you can use a local connection string like 'localhost:3306'. to connect to the remote db. It's good idea to put the ssh connection string in a .sh or .bat file because you'll have to run when you need to connect to the DB for the first time.
I'd like to connect to MySQL (deamon is running on my VPS) via HeidiSQL. I've created new user, commented bind-adress option and when I try to connect with it via HeidiSQL, I've got an error 2003: Can't connect to MySQL server on 'xxx.xxx.xxx.xxx' (10061).
What should I do?
It could be a number of factors.
See if a firewall is blocking your traffic to the other host
Can you simply ping the host from the client machine?
Can you also open a simple telnet session to the host on port 3306 ?
(If the telnet is accepted, you will probably see some characters appear and you will remain in the telnet session for a few seconds before the connection is closed. If not accepted, you will see the message Connection refused.)
There's a simple checklist for this:
Is your MySQL server bound to "localhost" only? It might be listening for connections only on 127.0.0.1 or ::1 instead of any which is usually 0.0.0.0. Try connecting on your server to your server with mysql --host=host_ip where host_ip is your network IP address. I think the default is localhost-only.
Is port 3306 firewalled? Many distributions allow only SSH by default, so you may need to open this up to your client machine. Try not to open this up to everyone on the internet as having an open MySQL port is asking for trouble. It's always best to limit access to a set of specific IPs if possible.
Can you connect via an SSH tunnel instead? This is far more secure as it means your 3306 port is properly firewalled. There are many tools for creating SSH tunnels, but the basic principle is to forward a local port of your choosing to the remote server's port 3306.