There are several good answers available for both of these scenarios - but not combined.
1. I need to make a call to an external API via whitelisted static
IP.
See: Associating Cloud Function egress with a static IP address
a) call the external API from the Cloud Function
b) route all egress from this Cloud Function through a VPC Connector on vpcnetwork-1 (IP address range = 10.8.10.0/28)
c) use a Cloud NAT
that routes all traffic on vpcnetwork-1 through [STATIC IP]
(whitelisted by external API)
2. Next, I need to take that API data and send it to a Cloud SQL
instance (MySQL in this case).
See: Connecting to Cloud SQL from Cloud Functions
a) create a UNIX socketpath
connection to [Cloud SQL Instance]
When I run with the VPC Connector (as shown above), I get:
1) SUCCESS! I've received the API data using my whitelisted IP address
2) CONNECTION REFUSED by [Cloud SQL Instance] - because I'm using a static external IP? Does socketpath use external, or connect within my Google Cloud Project?
If I remove the VPC Connector from my Cloud Function, then I get:
1) CONNECTION REFUSED - this IP is not whitelisted (because I'm no longer using the static IP)
2) SUCCESS! I'm now able to connect to [Cloud SQL Instance] (using UNIX socketpath, userid, password)
How can I get both of these to work from the same Cloud Function?
I see that I can "Route only requests to private IPs through the VPC connector" but I really want the opposite of that. I want to only route external requests to the VPC connector, to use my static IP, and then keep my private routing for connections within my GCP.
ADDED: I am using Javascript mysql to connect to Cloud SQL.
var pool = mysql.createPool({ socketPath: '/cloudsql/[instance_connection_name]',
user: uid,
password: pwd,
database: 'mysql_db' });
var result = pool.query(sql, {}, (err,result)=> {});
This works ok without using a VPC Connector. When I use a VPC Connector with a static external IP address, this connection is refused. Is it because the VPC Connector and Cloud SQL instance are not on the same VPC? I don't think Cloud SQL is linked to a VPC, is it?
The Cloud SQL Instance Overview dashboard lists both [Public IP Address] and [Instance Connection Name]
For a standard Cloud Functions connection, I use:
socketpath:[Instance Connection Name]
user: uid
password: pwd
database: 'mysql_db'
When using a VPC Connector, I use
host:[Private (or Public) IP Address]
user: uid
password: pwd
database: 'mysql_db'
Summary:
[Cloud Function] -> socketpath:[Instance Connection Name] => **SUCCESS**
[Cloud Function] -> host:[Public IP Address] => **FAIL** (Timeout - IP Not Allowed)
Private VPC Connections:
[Cloud Function] -> [VPC Connector] -> socketpath:[Instance Connection Name] => **FAIL** (Connection Refused)
[Cloud Function] -> [VPC Connector] -> host:[Private IP Address] => **SUCCESS** (Set up Private IP in GCP->SQL->Connections)
Public VPC Connection:
[Cloud Function] -> [VPC Connector] -> host:[Public IP Address] => **SUCCESS** (Only after allowing/whitelisting IP of the VPC Connector routed through Cloud NAT)
Google Cloud CLI instructions for Private IP setup:
CONNECTING FROM GOOGLE CLOUD FUNCTIONS TO CLOUD SQL USING TCP AND UNIX DOMAIN SOCKETS 2020
Related
I hope you are well.
I wan't to connect Google Datastream with MySQL database hosted into Cloud SQL by using private connection.
I have conected by the public internet using Public IP but I need to connect through VPC peering for security, documentation indicates that the following should be done: https://cloud.google.com/datastream/docs/private-connectivity
I have been tring to connect across Cloud SQL Auth Proxy follow this documentation and I have already connected the proxy to Cloud SQL: https://cloud.google.com/sql/docs/mysql/connect-admin-proxy#tcp-sockets
In every occasion I see the same error, no matter what ip configure it always changes to another ip.
For example, the follow image show the perfil configuration with Cloud SQL Auth Proxy internal IP 10.128.0.2 set up.
And, when I try to test, I've seen that the MySQL IP change for 192.168.5.236
In general, this is a MySQL client-side error code. The possible causes for this error are:
MySQL Server not running, or
Firewall configuration on the Windows server blocking access on port 3306
=> To troubleshoot the given error message, please follow the below mentioned steps:
Verify MySQL server is running and use the ping command to check the client-server connectivity. For Example: ping server_ip_address
To connect to a Cloud SQL instance using private IP, the Cloud SQL Auth proxy must be on a resource with access to the same VPC network as the instance.[1]
When you start the Cloud SQL Auth proxy, to ensure it is using the private IP, please make sure to pass the flag:
-ip_address_types=PRIVATE
Also,verify the firewall configuration and make sure the port 3306 is not blocked.
You can also view the mysql config file[2] and check if there is a bind-address relevant to it. If there is a bind-address, comment it out using the # character.
[1] https://cloud.google.com/sql/docs/mysql/connect-admin-proxy#private-ip
[2] How do I find the MySQL my.cnf location
In my case I have the Cloud SLQL Proxy mounted in compute engine and Mysql in Cloud SQL.
The solution was the following:
I needed to create a ingress firewall rule allow the ip range from the Datastream Private Connection (I needed to connect Datastream across a private connection for governance) to Cloud SQL Auth Proxy IP Range in tcp:3306.
You can test your connection throght Connectivity Tests
I am trying to migrate to use private IP:s for all our Cloud SQL instances. I have gotten it working for postgres, and am now trying to get access for our wordpress instances using MySQL.
The problem is I cant get the connection working from the running pods in our GKE cluster.
root#******:/var/www/app# mysql --host=10.**.**.* -u *_se -p
Enter password:
ERROR 2003 (HY000): Can't connect to MySQL server on '10.*.*.*' (110 "Connection timed out")
I have activated the private IP on the Cloud SQL instance, and added a private service connection to the VPC-network that is the same network that our cluster uses, but still cant connect in the same way as I did with postgres.
Does anyone know anything I could try to get this working?
There are specific network requirements Cloud SQL instances must
adhere to when communicating via a private connection. One of which
is that your Cloud SQL and GKE instances are located in the same
region and VPC network. Check this to configure private IP for MySQL.
The GKE cluster must be VPC-native and peered with the same VPC
network as the Cloud SQL instance. To connect MySQL from GKE, refer
to this Google documentation.
Note: If you are using shared VPC networks, you cannot assign a
private IP address in a shared VPC network to an existing Cloud SQL
instance. Also, VPC Network Peering uses private services access.
However, you do not create the VPC Network Peering explicitly, because
the peering is internal to Google Cloud.
Some possible causes of mentioned error are as follows:
Network failure especially if MySQL database server is running on a remote host.
No MYSQL server is running on the mentioned host.
Firewall blocking TCP-IP connection or other related reasons.
I would suggest you check the firewall setting and check if your MySQL server is listening on default port 3306. Also try to connect MySQL server on IP for which MySQL server is bound in 'my.cnf’. If it not so, run the following command to bind address if you are using Ubuntu:
1. Run the command vim /etc/mysql/my.cnf or vim /etc/mysql/mysqld.conf/mysqld.cnf
2. Comment bind-address = <Server IP> using the # symbol
3. Restart your MYSQL server once.
I want to connect Cloud SQL in Cloud Run Application. I used golang. this is the code around sql connect setting.
func getEnv(key, def string) string {
v := os.Getenv(key)
if v == "" {
return def
}
return v
}
DB: DB{
User: getEnv("DB_USER", "<user name>"),
Pass: getEnv("DB_PASS", "<password>"),
Host: getEnv("DB_HOST", "0.0.0.0"),
Port: getEnv("DB_PORT", "3306"),
Database: getEnv("DB_DATABASE", "<database name>"),
},
dsn := fmt.Sprintf("%s:%s#tcp(%s:%s)/%s?charset=utf8&parseTime=true",
config.DB.User, config.DB.Pass, config.DB.Host, config.DB.Port, config.DB.Database)
db, err := gorm.Open("mysql", dsn)
I set environment variable at Cloud Run setting console. After delpoy Application, Cloud Run console display Cloud Run error: Container failed to start. Failed to start and then listen on the port defined by the PORT environment variable. Logs for this revision might contain more information. and dial tcp <Cloud SQL Private IP> :3306: connect: connection timed out I wonder SQL connection is wrong...
You have not mentioned the word "VPC" in your question so I'm assuming you don't use it.
Cloud Run cannot directly connect to a private IP of a Cloud SQL instance. You need to configure a Serverless VPC Access Connector and specify it while deploying your Cloud Run app.
Cloud Run containers are not part of a VPC by default, so unless you do this, they will not have access to the private networks.
There is several way to connect your Cloud SQL database to Cloud Run. If it's MySQL, the easiest way is to follow the official documentation
If you want to use the IP with TCP connection, firstly, you can't use 0.0.0.0 as IP.
Use the Cloud SQL public IP (for this you have to authorize 0.0.0.0/0 network range on your Cloud SQL instance, and it's absolutely not recommended)
Plug your Cloud SQL to your VPC. And, as described by Ahmet, use serverless VPC connector to link Cloud Run with your VPC. Then add the private IP of your Cloud SQL in your code.
I have followed this guide https://cloud.google.com/community/tutorials/run-sailsjs-on-google-app-engine
on how to deployed Sails to Google App Engine.
However, my app used MySQL as the database and I have set it up like this in "connection.js".
mysql: {
adapter: 'sails-mysql',
host: '104.199.196.99', // Internal IP Address of the Cloud SQL Instance
user: 'root',
password: 'xxxxxx',
database: 'xxxxxxxx'
}
The error I got when trying to connect with Cloud SQL (when I make a requrest to log-in which involved connection to database) was:
Sending 500 ("Server Error") response:
Error (E_UNKNOWN) :: Encountered an unexpected error
Could not connect to MySQL:
Error: connect ETIMEDOUT
Did I set it up correctly?
For those who seek the solution for connecting to Google Cloud SQL using Google App Engine with Sails.js, I have figured it out.
Connecting to the Google Cloud SQL using host address doest not work here, because the Firewall Rules and the Access Control of the Cloud SQL will prevent this kind of connection to the whitelisted addresses. Unfortunately, Google App Engine could not be assigned with Static IP Address. Therefore, the App Engine could not be whitelisted by the Cloud SQL or you would need to whitelist 0.0.0.0/0 which is everything and that is not secure.
The solution is to use this configuration in your connection.js.
googleCloudSQL: {
adapter: 'sails-mysql',
socketPath: '/cloudsql/' + process.env.INSTANCE_CONNECTION_NAME,
user: 'XXXXX',
password: 'XXXXXXX',
database: 'XXXXXXXX'
}
By connecting to the Cloud SQL via socket path using the Instance Connection Name of the Cloud SQL, this is the native way of doing it. If your App Engine is already in the same project with your Cloud SQL then it will already be authorized to connect. However, if they are in different projects, you would need to set up service account and IAM.
I want to host my application based on node.js and MySql database. I try heroku hosting. I've create an application and add ClearDB add-on with Punch DB plan. The main restriction of this add-on that I cannot create any user-defined functions and events in this plan. It is sopported for expensive database plans.
Then I host my db on Google Cloud SQL. It allows to create user-defined functions and events. However I cannot access to the database from my heroku app because I don't know how to authorize external heroku network for my google cloud database. How can I do this?
Or may be there are any other hostings for NodeJS + MySql application where there are no the restrictions mentioned above?
You should be able to:
a) Use Cloud SQL by authorizing the external IP of your Heroku dyno via a proxy service like QuotaGuard Static or Proximo;
b) Do the same using Amazon RDS.
Additional details that may help:
1. SO question related to QuotaGuard
To clarify: you have to use a third-party add-on like this because Heroku won't guarantee a single IP or even a range of IPs for your dyno.
I add QuotaGuard Static add-on and it holds two ip adresses. Than I add this addresses to the Authorized Networks of my Google Cloud Sql. After then I try to connect to tha database using a guide. Here is my code:
exports.getConnection = () ->
dbConnParams =
host: 'google cloud sql ip'
port: 3306
proxy = url.parse process.env.QUOTAGUARDSTATIC_URL
auth = proxy.auth;
username = auth.split(":")[0]
pass = auth.split(":")[1]
sock_options =
host: proxy.hostname
port: proxy.port
user: username
pass: pass
sockConn = new SocksConnection dbConnParams, sock_options
dbConnection = mysql.createConnection({
user: 'root',
password: 'pwd',
stream: sockConn
});
return dbConnection
However I obtain an error: "connect ECONNREFUSED". Whats wrong with this approach?