I hope you are well.
I wan't to connect Google Datastream with MySQL database hosted into Cloud SQL by using private connection.
I have conected by the public internet using Public IP but I need to connect through VPC peering for security, documentation indicates that the following should be done: https://cloud.google.com/datastream/docs/private-connectivity
I have been tring to connect across Cloud SQL Auth Proxy follow this documentation and I have already connected the proxy to Cloud SQL: https://cloud.google.com/sql/docs/mysql/connect-admin-proxy#tcp-sockets
In every occasion I see the same error, no matter what ip configure it always changes to another ip.
For example, the follow image show the perfil configuration with Cloud SQL Auth Proxy internal IP 10.128.0.2 set up.
And, when I try to test, I've seen that the MySQL IP change for 192.168.5.236
In general, this is a MySQL client-side error code. The possible causes for this error are:
MySQL Server not running, or
Firewall configuration on the Windows server blocking access on port 3306
=> To troubleshoot the given error message, please follow the below mentioned steps:
Verify MySQL server is running and use the ping command to check the client-server connectivity. For Example: ping server_ip_address
To connect to a Cloud SQL instance using private IP, the Cloud SQL Auth proxy must be on a resource with access to the same VPC network as the instance.[1]
When you start the Cloud SQL Auth proxy, to ensure it is using the private IP, please make sure to pass the flag:
-ip_address_types=PRIVATE
Also,verify the firewall configuration and make sure the port 3306 is not blocked.
You can also view the mysql config file[2] and check if there is a bind-address relevant to it. If there is a bind-address, comment it out using the # character.
[1] https://cloud.google.com/sql/docs/mysql/connect-admin-proxy#private-ip
[2] How do I find the MySQL my.cnf location
In my case I have the Cloud SLQL Proxy mounted in compute engine and Mysql in Cloud SQL.
The solution was the following:
I needed to create a ingress firewall rule allow the ip range from the Datastream Private Connection (I needed to connect Datastream across a private connection for governance) to Cloud SQL Auth Proxy IP Range in tcp:3306.
You can test your connection throght Connectivity Tests
Hi in this instance database security group is open for all inbound traffic (ALL ports - All sources).
I am also able to connect to the database fine in mysql workbench or Datagrip which for sure use jdbc connection string.
2019-05-21 14:12:03 INFO CatalogClient:651 - Got connection 'tem-sas-main' info from Catalog with url: jdbc:mysql://my-database-example:3306/sas_tem_central
2019-05-21 14:12:03 INFO CatalogClient:684 - JDBC configuration for connection tem-sas-main: JDBCConfiguration(url=jdbc:mysql://my-database-example:3306/sas_tem_central, hostname=my-database-example, port=3306, databaseVendor=mysql, databaseVersion=null, connectionName=tem-sas-main, path=sas_tem_central, subnetId=subnet-0717c4db096e84393, availabilityZone=eu-west-1a, securityGroups=[sg-074b074ebc51c2315], enforceSSL=false)
2019-05-21 14:12:03 INFO JdbcConnection:42 - Starting connecter. driver com.mysql.jdbc.Driver#7e5d9a50
2019-05-21 14:12:03 INFO JdbcConnection:60 - Attempting to connect with SSL host matching: jdbc:mysql://my-database-example:3306/sas_tem_central
2019-05-21 14:14:15 INFO JdbcConnection:69 - SSL connection to data store using host matching failed. Retrying without host matching.
2019-05-21 14:14:15 INFO JdbcConnection:83 - Attempting to connect with SSL: jdbc:mysql://my-database-example:3306/sas_tem_central
2019-05-21 14:16:26 INFO JdbcConnection:88 - SSL connection to data store failed. Retrying without SSL.
2019-05-21 14:16:26 INFO JdbcConnection:102 - Attempting to connect without SSL: jdbc:mysql://my-database-example:3306/sas_tem_central
Check that your connection definition references your JDBC database with correct URL syntax, username, and password. Communications link failure
The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server.
Also note my JDBC connection string is
jdbc:mysql://my-database-example:3306/sas_tem_central
And Require SSL connection is set to false
After speaking to an AWS Architect, I can confirm that Glue does not work with MySql Version 8 at this time. This is NOT documented anywhere I could find.
Glue's supported Databases (not pointed out prominently in the Glue documentation / faqs)
The following table lists the JDBC driver versions that AWS Glue supports.
Microsoft SQL Server 6.x
MySQL 5.1
Oracle Database 11.2
PostgreSQL 42.x
Amazon Redshift 4.1
Now you can customize your own configuration to connect to MySQL 8 and other newer databases from AWS Glue Jobs.
Download the JDBC driver for MySQL 8
Upload to S3
Create a JDBC connection to your database via Glue - Tables - Connections
(Note this will not work using the "test connection" since the latest version supported is (currently) 5.7
Edit the Glue Job - Dependent jars path - e.g.
s3://yourS3bucket/path/mysql-connector-java-8.0.21.jar
Modify the Glue Job e.g. Python script
datasink4 = glueContext.write_dynamic_frame.from_jdbc_conf
(frame = dropnullfields3, catalog_connection = “GLUE-CONNECTION-NAME”,
connection_options = {
"customJdbcDriverS3Path": "s3://yourS3Bucket/path/mysql-connector-java-8.0.21.jar",
"customJdbcDriverClassName": "com.mysql.cj.jdbc.Driver",
"user": “#DBuserName#”, "password": “DbUsersPassword”,
"url":"jdbc:mysql://dbname.url.region.rds.amazonaws.com/schemaName,
"connectionType": "mysql", "dbtable": “tablenameInDb”,
"database": “schemaName”}, transformation_ctx = "datasink4")
(no new lines in that script)
Your connection string looks right.
I was struggling with this same issue and found that I had to troubleshoot a few issues:
Ensure that your security group that you are using for your RDS is self-referenced. This means that you should add the security group from your db instance on RDS to both the inbound and outbound rules for All TCP.
Ensure that you have created an endpoint for your VPC. To do this, navigate to your VPC, click on 'Endpoint', add an 'Endpoint' for the VPC that the RDS is in and create an endpoint for S3.
Finally, the version you create your MySQL RDS instance on should be version 5 (not version 8). I noticed that AWS Glue JDBC connector only worked with MySQL version 5 (5.7)
When you use the AWS RDS with the MySQL Engine, you don't need to create a JDBC connection. You can use the Amazon RDS when you are creating your connection.
New Connection Screen
I first tried with mysql8. It didn't work. It thrown same error.
Later i created mysql5.7 for testing connection. It worked now. I can able to connect Mysql5.7 from Glue. Makesure your VPC-s3 Endpoint are configured properly.
Thanks
Raaz
I created a database on AWS - RDS.And created a nodes app.
On MySql Workbench localhost I connected to database using endpoint and it was success and I got data on localhost/users
I deployed the nodejs app server to AWS but I did not get any data when I tried to call https://xalynj2ul4.execute-api.us-west-2.amazonaws.com/staging/users.
Try white-listing your node server in mysql host config file by entering your server ip in it
I am trying to connect to MySQL server at remote machine using spring-boot application.properties but it is failing with error o.s.b.a.orm.jpa.DatabaseLookup: Unable to determine jdbc url from datasource.
My database URL is in the correct format. Remote Server requires ssh connection and I have connected it successfully using MySQL Workbench. For connection via spring-boot app, I checked solutions and implemented JSCH solution as explained in post:
Spring Boot SSH Mysql? but still no success.
Remote Server SSH port is 202 instead of default 22. Any help on how to connect DB?
Please refer information mention here -
here
or
stackoverflow page
I am receiving an error when trying to load up my webpage
Failed to connect to MySQL: (2005) Unknown MySQL server host ':/cloudsql/testsite:europe-west1:testdatabase' (2)Error:
I have a Google Compute Engine VM set up with a LAMP stack (Apache/2.4.10 (Debian)/ Database client version: libmysql - 5.5.55 / PHP extension: mysqli)
I also have set up an instance on Google SQL with user credentials for aforementioned VM (i have set up both First Gen and Second Gen)
I can access both a local MySQL database on the VM as well as the Google SQL databases via phpAdmin installed locally
HOWEVER i appear to have an issue with the DB_HOST credentials in my config.php file when i run the script
path = /var/www/html/includes/config.php
I get
usually for local MYSQL databases i use
// The MySQL credentials
$CONF['host'] = 'localhost';
$CONF['user'] = 'YOURDBUSER';
$CONF['pass'] = 'YOURDBPASS';
$CONF['name'] = 'YOURDBNAME';
Documentation (and github links) recommend path
:/cloudsql/project-id:region:sql-db-instance-name
which is what i have done (see above) - but i keep getting the error message.
Am i typing the host description incorrectly? Or have i missed a configuration step?
Thanks in advance
It seems as if i have erred and that the credentials format i stated earlier are for Google App Engine
If you are on Google Compute Engine, you have two options:
Connect to the public IP address of your Cloud SQL instance. This requires you whitelist your GCE instance on the ACL for the Cloud SQL instance.
Use the Cloud SQL proxy. This is a extra daemon you run on your GCE instance that allows you to connect via TCP on localhost or a socket.