Why I can not use a 5.7 nor a 5.6 mysql docker image instead of mysql 8 with terraform in windows? - mysql

I am testing a mysql_database inside a docker_container.mysql using terraform in windows, but every time I try to use an image different from mysql:8 inside the docker_image.mysql used by docker_container.mysql, terraform takes 5 minutes to create the mysql_database resource and throws the following error:
Error: Could not connect to server: dial tcp 127.0.0.1:3306: connectex: No connection could be made because the target machine actively refused it.
on main.tf line 33, in resource "mysql_database" "test":
33: resource "mysql_database" "test" {
And here is main.tf:
provider "docker" {
host = "npipe:////.//pipe//docker_engine"
}
resource "docker_image" "mysql" {
name = "mysql:8"
//keep_locally = true
}
resource "docker_container" "mysql" {
name = "mysql"
image = docker_image.mysql.latest
restart = "always"
env = [
"MYSQL_ROOT_PASSWORD=root"
]
volumes {
volume_name = "mysql-vol"
container_path = "/var/lib/mysql"
}
ports {
internal = 3306
external = 3306
}
}
provider "mysql" {
endpoint = "127.0.0.1:3306"
username = "root"
password = "root"
}
resource "mysql_database" "test" {
name = "test"
depends_on = [docker_container.mysql]
}
I am testing mysql image tags shown at https://hub.docker.com/_/mysql, specifically 5.6, 5.7 and 8, but only using mysql:8 seems to work Is there an other way in which I should reference those mysql image tags?

I tried to verify the issue, and I observed the same error as yours only for mysql 5.7 and 5.6 when you keep the same volumes.
After removing the following section from the terraform script
volumes {
volume_name = "mysql-vol"
container_path = "/var/lib/mysql"
}
and removing existing mysql docker images, mysql 5.6, mysql 5.7 and 8 worked as expected.
Btw, the error leading to failed connection was:
ERROR 2013 (HY000): Lost connection to MySQL server at 'handshake: reading initial communication packet', system error: 11

Related

Problems connecting Cloud Run Application to Cloud SQL using Spring boot

I am trying to connect a Spring application (using Kotlin and Gradle) to a Google Cloud SQL instance and database. I am getting the error message
java.lang.RuntimeException: [<project-name>:europe-west1:<db-instance>] The Cloud SQL Instance does not exist or your account is not authorized to access it. Please verify the instance connection name and check the IAM permissions for project "<project-name>"
I have followed the guide on how to connect carefully, but to no avail.
Relevant files
src/main/resources/application.yml
server:
port: ${PORT:8080}
spring:
liquibase:
change-log: classpath:liquibase/db.changelog.xml
contexts: production
cloud:
appId: <project-id>
gcp:
sql:
instance-connection-name: <instance-connection-name>
database-name: <db-name>
jpa:
hibernate:
dialect: org.hibernate.dialect.MySQL8Dialect
default_schema: <schema>
show_sql: true
ddl-auto: none
datasource:
driver-class-name: com.mysql.cj.jdbc.Driver
continue-on-error: true
initialization-mode: always
url: jdbc:mysql:///<db-name>?cloudSqlInstance=<instance-connection-name>&socketFactory=com.google.cloud.sql.mysql.SocketFactory&user=<user>&password=<password>
username: <user>
password: <password>
---
spring:
config:
activate:
on-profile: dev
jpa:
hibernate:
ddl-auto: create-drop
spring.jpa.database-platform: org.hibernate.dialect.H2Dialect
datasource:
url: jdbc:h2:mem:mydb
username: sa
password: password
driverClassName: org.h2.Driver
cloud:
gcp:
sql:
enabled: false
build.gradle.kts
import org.jetbrains.kotlin.gradle.tasks.KotlinCompile
plugins {
id("org.springframework.boot") version "2.6.5"
id("io.spring.dependency-management") version "1.0.11.RELEASE"
kotlin("jvm") version "1.6.10"
kotlin("plugin.spring") version "1.6.10"
kotlin("plugin.allopen") version "1.4.32"
kotlin("plugin.jpa") version "1.4.32"
kotlin("kapt") version "1.4.32"
}
allOpen {
annotation("javax.persistence.Entity")
annotation("javax.persistence.Embeddable")
annotation("javax.persistence.MappedSuperclass")
}
group = "com.<company>"
version = "0.0.1-SNAPSHOT"
java.sourceCompatibility = JavaVersion.VERSION_17
repositories {
mavenCentral()
}
dependencies {
implementation("org.springframework.boot:spring-boot-starter-web:2.6.5")
implementation("org.springframework.boot:spring-boot-starter-webflux:2.6.5")
implementation("org.springframework.boot:spring-boot-starter-data-jpa:2.6.5")
implementation("org.springframework.cloud:spring-cloud-gcp-starter-sql-mysql:1.2.8.RELEASE")
implementation("org.jetbrains.kotlin:kotlin-reflect:1.6.10")
implementation("org.jetbrains.kotlin:kotlin-stdlib-jdk8:1.6.10")
implementation("com.fasterxml.jackson.dataformat:jackson-dataformat-xml:2.13.2")
implementation("com.fasterxml.jackson.core:jackson-annotations:2.13.2")
implementation("com.fasterxml.jackson.core:jackson-core:2.13.2")
implementation("com.fasterxml.jackson.core:jackson-databind:2.13.2.2")
implementation("com.fasterxml.jackson.module:jackson-module-kotlin:2.13.2")
implementation("com.fasterxml.jackson.datatype:jackson-datatype-jsr310:2.13.2")
implementation("org.hibernate:hibernate-core:5.6.7.Final")
implementation("javax.persistence:javax.persistence-api:2.2")
implementation( "commons-codec:commons-codec:1.15")
implementation("io.github.microutils:kotlin-logging-jvm:2.1.21")
implementation("ch.qos.logback:logback-classic:1.2.11")
implementation("com.google.cloud.sql:mysql-socket-factory-connector-j-8:1.4.4")
runtimeOnly("com.h2database:h2:2.1.210")
runtimeOnly("org.springframework.boot:spring-boot-devtools:2.6.5")
testImplementation("org.springframework.boot:spring-boot-starter-test:2.6.5")
}
tasks.withType<KotlinCompile> {
kotlinOptions {
freeCompilerArgs = listOf("-Xjsr305=strict")
jvmTarget = "17"
}
}
tasks.withType<Test> {
useJUnitPlatform()
}
Dockerfile
FROM openjdk:17-alpine
ENV USER=appuser
# <placeholder> Replace context path for your own application
ENV JAVA_HOME=/opt/openjdk-17 \
HOME=/home/$USER \
CONTEXT_PATH=/aws-service-baseline
RUN adduser -S $USER
# <placeholder> Add additional packages for the docker container here
RUN apk add --no-cache su-exec
# <placeholder> Replace baseline.jar with your applications JAR file (defined in build.gradle.kts)
COPY Docker/runapp.sh build/libs/<application-name>-0.0.1-SNAPSHOT.jar $HOME/
RUN chmod 755 $HOME/*.sh && \
chown -R $USER $HOME
WORKDIR /home/$USER
CMD [ "./runapp.sh"]
Docker/runapp.sh
#!/bin/sh
set -e
# The module to start.
# <placeholder> Replace this with your own modulename (from module-info)
APP_JAR="<application-name>-0.0.1-SNAPSHOT.jar"
JAVA_PARAMS="-XshowSettings:vm"
echo " --- RUNNING $(basename "$0") $(date -u "+%Y-%m-%d %H:%M:%S Z") --- "
set -x
/sbin/su-exec "$USER:1000" "$JAVA_HOME/bin/java" "$JAVA_PARAMS $JAVA_PARAMS_OVERRIDE" -jar -Dserver.port=$PORT "$APP_JAR"
GCP details
I have made sure the SQL instances connection is added to the Cloud Run Revisions. The IAM roles for the compute service account also seem to be right. See images
IAM: https://i.stack.imgur.com/yYaC5.png
Database: https://i.stack.imgur.com/NErad.png
Cloud Run connection https://i.stack.imgur.com/fKTSZ.png
Additional details
When running ./gradlew bootRun on my local machine (with GCP credentials present), the App works properly with an SQL connection. It also works after running ./gradle bootRun to build the JAR file and run the JAR directly. It does not work out of the box when running in Docker, but if I add the GCP credentials to the Docker container locally, it connects to the Database.
Does anyone have any suggestions on what might be wrong? Any help much appreciated!
I have tried connecting locally and locally in a Docker container.
Figured it out! Human error of course. The Cloud Run Service was initially configured with another Services Account, and not the default Compute Engine Service account.

MySql Setup in Linux Docker Container Via Terraform

Requirement: Need to automate MySQL installation & Database creation on Linux(Ubuntu)Docker Container via Terra form.
I am doing all this stuff on my local machine & below is the Terra form configuration.
Terra form file:
resource "docker_container" "db-server1" {
name = "db-server"
image = docker_image.ubuntu.latest
ports {
internal = 80
external = 9093
}
provisioner "local-exec" {
command = "docker container start dbs-my"
}
provisioner "local-exec" {
command = "docker exec dbs-my apt-get update"
}
provisioner "local-exec" {
command = "docker exec dbs-my apt-get -y install mysql-server"
}
}
But in container there is no mysql service present, when i am trying to launch mysql command, i am getting below error:
ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (2)
Using Terraform for this at all is a little unusual; you might look at more Docker-native tools like Docker Compose to set this up. There are also several anti-patterns in this example: you should generally avoid installing software in running containers, and avoid running long sequences of imperative commands via Terraform, and it's usually not useful to run the bare ubuntu Docker image as-is.
You can run the Docker Hub mysql image instead:
resource "docker_image" "mysql" {
name = "mysql:8"
}
resource "random_password" "mysql_root_password" {
length = 16
}
resource "docker_container" "mysql" {
name = "mysql"
image = "${docker_image.mysql.latest}"
env {
MYSQL_ROOT_PASSWORD = "${random_password.mysql_root_password.result}"
}
mounts {
source = "/some/host/mysql/data/path"
target = "/var/lib/mysql/data"
type = "bind"
}
ports {
internal = 3306
external = 3306
}
}
If you wanted to do further setup on the created database, you could use the MySQL provider
provider "mysql" {
endpoint = "127.0.0.1:3306" # the "external" port
username = "root"
password = "${random_password.mysql_root_password.result}"
}
resource "mysql_database" "db" {
name = "db"
}

docker and mysql: Got an error reading communication packets

I have a problem with connectivity in docker. I use an official mysql 5.7 image and Prisma server. When I start it via prisma cli, that uses docker compose underneath (described here) everything works.
But I need to start this containers programmatically via docker api and in this case connections from app are dropped with [Note] Aborted connection 8 to db: 'unconnected' user: 'root' host: '164.20.10.2' (Got an error reading communication packets).
So what I doo:
Creating a bridge network:
const network = await docker.network.create({
Name: manifest.name + '_network',
IPAM: {
"Driver": "default",
"Config": [
{
"Subnet": "164.20.0.0/16",
"IPRange": "164.20.10.0/24"
}
]
}});
Creating mysql container and attaching it to network
const mysql = await docker.container.create({
Image: 'mysql:5.7',
Hostname: manifest.name + '-mysql',
Names: ['/' + manifest.name + '-mysql'],
NetworkingConfig: {
EndpointsConfig: {
[manifest.name + '_network']: {
Aliases: [manifest.name + '-mysql']
}
}
},
Restart: 'always',
Args: [
"mysqld",
"--max-connections=1000",
"--sql-mode=ALLOW_INVALID_DATES,ANSI_QUOTES,ERROR_FOR_DIVISION_BY_ZERO,HIGH_NOT_PRECEDENCE,IGNORE_SPACE,NO_AUTO_CREATE_USER,NO_AUTO_VALUE_ON_ZERO,NO_BACKSLASH_ESCAPES,NO_DIR_IN_CREATE,NO_ENGINE_SUBSTITUTION,NO_FIELD_OPTIONS,NO_KEY_OPTIONS,NO_TABLE_OPTIONS,NO_UNSIGNED_SUBTRACTION,NO_ZERO_DATE,NO_ZERO_IN_DATE,ONLY_FULL_GROUP_BY,PIPES_AS_CONCAT,REAL_AS_FLOAT,STRICT_ALL_TABLES,STRICT_TRANS_TABLES,ANSI,DB2,MAXDB,MSSQL,MYSQL323,MYSQL40,ORACLE,POSTGRESQL,TRADITIONAL"
],
Env: [
'MYSQL_ROOT_PASSWORD=secret'
]
});
await network.connect({
Container: mysql.id
});
await mysql.start();
Then I wait Mysql to boot, create needed databases and needed Prisma containers from prismagraphql/prisma:1.1 and start them. App server resolves mysql host correctly, but connections are dropped by mysql.
Telnet from app container to mysql container in 3306 port responds correctly:
J
5.7.21U;uH Kem']#45T]2mysql_native_password
What am I doing wrong?
Check the below:
max_allowed_packets
wait_timeout
net_read_timeout
Also monitor MySQL process list during the issue to identify timeouts.
Can you try some wait, it could be possible that application try to connect to mysql server before its ready to accept connection. To test this, add some wait on startup or run mysql followed by application as different deployments.
The fix is to add --wait-timeout=28800 (or higher number) into MySQL arguments:
Args: [
"mysqld",
"--max-connections=1000",
"--sql-mode=ALLOW_INVALID_DATES,ANSI_QUOTES,ERROR_FOR_DIVISION_BY_ZERO,HIGH_NOT_PRECEDENCE,IGNORE_SPACE,NO_AUTO_CREATE_USER,NO_AUTO_VALUE_ON_ZERO,NO_BACKSLASH_ESCAPES,NO_DIR_IN_CREATE,NO_ENGINE_SUBSTITUTION,NO_FIELD_OPTIONS,NO_KEY_OPTIONS,NO_TABLE_OPTIONS,NO_UNSIGNED_SUBTRACTION,NO_ZERO_DATE,NO_ZERO_IN_DATE,ONLY_FULL_GROUP_BY,PIPES_AS_CONCAT,REAL_AS_FLOAT,STRICT_ALL_TABLES,STRICT_TRANS_TABLES,ANSI,DB2,MAXDB,MSSQL,MYSQL323,MYSQL40,ORACLE,POSTGRESQL,TRADITIONAL",
"--wait-timeout=28800" // 28800 sec = 8 hours
],
Reference: https://dev.mysql.com/doc/refman/5.7/en/server-system-variables.html#sysvar_wait_timeout
But maybe it's wiser to find out what is the root cause for idle connections.

Connection refused from MySQL runtime in Eclipse Che

I'm trying to connect to a db in a MySQL runtime from another NodeJS runtime in a multi-machine workspace.
In a test I'm calling the API http://localhost:3000/target with the list of target users. Code in this API runs a SELECT on the db:
...
exports.list = function(req, res) {
req.getConnection(function(err, connection) {
if (err) {
console.log("MySQL " + err);
} else {
connection.query('SELECT id FROM target', function(err, rows) {
if (err) {
console.log("Error Selecting : %s ", err);
} else {
...
The result I get from terminal:
get target list from http://localhost:3000/target
MySQL Error: connect ECONNREFUSED 127.0.0.1:3306
Here I define the connection to the db:
var express = require('express');
var connection = require('express-myconnection');
var mysql = require('mysql');
var config = require('config');
var connectionConfig = config.get('mysql');
var connectionInstance = connection(mysql, connectionConfig, 'request');
...
app.use(connectionInstance);
app.get('/', function(req, res) {
res.send('Welcome');
});
app.get('/target', target.list);
....
config:
{
"mysql": {
"host": "localhost",
"user": "[user]",
"password": "[password]",
"database": "[database]"
},
"app": {
"port": 3000,
"server": "http://localhost"
}
}
This is what I have in the configuration of the db machine in Eclipse Che:
snapshot of servers configuration
Here's my recipe:
services:
db:
image: eclipse/mysql
environment:
MYSQL_ROOT_PASSWORD: password
MYSQL_DATABASE: petclinic
MYSQL_USER: petclinic
MYSQL_PASSWORD: password
MYSQL_ROOT_USER: root
mem_limit: 1073741824
dev-machine:
image: eclipse/node
mem_limit: 2147483648
depends_on:
- db
elasticsearch:
image: florentbenoit/cdvy-ela-23
mem_limit: 2147483648
Can you share your recipe for the multi-machine workspace? That would help a lot in debugging it.
Just a guess: I think the problem with your setup is the use of localhost for your db connection. If you are running a multi-machine setup, the db is running in a different docker container and needs to be addressed by its name.
Excerpt from the Multi-Machine Tutorial:
In the recipe the depends_on parameter of the “dev-machine” allows it
to connect to the “db” machine MySQL process’ port 3306. The
“dev-machine” configures its MySQL client connection in the projects
source code at src/main/resources/spring/data-access.properties. The
url is defined by jdbc.url=jdbc:mysql://db:3306/petclinic which uses
the database machine’s name “db” and the MySQL server default port
3306.
You need to configure the open ports in your recipe.
Disclaimer: I am not directly affiliated with Eclipse Che, Codenvy or Red Hat, but we are building our own cloud IDE for C/C++ multicore optimization on top of Eclipse Che.

Unable to connect to any of the specified MySQL hosts in C# using SSH.Net Tunnel

I'm trying to connect to a remote mysql server using a SSH tunnel set up using OpenSSH key file and SSH.Net library as shown in the code section below...
try
{
PrivateKeyFile pkfile = new PrivateKeyFile("C:\\Users\\OpenSSHKey", "mypassword");
var client = new SshClient("servername.com", 22, "myusername", pkfile);
client.Connect();
if (client.IsConnected)
{
var local = new ForwardedPortLocal(22, "localhost", 3306);
client.AddForwardedPort(local);
try
{
local.Start();
}
catch (SocketException se)
{
}
string connStr = "server=localhost;user=mysql_username;database=database_name;port=3306;password=MyDBPassword;";
MySqlConnection sql = new MySqlConnection(connStr);
sql.Ping();
try
{
sql.Open();
}
catch (MySqlException se)
{
}
}
}
catch(Exception e)
{
}
the tunnel is definitely set up because it throws the following error "Unable to connect to any of the specified MySQL hosts" on the sql.Open() call...
Im sure the database is on the server named mysername because I can connect to it using putty or workbench with the credentials used in the MYSQL connection string... I hope someone can help
I've just solved exactly this problem myself. Simply change your ForwardedPortLocal to:
var local = new ForwardedPortLocal("127.0.0.1", 3306, "127.0.0.1", 3306);
In your original, you were giving the source port as 22. However, your MySql connection would have been connecting to port 3306 on your local machine (which you would then tunnel across to port 3306 on the destination), so you need to ensure that the forwarded port is listening locally on 3306 to start with.
In addition, I've changed localhost to 127.0.0.1. A secondary issue I tripped over was that localhost on my machine was actually resolving to ::1, an IPv6 address, and the destination server didn't 'speak' IPv6. By switching this to the 127.0.0.1 (IPv4's version of localhost), it ensures that IPv4 is used by default.