MySql Setup in Linux Docker Container Via Terraform - mysql

Requirement: Need to automate MySQL installation & Database creation on Linux(Ubuntu)Docker Container via Terra form.
I am doing all this stuff on my local machine & below is the Terra form configuration.
Terra form file:
resource "docker_container" "db-server1" {
name = "db-server"
image = docker_image.ubuntu.latest
ports {
internal = 80
external = 9093
}
provisioner "local-exec" {
command = "docker container start dbs-my"
}
provisioner "local-exec" {
command = "docker exec dbs-my apt-get update"
}
provisioner "local-exec" {
command = "docker exec dbs-my apt-get -y install mysql-server"
}
}
But in container there is no mysql service present, when i am trying to launch mysql command, i am getting below error:
ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (2)

Using Terraform for this at all is a little unusual; you might look at more Docker-native tools like Docker Compose to set this up. There are also several anti-patterns in this example: you should generally avoid installing software in running containers, and avoid running long sequences of imperative commands via Terraform, and it's usually not useful to run the bare ubuntu Docker image as-is.
You can run the Docker Hub mysql image instead:
resource "docker_image" "mysql" {
name = "mysql:8"
}
resource "random_password" "mysql_root_password" {
length = 16
}
resource "docker_container" "mysql" {
name = "mysql"
image = "${docker_image.mysql.latest}"
env {
MYSQL_ROOT_PASSWORD = "${random_password.mysql_root_password.result}"
}
mounts {
source = "/some/host/mysql/data/path"
target = "/var/lib/mysql/data"
type = "bind"
}
ports {
internal = 3306
external = 3306
}
}
If you wanted to do further setup on the created database, you could use the MySQL provider
provider "mysql" {
endpoint = "127.0.0.1:3306" # the "external" port
username = "root"
password = "${random_password.mysql_root_password.result}"
}
resource "mysql_database" "db" {
name = "db"
}

Related

Problems connecting Cloud Run Application to Cloud SQL using Spring boot

I am trying to connect a Spring application (using Kotlin and Gradle) to a Google Cloud SQL instance and database. I am getting the error message
java.lang.RuntimeException: [<project-name>:europe-west1:<db-instance>] The Cloud SQL Instance does not exist or your account is not authorized to access it. Please verify the instance connection name and check the IAM permissions for project "<project-name>"
I have followed the guide on how to connect carefully, but to no avail.
Relevant files
src/main/resources/application.yml
server:
port: ${PORT:8080}
spring:
liquibase:
change-log: classpath:liquibase/db.changelog.xml
contexts: production
cloud:
appId: <project-id>
gcp:
sql:
instance-connection-name: <instance-connection-name>
database-name: <db-name>
jpa:
hibernate:
dialect: org.hibernate.dialect.MySQL8Dialect
default_schema: <schema>
show_sql: true
ddl-auto: none
datasource:
driver-class-name: com.mysql.cj.jdbc.Driver
continue-on-error: true
initialization-mode: always
url: jdbc:mysql:///<db-name>?cloudSqlInstance=<instance-connection-name>&socketFactory=com.google.cloud.sql.mysql.SocketFactory&user=<user>&password=<password>
username: <user>
password: <password>
---
spring:
config:
activate:
on-profile: dev
jpa:
hibernate:
ddl-auto: create-drop
spring.jpa.database-platform: org.hibernate.dialect.H2Dialect
datasource:
url: jdbc:h2:mem:mydb
username: sa
password: password
driverClassName: org.h2.Driver
cloud:
gcp:
sql:
enabled: false
build.gradle.kts
import org.jetbrains.kotlin.gradle.tasks.KotlinCompile
plugins {
id("org.springframework.boot") version "2.6.5"
id("io.spring.dependency-management") version "1.0.11.RELEASE"
kotlin("jvm") version "1.6.10"
kotlin("plugin.spring") version "1.6.10"
kotlin("plugin.allopen") version "1.4.32"
kotlin("plugin.jpa") version "1.4.32"
kotlin("kapt") version "1.4.32"
}
allOpen {
annotation("javax.persistence.Entity")
annotation("javax.persistence.Embeddable")
annotation("javax.persistence.MappedSuperclass")
}
group = "com.<company>"
version = "0.0.1-SNAPSHOT"
java.sourceCompatibility = JavaVersion.VERSION_17
repositories {
mavenCentral()
}
dependencies {
implementation("org.springframework.boot:spring-boot-starter-web:2.6.5")
implementation("org.springframework.boot:spring-boot-starter-webflux:2.6.5")
implementation("org.springframework.boot:spring-boot-starter-data-jpa:2.6.5")
implementation("org.springframework.cloud:spring-cloud-gcp-starter-sql-mysql:1.2.8.RELEASE")
implementation("org.jetbrains.kotlin:kotlin-reflect:1.6.10")
implementation("org.jetbrains.kotlin:kotlin-stdlib-jdk8:1.6.10")
implementation("com.fasterxml.jackson.dataformat:jackson-dataformat-xml:2.13.2")
implementation("com.fasterxml.jackson.core:jackson-annotations:2.13.2")
implementation("com.fasterxml.jackson.core:jackson-core:2.13.2")
implementation("com.fasterxml.jackson.core:jackson-databind:2.13.2.2")
implementation("com.fasterxml.jackson.module:jackson-module-kotlin:2.13.2")
implementation("com.fasterxml.jackson.datatype:jackson-datatype-jsr310:2.13.2")
implementation("org.hibernate:hibernate-core:5.6.7.Final")
implementation("javax.persistence:javax.persistence-api:2.2")
implementation( "commons-codec:commons-codec:1.15")
implementation("io.github.microutils:kotlin-logging-jvm:2.1.21")
implementation("ch.qos.logback:logback-classic:1.2.11")
implementation("com.google.cloud.sql:mysql-socket-factory-connector-j-8:1.4.4")
runtimeOnly("com.h2database:h2:2.1.210")
runtimeOnly("org.springframework.boot:spring-boot-devtools:2.6.5")
testImplementation("org.springframework.boot:spring-boot-starter-test:2.6.5")
}
tasks.withType<KotlinCompile> {
kotlinOptions {
freeCompilerArgs = listOf("-Xjsr305=strict")
jvmTarget = "17"
}
}
tasks.withType<Test> {
useJUnitPlatform()
}
Dockerfile
FROM openjdk:17-alpine
ENV USER=appuser
# <placeholder> Replace context path for your own application
ENV JAVA_HOME=/opt/openjdk-17 \
HOME=/home/$USER \
CONTEXT_PATH=/aws-service-baseline
RUN adduser -S $USER
# <placeholder> Add additional packages for the docker container here
RUN apk add --no-cache su-exec
# <placeholder> Replace baseline.jar with your applications JAR file (defined in build.gradle.kts)
COPY Docker/runapp.sh build/libs/<application-name>-0.0.1-SNAPSHOT.jar $HOME/
RUN chmod 755 $HOME/*.sh && \
chown -R $USER $HOME
WORKDIR /home/$USER
CMD [ "./runapp.sh"]
Docker/runapp.sh
#!/bin/sh
set -e
# The module to start.
# <placeholder> Replace this with your own modulename (from module-info)
APP_JAR="<application-name>-0.0.1-SNAPSHOT.jar"
JAVA_PARAMS="-XshowSettings:vm"
echo " --- RUNNING $(basename "$0") $(date -u "+%Y-%m-%d %H:%M:%S Z") --- "
set -x
/sbin/su-exec "$USER:1000" "$JAVA_HOME/bin/java" "$JAVA_PARAMS $JAVA_PARAMS_OVERRIDE" -jar -Dserver.port=$PORT "$APP_JAR"
GCP details
I have made sure the SQL instances connection is added to the Cloud Run Revisions. The IAM roles for the compute service account also seem to be right. See images
IAM: https://i.stack.imgur.com/yYaC5.png
Database: https://i.stack.imgur.com/NErad.png
Cloud Run connection https://i.stack.imgur.com/fKTSZ.png
Additional details
When running ./gradlew bootRun on my local machine (with GCP credentials present), the App works properly with an SQL connection. It also works after running ./gradle bootRun to build the JAR file and run the JAR directly. It does not work out of the box when running in Docker, but if I add the GCP credentials to the Docker container locally, it connects to the Database.
Does anyone have any suggestions on what might be wrong? Any help much appreciated!
I have tried connecting locally and locally in a Docker container.
Figured it out! Human error of course. The Cloud Run Service was initially configured with another Services Account, and not the default Compute Engine Service account.

Problems with execution aws command via ssh jenkins

Good morning, how are you?
I have a problem with one execution via ssh in my jenkins.
Those characters that appear before did not appear, and we have not changed anything in the node..
The code we use is:
withCredentials([usernamePassword(credentialsId: 'id', passwordVariable: 'pass', usernameVariable: 'user')]) {
def remote = [:]
remote.name = 'id_nme'
remote.host = 'ip_node'
remote.user = user
remote.password = pass
remote.allowAnyHosts = true
remote.timeoutSec = 300
sshCommand remote: remote, command: "aws autoscaling describe-auto-scaling-groups --auto-scaling-group-name [name_asg]"
}
When the command is launched, the job is stuck and does not progress.

Why I can not use a 5.7 nor a 5.6 mysql docker image instead of mysql 8 with terraform in windows?

I am testing a mysql_database inside a docker_container.mysql using terraform in windows, but every time I try to use an image different from mysql:8 inside the docker_image.mysql used by docker_container.mysql, terraform takes 5 minutes to create the mysql_database resource and throws the following error:
Error: Could not connect to server: dial tcp 127.0.0.1:3306: connectex: No connection could be made because the target machine actively refused it.
on main.tf line 33, in resource "mysql_database" "test":
33: resource "mysql_database" "test" {
And here is main.tf:
provider "docker" {
host = "npipe:////.//pipe//docker_engine"
}
resource "docker_image" "mysql" {
name = "mysql:8"
//keep_locally = true
}
resource "docker_container" "mysql" {
name = "mysql"
image = docker_image.mysql.latest
restart = "always"
env = [
"MYSQL_ROOT_PASSWORD=root"
]
volumes {
volume_name = "mysql-vol"
container_path = "/var/lib/mysql"
}
ports {
internal = 3306
external = 3306
}
}
provider "mysql" {
endpoint = "127.0.0.1:3306"
username = "root"
password = "root"
}
resource "mysql_database" "test" {
name = "test"
depends_on = [docker_container.mysql]
}
I am testing mysql image tags shown at https://hub.docker.com/_/mysql, specifically 5.6, 5.7 and 8, but only using mysql:8 seems to work Is there an other way in which I should reference those mysql image tags?
I tried to verify the issue, and I observed the same error as yours only for mysql 5.7 and 5.6 when you keep the same volumes.
After removing the following section from the terraform script
volumes {
volume_name = "mysql-vol"
container_path = "/var/lib/mysql"
}
and removing existing mysql docker images, mysql 5.6, mysql 5.7 and 8 worked as expected.
Btw, the error leading to failed connection was:
ERROR 2013 (HY000): Lost connection to MySQL server at 'handshake: reading initial communication packet', system error: 11

How to access external database from Node container?

I have a nextjs app which is supposed to connect to an external MySQL database (not one from the same docker network). When running the app locally, it works correctly when connecting to DB, but when running it in a Docker container it keeps on trying to connect to 127.0.0.1, even though the environment variables are configured correctly in the container
Error: Error: connect ECONNREFUSED 127.0.0.1:3306
nextjs_1 | at connect (/opt/app/node_modules/serverless-mysql/index.js:80:15)
Dockerfile config:
FROM node:alpine
RUN mkdir -p /opt/app
RUN apk add --no-cache libc6-compat
ENV NODE_ENV production
ENV PORT 3000
EXPOSE 3000
WORKDIR /opt/app
COPY package.json /opt/app
COPY package-lock.json /opt/app
RUN npm install --no-optional
COPY . /opt/app
RUN npm run build
RUN npx next telemetry disable
RUN addgroup -g 1001 -S nodejs
RUN adduser -S nextjs -u 1001
USER nextjs
CMD [ "npm", "start" ]
Connection code:
const mysql = require('serverless-mysql')
const db = mysql({
config: {
host: process.env.MYSQL_HOST,
database: process.env.MYSQL_DATABASE,
user: process.env.MYSQL_USER,
password: process.env.MYSQL_PASSWORD,
},
})
exports.query = async (query) => {
try {
const results = await db.query(query)
await db.end()
return results
} catch (error) {
return { error }
}
}
Any ideas?

How to execute mysql script insertion on terraform user_data?

The last line of the script was not executed.
I tried to execute the code manually on the instance created and it was successful.
#!/bin/bash
#install tools
apt-get update -y
apt-get install mysql-client -y
#Create MySQL config file
echo "[mysql]" >> ~/.my.cnf
echo "user = poc5admin" >> ~/.my.cnf
echo "password = poc5password" >> ~/.my.cnf
#test
echo "endpoint = ${rds_endpoint}" >> ~/variables
hostip=$(hostname -I)
endpoint=${rds_endpoint}
echo "$hostip" >> ~/variables
#I have created a table here but I will remove the code since it is unnecessary...
#Create User
echo "CREATE USER 'poc5user'#'%' IDENTIFIED BY 'poc5pass';" >> ~/mysqlscript.sql
echo "GRANT EVENT ON * . * TO 'poc5user'#'%';" >> ~/mysqlscript.sql
cp mysqlscript.sql /home/ubuntu/mysqlscript.sql
mysql -h $endpoint -u poc5admin < ~/mysqlscript.sql
Expected result: There should be a Database, Table and User created on the RDS instance.
You can insert or create Database like this from the bash script but it is not recommended an approach to work with RDS. better to place your data over s3 and import from the s3.
Here is the example, that will create DB
resource "aws_db_instance" "db" {
allocated_storage = 20
storage_type = "gp2"
engine = "mysql"
engine_version = "5.7"
instance_class = "db.t2.micro"
name = "mydb"
username = "foo"
password = "foobarbaz"
parameter_group_name = "default.mysql5.7"
s3_import {
source_engine = "mysql"
source_engine_version = "5.6"
bucket_name = "mybucket"
bucket_prefix = "backups"
ingestion_role = "arn:aws:iam::1234567890:role/role-xtrabackup-rds-restore"
}
}
~/.my.cnf why you need this? better to place these script in the s3 file.
second thing, If you still interesting to run from your local environment then you can insert from local-exec
resource "null_resource" "main_db_update_table" {
provisioner "local-exec" {
on_failure = "fail"
interpreter = ["/bin/bash", "-c"]
command = <<EOT
mysql -h ${aws_rds_cluster.db.endpoint} -u your_username -pyour_password your_db < mysql_script.sql
EOT
}
}
But better to with s3.
If you want to import from remote, you can explore remote-exec.
With user-data, you can do this but it seems your MySQL script not generating properly. better to cp script to remote and then run with local exec in remote.
There is no such thing as terraform "user_data". User data is a bootstrap script for the EC2 instances which you can use to install software/binaries or to execute your script at the boot time.
The script will be executed by the cloud-init, not by the terraform itself. The responsibility of the terraform is to set user-data for the ec2 instances.
You may check the cloud-init output logs which should have the result of your user-data script also.
From your code, I am not able to understand which step you have copied the below file.
cp mysqlscript.sql /home/ubuntu/mysqlscript.sql
mysql -h $endpoint -u poc5admin < ~/mysqlscript.sql
I am assuming that you are creating a new server and it does not have any file.
Thank you for your inputs. I have found an answer by moving the config file to /etc/mysql/my.cnf and then executing
mysql -h $endpoint -u poc5admin < ~/mysqlscript.sql