Problems connecting Cloud Run Application to Cloud SQL using Spring boot - mysql

I am trying to connect a Spring application (using Kotlin and Gradle) to a Google Cloud SQL instance and database. I am getting the error message
java.lang.RuntimeException: [<project-name>:europe-west1:<db-instance>] The Cloud SQL Instance does not exist or your account is not authorized to access it. Please verify the instance connection name and check the IAM permissions for project "<project-name>"
I have followed the guide on how to connect carefully, but to no avail.
Relevant files
src/main/resources/application.yml
server:
port: ${PORT:8080}
spring:
liquibase:
change-log: classpath:liquibase/db.changelog.xml
contexts: production
cloud:
appId: <project-id>
gcp:
sql:
instance-connection-name: <instance-connection-name>
database-name: <db-name>
jpa:
hibernate:
dialect: org.hibernate.dialect.MySQL8Dialect
default_schema: <schema>
show_sql: true
ddl-auto: none
datasource:
driver-class-name: com.mysql.cj.jdbc.Driver
continue-on-error: true
initialization-mode: always
url: jdbc:mysql:///<db-name>?cloudSqlInstance=<instance-connection-name>&socketFactory=com.google.cloud.sql.mysql.SocketFactory&user=<user>&password=<password>
username: <user>
password: <password>
---
spring:
config:
activate:
on-profile: dev
jpa:
hibernate:
ddl-auto: create-drop
spring.jpa.database-platform: org.hibernate.dialect.H2Dialect
datasource:
url: jdbc:h2:mem:mydb
username: sa
password: password
driverClassName: org.h2.Driver
cloud:
gcp:
sql:
enabled: false
build.gradle.kts
import org.jetbrains.kotlin.gradle.tasks.KotlinCompile
plugins {
id("org.springframework.boot") version "2.6.5"
id("io.spring.dependency-management") version "1.0.11.RELEASE"
kotlin("jvm") version "1.6.10"
kotlin("plugin.spring") version "1.6.10"
kotlin("plugin.allopen") version "1.4.32"
kotlin("plugin.jpa") version "1.4.32"
kotlin("kapt") version "1.4.32"
}
allOpen {
annotation("javax.persistence.Entity")
annotation("javax.persistence.Embeddable")
annotation("javax.persistence.MappedSuperclass")
}
group = "com.<company>"
version = "0.0.1-SNAPSHOT"
java.sourceCompatibility = JavaVersion.VERSION_17
repositories {
mavenCentral()
}
dependencies {
implementation("org.springframework.boot:spring-boot-starter-web:2.6.5")
implementation("org.springframework.boot:spring-boot-starter-webflux:2.6.5")
implementation("org.springframework.boot:spring-boot-starter-data-jpa:2.6.5")
implementation("org.springframework.cloud:spring-cloud-gcp-starter-sql-mysql:1.2.8.RELEASE")
implementation("org.jetbrains.kotlin:kotlin-reflect:1.6.10")
implementation("org.jetbrains.kotlin:kotlin-stdlib-jdk8:1.6.10")
implementation("com.fasterxml.jackson.dataformat:jackson-dataformat-xml:2.13.2")
implementation("com.fasterxml.jackson.core:jackson-annotations:2.13.2")
implementation("com.fasterxml.jackson.core:jackson-core:2.13.2")
implementation("com.fasterxml.jackson.core:jackson-databind:2.13.2.2")
implementation("com.fasterxml.jackson.module:jackson-module-kotlin:2.13.2")
implementation("com.fasterxml.jackson.datatype:jackson-datatype-jsr310:2.13.2")
implementation("org.hibernate:hibernate-core:5.6.7.Final")
implementation("javax.persistence:javax.persistence-api:2.2")
implementation( "commons-codec:commons-codec:1.15")
implementation("io.github.microutils:kotlin-logging-jvm:2.1.21")
implementation("ch.qos.logback:logback-classic:1.2.11")
implementation("com.google.cloud.sql:mysql-socket-factory-connector-j-8:1.4.4")
runtimeOnly("com.h2database:h2:2.1.210")
runtimeOnly("org.springframework.boot:spring-boot-devtools:2.6.5")
testImplementation("org.springframework.boot:spring-boot-starter-test:2.6.5")
}
tasks.withType<KotlinCompile> {
kotlinOptions {
freeCompilerArgs = listOf("-Xjsr305=strict")
jvmTarget = "17"
}
}
tasks.withType<Test> {
useJUnitPlatform()
}
Dockerfile
FROM openjdk:17-alpine
ENV USER=appuser
# <placeholder> Replace context path for your own application
ENV JAVA_HOME=/opt/openjdk-17 \
HOME=/home/$USER \
CONTEXT_PATH=/aws-service-baseline
RUN adduser -S $USER
# <placeholder> Add additional packages for the docker container here
RUN apk add --no-cache su-exec
# <placeholder> Replace baseline.jar with your applications JAR file (defined in build.gradle.kts)
COPY Docker/runapp.sh build/libs/<application-name>-0.0.1-SNAPSHOT.jar $HOME/
RUN chmod 755 $HOME/*.sh && \
chown -R $USER $HOME
WORKDIR /home/$USER
CMD [ "./runapp.sh"]
Docker/runapp.sh
#!/bin/sh
set -e
# The module to start.
# <placeholder> Replace this with your own modulename (from module-info)
APP_JAR="<application-name>-0.0.1-SNAPSHOT.jar"
JAVA_PARAMS="-XshowSettings:vm"
echo " --- RUNNING $(basename "$0") $(date -u "+%Y-%m-%d %H:%M:%S Z") --- "
set -x
/sbin/su-exec "$USER:1000" "$JAVA_HOME/bin/java" "$JAVA_PARAMS $JAVA_PARAMS_OVERRIDE" -jar -Dserver.port=$PORT "$APP_JAR"
GCP details
I have made sure the SQL instances connection is added to the Cloud Run Revisions. The IAM roles for the compute service account also seem to be right. See images
IAM: https://i.stack.imgur.com/yYaC5.png
Database: https://i.stack.imgur.com/NErad.png
Cloud Run connection https://i.stack.imgur.com/fKTSZ.png
Additional details
When running ./gradlew bootRun on my local machine (with GCP credentials present), the App works properly with an SQL connection. It also works after running ./gradle bootRun to build the JAR file and run the JAR directly. It does not work out of the box when running in Docker, but if I add the GCP credentials to the Docker container locally, it connects to the Database.
Does anyone have any suggestions on what might be wrong? Any help much appreciated!
I have tried connecting locally and locally in a Docker container.

Figured it out! Human error of course. The Cloud Run Service was initially configured with another Services Account, and not the default Compute Engine Service account.

Related

How can I control services start sequence of asp.net web api and mysql db service?

I have a problem now. I have tried a few methods from Google to control services start sequence.for example, use wait-for-it.sh、 write shell script ,but all doesn't work for me.
I'm using macOS and my project is asp.net core web api in .net 6 core and using entity framework to connect and migrate to mysql database.
dockerfile
`# Get base sdk Image from Microsoft
FROM mcr.microsoft.com/dotnet/sdk:6.0 AS build-env
WORKDIR /app
#copy the csprog file and restore any dependecies
COPY *.csproj ./
RUN dotnet restore
#copy the project files and build our release
COPY . ./
RUN dotnet publish -c Release -o out
#Generate runtime image
FROM mcr.microsoft.com/dotnet/aspnet:6.0
WORKDIR /app
EXPOSE 80
COPY --from=build-env /app/out .
ENTRYPOINT ["dotnet", "SuperHeroAPI.dll"]`
docker-compose.yml
`version: '3.4'
services:
api:
build:
context: .
dockerfile: Dockerfile
ports:
- "8080:80"
container_name: super
depends_on:
- db
restart: always
environment:
- DBHOST=db
- ASPNETCORE_ENVIRONMENT=Development
db:
image: mysql:8.0.29
container_name: db
environment:
MYSQL_ROOT_PASSWORD: 123456
MYSQL_DATABASE: apidb
ports:
- "3306:3306"
`
program.cs
`
var builder = WebApplication.CreateBuilder(args);
// Add services to the container.
builder.Services.AddControllers();
builder.Services.AddCors();
var host = builder.Configuration["DBHOST"] ?? "db";
var port = builder.Configuration["DBPORT"] ?? "3306";
var password = builder.Configuration["DBPASSWORD"] ?? "123456";
builder.Services.AddDbContext<DataContext>(options =>
{
options.UseMySql($"server={host};userid=root;pwd={password};port={port};database=apidb", new MySqlServerVersion("8.0.29"));
});
var dataContext = builder.Services.BuildServiceProvider().GetService<DataContext>();
dataContext.Database.Migrate();
// Learn more about configuring Swagger/OpenAPI at https://aka.ms/aspnetcore/swashbuckle
builder.Services.AddEndpointsApiExplorer();
builder.Services.AddSwaggerGen();
builder.Services.AddScoped<ISuperHeroDao, SuperHeroImpl>();
builder.Services.AddScoped<SuperHeroService>();
var app = builder.Build();
// Configure the HTTP request pipeline.
if (app.Environment.IsDevelopment())
{
app.UseSwagger();
app.UseSwaggerUI();
}
app.UseHttpsRedirection();
app.UseAuthorization();
app.MapControllers();
app.Run();
Datacontext.cs
`using Microsoft.EntityFrameworkCore;
namespace SuperHeroAPI.Data
{
public class DataContext : DbContext
{
public DataContext(DbContextOptions<DataContext> options) : base(options) { }
public DbSet<SuperHero> SuperHeroes { get; set; }
}
}`
based on above all code , I can start my api service finally! but api would restart repeatedly until db service start successfully.I don't want that . I wanna start db service until succeed and then start api service so it won't restart repeatedly.
I'm sorry I'm not a native english speaker ,maybe I can't express my idea clearly , I would appreciate you if you could provide useful solutions and ideas.
expect to start db service first until succeed and then start api service. I've tried wait-for-it.sh. shell scripts .doesn't work.

Getting error while provide mySql Connection in spring boot application

I use sql work bench as sql editor . when I provide database
connection in my application I am get error . I get error when define
driver class name in yml file . I already added required dependency
in my gradlle.kt file. But I do not know why I am get this issue. here
is my db connection details
application.yml
spring:
datasource:
url: jdbc:mysql://localhost:3306/hospital-management
driverClassName: com.mysql.cj.jdbc.Driver
username: root
password: 12345678
jpa:
database-platform: org.hibernate.dialect.MySQL5Dialect
show-sql: true
hibernate:
ddl-auto: update
banner:
location: classpath:banner.txt
app:
jwt:
secret: ArbitrarySecretKey
expiration-in-ms: 864000000
token:
prefix: Bearer
header:
string: Authorization
gradle.kt
import org.jetbrains.kotlin.gradle.tasks.KotlinCompile
plugins {
id("org.springframework.boot") version "2.5.5"
id("io.spring.dependency-management") version "1.0.11.RELEASE"
kotlin("jvm") version "1.5.31"
kotlin("plugin.spring") version "1.5.31"
kotlin("plugin.jpa") version "1.5.31"
}
group = "com.nillmani"
version = "0.0.1-SNAPSHOT"
java.sourceCompatibility = JavaVersion.VERSION_11
repositories {
mavenCentral()
}
dependencies {
implementation("org.springframework.boot:spring-boot-starter-data-jpa")
implementation("org.springframework.boot:spring-boot-starter-web")
// https://mvnrepository.com/artifact/org.modelmapper/modelmapper
implementation("org.modelmapper:modelmapper:2.3.6")
implementation("org.springframework.boot:spring-boot-starter-security")
// https://mvnrepository.com/artifact/io.jsonwebtoken/jjwt
implementation("io.jsonwebtoken:jjwt:0.9.1")
implementation("com.fasterxml.jackson.module:jackson-module-kotlin")
implementation("org.jetbrains.kotlin:kotlin-reflect")
implementation("org.jetbrains.kotlin:kotlin-stdlib-jdk8")
runtimeOnly("mysql:mysql-connector-java")
testImplementation("org.springframework.boot:spring-boot-starter-test")
}
tasks.withType<KotlinCompile> {
kotlinOptions {
freeCompilerArgs = listOf("-Xjsr305=strict")
jvmTarget = "11"
}
}
tasks.withType<Test> {
useJUnitPlatform()
}
I get error exact at this point
driverClassName: com.mysql.cj.jdbc.Driver
I get error for the name defination(com.mysql.cj.jdbc.Driver)
It works fine for me While I run this application in windows system
using sql yog. But getting issue in Mac os , in mac os I use sql work
bench as editor. what is the reason for this issue
This type of error raises for versioning issue . Here I use MYSQL5 , I
upgrade MYSQL5 to 8 . All the errors in driver class name vanished.
This trick works for me .Here No issue in my db connection
Before my depedency
runtimeOnly("mysql:mysql-connector-java")
after upgrade to
runtimeOnly("mysql:mysql-connector-java:8.0.25")

Running database migrations during Google Cloud Build fails with ENOTFOUND error

I am trying to run migrations through Sequelize in Node JS on Google Cloud Run connecting to a MySQL Google Cloud SQL database. I followed
https://stackoverflow.com/a/58441728/4487248 to get the Google Cloud proxy setup. Given this log setting up the proxy connection to the database seems to have worked:
Step #2 - "migrate": Already have image (with digest): gcr.io/cloud-builders/yarn
Step #2 - "migrate": 2021/10/02 14:19:58 current FDs rlimit set to 1048576, wanted limit is 8500. Nothing to do here.
Step #2 - "migrate": 2021/10/02 14:19:58 Listening on /workspace/<MY-INSTANCE-NAME> for <MY-INSTANCE-NAME>
Step #2 - "migrate": 2021/10/02 14:19:58 Ready for new connections
Step #2 - "migrate": 2021/10/02 14:19:58 Generated RSA key in 74.706896ms
However, when I try to run migrations with yarn knex migrate:latest or ./node_modules/.bin/sequelize db:migrate I run into:
getaddrinfo ENOTFOUND /workspace/<MY-INSTANCE-NAME>
This seems to imply that the host could not be found.
Output / Logs
My cloudbuild.yaml (composed of https://stackoverflow.com/a/52366671/4487248 & https://stackoverflow.com/a/58441728/4487248):
steps:
# Install Node.js dependencies
- id: yarn-install
name: gcr.io/cloud-builders/yarn
waitFor: ["-"]
# Install Cloud SQL proxy
- id: proxy-install
name: gcr.io/cloud-builders/yarn
entrypoint: sh
args:
- "-c"
- "wget https://storage.googleapis.com/cloudsql-proxy/v1.25.0/cloud_sql_proxy.linux.amd64 -O /workspace/cloud_sql_proxy && chmod +x /workspace/cloud_sql_proxy"
waitFor: ["-"]
- id: migrate
name: gcr.io/cloud-builders/yarn
entrypoint: sh
args:
- "-c"
- "(/workspace/cloud_sql_proxy -dir=/workspace -instances=<MY-INSTANCE-NAME> & sleep 2) && ./node_modules/.bin/sequelize db:migrate"
timeout: "1200s"
waitFor: ["yarn-install", "proxy-install"]
timeout: "1200s"
My .sequelizerc (Documentation here):
const path = require('path');
module.exports = {
'config': path.resolve('config', 'config.js')
}
My config/config.js:
module.exports = {
production: {
username: process.env.PROD_DB_USERNAME,
password: process.env.PROD_DB_PASSWORD,
database: process.env.PROD_DB_NAME,
host: `/workspace/${process.env.INSTANCE_CONNECTION_NAME}`, // Replacing this line with `/workspace/cloudsql/${..}` or `/cloudsql/${..}` leads to the same error
dialect: 'mysql',
}
}
I did enable Public IP on the MySQL instance:
Setting the host to localhost and adding the instance path in socketPath in config.js fixed the issue:
module.exports = {
production: {
username: process.env.PROD_DB_USERNAME,
password: process.env.PROD_DB_PASSWORD,
database: process.env.PROD_DB_NAME,
host: localhost,
dialect: 'mysql',
dialectOptions: {
socketPath: `/workspace/${process.env.INSTANCE_CONNECTION_NAME}`,
},
}
}

How to access external database from Node container?

I have a nextjs app which is supposed to connect to an external MySQL database (not one from the same docker network). When running the app locally, it works correctly when connecting to DB, but when running it in a Docker container it keeps on trying to connect to 127.0.0.1, even though the environment variables are configured correctly in the container
Error: Error: connect ECONNREFUSED 127.0.0.1:3306
nextjs_1 | at connect (/opt/app/node_modules/serverless-mysql/index.js:80:15)
Dockerfile config:
FROM node:alpine
RUN mkdir -p /opt/app
RUN apk add --no-cache libc6-compat
ENV NODE_ENV production
ENV PORT 3000
EXPOSE 3000
WORKDIR /opt/app
COPY package.json /opt/app
COPY package-lock.json /opt/app
RUN npm install --no-optional
COPY . /opt/app
RUN npm run build
RUN npx next telemetry disable
RUN addgroup -g 1001 -S nodejs
RUN adduser -S nextjs -u 1001
USER nextjs
CMD [ "npm", "start" ]
Connection code:
const mysql = require('serverless-mysql')
const db = mysql({
config: {
host: process.env.MYSQL_HOST,
database: process.env.MYSQL_DATABASE,
user: process.env.MYSQL_USER,
password: process.env.MYSQL_PASSWORD,
},
})
exports.query = async (query) => {
try {
const results = await db.query(query)
await db.end()
return results
} catch (error) {
return { error }
}
}
Any ideas?

MySql Setup in Linux Docker Container Via Terraform

Requirement: Need to automate MySQL installation & Database creation on Linux(Ubuntu)Docker Container via Terra form.
I am doing all this stuff on my local machine & below is the Terra form configuration.
Terra form file:
resource "docker_container" "db-server1" {
name = "db-server"
image = docker_image.ubuntu.latest
ports {
internal = 80
external = 9093
}
provisioner "local-exec" {
command = "docker container start dbs-my"
}
provisioner "local-exec" {
command = "docker exec dbs-my apt-get update"
}
provisioner "local-exec" {
command = "docker exec dbs-my apt-get -y install mysql-server"
}
}
But in container there is no mysql service present, when i am trying to launch mysql command, i am getting below error:
ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (2)
Using Terraform for this at all is a little unusual; you might look at more Docker-native tools like Docker Compose to set this up. There are also several anti-patterns in this example: you should generally avoid installing software in running containers, and avoid running long sequences of imperative commands via Terraform, and it's usually not useful to run the bare ubuntu Docker image as-is.
You can run the Docker Hub mysql image instead:
resource "docker_image" "mysql" {
name = "mysql:8"
}
resource "random_password" "mysql_root_password" {
length = 16
}
resource "docker_container" "mysql" {
name = "mysql"
image = "${docker_image.mysql.latest}"
env {
MYSQL_ROOT_PASSWORD = "${random_password.mysql_root_password.result}"
}
mounts {
source = "/some/host/mysql/data/path"
target = "/var/lib/mysql/data"
type = "bind"
}
ports {
internal = 3306
external = 3306
}
}
If you wanted to do further setup on the created database, you could use the MySQL provider
provider "mysql" {
endpoint = "127.0.0.1:3306" # the "external" port
username = "root"
password = "${random_password.mysql_root_password.result}"
}
resource "mysql_database" "db" {
name = "db"
}