spring boot usage of placeholder in application properties - mysql

I have written a simple spring boot application(version springboot 2.0) which uses mysql(version 5.7).
application.properties snippet
spring.datasource.url = jdbc:mysql://localhost:3306/test?useSSL=false
spring.datasource.username = testuser
spring.datasource.password = testpassword
spring.datasource.driver-class-name=com.mysql.jdbc.Driver
When I run it locally, it works fine.
If I want to run this spring boot application in docker then I can change
spring.datasource.url = jdbc:mysql://mysql-container:3306/test?useSSL=false
mysql-container is run using mysql:5.7 image from dockerhub.
However I want to change value of host from some placeholder properties file. so that this looks something like:
spring.datasource.url = jdbc:mysql://${MYSQL_HOST}:3306/test?useSSL=false
note: I am not sure about placeholder format. Is it ${MYSQL_HOST} or #MYSQL_HOST# ?
you can name this placeholder file as placeholder.properties or placeholder.conf or .env or anything. The content of that file should be something like:
MYSQL_HOST=localhost
or
MYSQL_HOST=some ip address
I can create .env or .env.test or .env.prod and I can refer that env file based on where I want to run application.
UPDATE -
I have two questions:
Where should I keep placeholder.properties? Is it under /config/ or under some specific directory?
how to invoke placeholder inside application.properties ?
can someone suggest?

SUGGESTION: If you have a relatively small #/properties, why not just have a different application.properties file for each different environment?
You'd specify the environment at runtime with -Dspring.profiles.active=myenv.
Look here and here.
PS:
To answer your specific question: the syntax is ${MYSQL_HOST}

Thanks to answers by #Raheela Aslam and #paulsm4 and some more research found the issue.
What I was trying to achieve:
Deploy springboot application in docker and then to kubernetes.
I was using minikube for local testing and wanted to pass minikube ip to datasource url.
How I fixed it:
I created configmap for mysql_user, mysql_password, mysql_host with respective values.
kubectl create configmap mysql-config \
--from-literal=mysql_user=testuser \
--from-literal=mysql_password=testuserpass \
--from-literal=mysql_user=$(minikube ip)
and used these inside application.properties something like below
spring.datasource.url = jdbc:mysql://${MYSQL_HOST}:3306/test?useSSL=false
spring.datasource.username = ${MYSQL_USER}
spring.datasource.password = ${MYSQL_PASSWORD}
spring.datasource.driver-class-name=com.mysql.jdbc.Driver
and then used configmap values in deployment.yaml for kubernetes.
Then did start service for deployment.

If you project is Maven you can use maven filter:
<build>
<filters>
<filter>src/main/filters/myfilter.properties</filter>
</filters>
</build>
This generates the /target/classes/application.properties it has been filtered to contain the property values (with replaced placeholders)
http://www.avajava.com/tutorials/lessons/how-do-i-filter-resources-based-on-values-from-a-properties-file.html?page=1

Related

Use kubernetes with helm and provide my predefined user and password with bitnami correctly

I am using kubernetes with helm 3.
I need to create a kubernetes pod with sql - creating:
database name: my_database
user: root
password:12345
port: 3306
The steps:
creating chart by:
helm create test
after the chart is created, change the Chart.yaml file in test folder, by adding dependencies section.
apiVersion: v2
name: test3
description: A Helm chart for Kubernetes
version: 0.1.0
appVersion: "1.16.0"
dependencies:
name: mysql
version: 8.8.23 repository: "https://charts.bitnami.com/bitnami"
run:
helm dependencies build test
After that there is a compressed file tgz.
So I extracted it and there is tar file - I extracted it too, and leave only the final extracted folder.
I presume this isn't the best approach of changing parameter in yaml for bitnami,
and using also the security.yaml - I would like knowing that better approach too.
I need to change the user + password, and link to database,
so I changed the values.yaml directly (any better approach?), for values: auth:rootPassword and auth:my_database.
the another following steps:
helm build dependencies test
helm install test --namespace test --create-namespace
after that there are two pods created.
I could check it by:
kubectl get pods -n test
and I see two pods running (maybe replication).
one of the pod: test-mysql-0 (the other is with random parse).
run:
kubectl exec --stdin --tty test-mysql-0 --namespace test-mysql -- /bin/sh
did enter the pod.
run:
mysql -rroot -p12345;
and then:
show databases;
That did showing all the database, including seeing the created database: my_database, successfully.
When I tried openning the mysql database from 'mysql workbench', and test (same user: root, and password, and port: 3306, and localhost), I couldn't run test (test connection button in database properties returns: 'failed to connect to database').
Why cannot I run properly 'mysql workbench', while in the pad itself - without any particular problem?
Is there any better approach than extrating the tgz file as I described above, and can I pass in better way (some secured yaml) the user+password?
(Right now is only the root password)
Thanks.
It sounds like you're trying to set the parameters in the dependent chart (please correct me if I'm wrong)
If this is right, all you need to do is add another section in your chart's values.yaml
name-of-dependency:
user-name: ABC
password: abcdef
the "name-of-dependency" is specified in your Chart.yaml file when you declare your chart. For example, here's my redis dependency from one of my own charts
dependencies:
- name: redis
repository: https://charts.bitnami.com/bitnami/
version: x.x.x
Then when I install the chart, I can override the redis chart's settings by doing this in my own chart's values.yaml
redis:
architecture: standalone
auth:
password: "secret-password-here"

Flyway Migrations fails when passing Environment Variables to Docker

Performing database migrations using flyway. Using the dockerized version and using conf files for mirgations configurations.
Below is my config file
flyway.url = jdbc:mysql://${MYSQLHOST}:3306/myschema
flyway.user = myusername
flyway.password = mypassword
flyway.schemas = myschema
flyway.cleanDisabled = true
Am running the below command to perform migration
sudo docker run -e "MYSQLHOST=myhostip" --rm -it -v `pwd`/path/to/confi/:/flyway/conf/ -v `pwd`/path/to/migrations:/flyway/sql boxfuse/flyway:5.1.4 -configFiles=/flyway/conf/flyway.conf migrate
Am getting the below error
ERROR:
Unable to obtain connection from database (jdbc:mysql://${MYSQLHOST}:3306/myschema) for user 'myuser': Could not connect to address=(host=${MYSQLHOST})(port=3306)(type=master) : ${MYSQLHOST}
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
SQL State : 08
Error Code : -1
Message : Could not connect to address=(host=${MYSQLHOST})(port=3306)(type=master) : ${MYSQLHOST}
If I change the config file with my host ip details the migrations are successful without any errors. What am I doing wrongly?
Flyway allows you to pass in configuration parameters via a specific set of environment variables; it doesn't perform a complete variable substitution in the config file. See https://flywaydb.org/documentation/envvars.
What you could do is delete the first line of the config file, then
sudo docker run -e "FLYWAY_URL=jdbc:mysql://myhostip:3306/myschema" ...
I found it not very helpful to only see a page explaining that there are envVars -- but no mention of the exact env variable names to actually use.
Here is the definition of the env names / especially for the gradle plugin but I suspect that they are generally valid:
https://github.com/flyway/flyway/blob/master/flyway-core/src/main/java/org/flywaydb/core/internal/configuration/ConfigUtils.java#L139

Switch from docker mysql container to host's mysql without using network-mode="host"

My company has a docker-container based website (Ubuntu containers), deployed via GitHub => CircleCI => AWS. (This was set up by a consultant, who we are not currently working with. I'm trying to make sense out of everything on my own.)
I've copied the source files to my Windows 10 development PC.
Locally, the website is running successfully, until it tries to access MySQL data.
I've installed MySQL (8) locally, with default settings (port 3306) and location (C:\ProgramData\MySQL\MySQL Server 8.0\Data), and imported the database. I've created root user. Running test queries as root from MySQL Workbench works.
I've read Q&A's such as How to connect locally hosted MySQL database with the docker container.
I've successfully bridged to host's IP (on EthernetV; Docker running in Hyper-V) 172.26.92.81 for other purposes, specifically to have XDebug in my website container talk to phpstorm on my Windows 10 host. (I could use host.docker.internal for IP; but I'm making everything as "concrete" as possible, until everything works.)
I just don't understand what I have to change where, for redirecting MySQLi queries (in php as create web pages) to the (development pc) host. (Assume I'm clueless about both MySQL and Docker and Apache, until proven otherwise.)
Relevant parts of (original) docker-compose.yml:
mywebsite:
build:
context: ./mywebsite/
dockerfile: Dockerfile
...
depends_on:
- mysql
links:
- mysql
environment:
MYSQL_HOST: mysql
MYSQL_USER: root
MYSQL_PASS: xxx
MYSQL_SCHEMA: xxx
MYSQL_PORT: 3306
MYSQL_CHARSET: utf8
mysql:
image: mysql:5.7
ports:
- 3305:3306
command: mysqld --sql_mode=""
environment:
MYSQL_DATABASE: xxx
MYSQL_ROOT_PASSWORD: xxx
volumes:
- data:/var/lib/mysql
NOTE: Not going for "best practices" above; I need to see a simple approach working. Security comes later.
I'd prefer a solution that doesn't involve "network_mode: "host" -- solving it that way would avoid details that I need to understand. [The linked Q&A shows that host mode is the simplest solution, but that is "too simple" - obscures some important considerations.]
Host has MySQL on its port 3306. The mysql container shown above: is it using its own copy of MySQL? Or is that already attempting to connect to host's MySQL? And why does it have the port mapping 3305:3306? (I can't change that to 3306:3306; if do so, it is unable to assign the port number. I assume that is because host's MySQL already uses 3306, and that port line is exposing the mysql container's MySQL?)
The volume mapping data:... is where I should move the data to, if I want to use the mysql container as is? That's probably what I will do for now - which may make this SO question moot - but I'd still like to know how to do what I ask.
I'm assuming it is its own database, because it has its own version number (5.7).
From php inside mywebsite:
$connection = new \mysqli(
$_ENV["MYSQL_HOST"],
$_ENV["MYSQL_USER"],
$_ENV["MYSQL_PASS"],
'INFORMATION_SCHEMA',
$_ENV["MYSQL_PORT"]
);
$result = $connection->query("SELECT VERSION();");
# breakpoint: $result > one row > ['VERSION()'] = 5.7.25
which is the version number of that mysql image, not the host's mysql.
similarly, list of databases doesn't include some databases I see on the host mysql from Workbench.
What changes do I need to make in docker-compose.yml?
Do I also need to make changes in mywebsite's Dockerfile?
mywebsite is based on apache2 + php 7.3. Dockerfile:
FROM php:7.3.3-apache-stretch
...
RUN apt-get update && ... docker-php-ext-install ... mysqli \
&& docker-php-ext-enable mysqli ...
...
COPY /php.ini /usr/local/etc/php/
COPY /src/ /var/www/html/
First of all, the declaration:
links:
- mysql
in the mywebiste service is not needed and actually stays in the way of you solving your issue. You should remove it.
After doing that, you can try 2 things (both of them worked for me):
change MYSQL_HOST to point to your computer's IP (not 'localhost', not '127.0.0.1' but your local network IP address) and start only the mywebsite service.
in version 3.0 or higher of docker-compose file you can add extra hosts:
mywebsite:
...
extra_hosts:
- "mysql:<your ip>"
I hope this helps

Configuring Spring boot Docker and Mysql

I am new to Spring Boot and Docker.
I am trying to create a Spring Boot application connecting to mysql and using Docker to run both.
Steps I followed
Step1 - Created mysql image and started running it.
docker run --name=docker-mysql --env="MYSQL_ROOT_PASSWORD=root" --env="MYSQL_PASSWORD=root" --env="MYSQL_DATABASE=test" mysql
Step2 Created a SpringBoot application
docker build -f Dockerfile -t gradle-springboot-docker .
Step3 Ran the Spring Boot app and linked with Mysql
docker run -t --name gradle-springboot-docker --link docker-mysql:mysql -p 8080:8080 gradle-springboot-docker
It gives basic connection error to mysql. I have listed the below application.properties. Is my connection information correct since I am using Docker. What would be the host for mysql?
SSL properties
server.port=8080
#DataSource
datasource.driver = com.mysql.jdbc.Driver
datasource.url= jdbc:mysql://localhost:3306/test?
autoReconnect=true&useSSL=false
datasource.username=root
datasource.password=root
# Hibernate
hibernate.dialect = org.hibernate.dialect.MySQL5Dialect
hibernate.show_sql = true
hibernate.lazy = true
hibernate.max_fetch_depth = 3
hibernate.packagesToScan = com.springboot.poc
# Once DB is created change below property to 'update'
hibernate.hbm2ddl.auto = update
You should use the container name of mysql as the hostname, since these are linked the mysql container is discoverable by its name from spring. So you need to change datasource.url= jdbc:mysql://localhost:3306/test? to datasource.url= jdbc:mysql://docker-mysql:3306/test?
Or you can use the alias, as #g00glen00b suggested, like: datasource.url= jdbc:mysql://mysql:3306/test?

Access to mysql container from other container

I have setup docker container with mysql that expose 3306.
I've specified database user, database password and create a test db and give the privileges to new user.
In another container i want to accesso to this db.
So i set up new container with a simply php script that create new table in this db.
I know that mysql container's ip is 172.17.0.2 so :
$mysqli = new mysqli("172.17.0.2", "mattia", "prova", "prova");
Than using mysqli i create new table and all works fine.
But i think that connect to container using his ip address is not good.
Is there another way to specify db host? I tryed with the hostname of the mysql container but it doens't work.
The --link flag is considered a legacy feature, you should use user-defined networks.
You can run both containers on the same network:
docker run -d --name php_container --network my_network my_php_image
docker run -d --name mysql_container --network my_network my_mysql_image
Every container on that network will be able to communicate with each other using the container name as hostname.
You need to link your docker containers together with --link flag in docker run command or using link feature in docker-compose. For instance:
docker run -d -name app-container-name --link mysql-container-name app-image-name
In this way docker will add the IP address of the mysql container into /etc/hosts file of your application container.
For a complete document refer to:
MySQL Docker Containers: Understanding the basics
In your docker-compose.yml file add a link property to your webserver service:
https://docs.docker.com/compose/networking/#links
Then in your query string, the host parameter's value is your database service name:
$mysqli = new mysqli("database", "mattia", "prova", "prova");
If you are using docker-compose, than the database will be accessible under the service name.
version: "3.9"
services:
web:
build: .
ports:
- "8000:8000"
db:
image: postgres
ports:
- "8001:5432"
Then the database is accessible using: postgres://db:5432.
Here the service name is at the same time the hostname in the internal network.
Quote from docker docs:
When you run docker-compose up, the following happens:
A network called myapp_default is created.
A container is created using web’s configuration. It joins the network myapp_default under the name web.
A container is created using db’s configuration. It joins the network myapp_default under the name db.
Source:
https://docs.docker.com/compose/networking/