I am running Scorpio using the docker-compose file: docker-compose-aaio.yml but I want to use an RDS postgres instance rather than a container instance. I have updated the docker-compose-aaio.yml file as follows:
version: '3'
services:
zookeeper:
image: zookeeper
ports:
- "2181"
kafka:
image: wurstmeister/kafka
hostname: kafka
ports:
- "9092"
environment:
KAFKA_ADVERTISED_HOST_NAME: kafka
KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
KAFKA_ADVERTISED_PORT: 9092
volumes:
- /var/run/docker.sock:/var/run/docker.sock
depends_on:
- zookeeper
logging:
driver: none
scorpio:
image: scorpiobroker/scorpio:scorpio-aaio_1.0.2
ports:
- "9090:9090"
depends_on:
- kafka
environment:
spring_args: --maxLimit=1000 --reader.datasource.hikari.url=jdbc:postgresql//myrdshost.eu-west-2.rds.amazonaws.com:5432/ngb?ApplicationName=ngb_storagemanager_reader --writer.datasource.hikari.url=jdbc:postgresql//myrdshost.eu-west-2.rds.amazonaws.com:5432/ngb?ApplicationName=ngb_storagemanager_writer
However when I run this with: docker-compose -f docker-compose-aaio.yml up I get this error:
java.net.UnknownHostException: postgres
as though the Scorpio broker is still trying to use the default postgres database urls (i.e trying to use a containerised postgres instance). It's seems like the environment variables I set under spring_args are not getting applied.
I have followed the documentation in Chapters 4 and 5 here: https://scorpio.readthedocs.io/_/downloads/en/latest/pdf/.
Can you see anything I am doing wrong?
Thanks!
there is another config parameter in spring for postgres.
spring.datasource.hikari.url
This is missing in documentation. I'll update it for the next release.
For the full set of changeable options you can have a look in the application.yml
https://github.com/ScorpioBroker/ScorpioBroker/blob/development/AllInOneRunner/src/main/resources/application.yml
Good Luck
Benjamin
Regarding scorpio version scorpio-aaio-no-eureka_2.1.20 you just have to set the DBHOST and DBPORT environment variable.
scorpio:
image: scorpiobroker/scorpio-aaio-no-eureka_2.1.20
environment:
- DBHOST: "example.com"
- DBPORT: "5432"
The names of the environment variables follow the options from application.yml using underscores instead of period-separation.
Related
I have a Java Spring Boot app which works with a Postgres database. I want to use Docker for both of them. I initially put just the Postgres in Docker, and I had a docker-compose.yml file defined like this:
version: '2'
services:
db:
container_name: sample_db
image: postgres:9.5
volumes:
- sample_db:/var/lib/postgresql/data
environment:
- POSTGRES_PASSWORD=sample
- POSTGRES_USER=sample
- POSTGRES_DB=sample
- PGDATA=/var/lib/postgresql/data/pgdata
ports:
- 5432:5432
volumes:
sample_db: {}
Then, when I issued the commands sudo dockerd and sudo docker-compose -f docker-compose.yml up, it was starting the database. I could connect using pgAdmin for example, by using localhost as server and port 5432. Then, in my Spring Boot app, inside the application.properties file I defined the following properties.
spring.datasource.url=jdbc:postgresql://localhost:5432/sample
spring.datasource.username=sample
spring.datasource.password=sample
spring.jpa.generate-ddl=true
At this point I could run my Spring Boot app locally through Spring Suite, and it all was working fine. Then, I wanted to also add my Spring Boot app as Docker image. I first of all created a Dockerfile in my project directory, which looks like this:
FROM java:8
EXPOSE 8080
ADD /target/manager.jar manager.jar
ENTRYPOINT ["java","-jar","manager.jar"]
Then, I entered to the directory of the project issued mvn clean followed by mvn install. Next, issued docker build -f Dockerfile -t manager . followed by docker tag 9c6b1e3f1d5e myuser/manager:latest (the id is correct). Finally, I edited my existing docker-compose.yml file to look like this:
version: '2'
services:
web:
image: myuser/manager:latest
ports:
- 8080:8080
depends_on:
- db
db:
container_name: sample_db
image: postgres:9.5
volumes:
- sample_db:/var/lib/postgresql/data
environment:
- POSTGRES_PASSWORD=sample
- POSTGRES_USER=sample
- POSTGRES_DB=sample
- PGDATA=/var/lib/postgresql/data/pgdata
ports:
- 5432:5432
volumes:
sample_db: {}
But, now if I issue sudo docker-compose -f docker-compose.yml up command, the database again starts correctly, but I get errors and exit code 1 for the web app part. The problem is the connection string. I believe I have to change it to something else, but I don't know what it should be. I get the following error messages:
web_1 | 2017-06-27 22:11:54.418 ERROR 1 --- [ main] o.a.tomcat.jdbc.pool.ConnectionPool : Unable to create initial connections of pool.
web_1 |
web_1 | org.postgresql.util.PSQLException: Connection to localhost:5432 refused. Check that the hostname and port are correct and that the postmaster is accepting TCP/IP connections
Any ideas?
Each container has its own network interface with its own localhost. So change how Java points to Postgres:
spring.datasource.url=jdbc:postgresql://localhost:5432/sample
To:
spring.datasource.url=jdbc:postgresql://db:5432/sample
db will resolve to the proper Postgres IP.
Bonus. With docker-compose you don't need to build your image by hand. So change:
web:
image: myuser/manager:latest
To:
web:
build: .
I had the same problem and I lost some time to understand and solve this problem:
org.postgresql.util.PSQLException: Connection to localhost:5432 refused. Check that the hostname and port are correct and that the postmaster is accepting TCP/IP connections.
I show all the properties so that everyone understands.
application.properties:
spring.datasource.url=jdbc:postgresql://localhost:5432/testdb
spring.datasource.driver-class-name=org.postgresql.Driver
spring.datasource.username=postgres
spring.datasource.password=postgres
spring.jpa.properties.hibernate.dialect=org.hibernate.dialect.PostgreSQL82Dialect
spring.jpa.hibernate.ddl-auto=update
docker-compose.yml:
version: "3"
services:
springapp:
build: .
container_name: springapp
environment:
SPRING_DATASOURCE_URL: jdbc:postgresql://db:5432/testdb
ports:
- 8000:8080
restart: always
depends_on:
- db
db:
image: postgres
container_name: db
environment:
- POSTGRES_USER=postgres
- POSTGRES_PASSWORD=postgres
- POSTGRES_DB=testdb
- PGDATA=/var/lib/postgresql/data/pgdata
ports:
- 5000:5432
volumes:
- pgdata:/var/lib/postgresql/data
restart: always
volumes:
pgdata:
For start spring application with local database we use url localhost.
For connect to container with database we need change 'localhost' on your database service, in my case 'localhost' to 'db'.
Solution: add SPRING_DATASOURCE_URL environment in docker-compose.yml wich rewrite spring.datasource.url value for connect:
environment:
SPRING_DATASOURCE_URL: jdbc:postgresql://db:5432/testdb
I hope this helps someone save his time.
You can use this.
version: "2"
services:
sample_db-postgresql:
image: postgres:9.5
ports:
- 5432:5432
environment:
- POSTGRES_PASSWORD=sample
- POSTGRES_USER=sample
- POSTGRES_DB=sample
volumes:
- sample_db:/var/lib/postgresql/data
volumes:
sample_db:
You can use ENV variable to change the db address in your docker-compose.
Dockerfile:
FROM java:8
EXPOSE 8080
ENV POSTGRES localhost
ADD /target/manager.jar manager.jar
ENTRYPOINT exec java $JAVA_OPTS -jar manager.jar --spring.datasource.url=jdbc:postgresql://$POSTGRES:5432/sample
docker-compose:
`
container_name: springapp
environment:
- POSTGRES=db`
this is my docker-compose-yml file. first i tried to run the db service it work fined and docker running successfully. but when I run the app service, in the terminal it says connected to database but I get an error as (2003, "Can't connect to MySQL server on '127.0.0.1')
version: '4'
services:
app:
build: .
ports:
- "8000:8000"
env_file:
- .env
depends_on:
- db
links:
- db
db:
environment:
- MYSQL_ROOT_PASSWORD=root
-
image: mysql
ports:
- "3307:3307"
env_file:
- .env
enter image description here
What environment variable is the FastAPI app using to connect to the MySQL host?
If you're using docker-compose networking, services need to use the container name in order to reach each other (i.e. mysql://db:3307/db_name_here) not localhost.
By default Compose sets up a single network for your app. Each container for a service joins the default network and is both reachable by other containers on that network, and discoverable by them at a hostname identical to the container name.
More information on networking can be found in their docs here.
Also worth noting, since you're using links you can set aliases as well like so:
version: "3.9"
services:
web:
build: .
links:
- "db:database"
db:
image: postgres
Links Docs Source
I'm trying to connect a phpmyadmin-container to a mysql-container to view the databases. phpmyadmin web interface return error Cannot log in to the MySQL server with mysqli_real_connect(): (HY000/2002): No route to host.
What am I doing wrong? Thanks.
I use this operations:
docker-compose down && docker-compose up -d
Type to web browser http://localhost:8080
I use login as user root, password test
Get error described above
My docker-compose.yml
version: '2'
services:
db:
image: mysql
environment:
MYSQL_ROOT_PASSWORD: test
ports:
- "3306:3306"
phpmyadmin:
image: phpmyadmin/phpmyadmin
links:
- db
ports:
- "8080:80"
See official guide:
You need to specify some environment in order to link to an external mysql container:
Environment variables summary
PMA_ARBITRARY - when set to 1 connection to the arbitrary server will be allowed
PMA_HOST - define address/host name of the MySQL server
PMA_VERBOSE - define verbose name of the MySQL server
PMA_PORT - define port of the MySQL server
PMA_HOSTS - define comma separated list of address/host names of the MySQL servers
PMA_VERBOSES - define comma separated list of verbose names of the MySQL servers
PMA_PORTS - define comma separated list of ports of the MySQL servers
PMA_USER and PMA_PASSWORD - define username to use for config authentication method
PMA_ABSOLUTE_URI - define user-facing URI
For you, the mysql name is db, so you need to do next:
version: '2'
services:
db:
image: mysql
environment:
MYSQL_ROOT_PASSWORD: test
ports:
- "3306:3306"
command: --default-authentication-plugin=mysql_native_password
phpmyadmin:
image: phpmyadmin/phpmyadmin
ports:
- "8080:80"
environment:
PMA_HOSTS: db
Also, --link depracated, just delete it, docker-compose will set network for you automatically to let the containers see each other with dns help.
I wish to set up 2 docker containers: grafana and mysql and to allow queries from the grafana to the mysql db.
I have an AWS machine, in which I built the following folders structure:
- docker
- docker-compose.yml
- grafana:
- config.ini
- dashboards:
- grafana_dashboard.json
- provisioning:
- dashboards:
- all.yml
- datasources:
- all.yml
- Dockerfile
- mysql:
- dbcreation.sql
- Dockerfile
- dashboards:
- import.sh
the content of the docker-compose.yml is:
version: '2'
services:
db-service:
build: './mysql'
container_name: mysql
restart: always
ports:
- "3306:3306"
networks:
net:
ipv4_address: 172.16.1.3
grafana-service:
build: './grafana'
container_name: grafana
restart: always
ports:
- "3000:3000"
environment:
GF_SECURITY_ADMIN_PASSWORD: "XXX1"
GF_AUTH_PROXY_ENABLED: "true"
GF_SECURITY_DATA_SOURCE_PROXY_WHITELIST: 172.16.1.3:3306
GF_AUTH_ANONYMOUS_ENABLED: "true"
GF_LOG_LEVEL: "debug"
depends_on:
- db-service
networks:
net:
ipv4_address: 172.16.1.4
networks:
net:
external: true
volumes:
grafanadata:
driver: local
mysqldata:
The dockerfile for the grafana:
FROM grafana/grafana:5.2.2
ADD ./provisioning /etc/grafana/provisioning
ADD ./dashboards /var/lib/grafana/dashboards
ENV DS_DB "grafana"
The content of the mysql/Dockerfile is:
FROM mysql:8.0.12
ENV MYSQL_ROOT_PASSWORD="XXX2"
ENV MYSQL_DATABASE="grafana"
ADD ./dbcreation.sql /docker-entrypoint-initdb.d/dbcreation.sql
EXPOSE 3306
The grafana_dashboard.json file has the exported json from the Grafana I had set up locally on my own computer.
The dbcreation.sql file has the exported data from the local DB I had set up locally on my own computer.
I'm running the following commands:
docker network create --gateway 172.16.1.1 --subnet 172.16.1.0/24 net
docker-compose up --build
I'm getting an error: "The authentication plugin is not supported"
when turning the log level of Grafana to debug I'm seeing this:
t=2018-08-19T10:55:20+0000 lvl=dbug msg=getEngine logger=tsdb.mysql connection="root:XXX2#tcp(172.16.1.3:3306)/grafana?collation=utf8mb4_unicode_ci&parseTime=true&loc=UTC&allowNativePasswords=true"
t=2018-08-19T10:55:47+0000 lvl=eror msg="Request Completed" logger=context userId=1 orgId=1 uname=admin method=POST path=/api/tsdb/query status=500 remote_addr=XX.XXX.XXX.XXX time_ms=2 size=195 referer=http://XX.XXX.XXX.XXX:3000/datasources/edit/1
I have used the following sources already to set this up:
https://ops.tips/blog/initialize-grafana-with-preconfigured-dashboards/
https://storage.pardot.com/138181/61672/mysql_on_docker_how_to_containerize_your_database.pdf
Any help would be appreciated!
Thanks
As stated here:
go-mysql: authentication plugin not supported while connecting from go app container to mysql container
The problem was compatibility between the Grafana versions and the MySQL version.
Once moving to docker image mysql:5.7 (had to migrate the data too) - the issue was resolved (need to also change the COLLATION and CHAR_SET in the DB to downgrade from version 8.0.12 to 5.7).
I have a Docker web application with its database which I have set up:
-v /home/stephane/dev/php/learnintouch/docker/mysql/data:/usr/bin/mysql/install/data
It works fine but I wonder if that is the recommended way to go.
For I see we can also create a named volume by giving a name instead of an absolute path on the host:
-v learnintouch-data:/usr/bin/mysql/install/data
But then, how can I associate the volume name learnintouch-data with the host location at /home/stephane/dev/php/learnintouch/docker/mysql/data ?
Here is my current docker-compose.yml file:
learnintouch.com-startup:
image: stephaneeybert/learnintouch.com-startup
container_name: learnintouch.com-startup
ports:
- "80:80"
links:
- mysql
- redis
- nodejs-learnintouch
nodejs-learnintouch:
image: stephaneeybert/nodejs-learnintouch
container_name: nodejs-learnintouch
ports:
- "9001:9001"
links:
- redis
mysql:
image: stephaneeybert/mysql:5.6.30
container_name: mysql
ports:
- "3306:3306"
environment:
- MYSQL_ROOT_PASSWORD=root
volumes:
- "/home/stephane/dev/php/learnintouch/docker/mysql/data:/usr/bin/mysql/install/data"
redis:
image: stephaneeybert/redis:3.0.7
container_name: redis
ports:
- "6379:6379"
Using a named volume (or more specifically a volume created using the Docker Engine volume API) with a defined host path doesn't have much of an advantage over the method you've used. Technically, it is "easier" to create a new container, but only because you no longer have to remember the path. You can also use the volume API to "manage" the volume independently from the application container, but this is equally easy using the docker container API.
If you insist, to create a named volume with an absolute host path, you need to use a volume-driver. I would suggest local-persist. It is quite simple to install and works well.
https://github.com/CWSpear/local-persist