Connect SpringBoot to MySQL hosted in the cloud requires SSL - mysql

I am successful using MySQL Workbench to do full crud on a Bluemix hosted MySQL Compose service.
I then built a simple Microservice with SpringBoot on my local laptop with Apache Derby... successful.
My next step was to use the MySQL Compose hosted in Bluemix.
I edited application.properties and ran into this error
"PKIX path building failed: ...."
"SunCertPathBuilderException: unable to find valid certification path to request target"
application.properties file
spring.jpa.hibernate.ddl-auto=create
spring.jpa.database-platform=org.hibernate.dialect.MySQLDialect
spring.datasource.url=jdbc:mysql://somedomain:port/compose?useSSL=true?requireSSL=true
spring.datasource.username=myname
spring.datasource.password=mypassword
Bluemix provided me these credentials in json:
{
"db_type": "mysql",
"name": "bmix-dal-yp-xxxxxxx-",
"uri_cli": "mysql -u myname -p --host somedomain.com --port 5555 --ssl-mode=REQUIRED",
"ca_certificate_base64": "LS0tLS1CRUd......",
"deployment_id": "58fexxxxxxxxxxx",
"uri": "mysql://myname:mypassword#somedomain.com:55555/compose"
}
Am I supposed to use the ca certificate somewhere in my application.properties?
Do I need to enable ssl on my embedded tomcat server running by default with springBoot?
How can I configure my springBoot application to connect to my cloud providers MySQL instance with SSL with the json they provided?
Caused by: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target

Add the following to your pom.xml (or equivalent):
...
<repositories>
<repository>
<id>jcenter</id>
<url>http://jcenter.bintray.com </url>
<snapshots>
<enabled>true</enabled>
<updatePolicy>never</updatePolicy>
<checksumPolicy>warn</checksumPolicy>
</snapshots>
<releases>
<enabled>true</enabled>
<checksumPolicy>warn</checksumPolicy>
</releases>
</repository>
</repositories>
...
<dependency>
<groupId>com.orange.clara.cloud.boot.ssl-truststore-gen</groupId>
<artifactId>spring-boot-ssl-truststore-gen</artifactId>
<version>2.0.21</version>
</dependency>
...
Add the following to your manifest.yml
env:
# Add the certificate from VCAP_SERVICES ca_certificate_base64
# You need to base64 decode the certificate and add it below
# E.g. echo '<<ca_certificate_base64>>' | base64 -D
TRUSTED_CA_CERTIFICATE: |-
-----BEGIN CERTIFICATE-----
...
-----END CERTIFICATE-----
For more information, see https://github.com/orange-cloudfoundry/spring-boot-ssl-truststore-gen
Also see a minimal app here: https://github.com/snowch/hello-spring-cloud/tree/8b9728a826dcc1995a7ccb19a852ac8face21147
This is my first answer - this did not work. Ignore this section.
One option is:
Import the cert to Java truststore file, pack the file into Java
application and specify its path via JAVA_OPTS environment variable;
the truststore file can be placed under resource directory. This can
be used for single applications:
By using the 'cf set-env' command:
cf set-env <app> JAVA_OPTS '-Djavax.net.ssl.TrustStore=classpath:resources/config/truststore -Djavax.net.ssl.trustStorePassword=changeit'
or, by using manifest.yml
applications:
- name: java-app
...
env:
JAVA_OPTS: '-Djavax.net.ssl.TrustStore=classpath:resources/config/truststore -Djavax.net.ssl.trustStorePassword=changeit'
Note that the certificate in the field ca_certificate_base64 is base64 encoded so you will need to decode it before adding it to your truststore, e.g.
Decode the certificate:
echo '<<ca_certificate_base64>>' | base64 -D > ca_certificate.pem
Create a truststore:
keytool -import -trustcacerts -file ca_certificate.pem -alias compose_cert -keystore resources/config/truststore -storepass changeit -noprompt
Note that the keystore location (resources/config/truststore) and the storepass (changeit) are set in the JAVA_OPTS.
There are a few different options you can try. See this documentation for more information: https://discuss.pivotal.io/hc/en-us/articles/223454928-How-to-tell-application-containers-running-Java-apps-to-trust-self-signed-certs-or-a-private-or-internal-CA

Related

How do I get Keycloak to connect to MySQL DB?

I've been crawling a number of sites like this trying to get Keycloak working with a MySQL persistence layer. I am using docker, but I'm using my own images so it pulls passwords and other sensitive data from a secrets manager instead of environment variables or Docker secrets. The images are pretty close to stock besides that however.
Anyway, I have a MySQL 8 container up and running, and from within the Keycloak 12.0.3 container I can connect to the MySQL container fine:
# mysql -h mysql -u keycloak --password=somethingtochangelater -D keycloak -e "SHOW DATABASES;"
mysql: [Warning] Using a password on the command line interface can be insecure.
+--------------------+
| Database |
+--------------------+
| information_schema |
| keycloak |
+--------------------+
So there's no problems of connectivity between the instances, and that username/password has access to the keycloak database fine.
So then I ran several commands to configure the Keycloak instance (keycloak is installed at /opt/myco/bin/keycloak):
/opt/myco/bin/keycloak/bin/standalone.sh &
# Pausing for server startup
sleep 20
# Add mysql module - JDBC driver unpacked at /opt/myco/bin/keycloak-install/mysql-connector-java-8.0.23/mysql-connector-java-8.0.23.jar
/opt/myco/bin/keycloak/bin/jboss-cli.sh --connect --command="module add --name=com.mysql --dependencies=javax.api,javax.transaction.api --resources=/opt/myco/bin/keycloak-install/mysql-connector-java-8.0.23/mysql-connector-java-8.0.23.jar --module-root-dir=/opt/myco/bin/keycloak/modules/system/layers/keycloak/"
# Removing h2 datasource
/opt/myco/bin/keycloak/bin/jboss-cli.sh --connect --command="/subsystem=datasources/data-source=KeycloakDS:remove"
# Adding MySQL datasource
/opt/myco/bin/keycloak/bin/jboss-cli.sh --connect --command="/subsystem=datasources/jdbc-driver=mysql:add(driver-name=mysql,driver-module-name=com.mysql,driver-class-name=com.mysql.cj.jdbc.Driver)"
# TODO - add connection pooling options here...
# Configuring data source
/opt/myco/bin/keycloak/bin/jboss-cli.sh --connect --command="data-source add --name=KeycloakDS --jndi-name=java:jboss/datasources/KeycloakDS --enabled=true --password=somethingtochangelater --user-name=keycloak --driver-name=com.mysql --use-java-context=true --connection-url=jdbc:mysql://mysql:3306/keycloak?useSSL=false&characterEncoding=UTF-8"
# Testing connection
/opt/myco/bin/keycloak/bin/jboss-cli.sh --connect --command="/subsystem=datasources/data-source=KeycloakDS:test-connection-in-pool"
# Creating admin user
/opt/myco/bin/keycloak/bin/add-user-keycloak.sh -r master -u "admin" -p "somethingelse"
# Shutting down initial server
/opt/myco/bin/keycloak/bin/jboss-cli.sh --connect command=":shutdown"
This all appears to run fine. Note especially the test-connection-in-pool has no problems:
{
"outcome" => "success",
"result" => [true],
"response-headers" => {"process-state" => "reload-required"}
}
However, when I go to start the server back up again, it crashes with several exceptions, starting with:
22:31:52,484 FATAL [org.keycloak.services] (ServerService Thread Pool -- 56) Error during startup: java.lang.RuntimeException: Failed to connect to database
at org.keycloak.keycloak-model-jpa#12.0.3//org.keycloak.connections.jpa.DefaultJpaConnectionProviderFactory.getConnection(DefaultJpaConnectionProviderFactory.java:377)
at org.keycloak.keycloak-model-jpa#12.0.3//org.keycloak.connections.jpa.updater.liquibase.lock.LiquibaseDBLockProvider.lazyInit(LiquibaseDBLockProvider.java:65)
...
it keeps going, though I suspect that Exception ultimately to be fatal, and it eventually dies with:
22:31:53,114 ERROR [org.jboss.as.controller.management-operation] (ServerService Thread Pool -- 40) WFLYCTL0190: Step handler org.jboss.as.controller.AbstractAddStepHandler$1#33063168 for operation add at address [
("subsystem" => "jca"),
("workmanager" => "default"),
("short-running-threads" => "default")
] failed -- java.util.concurrent.RejectedExecutionException: java.util.concurrent.RejectedExecutionException
at org.jboss.threads#2.4.0.Final//org.jboss.threads.RejectingExecutor.execute(RejectingExecutor.java:37)
at org.jboss.threads#2.4.0.Final//org.jboss.threads.EnhancedQueueExecutor.rejectShutdown(EnhancedQueueExecutor.java:2029)
...
The module at /opt/myco/bin/keycloak/modules/system/layers/keycloak/com/mysql/main has the jar file and module.xml:
# ls
module.xml mysql-connector-java-8.0.23.jar
# cat module.xml
<?xml version='1.0' encoding='UTF-8'?>
<module xmlns="urn:jboss:module:1.1" name="com.mysql">
<resources>
<resource-root path="mysql-connector-java-8.0.23.jar"/>
</resources>
<dependencies>
<module name="javax.api"/>
<module name="javax.transaction.api"/>
</dependencies>
The standalone.xml file looks reasonable to me:
...
<subsystem xmlns="urn:jboss:domain:datasources:6.0">
<datasources>
...
<datasource jndi-name="java:jboss/datasources/KeycloakDS" pool-name="KeycloakDS" enabled="true" use-java-context="true">
<connection-url>jdbc:mysql://mysql:3306/keycloak?useSSL=false&characterEncoding=UTF-8</connection-url>
<driver>com.mysql</driver>
<security>
<user-name>keycloak</user-name>
<password>somethingtochangelater</password>
</security>
</datasource>
<drivers>
<driver name="h2" module="com.h2database.h2">
<xa-datasource-class>org.h2.jdbcx.JdbcDataSource</xa-datasource-class>
</driver>
<driver name="mysql" module="com.mysql">
<driver-class>com.mysql.cj.jdbc.Driver</driver-class>
</driver>
</drivers>
</datasources>
...
So.... anyone have any idea what's going on? What else do I need to do to get Keycloak talking properly to MySQL? Anything else I can do to debug what the issue is?
Not sure what is wrong with your particular case, but I used jboss/ keycloak image and it connects to MySQL just fine. Maybe you can derive your custom image from there. The full setup in my blog post https://link.medium.com/eK6IRducpeb
For standalone keycloak server you can try this command.
kc.bat start-dev --db postgres --db-url jdbc:postgresql://localhost:5432/keycloak-server --db-username postgres --db-password root

spring boot usage of placeholder in application properties

I have written a simple spring boot application(version springboot 2.0) which uses mysql(version 5.7).
application.properties snippet
spring.datasource.url = jdbc:mysql://localhost:3306/test?useSSL=false
spring.datasource.username = testuser
spring.datasource.password = testpassword
spring.datasource.driver-class-name=com.mysql.jdbc.Driver
When I run it locally, it works fine.
If I want to run this spring boot application in docker then I can change
spring.datasource.url = jdbc:mysql://mysql-container:3306/test?useSSL=false
mysql-container is run using mysql:5.7 image from dockerhub.
However I want to change value of host from some placeholder properties file. so that this looks something like:
spring.datasource.url = jdbc:mysql://${MYSQL_HOST}:3306/test?useSSL=false
note: I am not sure about placeholder format. Is it ${MYSQL_HOST} or #MYSQL_HOST# ?
you can name this placeholder file as placeholder.properties or placeholder.conf or .env or anything. The content of that file should be something like:
MYSQL_HOST=localhost
or
MYSQL_HOST=some ip address
I can create .env or .env.test or .env.prod and I can refer that env file based on where I want to run application.
UPDATE -
I have two questions:
Where should I keep placeholder.properties? Is it under /config/ or under some specific directory?
how to invoke placeholder inside application.properties ?
can someone suggest?
SUGGESTION: If you have a relatively small #/properties, why not just have a different application.properties file for each different environment?
You'd specify the environment at runtime with -Dspring.profiles.active=myenv.
Look here and here.
PS:
To answer your specific question: the syntax is ${MYSQL_HOST}
Thanks to answers by #Raheela Aslam and #paulsm4 and some more research found the issue.
What I was trying to achieve:
Deploy springboot application in docker and then to kubernetes.
I was using minikube for local testing and wanted to pass minikube ip to datasource url.
How I fixed it:
I created configmap for mysql_user, mysql_password, mysql_host with respective values.
kubectl create configmap mysql-config \
--from-literal=mysql_user=testuser \
--from-literal=mysql_password=testuserpass \
--from-literal=mysql_user=$(minikube ip)
and used these inside application.properties something like below
spring.datasource.url = jdbc:mysql://${MYSQL_HOST}:3306/test?useSSL=false
spring.datasource.username = ${MYSQL_USER}
spring.datasource.password = ${MYSQL_PASSWORD}
spring.datasource.driver-class-name=com.mysql.jdbc.Driver
and then used configmap values in deployment.yaml for kubernetes.
Then did start service for deployment.
If you project is Maven you can use maven filter:
<build>
<filters>
<filter>src/main/filters/myfilter.properties</filter>
</filters>
</build>
This generates the /target/classes/application.properties it has been filtered to contain the property values (with replaced placeholders)
http://www.avajava.com/tutorials/lessons/how-do-i-filter-resources-based-on-values-from-a-properties-file.html?page=1

Handshake error to mysql via ssl

tldr: I get the following error with a client/server setup with ssl:
Specifically: ssl configuration error: UseCertificateChainFile: SSL errors: PEM routines:PEM_read_bio:no start line SSL routines:SSL_CTX_use_certificate_chain_file:PEM lib
The details:
I am hosting something similar to a mysql server on ec2 (specifically mongosqld)
I need to connect to it via ssl, so I create certs using openssl on the machine using this set of instructions provided by mysql: https://dev.mysql.com/doc/refman/5.7/en/creating-ssl-files-using-openssl.html
I run the server with the following command
mongosqld --schema=schema.drdl \
--addr=0.0.0.0:3307 \
--auth \
--sslMode=allowSSL \
--sslCAFile=ca.pem \
--sslPEMKeyFile=server-key.pem
And on my machine, I attach to the server like so:
mysql --protocol tcp \
--host my.host.on.amazon.com --port 3307 \
--enable-cleartext-plugin \
--ssl-cert mongosqlcerts/client-cert.pem
--ssl-key mongosqlcerts/client-key.pem
--ssl-ca mongosqlcerts/ca.pem
On the client side I get the following error:
SSL connection error: error:00000001:lib(0):func(0):reason(1)
On the server logs, a bit more helpful:
mongosqld starting: version=v2.3.1 pid=11461 host=ip-xx-xx-xx-xx
I CONTROL [initandlisten] git version: fa3535342a4c5abe36e3cc28a2ecf72864dfc6fe
I CONTROL [initandlisten] OpenSSL version: OpenSSL 1.0.1e 11 Feb 2013
I CONTROL [initandlisten] options: {schema: {path: "schema.drdl"}, net: {bindIp: [0.0.0.0], ssl: {mode: "allowSSL", PEMKeyFile: "server-key.pem", CAFile: "ca.pem"}}, security: {enabled: true}}
I NETWORK [initandlisten] waiting for connections at [::]:3307
I NETWORK [initandlisten] waiting for connections at /tmp/mysql.sock
I NETWORK [conn1] connection accepted from 108.20.XXX.XXX:63353 #1 (1 connection now open)
E NETWORK [conn1] handshake error: ERROR 1043 (08S01): recv handshake response error: ERROR 1043 (08S01): ssl configuration error: UseCertificateChainFile: SSL errors: PEM routines:PEM_read_bio:no start line
SSL routines:SSL_CTX_use_certificate_chain_file:PEM lib
2017-12-01T22:24:50.394+0000 I NETWORK [conn1] end connection 108.20.XXX.XXX:63353 (0 connections now open)
Specifically: ssl configuration error: UseCertificateChainFile: SSL errors: PEM routines:PEM_read_bio:no start line
SSL routines:SSL_CTX_use_certificate_chain_file:PEM lib
Can anyone help me parse that error.
Thanks friends!
I got it!
so for the sslPEMKeyFile argument in mongosqld as stated in the docs you need to
Specifies the .pem file containing both the TLS/SSL certificate and key for MySQL clients. Specify the file name of the .pem file using relative or absolute paths.
So when I used openssl to create ca, server-cert and key files, I needed to create a new file that combined the private-key and the server-cert
so I created a new file called combined.pem with the following text in it:
-----BEGIN RSA PRIVATE KEY-----
(Your Private Key: your_domain_name.key)
-----END RSA PRIVATE KEY-----
-----BEGIN CERTIFICATE-----
(Your Primary SSL certificate: your_domain_name.crt)
-----END CERTIFICATE-----
and started the server with that and it worked!!!

cant list Hadoop filesystem

I'm trying to run Hadoop on a banana pi and I don't understand why my Hadoop isn't working.
I have an exception on running the very basic command
root#bananapi:/opt/hadoop# hadoop fs -ls
14/12/17 10:27:13 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
ls: Call From bananapi/10.0.2.150 to localhost:9000 failed on connection exception:
java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused
root#bananapi:/opt/hadoop#
I have installed the followed jdk
root#bananapi:/opt/hadoop# java -version
java version "1.7.0_65"
OpenJDK Runtime Environment (IcedTea 2.5.3) (7u71-2.5.3-2~deb7u1)
OpenJDK Zero VM (build 24.65-b04, mixed mode)
My etc/hosts file is looking like this:
root#bananapi:/opt/hadoop# cat /etc/hosts
127.0.0.1 localhost
10.0.2.150 bananapi # wlan0
10.0.2.119 bananapi # eth0
::1 localhost ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
The services (jps) is also running:
root#bananapi:/opt/hadoop# jps
3732 NodeManager
3317 NameNode
3644 ResourceManager
3401 DataNode
3513 SecondaryNameNode
3860 Jps
What I'm doinging wrong or why is commands not working?
P.S. I have also tried to run hdfs dfs -ls
UPDATE
my confs in the core-site.xml
<property>
<name>fs.default.name</name>
<value>hdfs://localhost:9000</value>
</property>
I also can't format the namenode
hadoop namenode -format

Digital Certificate: How to import .cer file in to .truststore file using?

Has anyone came across where they have to deal with .truststore file? and knowing how to import .cer into .truststore file?
I am not sure if I have to use Java Keytool or Linux command (such as openssl command).
Thanks
# Copy the certificate into the directory Java_home\Jre\Lib\Security
# Change your directory to Java_home\Jre\Lib\Security>
# Import the certificate to a trust store.
keytool -import -alias ca -file somecert.cer -keystore cacerts -storepass changeit [Return]
Trust this certificate: [Yes]
changeit is the default truststore password
Instead of using sed to filter out the certificate, you can also pipe the openssl s_client output through openssl x509 -out certfile.txt, for example:
echo "" | openssl s_client -connect my.server.com:443 -showcerts 2>/dev/null | openssl x509 -out certfile.txt
The way you import a .cer file into the trust store is the same way you'd import a .crt file from say an export from Firefox.
You do not have to put an alias and the password of the keystore, you can just type:
keytool -v -import -file somefile.crt -alias somecrt -keystore my-cacerts
Preferably use the cacerts file that is already in your Java installation (jre\lib\security\cacerts) as it contains secure "popular" certificates.
Update regarding the differences of cer and crt (just to clarify)
According to Apache with SSL - How to convert CER to CRT certificates? and user #Spawnrider
CER is a X.509 certificate in binary form, DER encoded.
CRT is a binary X.509 certificate, encapsulated in text (base-64) encoding.
It is not the same encoding.