Currently, I have this in my springboot application.dev.yaml:
datasource:
url: jdbc:mysql://mysql/$DB_HOST?useSSL=false&allowPublicKeyRetrieval=true
username: root
password: root
driver-class-name: com.mysql.cj.jdbc.Driver
I want to add a dynamic value in username, passwords, $DB_HOST field so that it can pick those values from the secrets file in Kubernetes.
The secrets file in Kubernetes is encrypted with base64
In Spring Boot, any property can be overridden by an environment variable of the same name, with the characters changed to upper case, and the dots changed to underscores.
For example datasource.url can be overridden by setting an environment variable like DATASOURCE_URL, which you define in Kubernetes
Source: https://developers.redhat.com/blog/2017/10/04/configuring-spring-boot-kubernetes-secrets#setup
Related
Apologies that I am working with inherited code and don't understand it well enough to create a minimal reproducible example.
The application is saving data to a MySQL database, through a JPA repository.
The application works, but the data is not apparent through a second application (eg DBeaver) until the MySQL server is restarted.
Is there something I need to set in MySlq (5.6.21) or is there some way I should be configuring the connection in my application to change this behavior?
I have the following in my application.yml:
spring:
lifecycle:
timeout-per-shutdown-phase: 5s
datasource:
password: MYPASS
url: jdbc:mysql://localhost:3306/myDB?useUnicode=true&useJDBCCompliantTimezoneShift=true&useLegacyDatetimeCode=false&serverTimezone=UTC&useServerPrepStmts=true
&rewriteBatchedStatements=true&cachePrepStmts=true
username: MYUSER
auto-commit: true
hikari:
auto-commit: true
jpa:
hibernate:
ddl-auto: none
use-new-id-generator-mappings: false
connection:
provider_disables_autocommit: false
I am trying to run Wordpress on Google App Engine standard environment. I have configured a Cloud SQL for MySQL Second Generation instance and can access it using Cloud SQL Proxy with this command:
cloud_sql_proxy -instances=my_project_id:us-central1:my_project=tcp:3306
The wp-config.php file:
if (isset($_SERVER['GAE_ENV'])) {
define('DB_HOST', ':/cloudsql/my_project_id:us-central1:my_project');
} else {
define('DB_HOST', '127.0.0.1');
}
Finally, I connect to the database using this:
$dbConn = mysqli_connect (DB_HOST, DB_USER, DB_PASSWORD) or die (mysqli_error($dbConn));
mysqli_select_db($dbConn, DB_NAME) or die(mysqli_error($dbConn));
This setup works perfectly from the local development environment, which is Cloud Shell. The website runs and I am able to query the database and insert records etc. My problem arises when I deploy to my_project_id.appspot.com using google app deploy. The website runs, but when I try to query the database I receive this error:
Warning: mysqli_connect(): php_network_getaddresses: getaddrinfo failed: Name or service not known in /srv/wp-content/themes/mytheme/system/db.php on line 14
Line 14 is $dbConn = mysqli_connect (DB_HOST, DB_USER, DB_PASSWORD)
so I am guessing that mysqli must not like the format of the specified DB_HOST which is :/cloudsql/my_project_id:us-central1:my_project.
In this Community Tutorial there is sample code which uses a unix socket and PDO to connect to the database. I don't know if I should be adding these lines to the app.yaml file and someone using this different connection string.
env_variables:
MYSQL_DSN: mysql:unix_socket=/cloudsql/my_project_id:us-central1:my_project;dbname=my_dbname
MYSQL_USER: username
MYSQL_PASSWORD: password
My apologies for the lengthy question, but I wanted to provide as much information as possible. Anyone have any ideas what I am doing wrong? Thanks.
It looks like you might be passing in the information incorrectly into mysqli_connect. If you take a look a the documentation for it, it actually takes in 6 parameters: host, username, passwd, dbname, port, socket.
Under host, you can read the following:
Passing the NULL value or the string "localhost" to this parameter, the local host is assumed. When possible, pipes will be used instead of the TCP/IP protocol.
Under socket, it clarifies it should be the socket path:
Specifies the socket or named pipe that should be used.
So you need to call mysqli_connect like this:
mysqli_connect (null, "user", "password", "database", 3306, "/cloudsql/<INSTANCE_CONNECTION_NAME>")
From Cloud Shell, your environment is all setup properly in order to directly connect to Cloud SQL. From AppEngine there's a few other steps necessary in order to connect.
Check out this documentation:
https://cloud.google.com/sql/docs/mysql/connect-app-engine
It should get you up and running.
The answer by #Kurtisvg is absolutely correct in terms of this being the proper format for connecting to Cloud SQL using mysqli_connect:
mysqli_connect (null, DB_USER, DB_PASSWORD, DB_NAME, 3306, "/cloudsql/<INSTANCE_CONNECTION_NAME>")
The original question, however, also mentioned that I was trying to get the connection working for a Wordpress installation. These are the two areas with additional information relevant to this tutorial on how to run Wordpress on Google App Engine standard.
1. Specifying correct mysqli_connect() parameters
Testing the app in the local development environment only required mysqli_connect to use these four parameters: DB_HOST, DB_USER, DB_PASSWORD, DB_NAME. Once the app is deployed to Google App Engine, the mysqli_connect has to use all six parameters: DB_HOST, DB_USER, DB_PASSWORD, DB_NAME, DB_PORT, DB_SOCK. This is the code in db.php that provides the correct parameters, depending on the environment:
if (isset($_SERVER['GAE_ENV'])) {
$dbConn = mysqli_connect (null, DB_USER, DB_PASSWORD, DB_NAME, DB_PORT, DB_SOCK);
} else { // local environment
$dbConn = mysqli_connect (DB_HOST, DB_USER, DB_PASSWORD, DB_NAME);
}
2. Specify correct Wordpress configuration in wp-config.php
What I was finding was that in order Wordpress to function correctly, it was necessary to not only define and use the socket DB_SOCK in mysqli_connect(), but I also had to define a DB_HOST for purposes of the Wordpress installation. This was the configuration that worked in my wp-config.php file:
define('DB_USER', 'user');
define('DB_PASSWORD', 'password');
define('DB_NAME', 'database');
define('DB_PORT', 3306);
// Check for Google App Engine Environment
if (isset($_SERVER['GAE_ENV'])) {
$onGae = true;
define('DB_HOST', ':/cloudsql/<INSTANCE_CONNECTION_NAME>');
define('DB_SOCK', '/cloudsql/<INSTANCE_CONNECTION_NAME>');
} else {
$onGae = false;
define('DB_HOST', '127.0.0.1');
}
In the above code, the variable for DB_HOST requires a full-colon : at the beginning of the socket. This DB_HOST variable is not used as one of the connection parameters of mysqli_connect when in the GAE environment. This variable does seem to be used elsewhere in Wordpress (such as setup-config.php), which is why it needs to be defined. The variable for DB_SOCK does not require the full-colon : in order to work in the GAE environment. This socket needs to be the last (6th) parameter of mysqli_connect, with the first parameter specified as null, which forces the connection to use the socket.
It took a while to get this sorted, but eventually got it working using these settings above. I wonder if anyone else had such a complicated experience as I did getting Wordpress on Google App Engine standard environment to connect to a Cloud SQL for MySQL Second Generation instance. I hope these comments help someone.
You are better off using wpdb class. Wpdb is a WordPress database access abstraction class. Check out this documentation
I have a very weird requirement in a project I am working on.
I have file stored in some location /etc/config/config.json which contains database information like Host, Port, Username or Passwordor roughly looks as below:
{
"mysql-db": {
"host": "172.17.0.27",
"port": 3306,
"password": "root",
"username": "root"
}
}
I am building my small web-app based on Sinatra and using sinatra-activerecord for working with MySql database.
My database.yml file looks like this:
development:
adapter: mysql2
database: toopaste
host: <%= ENV["MYSQL_DB_HOST"] %>
port: <%= ENV["MYSQL_DB_PORT"] %>
username: <%= ENV["MYSQL_DB_USERNAME"] %>
password: <%= ENV["MYSQL_DB_PASSWORD"] %>
What I was trying?
I created a executable file like setup.rb as:
#! /usr/bin/env ruby
require 'json'
FILE_PATH = "/etc/atlantis/config/konfig.json"
data = JSON.parse(File.read(FILE_PATH))
system("export MYSQL_DB_HOST=#{data['mysql-db']['host']}")
system("export MYSQL_DB_PORT=#{data['mysql-db']['port']}")
system("export MYSQL_DB_USERNAME=#{data['mysql-db']['username']}")
system("export MYSQL_DB_PASSWORD=#{data['mysql-db']['password']}")
This doesn't set env variables MYSQL_DB_HOST or others variables to be used by config/database.yml file.
Any idea how to accomplish such job?
One way I could think is "Dynamically generate whole database.yml file after reading config.json params.
But would like to know if there is better solution available.
I have a feeling that Sinatra doesn't process the environment variables into the database.yml file. Rails does... You can do this, but I think it's a bit of a faff. I think you have to put the YML file through as an ERB template or something.
Some other options:
Dynamically write the entire database.yml file from your setup.rb - although I wouldn't do this. The load it form there as normal.
Or, use the Sinatra config to set your Database connections details form your preferred file. Example in the sinatra-activerecord read me.
set :database, {adapter: 'mysql', database: ENV['MY_SQL_DB_HOST']}
This would seem cleaner to me. In fact, I would go a little further and use Sinatra config to do the whole thing (load the file, and get the parameters from there). This way the code becomes more explicit, and easier to change in the future. i.e., and this is just off the top of my head, so you may need to adjust:
configure do
FILE_PATH = "/etc/atlantis/config/konfig.json"
data = JSON.parse(File.read(FILE_PATH))
set :DB_PASSWORD = data['mysql-db']['password']
# ETC...
set :database, { ... }
end
Hope this helps.
I assume this configuration file is outside of your application's reach otherwise you could just read the json file directly. Otherwise you could have a parser of the JSON and transform that into YAML. Something like this would work:
require 'json'
require 'yaml'
json = JSON.parse(File.read('./test.json'))
database = Hash.new
database[:development] = json
File.open("./test.yaml","w"){|h| h.write database.to_yaml }
Your YAML should look like this.
$ cat test.yaml
---
:development:
mysql-db:
host: 172.17.0.27
port: 3306
password: root
username: root
I assume you can add the additional parameters to the hash to get something like the bellow configurations.
config/production.yaml
database:
adapter: mysql2
host: localhost
port: 3306
database: myappdb
username: myprodusername
password: myprodpassword
config/development.yaml
database:
adapter: mysql2
host: localhost
port: 3306
database: myappdb_dev
username: mydevuser
password: mydevpassword
And then load them into your app like this.
config.ru
require 'sinatra'
require 'yaml'
configure :production do
#config = YAML.load_file("config/#{ENV["RACK_ENV"]}.yaml")
#some other things that you do in prod
end
configure :development do
#config = YAML.load_file("config/#{ENV["RACK_ENV"]}.yaml")
#some other things that you only do in dev
end
Starting the app
$ RACK_ENV=development puma (or whatever other server you use like thin)
or for prod
$ RACK_ENV=production puma (or whatever other server you use like thin)
The system method creates a new subshell and executes the command in it. When the command sets an environment variable then that environment variable is set in that subshell, the change does not propagate back to the parent process. This means that those environment variables are not being set.
To set environment variables in the current process, you can simply access ENV directly. So instead of system("export MYSQL_DB_HOST=#{data['mysql-db']['host']}") do this:
ENV['MYSQL_DB_HOST'] = data['mysql-db']['host']
(and similarly for the other settings).
This should fix your problem, but since you are reading the settings yourself directly from the json config file there arguably isn’t much point using environment variables. It would be more direct to access the settings more directly from the Erb/Yaml file. Ideally you’d want to pass the settings to the Erb evaluation in some way, but Sinatra-ActiveRecord doesn’t appear to allow you to set the binding or context of the Erb evaluation in any way. An alternative could be to use a global variable for the data hash:
# When reading the json, use a global instead,
# and pick out the 'mysql-db' key
$data = JSON.parse(File.read(FILE_PATH))['mysql-db']
The in the database.yml:
development:
adapter: mysql2
database: toopaste
host: <%= $data['host'] %>
port: <%= $data['port'] %>
username: <%= $data['username'] %>
password: <%= $data['password'] %>
I have two databases, (MySQL and Oracle), I did the connection betweek sf2 and both databases, here is my config.yml file:
doctrine:
dbal:
default_connection: default
connections:
default:
driver: "%database_driver%"
host: "%database_host%"
port: "%database_port%"
dbname: "%database_name%"
user: "%database_user%"
password: "%database_password%"
charset: UTF8
# if using pdo_sqlite as your database driver, add the path in parameters.yml
# e.g. database_path: "%kernel.root_dir%/data/data.db3"
# path: "%database_path%"
sysman:
driver: %database_driver2%
host: %database_host2%
port: %database_port2%
dbname: %database_name2%
user: %database_user2%
password: %database_password2%
charset: UTF8
My question is, how can I run console command on the second database (Oracle), commands like (doctrine:database:create ...), and thanks
Use the --connection parameter:
php app/console doctrine:database:create --connection=default
or
php app/console doctrine:database:create --connection=sysman
You should first read a tutorial about commands and how to pass options and parameters to the commands. And how to distinguish between an option and a parameter.
If you want to make your own commands...
You will probably want to make it like this - if you do not pass an option (you will use the default database), if you pass it, you will make sure it is a valid option, and use the passed database connection name.
Doctrine is not tightly coupled with Mysql, you can use almost all most common available databases.
Also note, commands are container aware. That means you commands can access container, though which you have access to your services, such as doctrine:
protected function execute(InputInterface $input, OutputInterface $output)
{
$connection $input->getArgument('connection');
# Validate connection argument here and use it below
$container = $this->getContainer();
$em = $container->get('doctrine')->getManager(); // default db
$em = $container->get('doctrine')->getManager('sysman'); // another
return 1;
}
I wrote the code without testing, excuse me for any mistake I might have done.
php app/console doctrine:mapping:info --em=default (same without em option)
php app/console doctrine:mapping:info --em=sysman
How would I connect to my VPS based MySQL database remotely (from a cloud based app) using the Ruby Net::SSH or Net::SSH::Gateway gems and key, not password, authentication?
And then connect to the database with Sequel or DataMapper. I'm assuming that after I manage to get the SSH connection working, I would just setup a Sequel/DM connection to 'sql_user#localhost:3306/database'.
I did locate a couple of similar question here, but they all use password authentication, not keys, and only demonstrate executing raw commands to query the database.
UPDATE: I just cannot seem to get this (Net::SSH with key manager) to work.
UPDATE2: Alright I have managed to get authorization when logging in from a computer that has authorized keys stored in the users local .ssh folder, with the following (port is my custom SQL port on the VPS):
sql_gate = Net::SSH::Gateway.new('192.xxx.xxx.xx','sqluser', port: 26000)
However, I will not be able to create a .ssh folder in the app's VM, so I need to somehow pass the path and filename (I will be creating a public key just for SQL access for specified user) as an option ... but haven't been able to figure out how.
UPDATE: Just need to figure out DataMapper access now. Current code being tested (remote_user_sql is my Ubuntu user, sql_user is the MySQL database user with localhost/127.0.0.1 privileges):
require 'net/ssh/gateway'
require 'data_mapper'
require 'dm-mysql-adapter'
class User
include DataMapp......
.
.
end
ssh_gate = Net::SSH::Gateway.new('192.n.n.n','remote_user_sql', {port: 25000, keys: ["sql_rsa"], keys_only: true})
port = ssh_gate.open('localhost',3306,3307)
child = fork do
DataMapper.setup(:default, {
adapter: 'mysql',
database: 'sql_test',
username: 'sql_user',
password: 'passwd',
host: 'localhost',
port: port})
DataMapper.auto_upgrade!
exit
end
puts "child: #{child}"
Process.wait
ssh_gate.close(port)
My solution, in two parts:
Well I have figured how to make the Net::SSH::Gateway gem using a specified keyfile, and then connect to the VPS through ssh via a port other than 22:
Part 1: Net::SSH::Gateway key authentication
First you must generate the keyfiles you want to use, copy the .pub to the remove server and append it to the ~/.ssh/authorized_keys file (cat sql_rsa.pub >> authorized_keys), and then make sure user_sql (the user I created on the VPS to be used only for this purpose) has been added to AllowUsers list in sshd_config. Make note of port used for ssh (25000 for this example) and use the following code to establish the connection:
ssh_gate = Net::SSH::Gateway.new('192.n.n.n','user_sql', {port: 25000, keys: ["sql_rsa"], keys_only: true})
That will read the keyfile sql_rsa in the same directory as script file, then create a new ssh gateway for 'user_sql'#'192.n.n.n' on port 25000.
I can successfully execute raw shell commands on the remove VPS with:
ssh_gate.exec("ls -la")
To close:
ssh_gate.shutdown!
Unfortunately I am still having problems using DataMapper (do-mysql-adapter) to use the gateway. I will update this answer if I figure that part out, but at least the first half of the problem has been solved.
These are the errors that DataMapper::Logger has reported:
When 127.0.0.1 was used:
Can't connect to local MySQL server through socket '/tmp/mysql.sock' (2) (code: 2002, sql state: HY000, query: , uri: )
When localhost was used:
Access denied for user 'user_sql'#'localhost' (using password: YES) (code: 1045, sql state: 28000, query: , uri: )
When the VPS hostname was used:
Unknown MySQL server host 'hostname' (25) (code: 2005, sql state: HY000, query: , uri: )
UPDATE (No success yet): So far the only way I can access the remote MySQL database is by using Net::SSH::Gateway to establish a gateway, and then use the .sshmethod to open a new Net::SSH connection over that gateway, like so:
ssh_gate.ssh('192.n.n.n','user_sql',{port: 25000, keys: ["sql_rsa"], keys_only: true}) do |ssh|
ssh.exec("mysql -u sql_user -p'passwd' -h localhost -P 3306 -e 'SELECT DATABASE();'")
end
In other words, I can only execute SQL commands using the mysql command line. I cannot figure out how to get Sequel or DataMapper to use the gateway to connect.
Part 2: DataMapper/Sequel/mysql2 connection through Net::SSH::Gateway
Make sure your MySQL server is bound to 127.0.0.1 in /etc/mysql/my.cnf, setup your connection - DataMapper example:
DataMapper.setup(:default, {
adapter: 'mysql',
database: 'DATABASE',
username: 'username',
password: 'passwd',
host: '127.0.0.1',
port: 3307}) # local port being forwarded via Net::SSH:Gateway
Followed by any class table definitions and DataMapper.finalize if required. Note that DataMapper doesn't actually connect to the remote MySQL server until either an auto_upgrade!, auto_migrate!, or query is executed, so no need to create the forwarded port yet.
Then create a new Net::SSH::Gateway, and then whenever you need DataMapper/Sequel to access the remote database, just open a port for the process, like so:
port = ssh_gate.open('127.0.0.1',3306,3307)
child = fork do
DataMapper.auto_upgrade! # DM call that accesses MySQL server
exit
end
Process.wait
ssh_gate.close(port)
You may want to put the Net::SSH::Gateway/.open code in a begin..ensure..end block, ensure'ing the port closure and gateway shutdown.
I had to use a fork and Process.wait to establish the connection, without it the method just hangs.