How to store credentials in sql configuration file as environment variables - mysql

I've got an SQL configuration file that's something like this:
[client]
database = dev
host = my-host.com
user = dev
password = super-secret-password
default-character-set = utf8
Is there any way I can swap out the plaintext host and password with some sort of environment variable, so I don't have to push it to GitHub directly? To deploy, I've been pushing to GitHub, making a docker image of the code pushed, pulling it onto an AWS server, and running it.
I'd rather not push the plaintext config file directly so I was wondering how to get around this.

You can use Github Secret to store sensitive data for your projects .
Read more about it from here ; Creating encrypted secrets for a repository
Create Env Variable using Github Action:
steps:
- name: Execute script
env:
PASSWORD: ${{ secrets.SCRIPT_CREDENTIALS }}
run: # your script to connect the database here .
for example to use a PHP script you can follow this method :
<?php
$servername = "localhost";
$username = "username";
$password = getenv("PASSWORD");
$conn = new mysqli($servername, $username, $password);
if ($conn->connect_error) {
die("Connection failed: " . $conn->connect_error);
}
echo "Connected successfully";
?>
To make a change on a .cfg file you can also use githubaction like that :
steps:
- name: Edit your config file
env:
PASSWORD: ${{ secrets.SCRIPT_CREDENTIALS }}
run: echo "password = ${{ secrets.SCRIPT_CREDENTIALS }}" >> file.cfg

Update on this for anyone using Django and having a similar issue, I was able to figure it out like this.
Before, my database connection file was set up like this:
DATABASES = {
"default": {
"ENGINE": "django.db.backends.mysql",
"OPTIONS": {
"read_default_file": "local.cnf",
},
}
}
rather than doing this, it's easier to do something like:
DATABASES = {
"default": {
"ENGINE": "django.db.backends.mysql",
'NAME': 'dev',
'USER': 'dev',
'PASSWORD': os.environ['DEV_PASS'],
'HOST': os.environ['DEV_HOST']
}
}
so then you can specify your environment variables as usual.

Related

Problems connecting Cloud Run Application to Cloud SQL using Spring boot

I am trying to connect a Spring application (using Kotlin and Gradle) to a Google Cloud SQL instance and database. I am getting the error message
java.lang.RuntimeException: [<project-name>:europe-west1:<db-instance>] The Cloud SQL Instance does not exist or your account is not authorized to access it. Please verify the instance connection name and check the IAM permissions for project "<project-name>"
I have followed the guide on how to connect carefully, but to no avail.
Relevant files
src/main/resources/application.yml
server:
port: ${PORT:8080}
spring:
liquibase:
change-log: classpath:liquibase/db.changelog.xml
contexts: production
cloud:
appId: <project-id>
gcp:
sql:
instance-connection-name: <instance-connection-name>
database-name: <db-name>
jpa:
hibernate:
dialect: org.hibernate.dialect.MySQL8Dialect
default_schema: <schema>
show_sql: true
ddl-auto: none
datasource:
driver-class-name: com.mysql.cj.jdbc.Driver
continue-on-error: true
initialization-mode: always
url: jdbc:mysql:///<db-name>?cloudSqlInstance=<instance-connection-name>&socketFactory=com.google.cloud.sql.mysql.SocketFactory&user=<user>&password=<password>
username: <user>
password: <password>
---
spring:
config:
activate:
on-profile: dev
jpa:
hibernate:
ddl-auto: create-drop
spring.jpa.database-platform: org.hibernate.dialect.H2Dialect
datasource:
url: jdbc:h2:mem:mydb
username: sa
password: password
driverClassName: org.h2.Driver
cloud:
gcp:
sql:
enabled: false
build.gradle.kts
import org.jetbrains.kotlin.gradle.tasks.KotlinCompile
plugins {
id("org.springframework.boot") version "2.6.5"
id("io.spring.dependency-management") version "1.0.11.RELEASE"
kotlin("jvm") version "1.6.10"
kotlin("plugin.spring") version "1.6.10"
kotlin("plugin.allopen") version "1.4.32"
kotlin("plugin.jpa") version "1.4.32"
kotlin("kapt") version "1.4.32"
}
allOpen {
annotation("javax.persistence.Entity")
annotation("javax.persistence.Embeddable")
annotation("javax.persistence.MappedSuperclass")
}
group = "com.<company>"
version = "0.0.1-SNAPSHOT"
java.sourceCompatibility = JavaVersion.VERSION_17
repositories {
mavenCentral()
}
dependencies {
implementation("org.springframework.boot:spring-boot-starter-web:2.6.5")
implementation("org.springframework.boot:spring-boot-starter-webflux:2.6.5")
implementation("org.springframework.boot:spring-boot-starter-data-jpa:2.6.5")
implementation("org.springframework.cloud:spring-cloud-gcp-starter-sql-mysql:1.2.8.RELEASE")
implementation("org.jetbrains.kotlin:kotlin-reflect:1.6.10")
implementation("org.jetbrains.kotlin:kotlin-stdlib-jdk8:1.6.10")
implementation("com.fasterxml.jackson.dataformat:jackson-dataformat-xml:2.13.2")
implementation("com.fasterxml.jackson.core:jackson-annotations:2.13.2")
implementation("com.fasterxml.jackson.core:jackson-core:2.13.2")
implementation("com.fasterxml.jackson.core:jackson-databind:2.13.2.2")
implementation("com.fasterxml.jackson.module:jackson-module-kotlin:2.13.2")
implementation("com.fasterxml.jackson.datatype:jackson-datatype-jsr310:2.13.2")
implementation("org.hibernate:hibernate-core:5.6.7.Final")
implementation("javax.persistence:javax.persistence-api:2.2")
implementation( "commons-codec:commons-codec:1.15")
implementation("io.github.microutils:kotlin-logging-jvm:2.1.21")
implementation("ch.qos.logback:logback-classic:1.2.11")
implementation("com.google.cloud.sql:mysql-socket-factory-connector-j-8:1.4.4")
runtimeOnly("com.h2database:h2:2.1.210")
runtimeOnly("org.springframework.boot:spring-boot-devtools:2.6.5")
testImplementation("org.springframework.boot:spring-boot-starter-test:2.6.5")
}
tasks.withType<KotlinCompile> {
kotlinOptions {
freeCompilerArgs = listOf("-Xjsr305=strict")
jvmTarget = "17"
}
}
tasks.withType<Test> {
useJUnitPlatform()
}
Dockerfile
FROM openjdk:17-alpine
ENV USER=appuser
# <placeholder> Replace context path for your own application
ENV JAVA_HOME=/opt/openjdk-17 \
HOME=/home/$USER \
CONTEXT_PATH=/aws-service-baseline
RUN adduser -S $USER
# <placeholder> Add additional packages for the docker container here
RUN apk add --no-cache su-exec
# <placeholder> Replace baseline.jar with your applications JAR file (defined in build.gradle.kts)
COPY Docker/runapp.sh build/libs/<application-name>-0.0.1-SNAPSHOT.jar $HOME/
RUN chmod 755 $HOME/*.sh && \
chown -R $USER $HOME
WORKDIR /home/$USER
CMD [ "./runapp.sh"]
Docker/runapp.sh
#!/bin/sh
set -e
# The module to start.
# <placeholder> Replace this with your own modulename (from module-info)
APP_JAR="<application-name>-0.0.1-SNAPSHOT.jar"
JAVA_PARAMS="-XshowSettings:vm"
echo " --- RUNNING $(basename "$0") $(date -u "+%Y-%m-%d %H:%M:%S Z") --- "
set -x
/sbin/su-exec "$USER:1000" "$JAVA_HOME/bin/java" "$JAVA_PARAMS $JAVA_PARAMS_OVERRIDE" -jar -Dserver.port=$PORT "$APP_JAR"
GCP details
I have made sure the SQL instances connection is added to the Cloud Run Revisions. The IAM roles for the compute service account also seem to be right. See images
IAM: https://i.stack.imgur.com/yYaC5.png
Database: https://i.stack.imgur.com/NErad.png
Cloud Run connection https://i.stack.imgur.com/fKTSZ.png
Additional details
When running ./gradlew bootRun on my local machine (with GCP credentials present), the App works properly with an SQL connection. It also works after running ./gradle bootRun to build the JAR file and run the JAR directly. It does not work out of the box when running in Docker, but if I add the GCP credentials to the Docker container locally, it connects to the Database.
Does anyone have any suggestions on what might be wrong? Any help much appreciated!
I have tried connecting locally and locally in a Docker container.
Figured it out! Human error of course. The Cloud Run Service was initially configured with another Services Account, and not the default Compute Engine Service account.

Problems with execution aws command via ssh jenkins

Good morning, how are you?
I have a problem with one execution via ssh in my jenkins.
Those characters that appear before did not appear, and we have not changed anything in the node..
The code we use is:
withCredentials([usernamePassword(credentialsId: 'id', passwordVariable: 'pass', usernameVariable: 'user')]) {
def remote = [:]
remote.name = 'id_nme'
remote.host = 'ip_node'
remote.user = user
remote.password = pass
remote.allowAnyHosts = true
remote.timeoutSec = 300
sshCommand remote: remote, command: "aws autoscaling describe-auto-scaling-groups --auto-scaling-group-name [name_asg]"
}
When the command is launched, the job is stuck and does not progress.

How to execute mysql script insertion on terraform user_data?

The last line of the script was not executed.
I tried to execute the code manually on the instance created and it was successful.
#!/bin/bash
#install tools
apt-get update -y
apt-get install mysql-client -y
#Create MySQL config file
echo "[mysql]" >> ~/.my.cnf
echo "user = poc5admin" >> ~/.my.cnf
echo "password = poc5password" >> ~/.my.cnf
#test
echo "endpoint = ${rds_endpoint}" >> ~/variables
hostip=$(hostname -I)
endpoint=${rds_endpoint}
echo "$hostip" >> ~/variables
#I have created a table here but I will remove the code since it is unnecessary...
#Create User
echo "CREATE USER 'poc5user'#'%' IDENTIFIED BY 'poc5pass';" >> ~/mysqlscript.sql
echo "GRANT EVENT ON * . * TO 'poc5user'#'%';" >> ~/mysqlscript.sql
cp mysqlscript.sql /home/ubuntu/mysqlscript.sql
mysql -h $endpoint -u poc5admin < ~/mysqlscript.sql
Expected result: There should be a Database, Table and User created on the RDS instance.
You can insert or create Database like this from the bash script but it is not recommended an approach to work with RDS. better to place your data over s3 and import from the s3.
Here is the example, that will create DB
resource "aws_db_instance" "db" {
allocated_storage = 20
storage_type = "gp2"
engine = "mysql"
engine_version = "5.7"
instance_class = "db.t2.micro"
name = "mydb"
username = "foo"
password = "foobarbaz"
parameter_group_name = "default.mysql5.7"
s3_import {
source_engine = "mysql"
source_engine_version = "5.6"
bucket_name = "mybucket"
bucket_prefix = "backups"
ingestion_role = "arn:aws:iam::1234567890:role/role-xtrabackup-rds-restore"
}
}
~/.my.cnf why you need this? better to place these script in the s3 file.
second thing, If you still interesting to run from your local environment then you can insert from local-exec
resource "null_resource" "main_db_update_table" {
provisioner "local-exec" {
on_failure = "fail"
interpreter = ["/bin/bash", "-c"]
command = <<EOT
mysql -h ${aws_rds_cluster.db.endpoint} -u your_username -pyour_password your_db < mysql_script.sql
EOT
}
}
But better to with s3.
If you want to import from remote, you can explore remote-exec.
With user-data, you can do this but it seems your MySQL script not generating properly. better to cp script to remote and then run with local exec in remote.
There is no such thing as terraform "user_data". User data is a bootstrap script for the EC2 instances which you can use to install software/binaries or to execute your script at the boot time.
The script will be executed by the cloud-init, not by the terraform itself. The responsibility of the terraform is to set user-data for the ec2 instances.
You may check the cloud-init output logs which should have the result of your user-data script also.
From your code, I am not able to understand which step you have copied the below file.
cp mysqlscript.sql /home/ubuntu/mysqlscript.sql
mysql -h $endpoint -u poc5admin < ~/mysqlscript.sql
I am assuming that you are creating a new server and it does not have any file.
Thank you for your inputs. I have found an answer by moving the config file to /etc/mysql/my.cnf and then executing
mysql -h $endpoint -u poc5admin < ~/mysqlscript.sql

I provide password in envoy script but it prompts to enter password manually

With laravel 5.8 envoy command I deploy on remote server and I set password in command line, like:
envoy run Hostels2Deploy --lardeployer_password=111 --app_version=0.105a
and envoy file:
#setup
$server_login_user= 'lardeployer';
$lardeployer_password = isset($lardeployer_password) ? $lardeployer_password : "Not Defined";
#endsetup
#servers(['dev' => $server_login_user.':'.$lardeployer_password.'#NNN.NN.NNN.N'])
#task('clean_old_releases')
echo "Step # 81";
echo 'The password is: {{ $lardeployer_password }}';
echo 'The $server_login_user is: {{ $server_login_user }}';
echo "Step # 00 app_version ::{{ $app_version }}";
cd {{ $release_number_dir }}
# php artisan envoy:delete-old-versions Hostels2Deployed
#endtask
#macro('Hostels2Deploy',['on'=>'dev'])
clean_old_releases
#endmacro
With credentials in #servers block I expected I will not have to enter password manually, but in command line I see prompt to enter
password. I output $server_login_user and $lardeployer_password vars and they have valid values.
Which is valid path ?
I found a decision with ssh keys in /home/user/.ssh/config of my OS to add line :
Host laravelserver
IdentityFile ~/.ssh/id_rsa
HostName NNN.NN.NNN.N
Port 22
User lardeployer
and in envoy file to connect to this server like :
#servers(['dev' => ['laravelserver'] )
Aslo on remote user in file authorized_keys lardeployer's public key must be added.
and restart the sevice :
sudo systemctl restart ssh

How to access phpmyadmin cloud9 cakephp3?

Hi people I have a problem developing in cloud9. I followed the steps to configure mysql and phpmyadmin. So when I run the app I do it with the following line: bin/cake server -H 0.0.0.0 -p 8080. The apps run fine, but when a try to access phpmyadmin (https://james-mand-cortana.c9users.io/phpmyadmin/) shows an error:Error: PhpmyadminController could not be found.
But when I run the app by running the index.php file (without bin/cake server -H 0.0.0.0 -p 8080) works fine to access phpmyadmin.
So Basically this is my problem I want to run my application with the line bin / cake server -H 0.0.0.0 -p 8080 and access phpmyadmin without any problem.
Thanks for the help.
Here is an excerpt from the index.php:
<?php
if (php_sapi_name() === 'cli-server') {
$_SERVER['PHP_SELF'] = '/' . basename(__FILE__);
$url = parse_url(urldecode($_SERVER['REQUEST_URI']));
$file = DIR . $url['path'];
if (strpos($url['path'], '..') === false && strpos($url['path'], '.') !== false && is_file($file)) {
return false;
}
}
require dirname(DIR) . '/vendor/autoload.php';
use App\Application;
use Cake\Http\Server;
$server = new Server(new Application(dirname(DIR) . '/config'));
$server->emit($server->run());