My Google Compute Engine startup script has CommunicationsLinkFailures (SQL State 08S01 Error 0) even though it runs fine locally? - mysql

I'm deploying my Java8 SpringBoot App to a Google Compute Engine instance and trying to connect it to a Debian9 CloudSQL instance. I'm trying to get the instance to run with my startup-script.sh, but when it tries to boot up the SpringBoot Application, according to the daemon.log, when the .war is ran by "java -jar order-routing-0.0.1-SNAPSHOT.war" the startup fails with a "Unable to obtain connection from database: Communications link failure", with SQL State:08S01 and error code 0.
I mapped the GCE instance to a static external IP, as well as whitelisting that IP on the CloudSQL instance's connections configs. I also verify that the war file runs locally with "java -jar order-routing.war".
Here is my startup script.sh:
#!/usr/bin/env bash
# This script is passed to the GCE instance by the setup script. It is run on the instance when it is spun up.
# Derived from GCE Tutorial at https://cloud.google.com/java/docs/tutorials/bookshelf-on-compute-engine
# [START script]
set -e
set -v
# Talk to the metadata server to get the project id
PROJECTID=$(curl -s "http://metadata.google.internal/computeMetadata/v1/project/project-id" -H "Metadata-Flavor: Google")
BUCKET=$(curl -s "http://metadata.google.internal/computeMetadata/v1/instance/attributes/BUCKET" -H "Metadata-Flavor: Google")
echo "Project ID: ${PROJECTID}"
# get our file(s)
gsutil cp "gs://order-routing-install/gce/"** .
# Install dependencies from apt
apt-get update
apt-get install mysql-client -y
wget https://dl.google.com/cloudsql/cloud_sql_proxy.linux.amd64
mv cloud_sql_proxy.linux.amd64 cloud_sql_proxy
chmod +x cloud_sql_proxy
./cloud_sql_proxy -instances=instance-qa1:us-central1:instance-qa1-cloudsql-0=tcp:3307 &
apt-get install -yq default-jre
apt-get install -yq default-jdk
java -jar order-routing-0.0.1-SNAPSHOT.war
# [END script]
Here is the failing error log from the daemon.log:
ConfigServletWebServerApplicationContext : Exception encountered during context initialization - cancelling refresh attempt: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'flywayInitializer' defined in class path resource [org/springframework/boot/autoconfigure/flyway/FlywayAutoConfiguration$FlywayConfiguration.class]: Invocation of init method failed; nested exception is org.flywaydb.core.internal.exception.FlywaySqlException:
Jul 29 03:39:30 order-routing-group-hk66 startup-script: INFO startup-script: Unable to obtain connection from database: Communications link failure
Jul 29 03:39:30 order-routing-group-hk66 startup-script: INFO startup-script: 2019-07-29 03:39:30.435 INFO 9051 --- [ main] ConditionEvaluationReportLoggingListener :
Jul 29 03:39:30 order-routing-group-hk66 startup-script: INFO startup-script:n from database: Communications link failure
Jul 29 03:39:30 order-routing-group-hk66 startup-script: INFO startup-script: The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server.
Jul 29 03:39:30 order-routing-group-hk66 startup-script: INFO startup-script: SQL State : 08S01
Jul 29 03:39:30 order-routing-group-hk66 startup-script: INFO startup-script: Error Code : 0
Jul 29 03:39:30 order-routing-group-hk66 startup-script: INFO startup-script: Message : Communications link failure
Expected Result:
The GCE instance starts up the war file successfully and I can access my app using the external IP.
Actual Result:
Getting CommunicationsLinkFailures upon starting up Spring Boot Java app.

Related

AWS Beanstalk with working app fails to deploy app to new ec2 when current one is terminated

I have a new beanstalk that is a migration of an old one running an app under php5.6 platform on Amazon AMI Linux. The new beanstalk is running php7.3 on Amazon Linux2. I have worked through all the migration issues and the app is running correctly on my new beanstalk. I have a load-balancer (classic) and I run autoscaling with the max and min instance settings both set to 1.
The problem occurs when I terminate the ec2. The autoscaling is creating a new ec2 but it is't deploying the application to it.
Does anyone know why this might be, or where I can look to try and debug the issue?
What worked for me was to remove the old .ebextension config files related to cwlogs, and add the line
awslogs: []
to my config that does
packages:
yum:
then create a new conf file as follows
files:
"/tmp/start_aws_cloudwatch_service.sh":
content: |
#!/bin/sh
systemctl start awslogsd
systemctl status awslogsd
systemctl enable awslogsd.service
exit $?
mode : "000755"
owner : root
group : root
commands:
start_aws_cloudwatch_service:
cwd: /tmp
command: bash /tmp/start_aws_cloudwatch_service.sh
After this I could see that the service was up and running
$ systemctl status awslogsd
● awslogsd.service - awslogs daemon
Loaded: loaded (/usr/lib/systemd/system/awslogsd.service; enabled; vendor preset: disabled)
Active: active (running) since Wed 2020-10-14 14:08:19 UTC; 34min ago
Main PID: 4029 (aws)
CGroup: /system.slice/awslogsd.service
└─4029 /usr/bin/python2 -s /usr/bin/aws logs push --config-file /etc/awslogs/awslogs.conf --additional-configs-dir /etc/awslogs/c...
Oct 14 14:08:19 ip-xxx-xxx-30-7.eu-west-1.compute.internal systemd[1]: Started awslogs daemon.
Oct 14 14:08:19 ip-xxx-xxx-30-7.eu-west-1.compute.internal systemd[1]: Starting awslogs daemon...
See: https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/QuickStartEC2Instance.html

How to know where ejabberd is config

I am working on old project where ejabberd is used. In laravel env file this information is written
JABBER_SERVER=localhost
JABBER_ENV=dev
RPC_SERVER= http://xx.xx.xx.xxx:4560/RPC2
JABBER_HOST=localhost
when i try to find the ejabberd service on my ubuntu server i couldn't.I don't have any idea how i can find ejabberd information because i am getting an error.
2019-12-26 07:54:27.338 [info] <0.23787.5>#ejabberd_c2s:process_terminated:264 (tcp|<0.23786.5>) Closing c2s session for prod6067#localhost/121590970668513998469410: Connection failed: connection closed
2019-12-26 07:54:30.577 [info] <0.339.0>#ejabberd_listener:accept:300 (<0.23789.5>) Accepted connection 115.186.58.62:58101 -> 172.31.42.93:5222
2019-12-26 07:54:31.390 [info] <0.23790.5>#ejabberd_c2s:handle_auth_success:423 (tcp|<0.23789.5>) Accepted c2s SCRAM-SHA-1 authentication for prod6067#localhost by mnesia backend from
2019-12-26 07:54:32.166 [info] <0.23790.5>#ejabberd_c2s:bind:394 (tcp|<0.23789.5>) Opened c2s session for prod6067#localhost/151108589900049164879426
If you can run the ejabberdctl shell script, then it will show many information about where ejabberd is installed:
$ ejabberdctl
...
Optional parameters when starting an ejabberd node:
--config-dir dir Config ejabberd: /etc/ejabberd
--config file Config ejabberd: /etc/ejabberd/ejabberd.yml
--ctl-config file Config ejabberdctl: /etc/ejabberd/ejabberdctl.cfg
--logs dir Directory for logs: /usr/local/var/log/ejabberd
--spool dir Database spool dir: /usr/local/var/lib/ejabberd
--node nodename ejabberd node name: ejabberd#localhost

Gcloud sql proxy startup script runs but gives out an error

Here is my startup script:
#! /bin/sh
sudo apt-get update
sudo apt-get install mysql-client
wget https://dl.google.com/cloudsql/cloud_sql_proxy.linux.amd64 -O cloud_sql_proxy
chmod +x cloud_sql_proxy
sudo mkdir /cloudsql; sudo chmod 777 /cloudsql
./cloud_sql_proxy -dir=/cloudsql --instances=a:us-east4:b &
Here is the error I get:
Feb 19 22:12:38 instance-group-1-th2f startup-script: INFO startup-
script: 2019/02/19 22:12:38 errors parsing config:
Feb 19 22:12:38 instance-group-1-th2f startup-script: INFO startup-script: #011Get https://www.googleapis.com/sql/v1beta4/projects/a/instances/b?alt=json&prettyPrint=false: dial tcp: lookup www.googleapis.com on [::1]:53: read udp [::1]:52058->[::1]:53: read: connection refused
Feb 19 22:12:38 instance-group-1-th2f startup-script: INFO startup-script: 2019/02/19 22:12:38 Ready for new connections
If I open the instance manually and run the following, it will work:
./cloud_sql_proxy -dir=/cloudsql -instances=a:us-east4:b &
I am guessing that it is an authentication problem, but based on this:
Note: If your Compute Engine instance has either the Full API access
or Cloud SQL API scope, you can skip this step; you do not need to
provide a certificate file when you start the proxy.
It shouldn't have any problems since mine has a Full API access
I have a second gen instance and the Cloud SQL Admin API is enabled.
dial tcp: lookup www.googleapis.com on [::1]:53: read udp [::1]:52058->[::1]:53: read: connection refused
It doesn't look like your issue is authenticated related, but instead you are unable to either lookup or reach www.googleapis.com. Do you have a firewall that might be blocking the outgoing connection?

GitLab CI : My Test Job doesn't pickup the mysql container

I have a springboot application that is connected to a mysql database. I want to use the .gitlab-ci.yml to handle the pipeline. I created the following config file.
before_script:
- echo "Execute scripts which are required to bootstrap the application. !"
after_script:
- echo "Clean up activity can be done here !."
services:
- mysql
stages:
- build
- connect
- test
- package
- deploy
variables:
MAVEN_CLI_OPTS: "--batch-mode"
MAVEN_OPTS: "-Dmaven.repo.local=.m2/repository"
MYSQL_DATABASE_NAME: gyyconsortiumdb
MYSQL_DATABASE_SCHEMA: "$CI_PROJECT_DIR/src/main/resources/static/sql/gyyconsortiumdb.sql"
MYSQL_ROOT_PASSWORD: mysql
cache:
paths:
- .m2/repository/
- target/
build:
stage: build
image: maven:latest
script:
- mvn $MAVEN_CLI_OPTS clean compile
connect:
stage: connect
image: mysql
before_script:
- mysql --version
script:
- echo "create database $MYSQL_DATABASE_NAME;" | mysql --user=root --password="$MYSQL_ROOT_PASSWORD" --host=mysql
- mysql --user=root --password="$MYSQL_ROOT_PASSWORD" --host=mysql $MYSQL_DATABASE_NAME < $MYSQL_DATABASE_SCHEMA
- mysql --user=root --password="$MYSQL_ROOT_PASSWORD" --host=mysql -e "show databases; use $MYSQL_DATABASE_NAME; show tables;"
test:
stage: test
image: maven:latest
script:
- mvn $MAVEN_CLI_OPTS test
package:
stage: package
image: maven:latest
script:
- mvn $MAVEN_CLI_OPTS package
artifacts:
paths: [target/basecamp-0.0.1.war]
deploy_test:
stage: deploy
script:
- echo "######## To be defined ########"
environment: staging
deploy_prod:
stage: deploy
script:
- echo "######## To be defined ########"
only:
- master
environment: production
When the first job is running, it is pulling the mysql image from docker hub, and I don't know why? I thought the mysql should be built only on the connect job.
Also, after the connect job is success, the test job pull the mysql image again. And my test job doesn't recognize my database however the service was create successfully and I was able to see the database also created.
Build job console output:
Running with gitlab-runner 10.3.0 (5cf5e19a)
on docker-auto-scale (fa6cab46)
Using Docker executor with image maven:latest ...
Starting service mysql:latest ...
Pulling docker image mysql:latest ...
Using docker image mysql:latest ID=sha256:7d83a47ab2d2d0f803aa230fdac1c4e53d251bfafe9b7265a3777bcc95163755 for mysql service...
Waiting for services to be up and running...
Using docker image sha256:d6e999707ab00f954f4ab77c8ced4efce186099fb8318e1a0bc2f4fd8bb7bf6b for predefined container...
Pulling docker image maven:latest ...
Using docker image maven:latest ID=sha256:1f858e89a5843b0804a9e1498476a135f45e23a42fe673914f977e0882a6789e for build container...
Running on runner-fa6cab46-project-4899225-concurrent-0 via runner-fa6cab46-srm-1514081729-3d654341...
Cloning repository...
Cloning into '/builds/user/xxx'...
Checking out b9efb2da as 24-create-the-gitlab-ci-yml-file...
Skipping Git submodules setup
Checking cache for default...
Downloading cache.zip from http://runners-cache-3-internal.gitlab.com:444/runner/project/4899225/default
Successfully extracted cache
$ echo "Execute scripts which are required to bootstrap the application. !"
Execute scripts which are required to bootstrap the application. !
$ mvn $MAVEN_CLI_OPTS clean compile
[INFO] Scanning for projects...
[INFO]
[INFO] ------------------------------------------------------------------------
[INFO] Building xxx 0.0.1-SNAPSHOT
[INFO] ------------------------------------------------------------------------
[INFO]
[INFO] --- maven-clean-plugin:2.6.1:clean (default-clean) # xxx ---
[INFO] Deleting /builds/user/xxx/target
[INFO]
[INFO] --- maven-resources-plugin:2.6:resources (default-resources) # xxx ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Copying 4 resources
[INFO] Copying 108 resources
[INFO]
[INFO] --- maven-compiler-plugin:3.1:compile (default-compile) # xxx ---
[INFO] Changes detected - recompiling the module!
[INFO] Compiling 35 source files to /builds/user/xxx/target/classes
[WARNING] /builds/user/xxx/src/main/java/com//www/xxx/domain/Project.java: Some input files use unchecked or unsafe operations.
[WARNING] /builds/user/xxx/src/main/java/com/xxx/www/xxx/domain/Project.java: Recompile with -Xlint:unchecked for details.
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 6.111 s
[INFO] Finished at: 2017-12-24T02:18:07Z
[INFO] Final Memory: 30M/71M
[INFO] ------------------------------------------------------------------------
Running after script...
$ echo "Clean up activity can be done here !."
Clean up activity can be done here !.
Creating cache default...
.m2/repository/: found 1775 matching files
target/: found 197 matching files
Uploading cache.zip to http://runners-cache-3-
internal.gitlab.com:444/runner/project/4899225/default
Created cache
Job succeeded
Connect job console output
Running with gitlab-runner 10.3.0 (5cf5e19a)
on docker-auto-scale (e11ae361)
Using Docker executor with image mysql ...
Starting service mysql:latest ...
Pulling docker image mysql:latest ...
Using docker image mysql:latest ID=sha256:7d83a47ab2d2d0f803aa230fdac1c4e53d251bfafe9b7265a3777bcc95163755 for mysql service...
Waiting for services to be up and running...
Using docker image sha256:295a9e80fe6ae475bfeef8e318eb11db267d317fa6bc1ed8d72185dfd2adb8b7 for predefined container...
Pulling docker image mysql ...
Using docker image mysql ID=sha256:7d83a47ab2d2d0f803aa230fdac1c4e53d251bfafe9b7265a3777bcc95163755 for build container...
Running on runner-e11ae361-project-4899225-concurrent-0 via runner-e11ae361-srm-1514081811-5092ac06...
Cloning repository...
Cloning into '/builds/yimengael/basecamp'...
Checking out b9efb2da as 24-create-the-gitlab-ci-yml-file...
Skipping Git submodules setup
Checking cache for default...
Downloading cache.zip from http://runners-cache-5-internal.gitlab.com:444/runner/project/4899225/default
Successfully extracted cache
$ mysql --version
mysql Ver 14.14 Distrib 5.7.20, for Linux (x86_64) using EditLine wrapper
$ echo "create database $MYSQL_DATABASE_NAME;" | mysql --user=root --password="$MYSQL_ROOT_PASSWORD" --host=mysql
mysql: [Warning] Using a password on the command line interface can be insecure.
$ mysql --user=root --password="$MYSQL_ROOT_PASSWORD" --host=mysql $MYSQL_DATABASE_NAME < $MYSQL_DATABASE_SCHEMA
mysql: [Warning] Using a password on the command line interface can be insecure.
$ mysql --user=root --password="$MYSQL_ROOT_PASSWORD" --host=mysql -e "show databases; use $MYSQL_DATABASE_NAME; show tables;"
mysql: [Warning] Using a password on the command line interface can be insecure.
Database
information_schema
mydb
mysql
performance_schema
sys
Tables_in_mydb
message
persistent_logins
project
project_description
role
user
user_role
Running after script...
$ echo "Clean up activity can be done here !."
Clean up activity can be done here !.
Creating cache default...
.m2/repository/: found 1775 matching files
target/: found 197 matching files
Archive is up to date!
Created cache
Job succeeded
Test job failed :
2017-12-24 02:21:43.827 INFO 62 --- [ main] c.g.w.b.xxxWebApplicationTests : Starting xxxWebApplicationTests on runner-4e4528ca-project-4899225-concurrent-0 with PID 62 (started by root in /builds/yimengael/xxx)
2017-12-24 02:21:43.829 DEBUG 62 --- [ main] c.g.w.b.xxxWebApplicationTests : Running with Spring Boot v1.5.8.RELEASE, Spring v4.3.12.RELEASE
2017-12-24 02:21:43.829 INFO 62 --- [ main] c.g.w.b.xxxWebApplicationTests : The following profiles are active: dev
2017-12-24 02:21:45.101 INFO 62 --- [ main] o.s.w.c.s.GenericWebApplicationContext : Refreshing org.springframework.web.context.support.GenericWebApplicationContext#62e20a76: startup date [Sun Dec 24 02:21:45 UTC 2017]; root of context hierarchy
2017-12-24 02:21:49.515 INFO 62 --- [ main] f.a.AutowiredAnnotationBeanPostProcessor : JSR-330 'javax.inject.Inject' annotation found and supported for autowiring
2017-12-24 02:21:51.533 WARN 62 --- [ main] o.a.tomcat.jdbc.pool.ConnectionPool : maxIdle is larger than maxActive, setting maxIdle to: 50
2017-12-24 02:21:52.448 ERROR 62 --- [ main] o.a.tomcat.jdbc.pool.ConnectionPool : Unable to create initial connections of pool.
com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException: Unknown database 'mydb'
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
.....
2017-12-24 02:21:52.477 WARN 62 --- [ main] o.s.b.a.orm.jpa.DatabaseLookup : Unable to determine jdbc url from datasource
org.springframework.jdbc.support.MetaDataAccessException: Could not get Connection for extracting meta data; nested exception is
org.springframework.jdbc.CannotGetJdbcConnectionException: Could not get JDBC Connection; nested exception is com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException: Unknown database 'mydb'
at org.springframework.jdbc.support.JdbcUtils.extractDatabaseMetaData(JdbcUtils.java:339)
at org.springframework.jdbc.support.JdbcUtils.extractDatabaseMetaData(JdbcUtils.java:366)
at org.springframework.boot.autoconfigure.orm.jpa.DatabaseLookup.getDatabase(DatabaseLookup.java:72)
at org.springframework.boot.autoconfigure.orm.jpa.JpaProperties.determineDatabase(JpaProperties.java:139)
Could you please tell me why the test job does not pick up the mysql service the right way ?
Best,
G
When the first job is running, it is pulling the mysql image from
docker hub, and I don't know why?
Because you defined Mysql as a service. GitLab CI uses the services keyword to define what docker containers should be linked with your base image. So, it will be run for every stages.
Could you please tell me why the test job does not pick up the mysql
service the right way ?
Your database name in your gitlab config file is 'gyyconsortiumdb' which is different from what you defined in your spring config file as log shows you are trying to connect to 'mydb'
jdbc4.MySQLSyntaxErrorException: Unknown database 'mydb'
follow the link for more info
https://docs.gitlab.com/ce/ci/services/mysql.html

logstash-forwarder json invalid character

At some point my logstash forwarder used to work properly but at some point fail to start with this error message
Failed unmarshalling json: invalid character ']' looking for beginning of value logstash
This message appears only if i run forwarder with this command:
$ /opt/logstash-forwarder/bin/logstash-forwarder -config="/etc/logstash-forwarder"
If i run it as a service it fails without any error message, if i restart the service getting this
Restarting log shipper logstash-forwarder
start-stop-daemon: warning: failed to kill PID: No such process
...done.
The first time stopped forwarder working i found in syslog this message:
connection from UDP: [54.194.184.218]:63707->[10.38.143.181] Nov 21
12:41:38 logstash-forwarder[PID]: 2014/11/21 12:41:38.692517 Read
error looking for ack: EOF Nov 21 12:41:38 logstash-forwarder[PID]:
2014/11/21 12:41:38.692592 Setting trusted CA from file:
/etc/pki/tls/certs/logstash-forwarder.crt Nov 21 12:41:38
logstash-forwarder[PID]: 2014/11/21 12:41:38.692827 Connecting to
X.X.X.X:5000 (X.X.X.X) Nov 21 12:41:38 logstash-forwarder[PID]:
2014/11/21 12:41:38.694128 Failure connecting to X.X.X.X: dial tcp
X.X.X.X:5000: connection refused
At server side logstash and ES cluster seems to work fine..
I found many similar problems with mine but not any solution.