Silverstripe losing connection to MySQL server requiring a restart - mysql

Firstly please apologies for how vague this post may be but I'm completely at a loss as to what could be occurring here.
I've been having an issue with a site I've developed for a client where it seemingly, randomly loses connection to the MySQL server and requires a reboot of the server for the site to become available again.. I've a number of other SS sites in my portfolio and not a single other one has experienced this before.
The site can run for a few days or an hour or sometimes a couple of weeks before having this issue and I've not been able to pin down what actually triggers it, though it appears to be related to some kind of usage of the CMS. The client will report that it will lose the connection when he's trying to upload a photo or write a blog post. I've posted the error at the bottom of the post to not clutter things.
Thinking it was an issue with the server set up, I set up a brand new digital ocean droplet and migrated the site over to there, but the issue remains so it leads me to believe it's something within the framework or my site config (old server was running on CentOs and the new one is Ubuntu).
This is the only site I have set up using composer. A novice to composer, I followed the instructions on the SS site to achieve this. My composer.json is below. Does it look like I have any issues here?:
{
"name": "silverstripe/installer",
"description": "The SilverStripe Framework Installer",
"require": {
"php": ">=5.3.2",
"silverstripe/cms": "3.1.6",
"silverstripe/framework": "3.1.6",
"silverstripe-themes/simple": "*",
"silverstripe/blog": "dev-master",
"undefinedoffset/sortablegridfield": "dev-master"
},
"config": {
"process-timeout": 600
},
"minimum-stability": "dev"
}
I've not made any changes to the core. Nothing unusual in there but my _config.php is:
<?php
global $project;
$project = 'mysite';
global $databaseConfig;
$databaseConfig = array(
"type" => 'MySQLDatabase',
"server" => 'localhost',
"username" => 'root',
"password" => 'xxxxxx',
"database" => 'mysite',
"path" => '',
);
// Set the site locale
i18n::set_locale('en_US');
Director::set_environment_type('dev');
Security::setDefaultAdmin("fraser","xxxxxxx");
Can anyone think of any reason why the connection would be dropping to the MySQL server? Is there anything else I should check? Do you need any more information from me?
[Warning] mysqli::mysqli(): (HY000/2003): Can't connect to MySQL server on '127.0.0.1' (111)
GET /
Line 68 in /var/www/mysite/framework/model/MySQLDatabase.php
Source
59 * - password: The password to log on with
60 * - database: The database to connect to
61 * - timezone: (optional) The timezone offset. For example: +12:00, "Pacific/Auckland", or "SYSTEM"
62 */
63 public function __construct($parameters) {
64 if(!empty($parameters['port'])) {
65 $this->dbConn = new MySQLi($parameters['server'], $parameters['username'], $parameters['password'],
66 '', $parameters['port']);
67 } else {
68 $this->dbConn = new MySQLi($parameters['server'], $parameters['username'], $parameters['password']);
69 }
70
71 if($this->dbConn->connect_error) {
72 $this->databaseError("Couldn't connect to MySQL database | " . $this->dbConn->connect_error);
73 }
74
Trace
mysqli->mysqli(<filtered>,<filtered>,<filtered>)
MySQLDatabase.php:68
MySQLDatabase->__construct(Array)
DB.php:174
DB::connect(<filtered>)
main.php:127
{closure}(ErrorControlChain)
call_user_func(Closure,ErrorControlChain)
ErrorControlChain.php:125
ErrorControlChain->step()
ErrorControlChain.php:117
ErrorControlChain->execute()
main.php:154
require_once(/var/www/mysite/framework/main.php)
index.php:65
[User Error] Couldn't connect to MySQL database | Can't connect to MySQL server on '127.0.0.1' (111)
GET /
Line 598 in /var/www/mysite/framework/model/MySQLDatabase.php
Source
589 }
590
591 public function databaseError($msg, $errorLevel = E_USER_ERROR) {
592 // try to extract and format query
593 if(preg_match('/Couldn\'t run query: ([^\|]*)\|\s*(.*)/', $msg, $matches)) {
594 $formatter = new SQLFormatter();
595 $msg = "Couldn't run query: \n" . $formatter->formatPlain($matches[1]) . "\n\n" . $matches[2];
596 }
597
598 user_error($msg, $errorLevel);
599 }
600
601 /**
602 * Return a boolean type-formatted string
603 *
604 * #param array $values Contains a tokenised list of info about this data type
Trace
Couldn't connect to MySQL database | Can't connect to MySQL server on '127.0.0.1' (111)
MySQLDatabase.php:598
MySQLDatabase->databaseError(Couldn't connect to MySQL database | Can't connect to MySQL server on '127.0.0.1' (111))
MySQLDatabase.php:72
MySQLDatabase->__construct(Array)
DB.php:174
DB::connect(<filtered>)
main.php:127
{closure}(ErrorControlChain)
call_user_func(Closure,ErrorControlChain)
ErrorControlChain.php:125
ErrorControlChain->step()
ErrorControlChain.php:117
ErrorControlChain->execute()
main.php:154
require_once(/var/www/mysite/framework/main.php)
index.php:65

Related

Node.js GraphQL API Stops working as soon as I deploy it: "Error validating datasource `db`: the URL must start with the protocol `mysql://"

I build a GraphQL API with Apollo and Prisma ORM which is connected to my hosted MySQL Database (The Database has already content in it).
When I run it on my localhost everything works fine and I can query the Database with GraphQL statements.
As soon as I deploy my node.js project to DigitalOcean (auto deployed with GitHub) it stops working and I get the following error:
{
"errors": [
{
"message": "\nInvalid `prisma.content.findMany()` invocation in\n/workspace/src/schema.js:36:29\n\n 33 const resolvers = {\n 34 Query: {\n 35 memes: (parent, args) => {\nā†’ 36 return prisma.content.findMany(\n error: Error validating datasource `db`: the URL must start with the protocol `mysql://`.\n --> schema.prisma:7\n | \n 6 | provider = \"mysql\"\n 7 | url = env(\"DATABASE_URL\")\n | \n\nValidation Error Count: 1",
"locations": [
{
"line": 2,
"column": 3
}
],
"path": [
"memes"
],
"extensions": {
"code": "INTERNAL_SERVER_ERROR",
"exception": {
"clientVersion": "3.6.0",
"stacktrace": [
"Error: ",
"Invalid `prisma.content.findMany()` invocation in",
"/workspace/src/schema.js:36:29",
"",
" 33 const resolvers = {",
" 34 Query: {",
" 35 memes: (parent, args) => {",
"ā†’ 36 return prisma.content.findMany(",
" error: Error validating datasource `db`: the URL must start with the protocol `mysql://`.",
" --> schema.prisma:7",
" | ",
" 6 | provider = \"mysql\"",
" 7 | url = env(\"DATABASE_URL\")",
" | ",
"",
"Validation Error Count: 1",
" at cb (/workspace/node_modules/#prisma/client/runtime/index.js:38689:17)",
" at processTicksAndRejections (internal/process/task_queues.js:97:5)"
]
}
}
}
],
"data": null
}
Here is my schema.prisma file:
generator client {
provider = "prisma-client-js"
}
datasource db {
provider = "mysql"
url = env("DATABASE_URL")
}
...
The only thing that is different from the hosted project compared to the local project is that I put .env file and node_modules on the .gitignore file.
So it seems like the project is accessing the wrong DATABASE_URL, but how should the hosted project know the DATABASE_URL in my .env file when the .env file is on .gitignore?
Here is what I do:
Change the DATABASE_URL in my .env file to my local MySQL Database hosted on a docker container
Run npx prisma migrate dev --preview-feature to generate the migration files
Run git add .
Run git commit -m "New Commit"
Run DATABASE_URL=mysql://censored:censored#censored:3306/censored npx prisma migrate resolve --applied "my_migration_folder_name" --preview-feature which succeeds and tells me "Migration my_migration_folder_name marked as applied."
Run git push
I can see that the Migration is successfully created on my MySQL Database but as soon as I run the app and try to query the database it gives me that error.
The code has to be correct because it is working on my localhost even when querying the hosted MySQL Database.
I also double checked that the Model in the schema.prisma file is in sync with my hosted MySQL Database schema.
I'm running out of ideas on what I could try.
EDIT
I actually think it has something to do with the environment variables I set in the settings of my DigitalOcean application.
Before it was set to:
envs:
- key: DATABASE_URL
scope: RUN_AND_BUILD_TIME
value: ${db.DATABASE_URL}
Now I set it to:
envs:
- key: DATABASE_URL
scope: RUN_AND_BUILD_TIME
value: mysql://censored:cesnored#censored:3306/censored
I thought that this will fix the problem but now it tells me that the connection fails because of wrong database credentials even though it is the right link with the right credentials.
I fixed it by clicking "Force rebuild and deploy" on my digitalOcean app.

Can't connect to RDS via AWS.RDS.Signer

I'm trying to connect to my MySQL RDS from a Lambda via AWS.RDS.Signer with the following code and fake credentials:
1 const DB_REGION = 'ap-southeast-2a'
2 const DB_HOST = 'dbinstance.ddtev8utygt.ap-southeast-2.rds.amazonaws.com'
3 const DB_PORT = 3306
4 const DB_USER = 'anyuser'
5 const DB_NAME = 'anydb'
6
7 const signerOptions = {
8 region: DB_REGION,
9 hostname: DB_HOST,
10 port: DB_PORT,
11 username: DB_USER
12 }
13
14 const signer = new AWS.RDS.Signer(signerOptions)
15 const token = await signer.getAuthToken()
16
17 const config = {
18 host: DB_HOST,
19 user: DB_USER,
20 password: token, // "Password123"
21 database: DB_NAME,
22 ssl: 'Amazon RDS',
23 authPlugins: {
24 mysql_clear_password: () => () => token
25 }
26 }
but I always get this error
"Access denied for user 'anyuser'#'172.14.1.12' (using password: NO)"
I'm not entirely sure if that is needed for the AWS.RDS.Signer but I selected this option of my database:
Password and IAM database authentication
Authenticates using the database password and user credentials through AWS IAM users and roles.
NOTE: If I swap the password from token to "Password123" on line 20 I can successfully connect to my RDS.
Am I missing something here or does AWS.RDS.Signer only work with RDS Proxy?
By the way: the getAuthToken function gives me something like that (token truncated)
"dbinstance.ddtev8utygt.apsoutheast2.rds.amazonaws.com:3306/Action=connect&DBUser=anyuser&&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=ASIAU7VGXF6UCWYZCFEG%2F20210318%2Fap-southeast-2a%2Frds-db%2Faws4_request&X-Amz-Date=20210318T105145Z&X-Amz-Expires=900&X-Amz-Security-Token=IQoJb3JpZ2luX2VjEAsaDmFwLXNvdXRoZWFzdC0yIkgwRgIhAKg8ibwNJ4E3hSOuq7HtDFvqHxmTlpOUk3I6EH2%2B9VdOV3RQ%2F03xiVdvjhEBkHqEXHQ%3D&X-Amz-Signature=749d931f74873e6c2c0d4fec94f0743f42efd5aa95ca0ac0f05c4bef30e3bd4d&X-Amz-SignedHeaders=host"
I finally can connect to my RDS via Lamdba using IAM (aka AWS.RDS.Signer)
So what was the problem?
Short story:
I used the wrong region in my Policy and in the props for the AWS.RDS.Signer.
I assumed that the Availability zone (ap-southeast-2a) equals the region, but that's not true. The correct region is actually described when the AWS command line creds get created. To find out the region invoke cat ~/.aws/config from the terminal. My default region was actually region=ap-southeast-2.
Long story:
When I started with AWS.RDS.Signer I followed the instructions here
When it came to extracting the DB info for the Policy, I used this command
aws rds describe-db-instances --db-instance-identifier <MY INSTANCE NAME> --query "DBInstances[*].DbiResourceId" --region ap-southeast-2a
but I got this error Could not connect to the endpoint URL: "https://rds.ap-southeast-2a.amazonaws.com/"
After a bit of Googling, I realised that the region is not right. That gave me a hint to change the region in the policy and in the code of my Lambda function.
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"rds-db:connect"
],
"Resource": [
"arn:aws:rds-db:<my-region>:<my-account-id>:dbuser:<my-db-resource-id>/<my-db-username>"
]
}
]
}
Then I created an extra user (ssluser) in the DB to connect with the AWS.RDS.Signer token
CREATE USER 'ssluser' IDENTIFIED WITH AWSAuthenticationPlugin as 'RDS';
GRANT ALL PRIVILEGES ON <MY DB NAME>.* TO 'ssluser'#'%';
GRANT USAGE ON <MY DB NAME>.* TO 'ssluser'#'%' REQUIRE SSL;
The next step was to add the above policy to the EC2 instance, ssh into the instance, install the MySQL client yum install mysql and try to connect to the instance using the token
mysql --host=dbinstance.chteb5kjtggo.ap-southeast-2.rds.amazonaws.com --port=3306 --ssl-ca=rds-combined-ca-bundle.pem --enable-cleartext-plugin --user=ssluser --password=`aws rds generate-db-auth-token --hostname dbinstance.chteb5kjtggo.ap-southeast-2.rds.amazonaws.com --port 3306 --region ap-southeast-2 --username ssluser`
After successfully connect to the RDS without providing a password I only had to attach the policy to my Lambda and change the username and region in my Lambda code to
1 const DB_REGION = 'ap-southeast-2'
2 const DB_HOST = 'dbinstance.ddtev8utygt.ap-southeast-2.rds.amazonaws.com'
3 const DB_PORT = 3306
4 const DB_USER = 'ssluser'
5 const DB_NAME = 'anydb'
I hope that will help someone in the future.

Google cloud sql with php

I'm having a problem to connect my DB cloud sql with php.
I've tried every possible way to connect (pdo, mysqli & mysql)
none of them worked.
I've put an IPv4 for the could sql instance and authorized the compute engine IP in the allowed networks section.
When I'm trying to connect from the compute engine with
mysql --host=cloud_sql_IP --user=my_user --password
it's working and I'm able to see the tables.
On the php side I've put this code:
$db = mysql_connect(<CLOUD_SQL_IPV4>, 'root', '');
if (!$db){
die('Connect Error (' . mysql_error());
}
and I get "Connect Error (Permission denied)"
when I'm trying this way:
$conn = mysql_connect(":/cloudsql/<COMPUTE_INSTANCE>:<DB>", "root", "");
if (!$conn) {
die('Connect Error (' . mysql_error());
}
I'm getting:
"Connect Error (No such file or directory"
What else should I do?
Thanks!
Well after diggin into it for 2 days I managed to connect cloud DB
first run this for the permission fix (centos)
setsebool httpd_can_network_connect=1
And for the php it should be written this way:
new mysqli(
<IP_V4>, // host
'root', // username
'', // password
<DB_NAME>, // database name
null,
'/cloudsql/<GCE_INSTANCE>:<DB>'
);

I get error when I try to check my MySQLdb connector in Django shell

60 DATABASE_ENGINE = 'django.db.backends.mysql'
61 DATABASE_NAME = 'mysite'
62 DATABASE_USER = 'root'
63 DATABASE_PASSWORD = 'password'
64 DATABASE_HOST = '127.0.0.1'
65 DATABASE_PORT = '3306'
66
67 DATABASES = {
68 'default': {
69 'ENGINE': DATABASE_ENGINE,
70 'NAME': DATABASE_NAME,
71 'USER': DATABASE_USER,
72 'PASSWORD': DATABASE_PASSWORD,
73 'HOST': DATABASE_HOST,
74 'PORT': DATABASE_PORT,
75 }
76 }
Under the settings.py above in my django project,I tried to see if DB adapter(MySQLdb : http://sourceforge.net/projects/mysql-python/) is working OK with my django configuration.
But I ended up with meeting some kind of error I have never seen before like this.
python manage.py shell
>>> from django.db import connection
>>> a = connection()
Traceback (most recent call last):
File "<console>", line 1, in <module>
TypeError: 'DefaultConnectionProxy' object is not callable
I installed mysql on my machine, of course, and since I've done this configuration before and it worked at that time, I am confused what did I have done it wrong this time.
I know what error messages would be like when the error is something due to django's settings.py configuration. But I totally don't get what's going on with this error.
Please help me.
"The object django.db.connection represents the default database connection. To use the database connection, call connection.cursor() to get a cursor object. Then, call cursor.execute(sql, [params]) to execute the SQL and cursor.fetchone() or cursor.fetchall() to return the resulting rows."
https://docs.djangoproject.com/en/1.7/topics/db/sql/#executing-custom-sql-directly
So, if you need connection object you should do:
connection = django.db.connection
Django use persistent connection to database, so you no need to initialize it.

Intentionally get a "MySQL server has gone away" error

I'm trying to cope with MySQL's error MySQL server has gone away in a django env.
The quick workaround was to set the global wait_timeout MySQL variable to a huge value, but in the long run this would accumulate to many open connections.
I figured I'll get the wait_timeout variable and poll the server in smaller intervals. After implementing this I tried to test it but am failing to get the error.
I set global wait_timeout=15 and even set global interactive_timeout=15 but the connection refuses to disappear. I'm sure I'm polling the database in larger intervals than 15sec.
What could be the cause for not being able to recreate this error?
Run below dirty-and-quick script and test your django project or whatever. I think this is not an elegant approach, but it works well.
<?php
mysql_connect( "127.0.0.1", "id", "pw" ); // should be changed to yours
while(1)
{
$r = mysql_query( "SHOW PROCESSLIST" );
while( $d = mysql_fetch_row( $r ) )
{
if( $d[7] != "SHOW PROCESSLIST" )
{
mysql_query( "KILL ". $d[0] );
echo( $d[0]." was killed.\n" );
}
}
}
?>
And here is the result.
mysql> select * from foo;
ERROR 2006 (HY000): MySQL server has gone away
No connection. Trying to reconnect...
Connection id: 337
Current database: test
ERROR 2006 (HY000): MySQL server has gone away
No connection. Trying to reconnect...
Connection id: 338
Current database: test
ERROR 2006 (HY000): MySQL server has gone away
mysql>
I've made a few modifications to #lqez code to be used with mysqli:
<?php
$mysql = new mysqli("127.0.0.1", "user", "password", "database"); //should be changed to yours
while (1) {
$r = $mysql->query("SHOW PROCESSLIST");
while ($d = $r->fetch_row()) {
if ($d[7] != "SHOW PROCESSLIST") {
$mysql->query("KILL " . $d[0]);
echo($d[0] . " was killed.\n");
}
}
}
?>