Suddenly my instance stopped responding to my incoming connections for mysql connections - mysql

I had installed mysql on gcp instance for dev and test purposes. Everything was working perfect but suddenly started getting connection timed out error. Firewall is opened on for MySQL connections. It was working perfectly, suddenly it stopped.
Here is the log for the connection
{
inserted: "xxxxxx"
jsonPayload: {
connection: {
dest_ip: "10.142.0.2"
dest_port: 3306
protocol: 6
src_ip: "54.87.222.27"
src_port: 52638
}
disposition: "ALLOWED"
instance: {
project_id: "xxxxxxxx"
region: "us-east1"
vm_name: "stockarea-server"
zone: "us-east1-b"
}
remote_location: {
city: "Ashburn"
continent: "America"
country: "usa"
region: "Virginia"
}
rule_details: {
action: "ALLOW"
direction: "INGRESS"
ip_port_info: [
0: {
ip_protocol: "TCP"
port_range: [
0: "3306"
]
}
1: {
ip_protocol: "UDP"
port_range: [
0: "3306"
]
}
]
priority: 1000
reference: "network:default/firewall:heroku-sql"
source_range: [
0: "0.0.0.0/0"
]
}
vpc: {
project_id: "xxxxxxxx"
subnetwork_name: "default"
vpc_name: "default"
}
}
logName: "projects/pxxxxxxx/logs/compute.googleapis.com%2Ffirewall"
receiveTimestamp: "2020-05-22T18:25:57.341545693Z"
resource: {
labels: {
location: "us-east1-b"
project_id: "xxxxxxxx"
subnetwork_id: "5137290941342062105"
subnetwork_name: "default"
}
type: "gce_subnetwork"
}
timestamp: "2020-05-22T18:25:45.934585080Z"
}
I have MySQL users having permission to get access for different IP address:
+------------------+-----------+
| user | host |
+------------------+-----------+
| client | % |
| stockarea | % |
| debian-sys-maint | localhost |
| mysql.session | localhost |
| mysql.sys | localhost |
| root | localhost |
UPDATE:
I saw MySQL error.log file and saw somebody attacked and has somehow crashed the database. It was throwing error packet getting out of order. I reinstalled the MySQL for now to solve this and continue my development.

Hi I tried to ping your external IP and it works. So all 80, 443, and 3306. I believe your firewall is setup correctly, Please make sure your service is running and listening to those ports.
And finally, please be aware that you should sanitize private inforamtion such as ip address and project ID in public forum

Related

Cannot connect to AWS RDS MySql using IAM from EC2 instance

Using these commands
bash $ RDSHOST="xxxxxxx.mydbid.us-xxst-x.rds.amazonaws.com"
bash $ TOKEN="$(aws rds generate-db-auth-token --hostname $RDSHOST --port 3306 --region us-xxst-x --username some_name )"
bash $ mysql --host=$RDSHOST --port=3306 --ssl-ca=rds-ca-2019-root.pem --user=some_name --password=$TOKEN
I get this error
ERROR 1045 (28000): Access denied for user 'some_name'#'123.123.123.123' (using password: YES)
When logged into the database and running
SELECT USER, PLUGIN, HOST FROM mysql.user WHERE USER LIKE'%some_name%';
I get this:
+-----------+-------------------------+------+
| user | plugin | host |
+-----------+-------------------------+------+
| some_name | AWSAuthenticationPlugin | % |
+-----------+-------------------------+------+
1 row in set (0.02 sec)
This is the policy applied to the role
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"rds-db:connect"
],
"Resource": [
"arn:aws:rds-db:us-xxxx-x:123123123123:dbuser:db-QPASDASDASDASDASDASDASDASD/admin_user",
"arn:aws:rds-db:us-xxxx-x:123123123123:dbuser:db-QPASDASDASDASDASDASDASDASD/some_body",
"arn:aws:rds-db:us-xxxx-x:123123123123:dbuser:db-QPASDASDASDASDASDASDASDASD/some_name"
]
}
}
Also, IAM is enabled on the RDS database. The certificate .pem file is in the same folder as where I'm running the commands.
I've been over this guide, and this guide, and this guide.

ETIMEDOUT connecting to MySql from Lambda

I am trying to connect to a MySql RDS instance from a Lambda function and getting an ETIMEDOUT error
The Lambda is not part of a VPC
The RDS instance is available publicly, I can connect to it from my laptop using MySqlWorkbench
The RDS instance's security group has inbound rules configured for all ports and 0.0.0.0/0
The Lambda's execution role has many policies (probably too many) including RDSFullAccess, LambdaVPCAccessExecutionRole, ec2:*, even AdministratorAccess!)
Again, the code executes locally, connects to and queries the RDS instance just fine. Executing the same code in Lambda throws the ETIMEDOUT error
Similar posts are resolved by adding the Lambda to the RDS instance's VPC, or by configuring the inbound rules on the database's security group. Nothing seems to work.
Since I can connect to the database from my laptop just fine, my hunch is that it's a problem with the Lambda
Are there additional policies I should attach to the Lambda's execution role?
Is there any other reason the Lambda would time out connecting to a publicly available database?
Additional info:
The Lambda is not running in a VPC. It runs on Node and connects to MySql using the mysql package v2.18.1 and is deployed using Serverless with the following config:
foo:
handler: functions/handlers.foo
timeout: 20
events:
- http:
path: /path/{pathParameter}/foo
method: get
cors: true
caching:
enabled: true
ttlInSeconds: 3600
cacheKeyParameters:
- name: request.path.pathParameter
In the Lambda I try connecting with this function (which, again, works fine when I execute the function on my laptop):
function openDbConnection() {
let connection = mysql.createConnection({
host: 'db-name.cgwxrjuo6oyd.us-east-1.rds.amazonaws.com',
user: process.env.DB_USER,
password: process.env.DB_PASS,
database: 'db-name'
});
try {
connection.connect(function(err) {
console.log("Database is ", connection.state)
if (err) {
return console.error('error: ' + err.message);
}
console.log('Connected to the MySQL server.');
});
} catch (error) {
console.log("Database is ", connection.state)
console.log("Error connecting to MySql: ", error);
}
return connection;
}
The database username and password are retrieved from environment variables that are published to the Lambda with Serverless using a .env.yml file:
provider:
name: aws
runtime: nodejs12.x
lambdaHashingVersion: '20201221'
environment: ${file(.env.yml):}
Below are the Cloudwatch logs for a single execution, which I'm having trouble making sense of. Entries appear out of sequence:
| timestamp | message |
|---------------|----------------------------------------------------------------------------------------------------------------------------------------------------------|
| 1626209636788 | START RequestId: e3e3ceb7-bb55-4c3e-8392-38e08401f679 Version: $LATEST |
| 1626209636791 | 2021-07-13T20:53:56.790Z adfc1cd5-e4be-40b7-970c-d38acabeb199 INFO Database is disconnected |
| 1626209636791 | 2021-07-13T20:53:56.791Z adfc1cd5-e4be-40b7-970c-d38acabeb199 ERROR error: connect ETIMEDOUT |
| 1626209636791 | 2021-07-13T20:53:56.791Z adfc1cd5-e4be-40b7-970c-d38acabeb199 INFO An error occured querying MySql: connect ETIMEDOUT |
| 1626209636792 | 2021-07-13T20:53:56.792Z adfc1cd5-e4be-40b7-970c-d38acabeb199 INFO Database is disconnected |
| 1626209636792 | 2021-07-13T20:53:56.792Z adfc1cd5-e4be-40b7-970c-d38acabeb199 ERROR error: Connection lost: The server closed the connection. |
| 1626209636793 | 2021-07-13T20:53:56.792Z adfc1cd5-e4be-40b7-970c-d38acabeb199 INFO An error occured querying MySql: Connection lost: The server closed the connection. | 1626209636803 | END RequestId: e3e3ceb7-bb55-4c3e-8392-38e08401f679 |
| 1626209636803 | REPORT RequestId: e3e3ceb7-bb55-4c3e-8392-38e08401f679 Duration: 9.32 ms Billed Duration: 10 ms Memory Size: 1024 MB Max Memory Used: 79 MB |
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------

Why does truffe migration command doesn't work ? for me

Could not connect to your Ethereum client with the following parameters: - host > 127.0.0.1 - port > 7545 - network_id > * Please check that your Ethereum client: - is running - is accepting RPC connections (i.e., "--rpc" option is used in geth) - is accessible over the network - is properly configured in your Truffle configuration file (truffle-config.js)
Truffle v5.3.9 (core: 5.3.9) Node v16.3.0
This is from this video https://youtu.be/XLahq4qyors?t=1385
AND yes my ganache is opened , my truffle-config.json is configured properly with correct port and all tried other solutions from this work but unfortunately it did not work
My config
require('babel-polyfill');
networks: {
development: {
host: "127.0.0.1",
port: 7545,
network_id: "*" // Match any network id
},
},
contracts_directory: './src/contracts/',
contracts_build_directory: './src/abis/',
compilers: {
solc: {
optimizer: {
enabled: true,
runs: 200
},
evmVersion: "petersburg"
}
}
}```

Can't connect to RDS via AWS.RDS.Signer

I'm trying to connect to my MySQL RDS from a Lambda via AWS.RDS.Signer with the following code and fake credentials:
1 const DB_REGION = 'ap-southeast-2a'
2 const DB_HOST = 'dbinstance.ddtev8utygt.ap-southeast-2.rds.amazonaws.com'
3 const DB_PORT = 3306
4 const DB_USER = 'anyuser'
5 const DB_NAME = 'anydb'
6
7 const signerOptions = {
8 region: DB_REGION,
9 hostname: DB_HOST,
10 port: DB_PORT,
11 username: DB_USER
12 }
13
14 const signer = new AWS.RDS.Signer(signerOptions)
15 const token = await signer.getAuthToken()
16
17 const config = {
18 host: DB_HOST,
19 user: DB_USER,
20 password: token, // "Password123"
21 database: DB_NAME,
22 ssl: 'Amazon RDS',
23 authPlugins: {
24 mysql_clear_password: () => () => token
25 }
26 }
but I always get this error
"Access denied for user 'anyuser'#'172.14.1.12' (using password: NO)"
I'm not entirely sure if that is needed for the AWS.RDS.Signer but I selected this option of my database:
Password and IAM database authentication
Authenticates using the database password and user credentials through AWS IAM users and roles.
NOTE: If I swap the password from token to "Password123" on line 20 I can successfully connect to my RDS.
Am I missing something here or does AWS.RDS.Signer only work with RDS Proxy?
By the way: the getAuthToken function gives me something like that (token truncated)
"dbinstance.ddtev8utygt.apsoutheast2.rds.amazonaws.com:3306/Action=connect&DBUser=anyuser&&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=ASIAU7VGXF6UCWYZCFEG%2F20210318%2Fap-southeast-2a%2Frds-db%2Faws4_request&X-Amz-Date=20210318T105145Z&X-Amz-Expires=900&X-Amz-Security-Token=IQoJb3JpZ2luX2VjEAsaDmFwLXNvdXRoZWFzdC0yIkgwRgIhAKg8ibwNJ4E3hSOuq7HtDFvqHxmTlpOUk3I6EH2%2B9VdOV3RQ%2F03xiVdvjhEBkHqEXHQ%3D&X-Amz-Signature=749d931f74873e6c2c0d4fec94f0743f42efd5aa95ca0ac0f05c4bef30e3bd4d&X-Amz-SignedHeaders=host"
I finally can connect to my RDS via Lamdba using IAM (aka AWS.RDS.Signer)
So what was the problem?
Short story:
I used the wrong region in my Policy and in the props for the AWS.RDS.Signer.
I assumed that the Availability zone (ap-southeast-2a) equals the region, but that's not true. The correct region is actually described when the AWS command line creds get created. To find out the region invoke cat ~/.aws/config from the terminal. My default region was actually region=ap-southeast-2.
Long story:
When I started with AWS.RDS.Signer I followed the instructions here
When it came to extracting the DB info for the Policy, I used this command
aws rds describe-db-instances --db-instance-identifier <MY INSTANCE NAME> --query "DBInstances[*].DbiResourceId" --region ap-southeast-2a
but I got this error Could not connect to the endpoint URL: "https://rds.ap-southeast-2a.amazonaws.com/"
After a bit of Googling, I realised that the region is not right. That gave me a hint to change the region in the policy and in the code of my Lambda function.
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"rds-db:connect"
],
"Resource": [
"arn:aws:rds-db:<my-region>:<my-account-id>:dbuser:<my-db-resource-id>/<my-db-username>"
]
}
]
}
Then I created an extra user (ssluser) in the DB to connect with the AWS.RDS.Signer token
CREATE USER 'ssluser' IDENTIFIED WITH AWSAuthenticationPlugin as 'RDS';
GRANT ALL PRIVILEGES ON <MY DB NAME>.* TO 'ssluser'#'%';
GRANT USAGE ON <MY DB NAME>.* TO 'ssluser'#'%' REQUIRE SSL;
The next step was to add the above policy to the EC2 instance, ssh into the instance, install the MySQL client yum install mysql and try to connect to the instance using the token
mysql --host=dbinstance.chteb5kjtggo.ap-southeast-2.rds.amazonaws.com --port=3306 --ssl-ca=rds-combined-ca-bundle.pem --enable-cleartext-plugin --user=ssluser --password=`aws rds generate-db-auth-token --hostname dbinstance.chteb5kjtggo.ap-southeast-2.rds.amazonaws.com --port 3306 --region ap-southeast-2 --username ssluser`
After successfully connect to the RDS without providing a password I only had to attach the policy to my Lambda and change the username and region in my Lambda code to
1 const DB_REGION = 'ap-southeast-2'
2 const DB_HOST = 'dbinstance.ddtev8utygt.ap-southeast-2.rds.amazonaws.com'
3 const DB_PORT = 3306
4 const DB_USER = 'ssluser'
5 const DB_NAME = 'anydb'
I hope that will help someone in the future.

No remote stream using kurento docker image with kurento hello world example on host

I installed a KMS container on my server and I downloaded kurento hello world java application on my server but when I go to my java web application using my server IP adress I have to remote stream and and the following error (in firefox):
ICE failed, see about:webrtc for more details
in the about:webrtc It tells me that there is no STUN and no TURN server specified (and a lot of following output not very clear to me) The problem is that I specified a STUN server on the WebRtcEndpoint.conf.ini.
Here is my docker-compose.yml file:
kurento:
image: fiware/stream-oriented-kurento:latest
volumes:
- ./kurento.conf.json:/etc/kurento/kurento.conf.json:ro
- ./defaultCertificate.pem:/etc/kurento/defaultCertificate.pem:ro
- ./WebRtcEndpoint.conf.ini:/etc/kurento/modules/kurento/WebRtcEndpoint$
ports:
- "8888:8888"
- "8433:8433"
here is my kurento.conf.json file:
{
"mediaServer" : {
"resources": {
// //Resources usage limit for raising an exception when an object creatio$
// "exceptionLimit": "0.8",
// // Resources usage limit for restarting the server when no objects are $
// "killLimit": "0.7",
// Garbage collector period in seconds
"garbageCollectorPeriod": 240
},
"net" : {
"websocket": {
"port": 8888,
"secure": {
"port": 8433,
"certificate": "defaultCertificate.pem",
"password": ""
},
//"registrar": {
// "address": "ws://localhost:9090",
// "localAddress": "localhost"
//},
"path": "kurento",
"threads": 10
}
}
}
}
and my WebRtcEndpoint.conf.ini
; Only IP address are supported, not domain names for addresses
; You have to find a valid stun server. You can check if it works
; usin this tool:
; http://webrtc.github.io/samples/src/content/peerconnection/trickle-ice/
stunServerAddress=62.71.2.168
stunServerPort=3478
; turnURL gives the necessary info to configure TURN for WebRTC.
; 'address' must be an IP (not a domain).
; 'transport' is optional (UDP by default).
; turnURL=user:password#address:port(?transport=[udp|tcp|tls])
;pemCertificate is deprecated. Please use pemCertificateRSA instead
;pemCertificate=<path>
;pemCertificateRSA=<path>
;pemCertificateECDSA=<path>
and the certificate has been generated with :
certtool --generate-privkey --outfile defaultCertificate.pem
echo 'organization = your organization name' > certtool.tmpl
certtool --generate-self-signed --load-privkey defaultCertificate.pem \
--template certtool.tmpl >> defaultCertificate.pem
sudo chown kurento defaultCertificate.pem
and I went on my https://localhost:8433/kurento to validate the certificate
When I start the kurento container with docker-compose up I can see on the logs that my conf. file has been loaded:
kurento_1 | "websocket":
kurento_1 | {
kurento_1 | "port": "8888",
kurento_1 | "secure":
kurento_1 | {
kurento_1 | "port": "8433",
kurento_1 | "certificate":
"defaultCertificate.pem",
kurento_1 | "password": ""
kurento_1 | },
kurento_1 | "path": "kurento",
kurento_1 | "threads": "10"
kurento_1 | }
.....
kurento_1 | "WebRtcEndpoint":
kurento_1 | {
kurento_1 | "stunServerAddress": "62.71.2.168",
kurento_1 | "stunServerPort": "3478",
kurento_1 | "configPath":
"\/etc\/kurento\/modules\/kurento"
kurento_1 | },
and I start the hello world example with :
sudo mvn compile exec:java -Dkms.url=wss://localhost:8433/kurento
at this point everything seems to work OK, no error output.
When I try to access my web application from a client with https://:8443 the web page is loaded correctly and can start the stream. But I have no remote stream and have the error I printed at the beginning.
UPDATE 1
I changed the version of the kurento image in docker-compose.yml from
image: fiware/stream-oriented-kurento:latest
to:
image: fiware/stream-oriented-kurento:6.6.0
And now it is working sometimes. I have the same error (ICE failed, see about:webrtc for more details) but if I reload the page multiple time, it end up working after some reload. Any suggestion about what I am doing wrong?
UPDATE 2
I realized that when the web application start working (after multiple reload), the next time I access the web applicaiton, it will always work, until I restart the KMS. Then I have to reaload the page multiple time again to have the remote stream.
Now that I realized that, I tried again with image: fiware/stream-oriented-kurento:latest and it has the exact same behavior. I have to reload multiple time the page to make it work. I have no clue why is that, any idea?
Looking into your problem I feel the ICE candidates have not been created properly on both sides.
Have you configured the STUN / TURN in the webApplication (JS)?
If you haven't modify the example I believe they are not configured by default
Check into the options.configuration. Example:
var options = {
........ some options here ....
configuration: {
iceServers:[{
"url": "turn:xxx.xxx.xxx:port",
"username": "xxxxxx",
"credential": "xxxxxx"
}]
}
webRtcPeer = new kurentoUtils.WebRtcPeer.WebRtcPeerSendrecv(options,<callback-here>);
Can you provide logs for both the Firefox ICE candidates generation and KMS ICE generation?
In addition is KMS up and running in the same machine as the tutorial?
More information about kurento_utils used in the hello-world: http://doc-kurento.readthedocs.io/en/stable/mastering/kurento_utils_js.html#using-data-channels
General example of WebRTC configuration the STUN on the Web client side:
https://www.w3.org/TR/webrtc/#simple-peer-to-peer-example