Flask socket IO with Gunicorn, Monitor all existing Greenlets, because of a WORKER_TIME_OUT - gunicorn

So I have a Flask socket IO application start with Gunicorn with the worker class : geventwebsocket.gunicorn.workers.GeventWebSocketWorker and with 1 worker. And, I have find that in a particular case i am getting a [critical] WORKER_TIME_OUT and the API die ( there is probably other case ). I was able to reproduce the issue by doing that :
def test_get_all(self):
pool = ThreadPool(3)
entity_route = [self.API_ROUTE, self.API_ROUTE, self.API_ROUTE]
pool.map(self.get_entity, entity_route)
def get_entity(self, route):
rest_client.post(route, json={
"email": 'DEFAULT_USER_EMAIL',
"password": DEFAULT_USER_PASSWORD
}, status_code=200).json()
So i am calling the API_ROUTE 3 times in parallel. Inside the controller of API_ROUTE I am making a function call that update a field in an entity in the PSQL database and in a document in the Elastic Search instance. At that moment, The API froze, and after the default 30 sec timeout die. If i comment the call to ES or the PSQL. It pass no problem. I try to use the worker class eventlet and it fix it, but then other route failed, because it seem responses get mixed up. So i am not sure what worker class to use, because i need the web socket functionality.
I also try to use a lock around the function that call Elastic Search and PSQL. But it still fail. Something like this :
from gevent.threading import Lock
lock = Lock()
self.lock.acquire()
entity.update(**data)
self.lock.release()
If someone could point to me how to setup a monitoring on the greenlet with gunicorn and also explain to me what is happening it would be great. Also, the application is running in docker and kubernetes ( minikube locally )
Thank you !!

Related

How can I dynamically choose which MySQL server to point to?

TL;DR: Vertical or Horizontal scaling for this system design?
I have NGINX running as a load balancer for my application. It distributes across 4 EC2 (t2.micro's cuz I'm cheap) to route traffic and those are all currently hitting one server for my MySQL database (also a t2.micro, totalling 6 separate EC2 instances for the whole system).
I thinking about horizontally scale my database via Source/Replica distribution, and my thought is that I should route all read queries/GET requests (the highest traffic volume I'll get) to the Replicas and all write queries/POST requests to the Source db.
I know that I'll have to programmatically choose which DB my servers point to based on request method, but I'm unsure of how best to approach that or if I'm better off vertically scaling my DB at that point and investing in a larger EC2 instance.
Currently I'm connecting to the Source DB using an express server and it's handling everything. I haven't implemented the Source/Replica configuration just yet because I want to get my server-side planned out first.
Here's the current static connection setup:
const mysql = require('mysql2');
const Promise = require('bluebird');
const connection = mysql.createConnection({
host: '****',
port: 3306,
user: '****',
password: '*****',
database: 'qandapi',
});
const db = Promise.promisifyAll(connection, { multiArgs: true });
db.connectAsync().then(() =>
console.log(`Connected to QandApi as ID ${db.threadId}`)
);
module.exports = db;
What I want to happen is I want to either:
set up an express middleware function that looks at the request method and connects to the appropriate database by creating 2 configuration templates to put into the createConnection function (I'm unsure of how I would make sure it doesn't try to reconnect if a connection already exists, though)
if possible just open two connections simultaneously and route which database takes which method (I'm hopeful this option will work so that I can make things simpler)
Is this feasible? Am I going to see worse performance doing this than if I just vertically scaled my EC2 to something with more vCPUs?
Please let me know if any additional info is needed.
Simultaneous MySQL Database Connection
I would be hesitant to use any client input to connect to a server, but I understand how this could be something you would need to do in some scenarios. The simplest and quickest way around this issue would be to create a second database connection file. In order to make this dynamic, you can simply require the module based on conditions in your code, so sometimes it will be called and promised at only certain points, after certain conditions. This process could be risky and requires requiring modules in the middle of your code so it isn't ideal but can get the job done. Ex :
const dbConnection = require("../utils/dbConnection");
//conditional {
const controlledDBConnection = require("../utils/controlledDBConnection");
var [row] = await controlledDBConnection.execute("SELECT * FROM `foo`;")
}
Although using more files could potentially have an effect on space constraints and could potentially slow down code while waiting for a new promise, but the overall effect will be minimal. controlledDBConnection.js would just be something close to a duplicate to dbConnection.js with slightly different parameters depending on your needs.
Another path you can take if you want to avoid using multiple files is to export a module with a dynamically set variable from your controller file, and then import it into a standard connection file. This would allow you to change up your connection without rewriting a duplicate, but you will need diligent error checks and a default.
Info on modules in JS : https://javascript.info/import-export
Some other points
Use Environment Variables for your database information like host, etc. since this will allow for you to easily change information for your database all in one place, while also allowing you to include your .env file in .gitignore if you are using github
Here is another great stack overflow question/answer that might help with setting up a dynamic connection file : How to create dynamically database connection in Node.js?
How to set up .env files : https://nodejs.dev/learn/how-to-read-environment-variables-from-nodejs
How to set up .gitignore : https://stackabuse.com/git-ignore-files-with-gitignore/

Flask-SQLAlchemy connect to RDS-proxy over IAM

I have an application which needs to connect to and RDS (postgres) proxy with IAM. It makes use of the create_app method.
def create_app():
connex_app = connexion.App(__name__, specification_dir=base_apispec_dir)
connex_app.add_api("swagger.yaml", strict_validation=True)
app = connex_app.app
app.config.from_object(get_configuration())
ma.init_app(app)
db.init_app(app)
return connex_app
In this post there is an example of how to do this with SQLalchemy, but how do we connect with Flask-SQLAlchemy. In the example they use the #event.listen_for() event from SQLalchemy, but for that I need the engine, which I do not have.
It is possible to get it from the SQLalchemy object, but this gives to following error: No application found. Either work inside a view function or push.
Does anyone know how to make this connection working so the IAM-token can be refreshed every time it expires or just before that?

Server fails to launch in Google App Engine; OK in Localhost

I have a Flex App written in Go and React that is deployed to Google App engine. I would like it to interact with a MySql Database (2nd generation) on Google Cloud over a Unix socket. I believe the issue lies with the Go server not launching/responding to requests (see below for justification). The App is located at https://haveibeenexploited.appspot.com/
The project is simple. I have two routes in my Server:
server.go
package main
import (
"net/http"
"searchcontract"
)
func main() {
http.Handle("/", http.FileServer(http.Dir("./app/build")))
http.HandleFunc("/search", searchcontract.SearchContract)
http.ListenAndServe(":8080", nil)
}
The second route ("/search") is activated when a user hits the search button. Ideal behavior should return a row specifying the exploits available for the given "contract address" which React writes out to the screen.
searchcontract/searchcontract.go
//SearchContract is a handler that queries the DB for compromised contracts.
func SearchContract(w http.ResponseWriter, r *http.Request) {
var contractName contractID //Used for parsing in contractName
queryResult := getRow(&contractName.Name)
w.WriteHeader(200)
json.NewEncoder(w).Encode(queryResult)
}
//processRow queries the DB for a contract with ID value of name.
func getRow(contractName *string) *ContractVulnerabilityInfo {
var storage ContractVulnerabilityInfo //stores row to encode
//Login to database
...
scanErr := db.QueryRow("SELECT * FROM contracts WHERE ContractAddress=?;", &contractName).Scan(&storage.ContractAddress, &storage.IntegerOverflow, &storage.IntegerUnderflow, &storage.DOS, &storage.ExceptionState, &storage.ExternalCall, &storage.ExternalCallFixed, &storage.MultipleCalls, &storage.DelegateCall, &storage.PredictableEnv, &storage.TxOrigin, &storage.EtherWithdrawal, &storage.StateChange, &storage.UnprotectedSelfdestruct, &storage.UncheckedCall)
...
return &storage
}
My app.yaml file should allow me to deploy this flex app and does:
runtime: go1.12
env: flex
handlers:
- url: /.*
script: _server # my server.go file handles all endpoints
automatic_scaling:
max_num_instances: 1
resources:
cpu: 1
memory_gb: 0.5
disk_size_gb: 10
env_variables:
# user:password#unix(/cloudsql/INSTANCE_CONNECTION_NAME)/dbname
MYSQL_CONNECTION: root:root#unix(/cloudsql/haveibeenexploited:us-west1:hibe)/mythril
# https://cloud.google.com/sql/docs/mysql/connect-app-engine
beta_settings:
cloud_sql_instances: haveibeenexploited:us-west1:hibe
I am able to query the database successfully on localhost.Localhost correctly shows address
However, whenever I try to implement and push to AppEngine, when I query something that should be in the database, it does not show up in the remote App! App Engine does not show address in database. Furthermore, I get a status code of '0' returned, which indicates to me that the server function isn't even being called at all ('200' is what I expect if successful or some other error message.').
Summary
I can't wrap my head around this bug. What should work locally should work remotely. Also, I can't debug this app probably because Stackdriver does not support flex apps and the devserver Google Cloud provides does not support Go Apps.
I believe the primary issue is with Go not speaking to the React element correctly or the routing not being taken care of appropriately.
1) The problem does not lie with MySql connection/database access
- I changed my route to only be one page, turned off React, and included a hardcoded query. The result on localhost. The result on App Engine
2) There is an issue in either a) my routing or b) the interaction between React and Go.
3) Go seems to start correctly... at least when React is not started.
Any help is appreciated.
EDIT I believe that the go app indeed is still running, but the searchfunction is failing for whatever reason. The reason I believe this is because when I add another route for haveibeenexploited.com/hello, it works.

Google Cloud SQL No Response

We are running a Sails.js API on Google Container Engine with a Cloud SQL database and recently we've been finding some of our endpoints have been stalling, never sending a response.
I had a health check monitoring /v1/status and it registered 100% uptime when I had the following simple response;
status: function( req, res ){
res.ok('Welcome to the API');
}
As soon as we added a database query, the endpoint started timing out. It doesn't happen all the time, but seemingly at random intervals, sometimes for hours on end. This is what we have changed the query to;
status: function( req, res ){
Email.findOne({ value: "someone#example.com" }).then(function( email ){
res.ok('Welcome to the API');
}).fail(function(err){
res.serverError(err);
});
}
Rather suspiciously, this all works fine in our staging and development environments, it's only when the code is deployed in production that the timeout occurs and it only occurs some of the time. The only thing that changes between staging and production is the database we are connecting to and the load on the server.
As I mentioned earlier we are using Google Cloud SQL and the Sails-MySQL adapter. We have the following error stacks from the production server;
AdapterError: Invalid connection name specified
at getConnectionObject (/app/node_modules/sails-mysql/lib/adapter.js:1182:35)
at spawnConnection (/app/node_modules/sails-mysql/lib/adapter.js:1097:7)
at Object.module.exports.adapter.find (/app/node_modules/sails-mysql/lib/adapter.js:801:16)
at module.exports.find (/app/node_modules/sails/node_modules/waterline/lib/waterline/adapter/dql.js:120:13)
at module.exports.findOne (/app/node_modules/sails/node_modules/waterline/lib/waterline/adapter/dql.js:163:10)
at _runOperation (/app/node_modules/sails/node_modules/waterline/lib/waterline/query/finders/operations.js:408:29)
at run (/app/node_modules/sails/node_modules/waterline/lib/waterline/query/finders/operations.js:69:8)
at bound.module.exports.findOne (/app/node_modules/sails/node_modules/waterline/lib/waterline/query/finders/basic.js:78:16)
at bound [as findOne] (/app/node_modules/sails/node_modules/lodash/dist/lodash.js:729:21)
at Deferred.exec (/app/node_modules/sails/node_modules/waterline/lib/waterline/query/deferred.js:501:16)
at tryCatcher (/app/node_modules/sails/node_modules/waterline/node_modules/bluebird/js/main/util.js:26:23)
at ret (eval at <anonymous> (/app/node_modules/sails/node_modules/waterline/node_modules/bluebird/js/main/promisify.js:163:12), <anonymous>:13:39)
at Deferred.toPromise (/app/node_modules/sails/node_modules/waterline/lib/waterline/query/deferred.js:510:61)
at Deferred.then (/app/node_modules/sails/node_modules/waterline/lib/waterline/query/deferred.js:521:15)
at Strategy._verify (/app/api/services/passport.js:31:7)
at Strategy.authenticate (/app/node_modules/passport-local/lib/strategy.js:90:12)
at attempt (/app/node_modules/passport/lib/middleware/authenticate.js:341:16)
at authenticate (/app/node_modules/passport/lib/middleware/authenticate.js:342:7)
at Object.AuthController.login (/app/api/controllers/AuthController.js:119:5)
at bound (/app/node_modules/sails/node_modules/lodash/dist/lodash.js:729:21)
at routeTargetFnWrapper (/app/node_modules/sails/lib/router/bind.js:179:5)
at callbacks (/app/node_modules/sails/node_modules/express/lib/router/index.js:164:37)
Error (E_UNKNOWN) :: Encountered an unexpected error :
Could not connect to MySQL: Error: Pool is closed.
at afterwards (/app/node_modules/sails-mysql/lib/connections/spawn.js:72:13)
at /app/node_modules/sails-mysql/lib/connections/spawn.js:40:7
at process._tickDomainCallback (node.js:381:11)
Looking at the errors alone, I'd be tempted to say that we have something misconfigured. But the fact that it works some of the time (and has previously been working fine!) leads me to believe that there's some other black magic at work here. Our Cloud SQL instance is D0 (though we've tried upping the size to D4) and our activation policy is "Always On".
EDIT: I had seen others complain about Google Cloud SQL eg. this SO post and I was suspicious but we have since moved our database to Amazon RDS and we are still seeing the same issues, so it must be a problem with sails and the mysql adapter.
This issue is leading to hours of downtime a day, we need it resolved, any help is much appreciated!
This appears to be a sails issue, and not necessarily related to Cloud SQL.
Is there any way the QPS limit for Google Cloud SQL is being reached? See here: https://cloud.google.com/sql/faq#sizeqps
Why is my database instance sometimes slow to respond?
In order to minimize the amount you are charged for instances on per use billing plans, by default your instance becomes passive if it is not accessed for 15 minutes. The next time it is accessed there will be a short delay while it is activated. You can change this behavior by configuring the activation policy of the instance. For an example, see Editing an Instance Using the Cloud SDK.
It might be related to your policy setting. If you set it to ON_DEMAND, the instance will sleep to save your budget so that the first query to activate the instance is slow. This might cause the timeout.
https://cloud.google.com/sql/faq?hl=en

MySQL proxy redirect Read/Write

We have a system where we have a Master / Multiple Slaves .
Currently everything happens on the Master and the slaves are just here for backup .
We use Codeigniter as a development platform .
Now we decided to user the slaves for the Reads and the Master for the Write queries .
I have been told that this is not doable without modifying the source code because proxy can't know the type of the query .
Any idea how to proceed with this without causing too much damages for a perfectly working system ...
We will use this : http://dev.mysql.com/downloads/mysql-proxy/
It does exactly what we want :
More info here :
http://jan.kneschke.de/2007/8/1/mysql-proxy-learns-r-w-splitting/
http://www.infoq.com/news/2007/10/mysqlproxyrwsplitting
http://archive.oreilly.com/pub/a/databases/2007/07/12/getting-started-with-mysql-proxy.html
something i was also looking, few month back i did something like this but i added 3 web server with master slave mysql servers, first web server enabled with mod_proxy to redirect request to read and write server all request will come to this server, if post,put or delete request come to server it will go to write server, all get or normal request will go to read server
here you can find mod_proxy setting which i used
http://pastebin.com/a30BRHFq
here you can read about load balancing
http://www.rackspace.com/knowledge_center/article/simple-load-balancing-with-apache
still looking for better solution with less hardware involved
figure out another solution through CI, create two database connections in database.php file keep save mysql server as default database connection and other connection for write only server
you can use this base model extend
https://github.com/jamierumbelow/codeigniter-base-model
you need to extend your models with this model and need to extend you model with this, it has functionality for callbacks before and after insert,update, delete and get queries, only you need to add one custom method or callback change_db_group
//this method in MY_Model
function change_db_group{
$this->_database = $this->load->database('writedb', TRUE)
}
no your example model
class Example_Model extends MY_Model{
protected $_table = 'example_table';
protected $before_create = array('change_db_group');
protected $before_update = array('change_db_group');
protected $before_delete = array('change_db_group');
}
you database connection will be changed before executing insert,update or delete queries