I am creating a rest api that uses mysql as data base. My confusion is that should i connect to database in every request and release the connection at the end of the operation. Or should i connect the database at the start of the server and make it globally available and forget about releasing the connection
I would caution that neither option is quite wise.
The advantage of creating one connection for each request is that those connections can interact with your database in parallel, this is great when you have a lot of requests coming through.
The disadvantage (and the reason you might just create one connection on startup and share it) is obviously the setup cost of establishing a new connection each time.
One option to look into is connection pooling https://en.wikipedia.org/wiki/Connection_pool.
At a high level you can establish a pool of open connections on startup. When you need to make a request remove one of those connections from the pool, use it, and return it when done.
There are a number of useful Node packages that implement this abstraction, you should be able to find one if you look.
Related
I have an RDS instance hosting a mySQL database. Instance size is db.t2.micro
I also have an ExpressJS backend connecting to the mySQL RDS instance via a connection pool:
Additionally i have a mobile app, the client, feeding off the ExpressJS API.
The issue i'm facing is, either via the mobile app or via Postman, there are times where i get a 'Too many connections' error and therefore several requests fail:
On the RDS instance. On current activity i sometimes get 65 connections, showing it's reaching the limit. What i need clarity on is:
When 200 mobile app instances connect to the API, to the RDS instance, does it register as 200 connections or 1 connection from ExpressJS?
Is it normal to be reaching the RDS instance 65 connection limit?
Is this just a matter of me using db.t2.micro instance size which is not recommended for prod? Will upgrading the instance size resolve this issue?
Is there something i'm doing wrong with my requests?
Thank you and your feedback is appreciated.
If your app creates a connection pool of 100, that's the number of database connections it will try to open. It must be lower than your MySQL connection limit.
Typically connection pools open all the connections for the pool, so they are ready when a client calls the http API. The connections might normally be running no SQL queries, if there are not many clients using the API at a given moment. The database connections are nevertheless connected.
Sort of like when you ssh to a remote linux server but you just sit there at a shell prompt for a while before running any command. You're still connected.
You asked if a db.t2.micro instance was not recommended for production. Yes, I would agree with that. It's tempting to use the smallest instance possible to save money, but a db.t2.micro is too small for anything but light testing, in my opinion.
In fact, I would not use any t2 instance for production, regardless of size. The t2 type uses "burstable" performance. This means it can provide only brief periods of good performance. Once the instance depletes its performance credits, they recharge slowly, and while they recharge, the performance of that instance is very low. This is okay for testing, but not for production, if you expect to provide consistent performance at any time.
i have a database that thousands of users need to connect to (via ODBC) for very brief periods (it's a subscription licensing database for a win32 desktop app). They connect, get their approval to run and disconnect).
max_connections is set to 1000 but am not seeing the re-use i would expect server side. i.e. server currently has about 800 processes/connections sleeping (and another 200 connected to real data in other databases on the same server) .... yet a new attempt by a client app was rejected 'too many connections'.
What am i missing?
have increased the max_connections for now to 1500 but if that just means another 500 sleeping connections it's not a long term solution. pretty sure clients are disconnecting properly but am adding some diagnostics to the win32 app just in case.
MariaDB 10.3.11
with MySQL ODBC 5.3 ANSI Driver
It's normal to see a lot of sessions "Sleeping". That means the client is connected, but not executing a query at this moment. The client is likely doing other tasks, before or after running an SQL query. Just like if you are logged into a server with ssh, most of the time you're just sitting at the shell prompt not running any program.
It's up to you to design your clients to wait to connect until they need data, then disconnect promptly after getting their data. It's pretty common in apps that they connect to the database at startup, and remain connected. It's also pretty common in some frameworks to make multiple connections at startup, and treat them as a pool that can be used by multiple threads of the client app. It's your app, so you should configure this as needed.
Another thing to try is to enable the thread pool in the MariaDB server. See https://mariadb.com/kb/en/thread-pool-in-mariadb/
This is different from a client-side connection pool. The thread pool allows many thousands of clients to think they're connected, without allocating a full-blown thread in the MariaDB server for every single connection. When a client has something to query, at that time it is given one of the threads. When that client is done, it may continue to maintain a connection, but the thread in the MariaDB server is reallocated to a different client's request.
This is good for "bursty" workloads by many clients, and it sounds like your case might be a good candidate.
I want to test Node and Deno and try to redirect users via proxy to one MySQL DB.
How will it impact the database?
Can some timestamp conflicts via CRUD operations arise or does MySQL have some mechanism to cope with connections from multiple servers?
What about performance or memory footprint of the database in RAM? Will it be occupying the same amount of space as if there was only one server requesting the database to CRUD something?
What would happen if I added another server that will connect to the DB, for example, java or Go server?
It will virtually have no impact on the database other than having any other concurrent processes connecting to it.
This is not a deno issue but rather a database issue.
The exact same problems can happen even with your current single Node.js instance, because the nature of all systems these days is concurrent/parallel.
You might as well replace the Deno app with another Node.js instance, Java, etc. Or even your current Node.js app.
Data in a database can change once you loaded it to the client, and it is up to you to implement the code that will handle such scenarios.
The fact that MySQL is not "ACID" is neither negative nor relevant in and of itself because it is doesn't have context.
If you need complete absolute integrity on a registry make sure you lock it when you select it, but there will be a trade off.
I am connecting to a remote MySQL database in my node.js server code for a web app. Is there any advantage to using a connection pool when I only have a single instance of a node.js application server running?
Connection pools are per instance of application. When you connect to the db, you are doing it from that particular instance and hence the pool is in the scope of that instance. The advantage of creating a pool is that you don't create / close connections very often, as this is, in general, a very expensive process. Rather, you maintain a set of connections open, in idle state, ready to be used if there is a need.
Update
In node there is async.parallel() construct which allows you to launch a set of tasks in async manner. Immagine that those tasks represent each one a single query. If you have a single connection to use, each process should use that same one, and it will quickly become a bottelneck. Instead, if you have a pool of available connections, each task can use a separate connection until the pool is completely used. Check this for more detailed reference.
Is there anyway I could keep the DBI session active even after the script exits?
http://mysqlresources.com/documentation/perl-dbi/connect
Basically I need to call the perl (DBI) script multiple times with different parameters (decide pass/fail after it completes). Each time its called Perl is making new connection to Mysql and destroys while exiting which itself is adding considerable amount of delay.
Just wondering if there is any way I could store and use the session for future?
Your connection and its associated socket is process specific, so there's no way of keeping it alive after your process terminates.
You should be able to better tune your server so that connecting is faster. A common issue is doing a reverse IP lookup by enabling the skip-name-resolve configuration parameter in my.cnf.
Barring that, what you might do is use either MySQL Proxy to keep a pool of warm connections, or to combine all your various operations into a single script that can run several stages without terminating.