I'm working on my first node.js-socket.io project. Until now i coded only in PHP. In PHP it is common to close the mysql connection, when it is not needed any more.
My Question: Does it make sense to keep just one mysql-connection during server is running open, or should i handle this like PHP.
Info: In the happy hours i will have about 5 requests/seconds from socket clients and for almost all of them i have to make a mysql_crud.
Which one would you prefer?
io = require('socket.io').listen(3000); var mysql = require('mysql');
var connection = mysql.createConnection({
host:'localhost',user:'root',password :'pass',database :'myDB'
});
connection.connect(); // and never 'end' or 'destroy'
// ...
or
var app = {};
app.set_geolocation = function(driver_id, driver_location) {
connection.connect();
connection.query('UPDATE drivers set ....', function (err) {
/* do something */
})
connection.end();
}
...
The whole idea of Node.js is async io (that includes db queries).
And the rule with a mysql connection is that you can only have one query per connection at a time. So you either make a queue and have a single connection, as in the first option or create a connection each time as with option 2.
I personally would go with option 2, as opening and closing connections are not such a big overhead.
Here are some code samples to help you out:
https://codeforgeek.com/2015/01/nodejs-mysql-tutorial/
Related
I am trying to query a comments table from mysql database by language.
Whenever I query by language to fetch chinese comments it displays encoded gibberish characters. but whenever I use python to query, it works.
Cloud Platform: Google Cloud SQL
Database location: Google Cloud SQL
Programming Language: Nodejs
Below is my code
// Require process, so we can mock environment variables
const process = require('process');
const Knex = require('knex');
const express = require('express');
const app = express();
const config = {
user: process.env.SQL_USER,
password: process.env.SQL_PASSWORD,
database: process.env.SQL_DATABASE,
socketPath: `/cloudsql/${process.env.INSTANCE_CONNECTION_NAME}`
};
var knex = Knex({
client: 'mysql',
connection: config
});
app.get('/', (req, res) => {
knex.select('post')
.from('comment')
.where({
'language': 'zh'
}).limit(1).then((rows) => {
res.send(rows);
}).catch((err) => {
res.send(err);
});
});
This is my query result:
"post": "最白痴的部长ï¼æœ€åŸºæœ¬çš„常识和逻辑都没有。真丢人ï¼"
please help.....
The text "最白痴的部长ï¼æœ€åŸºæœ¬çš„常识和逻辑都没有。真丢人ï¼" is what you get if "最白痴的部长基本的常识和逻辑都没有。真丢人" is sent encoded as UTF-8, but is then read and decoded as windows-1252 character set.
There are several different places this mis-decoding could happen:
From the client to the application writing to the database when the data was first added
Between the application and MySQL when adding the data
Across a configuration change in MySQL that wasn't applied correctly.
Between MySQL and the application reading the data.
Between the application and the end client displaying the data to you.
To investigate, I suggest being systematic. Start by accessing the data using other tools, e.g. PHPMyAdmin or the mysql command line in Cloud Shell. If you see the right data, you know the issue is (4) or (5). If the database definitly has the wrong data in it, then it's (1), (2) or (3).
The most common place for this error to happen is (5), so I'll go into that a bit more. This is because often websites set the character set to something wrong, or not at all. To fix this, we must make the character set explicit. You can do this in express.js by adding:
res.set('Content-Type', 'text/plain; charset=utf-8')
Hi i've just read the docs of mysql package for nodejs. Lil bit not sure of how is the best practice to work with pooling.
var mysql = require('mysql');
var pool = mysql.createPool(...);
pool.getConnection(function(err, connection) {
// Use the connection
connection.query('SELECT something FROM sometable', function (error, results, fields) {
// And done with the connection.
connection.release();
// Handle error after the release.
if (error) throw error;
// Don't use the connection here, it has been returned to the pool.
});
});
Do we have to call release() method everytime we have performed query?
And one more.
What is the difference between using pool directly to perform the query vs. using getConnection method then perform the query?
Code using pool direcly:
var pool = mysql.createPool(...);
pool.query(...)
Using getConnection method then perform the query:
pool.getConnection(function(err, connection) {
connection.query(....);
});
If you ask for getting a connection, you basically reserve that connection for a little while. This is important for 2 reasons:
Only 1 query can be done on a connection at a time, never in parallel. So this prevents 2 things from using the same connection.
Transactions are connection-based and all queries within the transaction must happen on that connection object.
The mysql library would have no way to predict that you are 'done' your transaction, this is why you need to release it.
Aside: You should consider looking into mysql2 for a similar library that's more powerful, and use promises instead of this callback pattern.
Update based on comment
When you do query directly on the pool, the pool will automatically get the connection, run the query and release it for you.
This is useful if you just need to do a single query and don't care about transactions.
Context:
I'm building a web application that calls data from a large db (several millions of rows for table); sometimes a user can change his mind and call for new data before the query on the db has been completed.
Technical question:
I tried to kill the query in this cases using:
app.get("/data", function(req, res) {
if (req.query.killQuery == "true") {
con.query("KILL \"" + threadId + "\"", function(err) {
if (err) throw err;
console.log("I have interrupted the executing query for a new request");
giveData(req, res); //The function that will execute a new query
});
return;
}
giveData(req, res); //The function that will execute a new query
});
Now I have several doubts about this code:
I had to use a second connection to kill the thread of the first, since the first was unable to perform new queries before the first was completed. Is this a Node.js behaviour or is it the right way to do this kind of things?
The KILL thread_id statement closes the whole connection instead of stopping the single query. Again, is it Node.js behaviour, or is it MySQL itself? Should I really disconnect and reconnect to stop a query and start with an other?
If you have a modern version of MySQL, you can use KILL QUERY <threadId> instead which will only kill the currently executing query on that connection but leave the connection intact.
I am using felixge/node-mysql. Also I am using express-myconnection which prevents mysql timeout and in turn prevents killing of node server. What I am doing is logging the activities in mysql. The scenario is I have a file upload functionality once the file is uploaded I am performing different operations on the file. During every stage of processing I am logging those activities in database. This works fine if the file is small. If the file is large say 100 MB it takes some time to load so in the mean time the mysql server reconnects and creates a new connection but the logging code still uses the old reference. Error: Cannot enqueue Query after fatal error. So, my question is is there a way that i can use the new connection reference instead of the old one. There is a single function in which all the different phases of activities regarding file takes place. Any help is greatly appreciated. thanks
Hi #paul, if you have seen the gist link you can see that I have the upload.on('begin', function (fileInfo, reqs, resp) { } where I have logged the activity that file upload process has begin. Once the file is uploaded upload.on('end', function (fileInfo,request,response) { } is triggered. I am also logging some activity here. As I said in my question, if the file is big the upload takes time. In the mean time a new MySql connection is created but the insert query in 'end' event still refers to the old myconnection. So, I wanted to know how can I use the new mysql connection reference in this scenario? I hope this has explained the scenario better.
Actually, I decided to google your error for you, and after reading this thread: https://github.com/felixge/node-mysql/issues/832 I realized that you're not releasing the connection after the first query completes, and so the pool never tries to issue you a new one. You were correct that the stale connection might be the problem. here's how you fix that if it is:
upload.on('begin', function (fileInfo, reqs, resp) {
var fileType = resp.req.fields.file_type;
var originalFileName = fileInfo.name;
var renamedFilename = file.fileRename(fileInfo,fileType);
/*renaming the file */
fileInfo.name = renamedFilename;
/*start: log the details in database;*/
var utcMoment = conf.moment.utc();
var UtcSCPDateTime = new Date( utcMoment.format() );
var activityData = {
activity_type : conf.LIST_UPLOAD_BEGIN,
username : test ,
message : 'test has started the upload process for the file',
activity_datetime : UtcSCPDateTime
};
reqs.params.activityData = activityData;
req.getConnection(function(err,connection) {
var dbData = req.params.activityData;
var activity_type = dbData.activity_type;
console.dir("[ Connection ID:- ] "+connection.threadId+' ] [ Activity type:- ] '+activity_type);
var insertQuery = connection.query("INSERT INTO tblListmanagerActivityLog SET ? ",dbData, function(err, result) {
if (err) {
console.log("Error inserting while performing insert for activity "+activity_type+" : %s ",err );
} else {
console.log('Insert successfull');
}
/// Here is the change:
connection.release();
});
});
/*end: log the details in database;*/
});
I fixed mine. I always suspect that when errors occur along with my queries it has something to do with the MySQL80 Service being stopped in the background. In case other solutions failed. Try going to the task manager, head to the services, find MySQL80 and check if it is stopped, when it is, click start or set it as automatic so that it will start at the moment desktop is running.
For pool connection to work we need to comment out https://github.com/pwalczyszyn/express-myconnection/blob/master/lib/express-myconnection.js#L84 this line. Hope it helps anyone facing the same issue.
Also we can use single connection option.
I had the same issue. Came across one solution -
Re-Install MySQl and while doing so, in the configuration step, select "legacy encryption" option instead and finish the installation.
Hope this helps!
This question is mainly about the best practice of writing queries in nodejs. We had referred several tutorials, but were not able to reach a conclusion.
We have a node js API layer which is mainly used for reading and writing to database. Here is a sample code:
pool.query("update node SET changed = " + params.updationTime + " where nid = " + params.nid);
pool.query("update node_revision SET timestamp = " + params.updationTime +" where nid = " + params.nid);
pool.end();
Is this a correct way of writing code or should we write the sql queries in async format itself.
If your pool configuration allows more than one connection then likely both queries are executed in parallel. Type of call itself does not matter. This example takes 2 seconds to finish:
connection.query('select sleep(1)');
connection.query('select sleep(1)', function() { console.log('done!') });
As well as this one:
connection.query('select sleep(1)', function() {
connection.query('select sleep(1)', function() {
console.log('done!')
});
});
because mysql protocol itself is "sequential" (that is, client is allowed to send next query only after result of previous is fully received). Most async clients hide this limitation by queueing commands internally. In case of two connections, queries actually go in parallel:
connection1.query('select sleep(1)', function() { console.log('done1') });
connection2.query('select sleep(1)', function() { console.log('done2') });
"done1" and "done2" are both going to appear on screen in approximately 1 second
pool.query is a shorlcut for pool.getConnection() + connection.query() + connection.release() - see readme
When writing SQL queries in NodeJS, I cannot promote Knex.js enough!
Programatic way to build dynamic queries. (writing dynamic raw SQL strings is a very manual process)
Connection pools.
Transaction support.
String escaping.
And on and on.
For your specific question, you just make the queries and execute them (using callbacks or Promises), the Knex connection pool will handle all the pooling, and generally things will just work for you.
You'll like it, give it a try : )
I suggest you to use sails.js (http://sailsjs.org/#/) framework, which uses Waterline Query Language(http://sailsjs.org/#/documentation/concepts/ORM/Querylanguage.html) to retrieve data from mySQL/mongodb/Redis database.