I want to develop an API in Node.JS with only one endpoint taking 2 parameters : a number and a datetime.
This endpoint will return the result of a request in a MySql database, in json.
But my problem is : I don't know if I need to define the models in my code. Indeed, my database is already created, I am connected to it and I only need to return the result of one SQL request with the 2 parameters.
According to me, I think there would be a solution to just call the database and directly return the result.
Is it possible ?
Thank you in advance !!
Not sure if I understand correctly the issue, but from the looks of it, maybe you are bound to some kind of ORM. In any case, most ORMs or the underlying database drivers allow you to send raw SQL queries to the MySQL server without the need for any kind of models or schemas.
For instance, using the mysql package from npm (sample taken from the official repo):
var mysql = require('mysql');
var connection = mysql.createConnection({
host : 'localhost',
user : 'me',
password : 'secret',
database : 'my_db'
});
connection.connect();
connection.query('SELECT 1 + 1 AS solution', function (error, results, fields) {
if (error) throw error;
console.log('The solution is: ', results[0].solution);
});
connection.end();
Hi i've just read the docs of mysql package for nodejs. Lil bit not sure of how is the best practice to work with pooling.
var mysql = require('mysql');
var pool = mysql.createPool(...);
pool.getConnection(function(err, connection) {
// Use the connection
connection.query('SELECT something FROM sometable', function (error, results, fields) {
// And done with the connection.
connection.release();
// Handle error after the release.
if (error) throw error;
// Don't use the connection here, it has been returned to the pool.
});
});
Do we have to call release() method everytime we have performed query?
And one more.
What is the difference between using pool directly to perform the query vs. using getConnection method then perform the query?
Code using pool direcly:
var pool = mysql.createPool(...);
pool.query(...)
Using getConnection method then perform the query:
pool.getConnection(function(err, connection) {
connection.query(....);
});
If you ask for getting a connection, you basically reserve that connection for a little while. This is important for 2 reasons:
Only 1 query can be done on a connection at a time, never in parallel. So this prevents 2 things from using the same connection.
Transactions are connection-based and all queries within the transaction must happen on that connection object.
The mysql library would have no way to predict that you are 'done' your transaction, this is why you need to release it.
Aside: You should consider looking into mysql2 for a similar library that's more powerful, and use promises instead of this callback pattern.
Update based on comment
When you do query directly on the pool, the pool will automatically get the connection, run the query and release it for you.
This is useful if you just need to do a single query and don't care about transactions.
We are using mysql module for node and i was just wondering if this approach is good or does it have any bad effects on our application, consider this situation
dbPool.getConnection(function(err, db) {
if(err) return err;
db.query()
Here i am calling the dbPool object and requesting a connection from the pool then using it. However i found another implementation (which is the one i am asking about) which uses the dbPool object directly like:
dbPool.query('select * from test where id = 1' function(err, rows) {})
so i was wondering what does the second implementation does exactly, does it automatically return a free connection and use it ? can explain what is happening exactly in the second case and if it has any effect + or - on my application ? Thank you.
So this is so what called callback chaining. In NodeJS you have a lot of asynchronous calls going around. But sometimes you want to do something when the connection is done with MySQL. That's why the getConnection functionality has a callBack feature.
dbPool.getConnection(function(err, db) {
if(err) return err;
db.query()
Is equal to this:
dbPool.query('select * from test where id = 1' function(err, rows) {})
dbPool.query() will wait for the connection to be open, you don't have to put all your queries inside the getConnection to make it work. This is why it also has a callBack feature.
Tell me if I'm wrong. I hope this solves your question.
This question is mainly about the best practice of writing queries in nodejs. We had referred several tutorials, but were not able to reach a conclusion.
We have a node js API layer which is mainly used for reading and writing to database. Here is a sample code:
pool.query("update node SET changed = " + params.updationTime + " where nid = " + params.nid);
pool.query("update node_revision SET timestamp = " + params.updationTime +" where nid = " + params.nid);
pool.end();
Is this a correct way of writing code or should we write the sql queries in async format itself.
If your pool configuration allows more than one connection then likely both queries are executed in parallel. Type of call itself does not matter. This example takes 2 seconds to finish:
connection.query('select sleep(1)');
connection.query('select sleep(1)', function() { console.log('done!') });
As well as this one:
connection.query('select sleep(1)', function() {
connection.query('select sleep(1)', function() {
console.log('done!')
});
});
because mysql protocol itself is "sequential" (that is, client is allowed to send next query only after result of previous is fully received). Most async clients hide this limitation by queueing commands internally. In case of two connections, queries actually go in parallel:
connection1.query('select sleep(1)', function() { console.log('done1') });
connection2.query('select sleep(1)', function() { console.log('done2') });
"done1" and "done2" are both going to appear on screen in approximately 1 second
pool.query is a shorlcut for pool.getConnection() + connection.query() + connection.release() - see readme
When writing SQL queries in NodeJS, I cannot promote Knex.js enough!
Programatic way to build dynamic queries. (writing dynamic raw SQL strings is a very manual process)
Connection pools.
Transaction support.
String escaping.
And on and on.
For your specific question, you just make the queries and execute them (using callbacks or Promises), the Knex connection pool will handle all the pooling, and generally things will just work for you.
You'll like it, give it a try : )
I suggest you to use sails.js (http://sailsjs.org/#/) framework, which uses Waterline Query Language(http://sailsjs.org/#/documentation/concepts/ORM/Querylanguage.html) to retrieve data from mySQL/mongodb/Redis database.
is there any easy way to change the database from mysql to mongoDB ?
or better any one suggest me good tutorial do it
is there any easy way to change the database from mysql to mongoDB ?
Method #1: export from MySQL in a CSV format and then use the mongoimport tool. However, this does not always work well in terms of handling dates of binary data.
Method #2: script the transfer in your language of choice. Basically you write a program that reads everything from MySQL one element at a time and then inserts it into MongoDB.
Method #2 is better than #1, but it is still not adequate.
MongoDB uses collections instead of tables. MongoDB does not support joins. In every database I've seen, this means that your data structure in MongoDB is different from the structure in MySQL.
Because of this, there is no "universal tool" for porting SQL to MongoDB. Your data will need to be transformed before it reaches MongoDB.
If you're using Ruby, you can also try: Mongify
It's a super simple way to transform your data from a RDBS to MongoDB without losing anything.
Mongify will read your mysql database, build a translation file for you and all you have to do is map how you want your data transformed.
It supports:
Auto updating IDs (to BSON ObjectID)
Updating referencing IDs
Type Casting values
Embedding tables into other documents
Before save filters (to allow changes to the data manually)
and much much more...
Read more about it at: http://mongify.com/getting_started.html
There is also a short 5 min video on the homepage that shows you how easy it is.
Here's what I did it with Node.js for this purpose:
var mysql = require('mysql');
var MongoClient = require('mongodb').MongoClient;
function getMysqlTables(mysqlConnection, callback) {
mysqlConnection.query("show full tables where Table_Type = 'BASE TABLE';", function(error, results, fields) {
if (error) {
callback(error);
} else {
var tables = [];
results.forEach(function (row) {
for (var key in row) {
if (row.hasOwnProperty(key)) {
if(key.startsWith('Tables_in')) {
tables.push(row[key]);
}
}
}
});
callback(null, tables);
}
});
}
function tableToCollection(mysqlConnection, tableName, mongoCollection, callback) {
var sql = 'SELECT * FROM ' + tableName + ';';
mysqlConnection.query(sql, function (error, results, fields) {
if (error) {
callback(error);
} else {
if (results.length > 0) {
mongoCollection.insertMany(results, {}, function (error) {
if (error) {
callback(error);
} else {
callback(null);
}
});
} else {
callback(null);
}
}
});
}
MongoClient.connect("mongodb://localhost:27017/importedDb", function (error, db) {
if (error) throw error;
var MysqlCon = mysql.createConnection({
host: 'localhost',
user: 'root',
password: 'root',
port: 8889,
database: 'dbToExport'
});
MysqlCon.connect();
var jobs = 0;
getMysqlTables(MysqlCon, function(error, tables) {
tables.forEach(function(table) {
var collection = db.collection(table);
++jobs;
tableToCollection(MysqlCon, table, collection, function(error) {
if (error) throw error;
--jobs;
});
})
});
// Waiting for all jobs to complete before closing databases connections.
var interval = setInterval(function() {
if(jobs<=0) {
clearInterval(interval);
console.log('done!');
db.close();
MysqlCon.end();
}
}, 300);
});
MongoVUE's free version can do this automatically for you.
It can connect to both databases and perform the import
I think one of the easiest ways is to export the MySQL database to JSON and then use mongorestore to import it to a MongoDB database.
Step 1: Export the MySQL database to JSON
Load the mysql dump file into a MySQL database if necessary
Open MySQL Workbench and connect to the MySQL database
Go to the Schema viewer > Select database > Tables > right-click on the name of the table to export
Select 'Table Data Export Wizard'
Set the file format to .json and type in a filename such as tablename.json
Note: All tables will need to be exported individually
Step 2: Import the JSON files to a MongoDB using the mongorestore command
The mongorestore command should be run from the server command line (not the mongo shell)
Note that you may need to provide the authentication details as well as the --jsonArray option, see the mongorestore docs for more information
mongoimport -d dbname -u ${MONGO_USERNAME} -p ${MONGO_PASSWORD} --authenticationDatabase admin -c collectionname --jsonArray --file tablename.json
Note: This method will not work if the original MySQL database has BLOBs/binary data.
I am kind of partial to TalendOpenStudio for those kind of migration jobs. It is an eclipse based solution to create data migration "scripts" in a visual way. I do not like visual programming, but this is a problem domain I make an exception.
Adrien Mogenet has create a MongoDBConnection plugin for mongodb.
It is probably overkill for a "simple" migration but ut is a cool tool.
Mind however, that the suggestion of Nix will probably save you time if it is a one-of migration.
You can use QCubed (http://qcu.be) framework for that. The procedure would be something like this:
Install QCubed (http://www.thetrozone.com/qcubed-installation)
Do the codegen on your database. (http://www.thetrozone.com/php-code-generation-qcubed-eliminating-sql-hassle)
Take your database offline from the rest of the world so that only one operation runs at a time.
Now write a script which will read all rows from all tables of the database and use the getJson on all objects to get the json. You can then use the data to convert to array and push it into the mongoDB!
If anyone's still looking for a solution, i found that the easiest way is to write a PHP script to connect to your SQL DB, retrieve the information you want using the usual Select statement, transform the information into JSON using the PHP JSON Encode functions and simply output your results to file or directly to MongoDB. It's actually pretty simple and straight forward, the only thing to do is to double check your output against a Json validator, you may have to use functions such as explode to replace certain characters and symbols to make it valid. I have done this before however i currently do not have the script at hand but from what i can remember it was literally half a page of code.
Oh also remember Mongo is a document store so some data mapping is required to get it to be acceptable with mongo.
For those coming to this with the same problem, you can check out this Github project. This is an ongoing development that will help you migrate data from MySQL database to MongoDB by simply running a simple command.
It will generate MongoDB Schemas in TypeScript so you can use them later in your project. Each MySQL table will be a MongoDB collection, and datatypes will be efficiently converted to their MongoDB compatibles.
The documentation for the same can be found in the project's README.md. Feel free to come in and contribute. Would like to help if need be.
If you are looking for a tool to do it for you, good luck.
My suggestion is to just pick your language of choice, and read from one and write to another.
If I could quote Matt Briggs (it solved my roblem one time):
The driver way is by FAR the most straight forward. The import/export tools are fantastic, but only if you are using them as a pair. You are in for a wild ride if your table includes dates and you try to export from the db and import into mongo.
You are lucky too, being in c#. We are using ruby, and have a 32million row table we migrated to mongo. Our ending solution was to craft an insane sql statement in postgres that output json (including some pretty kludgy things to get dates going properly) and piped the output of that query on the command line into mongoimport. It took an incredibly frustrating day to write, and is not the sort of thing that can ever really be changed.
So if you can get away with it, use ado.net with the mongo driver. If not, I wish you well :-)
(note that this is coming from a total mongo fanboi)
MySQL is very similar to other SQL databases, so I send You to the topić:
Convert SQL table to mongoDB document
You can use the following project.It requires solr like configuration file to be written.Its very simple and straight forward.
http://code.google.com/p/sql-to-mongo-importer/
Try this:
Automated conversion of MySQL dump to Mongo updates using simple r2n mappings.
https://github.com/virtimus/mysql2mongo