Sails js: select records from mysql? - mysql

I am trying to select all records from table called fairCustomer where business_type_id equal array of value.
I am using sailsjs for server and mysql for DB, in frontend I am using nativescript with typescript.
and this is my code:
filterShopType(){
this.service.serviceName = 'fairCustomers';
this.service.Get({ populate:'country_id,business_type_id',where:{ business_type_id:{ in:this.business_ids_filter } }, limit:20 }).then((data: any) => {
this.fairCustomers.splice(0, this.fairCustomers.length);
this.fairCustomers.push(data);
this.Refresh('fairCustomers');
}).catch((err) => {
console.log('failed get fairCustomers from server ', err);
});
}
service refer to http.request(), where i am using it in another place in my code.
business_ids_filter is an array of ids.
When I run this code the I am getting this error:
"message": "Could not parse the provided where clause. Refer to the Sails documentation for up-to-date info on supported query language syntax:\n(http://sailsjs.com/documentation/concepts/models-and-orm/query-language)\nDetails: Unrecognized sub-attribute modifier (in) for business_type_id. Make sure to use a recognized sub-attribute modifier such as startsWith, <=, !, etc. )",
and if I removed where I got an error result.
please anyone have any idea or solution?

you may try with a native query as described here, https://sailsjs.com/documentation/reference/waterline-orm/datastores/send-native-query

As far as I can tell, you could simply provide an array of items. Matches against ANY of the items in the array means that the record should be returned as part of the result. Not sure if it works in a where-clause though.
Docs:Query Language (scroll to: In Modifier)
I do share your confusion as to why the given query does not work though, as the docs state that in should be valid, but perhaps it's not valid inside where:{...}? And I am assuming you are using .find() inside .Get(...)? Simply proxying the query unchanged through to .find()?
filterShopType(){
this.service.serviceName = 'fairCustomers';
this.service.Get({
populate:'country_id, business_type_id',
business_type_id: this.business_ids_filter,
limit:20
})
.then((data: any) => {
this.fairCustomers.splice(0, this.fairCustomers.length);
this.fairCustomers.push(data);
this.Refresh('fairCustomers');
}).catch((err) => {
console.log('failed get fairCustomers from server ', err);
});
}
Now, you of course need to make sure that the array actually is an array if you use in:.

It is working with me by using [or] instead of [in], this is the code
filterShopType(){
if(this.business_ids_filter){
this.service.serviceName = 'fairCustomers';
var arr = [];
this.business_ids_filter.forEach( (id) => {
arr.push({ business_type_id: id })
});
let where = {};
if(arr.length > 0){
where = {or: arr};
}
this.service.Get({ populate: 'country_id,business_type_id',where:where , limit:20 }).then((data: any) => {
this.filterItems.splice(0, this.filterItems.length);
this.filterItems.push(data);
this.Refresh('filterItems');
}).catch((err) => {
console.log('failed get fair Customers from server ', err);
});
}else{
this.set('isFilter', false);
}
}

Related

Sequelize raw queries TextRow and getting data out of it

Given this query here,
let output = [];
const sql = `select * from coredb.account LIMIT ${offset},${limit}`;
let data = await sequelize.query(sql, null, {raw: true, type: sequelize.QueryTypes.SELECT});
data.forEach((item) => {
console.log(item['id'], item.id); // <-- output says "undefined, undefined"
});
the data variable is indeed hydrated with the right row data when using console.log to inspect it.
But, when I try to access the individual properties, they only ever come back as undefined. This TextRow object that Sequelize seems to return the result in doesn't seem to want to let me access then explicit rows.
Just curious what i'm missing here, am I missing an option?
I agree, Sequalize raw queries are not intuitive. You don't need the null or raw: true flag. Something like this should work:
let data = await sequelize.query(sql, {type: sequelize.QueryTypes.SELECT});
When I tried this, "data" was an array of two objects, each being the query result. So, the properties can be accessed by using index [0].... e.g.
data[0].forEach((item) => {
console.log(item['id'], item.id); // <-- output says "undefined, undefined"
});
Not yet sure WHY this occurs!
EDIT - it's because .query() should have only two arguments. Changing the call to: sequelize.query(sql, {raw: true, type: sequelize.QueryTypes.SELECT}) resulted in data being a single array (as expected).
Finally I was able to find the solution for it.
You just need to make a new array and push data into it by finding bases on key name like this:
suppose we have data in students object:
let finalArray = new Array();
for (var k in students ) {
finalArray.push(students[k])
}
console.log(finalArray) // Normal JSON array object :)
m.sequelize.query(sql, {
model,
mapToModel: true
})
.then(model => res.status(200).send(model))
.catch(error => res.status(400).send(error.toString())
})

SQL result lengths and showing nothing when searched

I want my Discord bot to check if a user that joined exists in a MySQL table. However, when it sends the query, it's basically telling me that it doesn't exist even though it should.
This is my current code:
bot.on('guildMemberAdd', async (member) => {
console.log(member.id)
let query = `SELECT userId FROM QR5PVGPh1D.users WHERE userId = '${member.id}'`
let result = connection.query(query)
if(result.length > 0){
console.log("It works!")
}
})
Node is asynchronous, so here you try to console.log result before it has been populated...
You'll find more info in this here :
Node JS MySQL query function not returning result
How do I return callback of MySQL query and push to an array in Node.js?
Here is the code with a callback function. Try it, it should work:
let query = `SELECT userId FROM QR5PVGPh1D.users WHERE userId = '${member.id}'`
connection.query(query, function (err, result) {
if (!err) {
if (result.length > 0) {
console.log("It works!")
}
}
});
Explanation:
As BadSpencer has stated, the mysql driver is asynchronous, and based around callbacks.
Say you're planning on picking your friend up to go to a sporting event. You're not sure when they want you to come, so you call them on the phone and ask them. They think about it for a while, and then tell you a time. You got the information you requested, so you hang up. In programming terms, this would be an example of synchronous code (sometimes thought of as "normal" code in Node.js).
Put yourself back in the same situation. However, when you call your friend this time, they're very busy. You don't want to bother them so you ask them to call you later. You hang up, but now you wait. An hour later, they call you back and tell you the time. This is the thought process of asynchronous code.
There's a lot more that goes on behind the screen, but for simplicity's sake, I'm not going to bombard you with all that information in this answer.
Solutions:
You should pass a function to act as a callback which will use the returned data. Consider this example:
let query = `SELECT userId FROM QR5PVGPh1D.users WHERE userId = '${member.id}'`;
// Passing the callback function as the second parameter.
connection.query(query, (err, result) => {
if (err) return console.error(err);
if (result.length > 0) console.log('It works (it actually does).');
});
However, this callback-based nature can become a nightmare due to the scope of the result and subsequent flow of the code. After a few queries, your code can become messy. To prevent this, you can wrap the query in your own Promise (or use a Promise-based version of the mysql package, like promise-mysql) and await the calls.
Here's an example setup:
// Defining 'connection' as a parameter so
// you can export this function and use it
// anywhere.
function query(connection, sql) {
return new Promise((resolve, reject) => {
connection.query(sql, (err, result) => {
if (err) reject(err);
else resolve(result);
});
});
}
// Asynchronous context needed for 'await' (this needs to be within an async function).
let query = `SELECT userId FROM QR5PVGPh1D.users WHERE userId = '${member.id}'`;
let result = await query(connection, query)
.catch(console.error);

Getting Error When I Start multiple select query inside loop TimeoutError: ResourceRequest timed out

I'm using nodeJs Express Framework.
I'm using mysql database with sequelizejs library and using querying for retrieve data.
I am getting timeout error when I fired select query for almost 50,00,000 records.
I have done the server timeout but not worked.
I have done the pooling method in sequlizeJs But not worked.
function fetchNamesData(req, name) {
return new Promise((resolve, reject) => {
const names = req.app.locals.models.names_data;
names.findAll({
where: {
name: name
},
order: [['date', 'DESC']],
limit: 50
})
.then(function (dbRes) {
console.log(dbRes.length);
resolve(dbRes);
})
.catch(function (dbErr) {
console.log(dbErr);
return reject(dbErr);
});
});
}
allNames.forEach(element => {
//console.log(element.dataValues.name);
fetchNamesData(req, element.dataValues.name).then((dbRes) => {
//here I will have all the records
}).catch((dbErr) => { console.log(dbErr) });
var allNames = {having almost 7000 names}
now I iterate this obj and each names having 50 record in database
I want to get that all record like 50*7000 = 3,50,000.
What happens in your case is :
Looping through 7000 names and at same time hitting 7000 queries in mySql , and mysql will create queue for executing 7000 queries at same time cause load on machine. Either you can update your configuration to handle such load OR
Solution to this : Try to put some timeout b/w each queries , this way you will be able to fetch more records ,
allNames.forEach(element => {
setTimeout(() => { // <----------- HERE -------------
fetchNamesData(req, element.dataValues.name).then((dbRes) => {
//here I will have all the records
}).catch((dbErr) => {
console.log(dbErr)
});
},500); // <----------- HERE -------------
});
I have found the solution like
- Remove the unwanted console.log() and
- Also your hardware configuration depend upon it for timeout error.
- When query is firing do not start or run any other work it will lead to time out error[when there are multiple crud operation is going on].
- Also give index to table field when particular field is going to use in where clause.

Error from ending connection from 'query inside query'

I am using mysql node module and having an error of Error: Cannot enqueue Query after invoking quit. from my source code which has a block that does 'update if exists, otherwise insert' (similar logic as in another question I asked).
The reason I include it in node is that the definition of 'duplicates' is customized.
I have tested my source code and it was able to successfully perform 'update if exists, otherwise insert' to my table, but I am not able to close my connection, as for my insert/update query reside in the search query.
const insertQueue = (results) => {
_.forEach(results, (values, key) => {
_.map(values, (value) => {
let query = 'SELECT index FROM table WHERE (document=? AND ((document NOT like \'PERSONAL:%\' AND username=?) OR (document like \'PERSONAL:%\' AND serial=?)))';
connection.query(query, [value.document, key, value.serial], function(error, rows){
if (error) {
console.log(error);
}
else {
//Ideally there should be only one entry, but still return an array to make the function error-safe
let indices = _.map(rows,'index');
if (_.isEmpty(indices)) {
//sqlInsert and sqlUpdate are defined elsewhere and has sql query defined with 'connection.query(..)' similar to this
sqlInsert(key, value.serial, value.document, value.lastRun);
}
else {
_.each(indices, (index) => {
sqlUpdate(index,value.lastRun);
});
}
}
});
});
});
connection.end();
}
sqlInsert and sqlUpdate are defined elsewhere and has sql query defined with connection.query(..) similar to the function above.
I understand that connection.query(..) is asynchronous, and I put my connection.end() in the very end of the function. But I don't understand why I am still getting the error of Cannot enqueue Query after invoking quit every time I call my insertQueue.
Also if I replace my sqlInsert(...); and sqlUpdate(...) with some testing command (with no db sql query execution), the error will be gone.
Is there any reasons for this?
first thing you should not call async function inside loop (map, forEach, for, while or anyone).
second it's worst idea to fire query in loop.
mysql.js provide method for bulk insert
from official doc https://github.com/mysqljs/mysql
Nested arrays are turned into grouped lists (for bulk inserts), e.g.
[['a', 'b'], ['c', 'd']] turns into ('a', 'b'), ('c', 'd')
take an example
connection.query('INSERT INTO posts SET ?', [['a', 'b'], ['c', 'd']], function (error, results, fields) {
if (error) throw error;
// ...
});
third i think lastRun is datetime. you can choose mysql type current timestamp and now no need to worry this in query,

Mysql native spatial functions with sequelize and node js

I'm storing a location of type POINT in my mysql database:
module.exports = function (sequelize, DataTypes) {
var Promotion = sequelize.define('promotion', {
userId: {
type: DataTypes.STRING,
allowNull: false
},
location: {
type: DataTypes.GEOMETRY('POINT'),
allowNull: false
}
It looks like the point is being stored correctly:
I'm trying find all rows within a set of bounds using the mysql native ST_MakeEnvelope function. I was using this with postgresql and it worked perfectly, but when I switch to mysql it start throwing errors:
if (northEastLng &&
northEastLat &&
southWestLat &&
southWestLng)
{
where.location = {
$overlap: db.sequelize.fn('ST_MakeEnvelope', southWestLng, southWestLat, northEastLng, northEastLat)
}
}
promotion.findAll({
where,
})
.then(promoters => {
res.json(promoters)
})
.catch(err => {
console.log('ERROR: ', err)
})
The error:
Error: ER_WRONG_PARAMCOUNT_TO_NATIVE_FCT:
Incorrect parameter count in the call to native function 'ST_MakeEnvelope'
So I tried passing in two points instead:
const POINT_ONE = `POINT(${southWestLng} ${southWestLat})`
const POINT_TWO = `POINT(${northEastLng} ${northEastLat})`
where.location = {
$overlap: db.sequelize.fn('ST_MakeEnvelope', POINT_ONE, POINT_TWO)
}
Now I'm getting this error:
SequelizeDatabaseError: UNKNOWN_CODE_PLEASE_REPORT:
Geometry byte string must be little endian.
Been searching around, and not too sure where to go from here. How do I use the ST_MakeEnvelope function with sequelize to query?
Edit
Adding the generated sql from piotrbienias response:
SELECT `promotion`.`id`, `promotion`.`userId`, `promotion`.`title`, `promotion`.`description`, `promotion`.`startDate`, `promotion`.`endDate`, `promotion`.`isIndefinite`, `promotion`.`isApproved`, `promotion`.`status`, `promotion`.`reach`, `promotion`.`trustRanking`, `promotion`.`isLocationBased`, `promotion`.`address`, `promotion`.`city`, `promotion`.`state`, `promotion`.`zip`, `promotion`.`location`, `promotion`.`createdAt`, `promotion`.`updatedAt`, `promotion`.`categoryId
`, `promotionImages`.`id` AS `promotionImages.id`, `promotionImages`.`url` AS `promotionImages.url`, `promotionImages`.`publicId` AS `promotionImages.publicId`, `promotionImages`.`secureUrl` AS `promotionImages.secureUrl`, `promotionImages`.`isApproved` AS `promotionImages.isApproved`, `promotionImages`.`createdAt` AS `promotionImages.createdAt`, `promotionImages`.`updatedAt` AS `promotionImages.updatedAt`, `promotionImages`.`promotionId` AS `promotionImages.promotionId`, `category`.`id` AS `cat
egory.id`, `category`.`title` AS `category.title` FROM `promotions` AS `promotion` LEFT OUTER JOIN `promotionImages` AS `promotionImages` ON `promotion`.`id` = `promotionImages`.`promotionId` LEFT OUTER JOIN `categories` AS `category` ON `promotion`.`categoryId` = `category`.`id` WHERE `promotion`.`location` && ST_MakeEnvelope(ST_GeomFromText('POINT(-80.30252222253421 25.802030960352745)'), ST_GeomFromText('POINT(-80.30252222253421 25.802030960352745)'));
I am not familiar with the Geometry part of MySQL, but shouldn't you use the ST_GeomFromText function on those points before using them?
where.location = {
$overlap: db.sequelize.fn(
'ST_MakeEnvelope',
db.sequelize.fn('ST_GeomFromText', POINT_ONE),
db.sequelize.fn('ST_GeomFromText', POINT_TWO),
)
}
};
Just as it is presented in the example of ST_MakeEnvelope function (and it takes two parameters, so that may be the reason of your first error Incorrect parameter count in the call to native function 'ST_MakeEnvelope'). And, as I said on the beginning, I am definitely not an expert in these, just my suggestion after taking a look at the MySQL documentation.
EDIT
Below is description of $overlap from Sequelize documentation
$overlap: [1, 2] // && [1, 2] (PG array overlap operator)
I think that you should construct your Sequelize query differently, without the $overlap. I suppose that you should use ST_Contains function and your where object should be as follows
{
where: db.sequelize.where(
db.sequelize.fn(
'ST_Contains',
db.sequelize.fn(
'ST_MakeEnvelope',
db.sequelize.fn('ST_GeomFromText', POINT_ONE),
db.sequelize.fn('ST_GeomFromText', POINT_TWO)
),
db.sequelize.col('promotion.location')
),
'=',
1
)
}
In my opinion the SQL you want should be something like:
WHERE ST_Contains(POLYGON(...), location)
Above Sequelize query would generate
WHERE ST_Contains(ST_MakeEnvelope(ST_GeomFromText(POINT_ONE), ST_GeomFromText(POINT_TWO)), location) = 1;
Which should check if polygon created from two points contains value of location column. ST_Contains returns 1 or 0, so I think that = 1 condition should be ok in this case.