In my mysql table ,i have a column of JSON type
const monthlyProgress = sequelize.define("monthlyprogress", {
.
.
.
duration: {
type: Sequelize.JSON
}
});
return monthlyProgress;
in this "duration" column i have a JSON array of objects
[{"time":"nov-2020"},{"time":"dec-2020"}]
Now i need a sequelize findall to fetch the rows where the condition will e
time = "nov-2020" of that "duration" column
i have tried i many ways but failed everyime. Below i am showing one of the ways that i found on stackoverflow. But its also a failed attepmt
Its giving me output of those rows who doesnt have time="nov-2020"
await monthlyProgress.findAll({
where: {
duration: [
fn('JSON_CONTAINS', col('duration'), cast('{"time": "nov-2020"}', 'CHAR CHARACTER SET utf8'))
]
}
})
It's another attempt thats is showing this error message UnhandledPromiseRejectionWarning: ReferenceError: JSON_CONTAINS is not defined
await monthlyProgress.findAll({
where: {
[Op.and]: db.Sequelize.literal(JSON_CONTAINS(`duration`, '{\"time\": nov-2020}')),
}
})
Related
I want to get id of affected row in update action using Sequlaize.update method.
const result = await Voucher.update(
{
used: "1",
},
{
where: {
voucher_group_id: 5,
used: "0",
},
limit: 1,
returning: true,
}
);
when I set returning to true, the result was [null, 1]. but the affected row id is other value like: 72
The returning option is only for the Postgres dialect, as specified in the Sequelize v6 Documentation for Model#update.
If you are using a Dialect like MySQL, you could load in the record you want to update via Model#findOne first and then use instance#update.
const voucher = await Voucher.findOne({
where: {
voucher_group_id: 5,
used: "0"
}
});
// voucher is potentially null at this point
voucher.update({ used: "1" });
Side note: If your used field on your Voucher model is a boolean, you can use true/false and Sequelize will automatically convert these to the tiny int equivalent values for your dialect.
Given this query here,
let output = [];
const sql = `select * from coredb.account LIMIT ${offset},${limit}`;
let data = await sequelize.query(sql, null, {raw: true, type: sequelize.QueryTypes.SELECT});
data.forEach((item) => {
console.log(item['id'], item.id); // <-- output says "undefined, undefined"
});
the data variable is indeed hydrated with the right row data when using console.log to inspect it.
But, when I try to access the individual properties, they only ever come back as undefined. This TextRow object that Sequelize seems to return the result in doesn't seem to want to let me access then explicit rows.
Just curious what i'm missing here, am I missing an option?
I agree, Sequalize raw queries are not intuitive. You don't need the null or raw: true flag. Something like this should work:
let data = await sequelize.query(sql, {type: sequelize.QueryTypes.SELECT});
When I tried this, "data" was an array of two objects, each being the query result. So, the properties can be accessed by using index [0].... e.g.
data[0].forEach((item) => {
console.log(item['id'], item.id); // <-- output says "undefined, undefined"
});
Not yet sure WHY this occurs!
EDIT - it's because .query() should have only two arguments. Changing the call to: sequelize.query(sql, {raw: true, type: sequelize.QueryTypes.SELECT}) resulted in data being a single array (as expected).
Finally I was able to find the solution for it.
You just need to make a new array and push data into it by finding bases on key name like this:
suppose we have data in students object:
let finalArray = new Array();
for (var k in students ) {
finalArray.push(students[k])
}
console.log(finalArray) // Normal JSON array object :)
m.sequelize.query(sql, {
model,
mapToModel: true
})
.then(model => res.status(200).send(model))
.catch(error => res.status(400).send(error.toString())
})
I'm using nodeJs Express Framework.
I'm using mysql database with sequelizejs library and using querying for retrieve data.
I am getting timeout error when I fired select query for almost 50,00,000 records.
I have done the server timeout but not worked.
I have done the pooling method in sequlizeJs But not worked.
function fetchNamesData(req, name) {
return new Promise((resolve, reject) => {
const names = req.app.locals.models.names_data;
names.findAll({
where: {
name: name
},
order: [['date', 'DESC']],
limit: 50
})
.then(function (dbRes) {
console.log(dbRes.length);
resolve(dbRes);
})
.catch(function (dbErr) {
console.log(dbErr);
return reject(dbErr);
});
});
}
allNames.forEach(element => {
//console.log(element.dataValues.name);
fetchNamesData(req, element.dataValues.name).then((dbRes) => {
//here I will have all the records
}).catch((dbErr) => { console.log(dbErr) });
var allNames = {having almost 7000 names}
now I iterate this obj and each names having 50 record in database
I want to get that all record like 50*7000 = 3,50,000.
What happens in your case is :
Looping through 7000 names and at same time hitting 7000 queries in mySql , and mysql will create queue for executing 7000 queries at same time cause load on machine. Either you can update your configuration to handle such load OR
Solution to this : Try to put some timeout b/w each queries , this way you will be able to fetch more records ,
allNames.forEach(element => {
setTimeout(() => { // <----------- HERE -------------
fetchNamesData(req, element.dataValues.name).then((dbRes) => {
//here I will have all the records
}).catch((dbErr) => {
console.log(dbErr)
});
},500); // <----------- HERE -------------
});
I have found the solution like
- Remove the unwanted console.log() and
- Also your hardware configuration depend upon it for timeout error.
- When query is firing do not start or run any other work it will lead to time out error[when there are multiple crud operation is going on].
- Also give index to table field when particular field is going to use in where clause.
I'm storing a location of type POINT in my mysql database:
module.exports = function (sequelize, DataTypes) {
var Promotion = sequelize.define('promotion', {
userId: {
type: DataTypes.STRING,
allowNull: false
},
location: {
type: DataTypes.GEOMETRY('POINT'),
allowNull: false
}
It looks like the point is being stored correctly:
I'm trying find all rows within a set of bounds using the mysql native ST_MakeEnvelope function. I was using this with postgresql and it worked perfectly, but when I switch to mysql it start throwing errors:
if (northEastLng &&
northEastLat &&
southWestLat &&
southWestLng)
{
where.location = {
$overlap: db.sequelize.fn('ST_MakeEnvelope', southWestLng, southWestLat, northEastLng, northEastLat)
}
}
promotion.findAll({
where,
})
.then(promoters => {
res.json(promoters)
})
.catch(err => {
console.log('ERROR: ', err)
})
The error:
Error: ER_WRONG_PARAMCOUNT_TO_NATIVE_FCT:
Incorrect parameter count in the call to native function 'ST_MakeEnvelope'
So I tried passing in two points instead:
const POINT_ONE = `POINT(${southWestLng} ${southWestLat})`
const POINT_TWO = `POINT(${northEastLng} ${northEastLat})`
where.location = {
$overlap: db.sequelize.fn('ST_MakeEnvelope', POINT_ONE, POINT_TWO)
}
Now I'm getting this error:
SequelizeDatabaseError: UNKNOWN_CODE_PLEASE_REPORT:
Geometry byte string must be little endian.
Been searching around, and not too sure where to go from here. How do I use the ST_MakeEnvelope function with sequelize to query?
Edit
Adding the generated sql from piotrbienias response:
SELECT `promotion`.`id`, `promotion`.`userId`, `promotion`.`title`, `promotion`.`description`, `promotion`.`startDate`, `promotion`.`endDate`, `promotion`.`isIndefinite`, `promotion`.`isApproved`, `promotion`.`status`, `promotion`.`reach`, `promotion`.`trustRanking`, `promotion`.`isLocationBased`, `promotion`.`address`, `promotion`.`city`, `promotion`.`state`, `promotion`.`zip`, `promotion`.`location`, `promotion`.`createdAt`, `promotion`.`updatedAt`, `promotion`.`categoryId
`, `promotionImages`.`id` AS `promotionImages.id`, `promotionImages`.`url` AS `promotionImages.url`, `promotionImages`.`publicId` AS `promotionImages.publicId`, `promotionImages`.`secureUrl` AS `promotionImages.secureUrl`, `promotionImages`.`isApproved` AS `promotionImages.isApproved`, `promotionImages`.`createdAt` AS `promotionImages.createdAt`, `promotionImages`.`updatedAt` AS `promotionImages.updatedAt`, `promotionImages`.`promotionId` AS `promotionImages.promotionId`, `category`.`id` AS `cat
egory.id`, `category`.`title` AS `category.title` FROM `promotions` AS `promotion` LEFT OUTER JOIN `promotionImages` AS `promotionImages` ON `promotion`.`id` = `promotionImages`.`promotionId` LEFT OUTER JOIN `categories` AS `category` ON `promotion`.`categoryId` = `category`.`id` WHERE `promotion`.`location` && ST_MakeEnvelope(ST_GeomFromText('POINT(-80.30252222253421 25.802030960352745)'), ST_GeomFromText('POINT(-80.30252222253421 25.802030960352745)'));
I am not familiar with the Geometry part of MySQL, but shouldn't you use the ST_GeomFromText function on those points before using them?
where.location = {
$overlap: db.sequelize.fn(
'ST_MakeEnvelope',
db.sequelize.fn('ST_GeomFromText', POINT_ONE),
db.sequelize.fn('ST_GeomFromText', POINT_TWO),
)
}
};
Just as it is presented in the example of ST_MakeEnvelope function (and it takes two parameters, so that may be the reason of your first error Incorrect parameter count in the call to native function 'ST_MakeEnvelope'). And, as I said on the beginning, I am definitely not an expert in these, just my suggestion after taking a look at the MySQL documentation.
EDIT
Below is description of $overlap from Sequelize documentation
$overlap: [1, 2] // && [1, 2] (PG array overlap operator)
I think that you should construct your Sequelize query differently, without the $overlap. I suppose that you should use ST_Contains function and your where object should be as follows
{
where: db.sequelize.where(
db.sequelize.fn(
'ST_Contains',
db.sequelize.fn(
'ST_MakeEnvelope',
db.sequelize.fn('ST_GeomFromText', POINT_ONE),
db.sequelize.fn('ST_GeomFromText', POINT_TWO)
),
db.sequelize.col('promotion.location')
),
'=',
1
)
}
In my opinion the SQL you want should be something like:
WHERE ST_Contains(POLYGON(...), location)
Above Sequelize query would generate
WHERE ST_Contains(ST_MakeEnvelope(ST_GeomFromText(POINT_ONE), ST_GeomFromText(POINT_TWO)), location) = 1;
Which should check if polygon created from two points contains value of location column. ST_Contains returns 1 or 0, so I think that = 1 condition should be ok in this case.
I imported my MySQL database to MongoDB using MongoVUE.
First let me make an example of my tables:
Table0 = 1,500 entries
Table1 = 120,000 entries
Table2 = 18,000,000 entries
Table0 -> hasMany -> Table1 entries
Table1 -> hasMany -> Table2 entries
All tables have a _id key now but after the import both tables still have an id key from MySQL.
How do I updated table2's keys table1_id to match table1's _id key? Is it doable using a Mongo query or should I have to write a script for that? (The only language I know is PHP and Javascript/NodeJS)
Update 1
Using user #profesor79 answer I made this query where table1 = market_item_histories and table2 = market_items
db.market_item_histories.aggregate([
{
$lookup: {
from:"market_items",
localField: "market_item_id",
foreignField: "id",
as: "market_items_docs"
}
},
{
$unwind:"$market_items_docs"
},
{
$project: {
_id:1,
oldId:"$market_item_id",
market_item_id:"$market_items_docs._id",
date:1,
price:1,
amount:1,
created_at:1,
updated_at:1
}
},
{
$out:"marketItemHistories"
}
])
When running that code I get this Error:
assert: command failed: {
"errmsg" : "exception: Unrecognized pipeline stage name: '$lookup'",
"code" : 16436,
"ok" : 0
} : aggregate failed
Error: command failed: {
"errmsg" : "exception: Unrecognized pipeline stage name: '$lookup'",
"code" : 16436,
"ok" : 0
} : aggregate failed
at Error (<anonymous>)
at doassert (src/mongo/shell/assert.js:11:14)
at Function.assert.commandWorked (src/mongo/shell/assert.js:254:5)
at DBCollection.aggregate (src/mongo/shell/collection.js:1278:12)
at (shell):1:26
2016-04-29T14:13:48.223+0000 E QUERY Error: command failed: {
"errmsg" : "exception: Unrecognized pipeline stage name: '$lookup'",
"code" : 16436,
"ok" : 0
} : aggregate failed
at Error (<anonymous>)
at doassert (src/mongo/shell/assert.js:11:14)
at Function.assert.commandWorked (src/mongo/shell/assert.js:254:5)
at DBCollection.aggregate (src/mongo/shell/collection.js:1278:12)
at (shell):1:26 at src/mongo/shell/assert.js:13
this is a nice real life problem.
To get this done we can use aggregation framework and "join" tables, then write results in new collection.
After that the source can be renamed/deleted and our out can be renamed too.
This was done using mongo console.
Please find solution for joining table1 with table0, and use this to perform on others joins.
db.table1.aggregate([
{
$lookup:{
from:"table0",
localField: "table0_Id", // this is our join source
foreignField: "id", // this id field in table0 collection
as: "table0_docs"
}
},
{
$unwind:"$table0_docs"
},
{
$project:{
// very important list all fields here
_id:1,
data:1,
oldId:"$table0_Id",
referenceID:"$table0_docs._id",
}
},
{
$out:"newCollectionName"
}
])
AND OUTPUT DOCUMENT
{
"_id" : ObjectId("57234f5de63d33670e521892"),
"data" : "22",
"oldId" : 1,
"referenceID" : ObjectId("57234f33e63d33670e52188e")
}
Any comments welcome!