I have got 3 entities in my database, we can imagine these 3(I kept only important columns)
User
#PrimaryColumn()
nick: string
#OneToMany(type => Connection, connection => connection.user)
connections:Connection[]
Connection
#PrimaryGeneratedColumn()
id: number;
#ManyToOne(type => User,user => user.connections)
user: User;
#ManyToOne(type => Stream)
stream: Stream;
Stream
#PrimaryGeneratedColumn()
id: number;
I think that I have some skills with building Queries in typeorm, but I am stucked right now on using subquery.
What I want to do?
Join all User entites on part of Connection entites(Select few of Connection entities as subquery => make leftJoin User-subquery => filter results to get connections which didnt find right(Connection) side so Connection ID is NULL ).
What I tried?
let query = await connectionRepository
.createQueryBuilder('connection')
.subQuery()
.where("connection.id = :id",{id:1})
.select(['connection.id','connection.user AS nick'])
.getQuery();
const result = await userRepository.createQueryBuilder("user")
.leftJoinAndSelect("user.permissions", '(' + query + ')')
.where("user.nick = :nick")
.andWhere(`(${query}).id is null`)
.getMany();
I also tried to use subquery without .getQuery() but It didnt work anyway, I think that I can write Ok subquery, but then I have problem to use it in the second query(with subquery attributes etc.)
I have also seen some examples with lambda functions, but lambda was always on first position in the leftJoin.
Thank you for your answer:)
You need something like that:
const result = await userRepository.query(`
SELECT user.* FROM user
LEFT JOIN connection connection ON user.id = connection.user_id
WHERE connection.user_id IS NULL
AND connection.id = ?;
`, [id])
Related question: MYSQL Left Join how do I select NULL values?
Related
I have a table, called "bsService", where I save my created services. Those services have some relations, like categories, activities and others.
I'm trying to get services where categories was softDeleted. Example: service 1 relates to category 1, I softDeleted category 1, and now this service doesn’t return on findAll even if add 'withDeleted: true' on the query.
Here's my findAll method. I want all data even if a relation is softDeleted.
findAll = async (
where?: WhereConditions,
transactionEntityManager: EntityManager = getManager(),
order?: 'ASC' | 'DESC',
withDeleted?: boolean,
): Promise<BSService[]> => transactionEntityManager.find(BSService, {
withDeleted,
where,
relations: ['sla', 'activity', 'activity.category', 'department', 'department.company', 'attendance', 'logs', 'logs.user', 'requestingAgent', 'alocatedAgent', 'category', 'requestingAgent.jobs', 'alocatedAgent.jobs'],
order: {
updateAt: order,
},
});
The 'withDeleted' becomes true, depending on which page client is using. For that example, it is always true.
withDeleted in .find() only applies to the top layer, not relations.
Here's someone with a similar issue.
Here's a PR that shows how you can solve it by using a query builder with .withDeleted() before the relations you want to include. There's some discussion about supporting this with .find() but it seems like it won't be any time soon.
await manager
.getRepository(BSService)
.createQueryBuilder('bsservice')
.withDeleted() // Above innerJoinAndSelects
.innerJoinAndSelect('bsservice.sla', 'sla')
.innerJoinAndSelect('bsservice.activity', 'activity')
// ... more inner joins ...
.getMany();
Problem:
I'm working on a project that consists of multiple studies and a group of users that each participates in one of the studies. Every study divides the participants in two groups based on a list that is generated using some randomization algorithm. Upon signing up each user is assigned to a study and their group is determined by the order of signup and the corresponding index in the groups list. For example if study A has total seats of 4 and the groups list is [0, 1, 1, 0] the first user is assigned to group 0, the second one to 1 and so on until the study is full.
There are other user roles defined in the project which are the admins and can be assigned to multiple studies without occupying a position in the study. That means the relation of users to studies is n:m.
The problem that occurs in the current implementation is a race condition when assigning users to studies and study groups. The code is provided below and the way it works is that it overrides the addUser of Study model and whenever a user is added to a study it checks how many users are already in the study and gives the user the current index of the group list which is the seatsTaken number. This works so long as the users are added to the study in intervals. But whenever multiple users are added at the same time the asynchronous queries cause a race condition and the seatsTaken count is affected by other users signing up at the same time.
In the example below the users which are assigned to study A in intervals have the correct groups assigned but study B with simultaneous queries has incorrect groups assignment.
const Sequelize = require('sequelize');
const assert = require('assert');
const sequelize = new Sequelize({
database: 'database',
username: 'username',
password: 'password',
dialect: process.env.DB_DIALECT || 'sqlite',
storage: 'db.sqlite',
logging: false
});
const User = sequelize.define('user', {
id: {
type: Sequelize.INTEGER,
autoIncrement: true,
primaryKey: true,
},
group: {
type: Sequelize.INTEGER,
allowNull: true,
defaultValue: null
}
});
// Groups list for studies 'A' and 'B'
const groupLists = {
a: [0, 1, 1, 0],
b: [1, 0, 1, 0]
}
const Study = sequelize.define('study', {
id: {
type: Sequelize.INTEGER,
autoIncrement: true,
primaryKey: true,
},
name: {
type: Sequelize.STRING,
allowNull: false
},
seatsTotal: {
type: Sequelize.INTEGER,
defaultValue: 0
}
});
// n:m relation between users and studies
User.belongsToMany(Study, {through: 'UserStudy'});
Study.belongsToMany(User, {through: 'UserStudy'});
// Overridden 'addUser' method for groups assignment
Study.prototype.addUser = async function(user) {
// Count already occupied seats
const seatsTaken = await User.count({
include: [{
model: Study,
where: {
name: this.name
}
}]
});
// Add the user to study
await Study.associations.users.add(this, user);
// Assign the group of the user based on the seatsTaken
await user.update({ group: groupLists[this.name][seatsTaken] });
}
sequelize.sync({force: true}).then(async () => {
// Studies 'A' and 'B' with 4 seats
await Study.bulkCreate([{name: 'a', seatsTotal: 4}, {name: 'b', seatsTotal: 4}]);
// 8 users
await User.bulkCreate(new Array(8).fill(0).map(() => ({})));
const studies = await Study.findAll();
const users = await User.findAll();
// Assign half of the users to study 'A' in intervals
users.filter((_, idx) => idx % 2 === 0).forEach((user, idx) => {
setTimeout(() => {
studies[0].addUser(user);
}, 100*idx);
});
// Assign the other half to study 'B' at the same time
await Promise.all(users.filter((_, idx) => idx % 2 === 1).map(user => {
return studies[1].addUser(user);
}));
setTimeout(async () => {
// Wait for all queries to finish and assert the results
const userStudies = await User.findAll({
include: [Study]
});
const studyUsersA = userStudies.filter(u => u.studies.some(s => s.name === 'a'));
const studyUsersB = userStudies.filter(u => u.studies.some(s => s.name === 'b'));
try {
console.log('Group list A actual:', studyUsersA.map(u => u.group), 'expected:', groupLists['a']);
assert.deepEqual(studyUsersA.map(u => u.group).sort((a, b) => a-b), groupLists['a'].sort((a, b) => a-b), 'Group list A is not assigned correctly');
console.log('Group list B actual:', studyUsersB.map(u => u.group), 'expected:', groupLists['b']);
assert.deepEqual(studyUsersB.map(u => u.group).sort((a, b) => a-b), groupLists['b'].sort((a, b) => a-b), 'Group list B is not assigned correctly');
console.log(`Passed: Group lists are assigned correctly.`);
} catch (e) {
console.log(`Failed: ${e.message}`);
}
}, 500);
});
Related questions that I could find are either about incrementing one value in one table or they just mention transactions and locks without providing example code:
Avoiding race condition with Nodejs Sequelize
How to lock table in sequelize, wait until another request to be complete
Addition and Subtraction Assignment Operator With Sequelize
Database race conditions
Limitations:
The project stack is nodejs, expressjs and sequelize with
mysql database for production and sqlite for development and
tests.
The solution should work for both sqlite and mysql.
It is preferred that the groups lists are not stored in the database. The lists are generated by an algorithm and randomization seeds but in the example code they're hardcoded.
The solution should be a sequelize solution and not throttling or queuing user requests in express server.
In the case of simultaneous requests it is not strictly required that the exact order of users signup is preserved since it's not really verifiable which user is added to the study first, but the final result must have the correct number of 0s and 1s which are the assigned groups.
I've tried sequelize transactions but I had lots of issues with sqlite compatibilities and also express requests failing because of database locks but that might have been because of my lack of knowledge on how to do it properly. The limitation here is that requests should not fail because of database locks.
The provided code is a minimal example reproducing the issue. Please use it as a base.
To run the code
npm install sequelize sqlite3 mysql2
sqlite:
node index.js
mysql (using docker):
docker run -d --env MYSQL_DATABASE=database --env MYSQL_USER=username --env MYSQL_PASSWORD=password --env MYSQL_RANDOM_ROOT_PASSWORD=yes -p 3306:3306 mysql:5.7
DB_DIALECT=mysql node index.js
Note:
The example code is only for demonstration of the issue in the current implementation and the intervals and timeouts are there to simulate the users interaction with the server. Please do not focus on the patterns in the example being wrong and rather focus on the problem itself and how it can be approached in a better way while meeting the requirements mentioned in the limitations section.
This is part of a fairly big project and I might update the requirements based on the actual project requirements and the feedbacks that I receive here.
Please let me know if I should provide any other information. Thank you in advance.
I'm afraid this is expected behavior.
You declare seatsTaken as an asynchronously computed property.
You insert several users asynchronously, too.
You don't isolate each user creation within its own transaction.
Because of this, you see the changing state of one transaction, and it's changing rather chaotically because you do not specify any certain order. Eventually the state becomes consistent, but your way to achieve that state was just to wait for some time.
I suppose the easiest way to achieve consistency would be wrapping each insertion in a transaction.
If a transaction per insert is too slow, you can bulk insert all user records in one transaction, and then count seats taken in another, or even just do everything synchronously.
In any case, if you want consistency, you need logical serialization, a clear "before-after" relation. Currently your code lacks it, AFAICT.
I'm working with two tables in particular. Users and Friends. Users has a bunch of information that defines the User whereas Friends has two columns aside from id: user_id and friend_id where both of them are a reference to the User table.
I'm trying to find all of the users friends in as little calls to the db as possible and I currently have 2. One to retrieve the id of a user first from a request, then another to Friends where I compare the IDs from the first call and then a third call that passes the array of friends and find all of them in the Users table. This already feels like overkill and I think that with associations, there has to be a better way.
Modification of the tables unfortunately is not an option.
One thing that I saw from "http://docs.sequelizejs.com/manual/querying.html#relations---associations"
I tried but got an interesting error.. when trying to repurpose the code snippet in the link under Relations/Associations, I get "user is associated to friends multiple times. To identify the correct association, you must use the 'as' keyword to specify the alias of the association you want to include."
const userRecord = await User.findOne({
where: { id }
})
const friendsIDs = await Friends.findAll({
attributes: ["friend_id"],
where: {
user_id: userRecord.id
}
}).then(results => results.map(result => result.friend_id));
const Sequelize = require("sequelize");
const Op = Sequelize.Op;
return await User.findAll({
where: {
id: { [Op.in]: friendsIDs }
},
});
Above for my use case works. I'm just wondering if there are ways to cut down the number of calls to the db.
Turns out Sequelize handles this for you if you have the proper associations in place so yes, it was a one liner user.getFriends() for me.
I have two SQL queries which i am having trouble with converting to Sequelieze Queries.
I have a model named UserTeam having columns id,team_id and isAdmin.
The first query is
Select distinct('team_id') from UserTeam where id==user.id and idAdmin==true
The second one is similar by with a little addition of group by
Select count('isAdmin') from UserTeam where id==user.id group by 'team_id';
I have written this but is incorrect:
const teamsMember = await UserTeam.findAll({where: { id: user.id, isAdmin : true}, sequelize.fn('distinct', sequelize.col('UserTeam.team_id'))});
const adminCount = await UserTeam.findAll({where: {id: user.id }, sequelize.fn('count',sequelize.col('isAdmin')), group: ['UserTeam.team_id']]});
And is there any way to get the generated query from these sequelize queries?
I am trying to run the following Linq query from MySQL client
query = query.Where(c => c.CustomerRoles
.Select(cr => cr.Id)
.Intersect(customerRoleIds)
.Any()
);
This code looks okay, but gives the error:
System.NotSupportedException: Specified method is not supported.at MySql.Data.Entity.SqlGenerator.Visit(DbIntersectExpression expression)
This looks to me like an issue with .Intersect. Can anybody tell me the cause of this error and how to fix it?
i think #GertArnold's post is a correct and best of the answers, but i'm wonder why have you gotten NotSupportedException yet ? so the problem should not be from intersect probably.
where is customerRoleIds come from ? is it IQueryable<T> ?
break the query, and complete it step by step.
if you don't get exception at this lines:
var a = query.Select(c => new {
c,
CustomerRoleIDList = c.CustomerRoles.Select(cr => cr.Id).AsEnumerable()
})
.ToList();
var b = customerRoleIds.ToList();
you must get the result by this:
var b = query.Where(c => c.CustomerRoles.any(u => customerRoleIds.Contains(u.Id)))
.ToList();
if you get exception by above query, you can try this final solution to fetch data first, but note by this, all data will be fetched in memory first:
var a = query.Select(c => new {
c,
CustomerRoleIDList = c.CustomerRoles.Select(cr => cr.Id).AsEnumerable()
})
.ToList();
var b = a.Where(c => c.CustomerRoleIDList.any(u => customerRoleIds.Contains(u)))
.Select(u => u.c)
.ToList();
Using Intersect or Except is probably always troublesome with LINQ to a SQL backend. With Sql Server they may produce horrible SQL queries.
Usually there is support for Contains because that easily translates to a SQL IN statement. Your query can be rewritten as
query = query.Where(c => c.CustomerRoles
.Any(cr => customerRoleIds.Contains(cr.Id)));
I don't think that customerRoleIds will contain many items (typically there won't be hundreds of roles), otherwise you should take care not to hit the maximum number of items allowed in an IN statement.
query.Where(c => c.CustomerRoles
.Any(v=>customerRoleIds.Any(e=>e == v.Id))
.Select(cr => cr.Id))
.ToList();
Try adding toList() before intersect, that should enumerate results locally instead running on MySql, you will get performance hit thought.
query = query.Where(c => c.CustomerRoles.Select(cr => cr.Id)).ToList().Intersect(customerRoleIds);