CakePHP 4: How to add count of association results to where conditions - mysql

I'm trying to add the count results of an association to the where conditions, for example:
$findQuery = $this->Products->find('all');
$findQuery->leftJoinWith('Synonyms', function($q) {
return $q->where(['Synonyms.title LIKE' => '%TEST%']);
});
$findQuery->select([
'amount_of_matching_synonyms' => $findQuery->func()->count('Synonyms.id')
]);
$findQuery->where([
'OR' => [
'Products.description LIKE' => '%TEST%',
'amount_of_matching_synonyms >' => 0
]
]);
What happens now is that I'm getting 1 result returned with the 'amount_of_matching_synonyms' field. But this appears to have a cumulated result of all the records it should return.
Please help me out!

You should first figure out how to do these things in plain SQL, it will then be much easier to translate things to the query builder.
Counting related data requires joining in the realted data and creating groups on which aggregate functions can be used, and you're missing the latter. Furthermore you cannot use aggregates in the WHERE clause, as grouping happens after the WHERE clause is applied, you would have to use the HAVING clause instead.
The basic SQL to filter on such counting would look something like this:
SELECT
COUNT(synonyms.id) amount_of_matching_synonyms
FROM
products
LEFT JOIN
synonyms ON synonyms.id = synonyms.product_id
GROUP BY
products.id
HAVING
amount_of_matching_synonyms > 0
Translating this into the query builder would be fairly simple, you'd just need group() and having(), something like this:
$findQuery = $this->Products
->find()
->select([
'Products.description',
'amount_of_matching_synonyms' => $findQuery->func()->count('Synonyms.id')
])
->leftJoinWith('Synonyms', function(\Cake\ORM\Query $q) {
return $q->where(['Synonyms.title LIKE' => '%TEST%']);
})
->group('Products.id')
->having([
'OR' => [
'Products.description LIKE' => '%TEST%',
'amount_of_matching_synonyms >' => 0
],
]);
Note that you need to select the description, otherwise the condition in the having clause would fail.
The resulting SQL would look something like this:
SELECT
products.description,
COUNT(synonyms.id) amount_of_matching_synonyms
FROM
products
LEFT JOIN
synonyms ON
synonyms.product_id = products.id
AND
synonyms.title LIKE '%TEST%'
GROUP BY
products.id
HAVING
products.description LIKE '%TEST%'
OR
amount_of_matching_synonyms > 0
See also
MySQL 5.7 Reference Manual / Functions and Operators / Aggregate Functions
Cookbook > Database Access & ORM > Query Builder > Aggregates - Group and Having
Cookbook > Database Access & ORM > Query Builder > Subqueries

Related

Add multiple count fields of contained model to query in cakephp 3

I want to have in my model, instead of the complete set of entries of the models contained by another one, just the quantity of them. I could do this by adding the "size" field to the resultset, but I'd like to do this in the query, as I want to paginate and order the results dinamically. I am using this code, but for some reason, if the first count is different from zero, both count fields have the same value, which is the value of the second count field.
$query = $this->Users->find('all')
->contain(['Likes','Favs']);
$query
->select(['like_count' => $query->func()->count('Likes.id')])
->leftJoinWith('Likes')
->group(['Users.id'])
->autoFields(true);
$query
->select(['fav_count' => $query->func()->count('Favs.id')])
->leftJoinWith('Favs')
->group(['Users.id'])
->autoFields(true);
$this->paginate['sortWhitelist'] = [
'name',
'email',
'last_login',
'fav_count',
'like_count',
];
I would like to know why this happens and if there is any other way to do what I attempt, which would be to have ('name', email, 'last_login', quantity of entries in Likes with the user's id, quantity of entries in Favs with the user's id). I have tried using join() to do the left join, but I haven't been able to obtain the result I want.
If you have multiple joins, then you need to count with DISTINCT, eg:
COUNT(DISTINCT Likes.id)
This is because your result will contain Likes * Favs number of rows, as for every joined like, all favs will be joined, ie for 2 likes and 10 favs you'd end up with 20 rows in total. A regular COUNT() would include every single one of those rows.
Also note that you don't need to repeat all that grouping, etc stuff, and you can use a callable for select() to avoid breaking up the builder.
$query = $this->Users
->find('all')
->select(function (\Cake\ORM\Query $query) {
return [
'like_count' => $query->func()->count(
$query->func()->distinct(['Likes.id' => 'identifier'])
),
'fav_count' => $query->func()->count(
$query->func()->distinct(['Favs.id' => 'identifier'])
),
];
})
->autoFields(true)
->contain(['Likes', 'Favs'])
->leftJoinWith('Likes')
->leftJoinWith('Favs')
->group(['Users.id']);
Using the functions builder to generate the DISTINCT is a workaround, as there is no API yet that would allow to specifically generate a keyword. The result will be something like DISTINCT(Likes.id), but it will work fine, the parentheses will be interpreted as part of the expression after the keyword, not as part of a function call.

Query matching() causing duplicate rows and distinct() not working

I am trying to filter one table Payments by a field on the associated table Invoices.
Using the function matching() on the query object filters correctly but causes duplicate rows. It seemed like the solution was using distinct(), but calling distinct(Payments.id) results in an invalid query. I'm doing the following in a controller action.
$conditions = [
'Payments.is_deleted =' => false
];
$args = [
'conditions' => $conditions,
'contain' => ['Invoices', 'Invoices.Clients'],
];
$payments = $this->Payments->find('all', $args);
if($issuer) {
// This causes duplicate rows
$payments->matching('Invoices', function ($q) use ($issuer) {
return $q->where(['Invoices.issuer_id' => $issuer['id']]);
});
// $payments->distinct('Payments.id'); // Causes a mysql error
}
Am I correct in thinking that distinct() is what I need, and if so any idea what's missing to make it work?
I'm getting the following mysql error when uncommenting the line above:
Error: SQLSTATE[42000]: Syntax error or access violation: 1055 Expression #8 of SELECT list is not in GROUP BY clause and contains nonaggregated column 'InvoicesPayments.id' which is not functionally dependent on columns in GROUP BY clause; this is incompatible with sql_mode=only_full_group_by
Full query:
SELECT
PAYMENTS.ID AS `PAYMENTS__ID`,
PAYMENTS.CREATED AS `PAYMENTS__CREATED`,
PAYMENTS.MODIFIED AS `PAYMENTS__MODIFIED`,
PAYMENTS.DATE_REGISTERED AS `PAYMENTS__DATE_REGISTERED`,
PAYMENTS.USER_ID AS `PAYMENTS__USER_ID`,
PAYMENTS.AMOUNT AS `PAYMENTS__AMOUNT`,
PAYMENTS.IS_DELETED AS `PAYMENTS__IS_DELETED`,
INVOICESPAYMENTS.ID AS `INVOICESPAYMENTS__ID`,
INVOICESPAYMENTS.INVOICE_ID AS `INVOICESPAYMENTS__INVOICE_ID`,
INVOICESPAYMENTS.PAYMENT_ID AS `INVOICESPAYMENTS__PAYMENT_ID`,
INVOICESPAYMENTS.PART_AMOUNT AS `INVOICESPAYMENTS__PART_AMOUNT`,
INVOICES.ID AS `INVOICES__ID`,
INVOICES.CREATED AS `INVOICES__CREATED`,
INVOICES.MODIFIED AS `INVOICES__MODIFIED`,
INVOICES.IS_PAID AS `INVOICES__IS_PAID`,
INVOICES.IS_DELETED AS `INVOICES__IS_DELETED`,
INVOICES.CLIENT_ID AS `INVOICES__CLIENT_ID`,
INVOICES.ISSUER_ID AS `INVOICES__ISSUER_ID`,
INVOICES.NUMBER AS `INVOICES__NUMBER`,
INVOICES.SUBTOTAL AS `INVOICES__SUBTOTAL`,
INVOICES.TOTAL AS `INVOICES__TOTAL`,
INVOICES.DATE_REGISTERED AS `INVOICES__DATE_REGISTERED`,
INVOICES.CURRENCY AS `INVOICES__CURRENCY`,
INVOICES.RECEIVER_NAME AS `INVOICES__RECEIVER_NAME`,
INVOICES.RECEIVER_RFC AS `INVOICES__RECEIVER_RFC`,
INVOICES.EMAIL_SENDER AS `INVOICES__EMAIL_SENDER`,
INVOICES.PDF_PATH AS `INVOICES__PDF_PATH`
FROM
PAYMENTS PAYMENTS
INNER JOIN
INVOICES_PAYMENTS INVOICESPAYMENTS
ON PAYMENTS.ID = (
INVOICESPAYMENTS.PAYMENT_ID
)
INNER JOIN
INVOICES INVOICES
ON (
INVOICES.ISSUER_ID = :C0
AND INVOICES.ID = (
INVOICESPAYMENTS.INVOICE_ID
)
)
WHERE
(
PAYMENTS.IS_DELETED = :C1
AND PAYMENTS.DATE_REGISTERED >= :C2
AND PAYMENTS.DATE_REGISTERED <= :C3
)
GROUP BY
PAYMENT_ID
ORDER BY
PAYMENTS.DATE_REGISTERED ASC
That behavior is expected, as matching will use an INNER join, and yes, grouping is how you avoid duplicates:
As this function will create an INNER JOIN, you might want to consider calling distinct on the find query as you might get duplicate rows if your conditions don’t exclude them already. This might be the case, for example, when the same users comments more than once on a single article.
Cookbook > Database Access & ORM > Query Builder > Loading Associations > Filtering by Associated Data
As the error message states, your MySQL server is configured to use the strict only_full_group_by mode, where your query is invalid. You can either disable that strict mode as mentioned by Akash prajapati (which can come with its own problems, as MySQL is then allowed to pretty much pick values of a group at random), or you could change how you query things in order to conform to the strict mode.
In your case where you need to group on the primary key, you could simply switch to using innerJoinWith() instead, unlike matching() this will not add any fields of that association to the SELECT list, and things should be fine in strict mode, as everything else is functionally dependent:
In cases where you would group on a key that would break functional dependency detection, one way to solve that could for example be to use a subquery for filtering, one that only selects that key, something along the lines of this:
$conditions = [
'Payments.is_deleted =' => false
];
$payments = $this->Payments
->find()
->contain(['Invoices.Clients']);
if($issuer) {
$matcherQuery = $this->Payments
->find()
->select(['Payments.some_other_field'])
->where($conditions)
->matching('Invoices', function ($q) use ($issuer) {
return $q->where(['Invoices.issuer_id' => $issuer['id']]);
})
->distinct('Payments.some_other_field');
$payments->where([
'Payments.some_other_field IN' => $matcherQuery
]);
} else {
$payments->where($conditions);
}
This will result in a query similar to this, where the outer query can then select all the fields you want:
SELECT
...
FROM
payments
WHERE
payments.some_other_field IN (
SELECT
payments.some_other_field
FROM
payments
INNER JOIN
invoices_payments ON
payments.id = invoices_payments.payment_id
INNER JOIN
invoices ON
invoices.issuer_id = ...
AND
invoices.id = invoices_payments.invoice_id
WHERE
payments.is_deleted = ...
GROUP BY
payments.some_other_field
)
The problem with sql_mode value in mysql so you need to set the sql_mode value as blank and then you can try and working fine for you.
SET GLOBAL sql_mode=(SELECT REPLACE(##sql_mode,'ONLY_FULL_GROUP_BY',''));
Please let me know still anything else.
I had the same issue, but was too afraid to set the sql_mode as mentioned by #Akash and also too much in a hurry to restructure the query. So I decided to use the inherited Collection method indexBy()
https://book.cakephp.org/4/en/core-libraries/collections.html#Cake\Collection\Collection::indexBy
$resultSetFromYourPaymentsQuery = $resultSetFromYourPaymentsQuery->indexBy('id');
It worked like a charm and it is DB independent.
EDIT: After some more tinkering, this might not be practical for all use cases. Replacing matching with innerJoinWith as proposed in the accepted answer will probably solve it in more generalized manner.

Zend 2 subquery columns

I want to create a SQL(MySQL) query in Zend Framework 2 like:
SELECT a.id,
a.name,
a.age,
(SELECT MAX(score)
FROM scores AS s
WHERE s.user_id = a.id) AS max_score,
(SELECT SUM(time)
FROM games_played_time AS gpt
WHERE gpt.user_id = a.id) AS time_played
FROM users AS a
ORDER BY last_visited DESC
LIMIT 0, 100
Mind that this is an artificial example of existing query.
I tried creating sub-queries and then creating main select query where when I use:
$select->columns(
array(
'id',
'name',
'age',
'max_score' => new Expression('?', array($sub1),
'time_played' => new Expression('?', array($sub2)
)
I also tried using:
$subquery = new \Zend\Db\Sql\Expression("({$sub->getSqlString()})")
And even lambda functions like suggested here: http://circlical.com/blog/2014/1/27/zend-framework-2-subqueries-subselect-and-table-gateway
Still no luck because all the time I keep getting errors like:
No data supplied for parameters in prepared statement
And when I succeed in making the query work, it ends up that column contains the text of sub-queries. It starts to look that it is not possible to make multiple expressions in columns method. Any ideas?
SOLVED:
I rewrote query by query as #Tim Klever proposed. Everythin worked except one query. It turns out there is some kind of issue when using limit in subquery and in main query. In my case one of the subqueries returns multiple rows, so I ussed limit(1) to force return of a single value. But using that turned out to produce error:
No data supplied for parameters in prepared statement
I changed the query to use MAX instead of limit and now it works. Later will try to debug why this is happening.. Thank you!
The following worked for me to produce the query you listed
$maxScoreSelect = new Select();
$maxScoreSelect->from(array('s' => 'scores'));
$maxScoreSelect->columns(array(new Expression('MAX(score)')));
$maxScoreSelect->where->addPredicates('s.user_id = a.id');
$sumTimeSelect = new Select();
$sumTimeSelect->from(array('gpt' => 'games_played_time'));
$sumTimeSelect->columns(array(new Expression('SUM(time)')));
$sumTimeSelect->where->addPredicates('gpt.user_id = a.id');
$select = new Select();
$select->from(array('a' => 'users'));
$select->columns(array(
'id',
'name',
'age',
'max_score' => new Expression('?', array($maxScoreSelect)),
'time_played' => new Expression('?', array($sumTimeSelect))
));
$select->order('last_visited DESC');
$select->limit(100);
$select->offset(0);

Selecting distinct values from key/value pairs in a seperate table

I have 2 database tables
JOBS(JOB_ID, JOB_TIME, JOB_NAME,...), JOB_PARAMETERS(JOB_ID,NAME,VALUE)
where JOB_PARAMETERS is essentially a map containing job parameter key value pairs.
Every job may have a unique parameter key/value pairs.
I am looking to pragmatically build a query that will return distinct job id's that contain key/value combinations. Where the values are actually a list of values, comparison operators.
For example:
JOB_PARAMETERS: NAME = 'OUTPUT_FILENAME', VALUE LIKE "ALEX%", "JAX%"
NAME = 'PRIORITY' , VALUE > 7
The above example would automatically filter out all jobs that don't have the OUTPUT_FILENAME and PRIORITY key. Returning All jobs that meet both conditions.
I also need to be able to support pagination and order by.
I was planning on using Perl with DBIx::Class, But I can do it in pure Perl/SQL as well.
I am open to changing the database schema, but every job can have different key/value pairs, so I cant just make them columns in the jobs table.
Thanks in advance.
When using DBIx::Class you can generate a DBIC schema by using Schema::Loader.
After connecting to the database you get a $schema object you can use to get a ResultSet filtered to return the Result objects you want:
my $rs_job_parameters = $schema->resultset('Job_Parameters')->search({
-or => [
{
'name' => 'OUTPUT_FILENAME',
'value' => [{ like => 'ALEX%'}, { like => 'JAX%' }].
},
{
'name' => 'PRIORITY',
'value' => [{ '>' => 7}].
}
]},
{
columns => [qw( job_id )],
group_by => [qw( job_id )], # alternative you can use distinct => 1 to group_by all selected columns
having => \[ 'COUNT(*) = ?', [ 2 ] ],
}
);
my #job_ids = $rs_job_parameters->get_column('job_id')->all;
One can do it in SQL, by grouping JOB_PARAMETERS by JOB_ID and filtering the groups accordingly. For example, if there is a uniqueness constraint over (JOB_ID, NAME), one can query as follows:
SELECT JOB_ID
FROM JOB_PARAMETERS
WHERE (NAME='OUTPUT_FILENAME' AND (VALUE LIKE 'ALEX%' OR VALUE LIKE 'JAX%'))
OR (NAME='PRIORITY' AND VALUE > 7)
GROUP BY JOB_ID
HAVING COUNT(*) = 2
Absent such a uniqueness constraint, COUNT(*) would have to be replaced e.g. with COUNT(DISTINCT NAME).

Zend framework order before group

Any one have a example how to order before group with leftjoin in zend paginator adapter ?
new Zend_Paginator_Adapter_DbSelect($this->db->select()
->from(array( 'a' => $this->prefix.'contest' ), array('id'))
->joinLeft(array( 'b' => $this->prefix.'logo_to_contest'),'a.id=b.contest_id', array('img'))
->group('a.id')
->order('a.end','a.date_start DESC','b.id RAND()')
)
From mysql manuel
In general, clauses used must be given in exactly the order shown in
the syntax description. For example, a HAVING clause must come after
any GROUP BY clause and before any ORDER BY clause. The exception is
that the INTO clause can appear either as shown in the syntax
description or immediately following the select_expr list.
and in the syntax description group comes before order so it has nothing to do with zend
it's mysql that requires that you put group before order.
However to get around this issue and group after ordering you can select with a subquery with order then group on a new select like :
$subselect = $db->select()
->from(array( 'a' => $this->prefix.'contest' ), array('id'))
->joinLeft(array( 'b' => $this->prefix.'logo_to_contest'),'a.id=b.contest_id', array())
->order('a.end','a.date_start DESC','b.id RAND()');
$select = $db->select()->from(a(array( 'a' => $this->prefix.'contest' ), array('id'))
->joinLeft(array( 'b' => $this->prefix.'logo_to_contest'),'a.id=b.contest_id', array('img'))
->where("a.id in ($subselect)")
->group('a.id');