I have an eloquent carry that has to filter through 30+ million records. Everything goes well speed wise.. but whenever i try to do a nested filter query it will keep loading and loading until it crashes...
This is the query:
Balance::orderBy('id', 'asc')->maincheck()->simplePaginate(300);
this below is the same query but without the "simplePaginate" behind it
SELECT *
FROM `balances`
WHERE ((
`balances`.`balanceable_type` = ?
AND EXISTS(SELECT *
FROM `balance_accounts`
WHERE `balances`.`balanceable_id` = `balance_accounts`.`id`
AND `maincheck` = ?)
))
ORDER BY `id` ASC
Does someone have an idea on how to fix this with an other way. I have tried using chunks but it sucks. I have to use eloquent.
Subselect will be significantly slower in this case as it will execute N queries.
You can try using JOIN instead but make sure there is a foreign key between balance_accounts.id and balances.balanceable_id and balanceable_type is indexed:
SELECT *
FROM `balances`
INNER JOIN `balance_accounts`
ON `balance_accounts`.`id` = `balances`.`balanceable_id`
AND `maincheck` = ?
WHERE `balances`.`balanceable_type` = ?
ORDER BY `id` ASC
I'm not really sure if this can be implemented with Eloquent filters but you can easily achieve it using Eloquent query builder.
Related
I have what seems like an easy many-to-many relationship query with pagination. It works fine, but the downside is the time it takes. On the prod server, it's more than 20 seconds. On my development environment, 13 seconds.
Here is the code:
$query = $this->excerpt->orderBy($sort, $order);
$excerpts = $query->with('source.authors')
->with('excerptType')
->with('tags')
->whereHas('tags', function($q) use ($tagId){
$q->where('tag_id', $tagId);
})
->paginate($this->paginateCount);
These two queries take the longest
select count(*) as aggregate
from `excerpt`
where (select count(*)
from `tags`
inner join `excerpt_tag`
on `tags`.`id` = `excerpt_tag`.`tag_id`
where `excerpt_tag`.`excerpt_id` = `excerpt`.`id`
and `tag_id` = '655') >= 1
2.02 secs
select *
from `excerpt`
where (select count(*) from `tags`
inner join `excerpt_tag`
on `tags`.`id` = `excerpt_tag`.`tag_id`
where `excerpt_tag`.`excerpt_id` = `excerpt`.`id`
and `tag_id` = '655') >= 1
order by `created_at` desc limit 15 offset 0
2.02 secs
I was thinking of changing this to a simple query with inner joins, like:
select *
from `excerpt`
inner join excerpt_tag on excerpt.id = excerpt_tag.excerpt_id
inner join tags on excerpt_tag.tag_id = tags.id
where tags.id = 655
limit 10 offset 0
But then I lose the advantage of eager loading and so on.
Does anyone have an idea on what the best way to speed this up would be?
Change
( SELECT COUNT(*) ... ) > 0
to
EXISTS ( SELECT 1 ... )
Follow the instructions here for index tips in many:many tables.
If a tag is just a short string, don't bother having a table (tags) for them. Instead, simply have the tag in the excerpt_tag and get rid of tag_id.
A LIMIT without an ORDER BY is somewhat meaningless -- which 10 rows you get will be unpredictable.
Well I have a solution that has led to a significant improvement and only added a few lines of code and only 1 or maybe 2 extra sql queries.
I decided to query the tags first and find out which excerpts were connected and then use a whereIn to then query all information from the excerpts, thus hoping to still make use of the with function and eager loading. At least keep the number of queries down to an absolute minimum.
Here is the code with the solution:
// workaround to make excerpt query faster
$excerptsWithTag = $this->tag->with(['excerpts' => function($query) {
$query->select('excerpt.id');
}])->find($tagId,['tags.id']);
// actual excrpt query
$excerptIds = array_column($excerptsWithTag->excerpts()->get()->toArray(), 'id');
$query = $this->excerpt->orderBy($sort, $order);
$excerpts = $query->with([
'source.authors',
'excerptType',
'tags'
])
->whereIn('excerpt.id', $excerptIds)
->paginate($this->paginateCount);
There is very likely a much more eloquent way to solve this problem, but this works and I'm happy.
I have this query with subquery.
SELECT * FROM
(SELECT module_id FROM an_modules AS m LIMIT 20 OFFSET 0) AS m
LEFT JOIN an_module_sites AS ms ON (m.module_id = ms.module_id)
How to use DBAL to build subquery like this?
This doesn't seem to be work.
$qb->select('*')
->from(
$qb->select('module_id')
->from($this->Db->getTableName('modules'), 'm')
, 'm')
->leftJoin('m', $this->Db->getTableName('module_sites'), 'ms', 'm.module_id = ms.module_id');
$stmt = $qb->execute();
$result = $stmt->fetchAll();
I recently needed to do this to implement a pagination/sorting helper. As part of this I would take a querybuilder executed by my model and and generically count the total rows it would produce if unlimited.
To be cross platform I couldn't use rowCount and potential grouping meant I couldn't just change the fields in the select - so the best option was to remove the limits and count it as a subquery. This is what I came up with:
<?php
$totalResults = $qb->getConnection()->createQueryBuilder()
->select('COUNT(*)')
->from(
'('
.$qb
->setFirstResult(null)
->setMaxResults(null)
->resetQueryPart('orderBy')
->getSQL()
.')',
'tmp')
->execute()
->fetch(\PDO::FETCH_COLUMN);
I'm not sure how doctrine ORM handles this, but in pure DBAL at least this seems to work.
I'm using this kind of queries with different parameters :
EXPLAIN SELECT SQL_NO_CACHE `ilan_genel`.`id` , `ilan_genel`.`durum` , `ilan_genel`.`kategori` , `ilan_genel`.`tip` , `ilan_genel`.`ozellik` , `ilan_genel`.`m2` , `ilan_genel`.`fiyat` , `ilan_genel`.`baslik` , `ilan_genel`.`ilce` , `ilan_genel`.`parabirimi` , `ilan_genel`.`tarih` , `kgsim_mahalleler`.`isim` AS mahalle, `kgsim_ilceler`.`isim` AS ilce, (
SELECT `ilanresimler`.`resimlink`
FROM `ilanresimler`
WHERE `ilanresimler`.`ilanid` = `ilan_genel`.`id`
LIMIT 1
) AS resim
FROM (
`ilan_genel`
)
LEFT JOIN `kgsim_ilceler` ON `kgsim_ilceler`.`id` = `ilan_genel`.`ilce`
LEFT JOIN `kgsim_mahalleler` ON `kgsim_mahalleler`.`id` = `ilan_genel`.`mahalle`
WHERE `ilan_genel`.`ilce` = '703'
AND `ilan_genel`.`durum` = '1'
AND `ilan_genel`.`kategori` = '1'
AND `ilan_genel`.`tip` = '9'
ORDER BY `ilan_genel`.`id` DESC
LIMIT 225 , 15
and this is what i get in explain section:
these are the indexes that i already tried to use:
any help will be deeply appreciated what kind of index will be the best option or should i use another table structure ?
You should first simplify your query to understand your problem better. As it appears your problem is constrained to the ilan_gen1 table, the following query would also show you the same symptoms.:
SELECT * from ilan_gene1 WHERE `ilan_genel`.`ilce` = '703'
AND `ilan_genel`.`durum` = '1'
AND `ilan_genel`.`kategori` = '1'
AND `ilan_genel`.`tip` = '9'
So the first thing to do is check that this is the case. If so, the simpler question is simply why does this query require a file sort on 3661 rows. Now the 'hepsi' index sort order is:
ilce->mahelle->durum->kategori->tip->ozelik
I've written it that way to emphasise that it is first sorted on 'ilce', then 'mahelle', then 'durum', etc. Note that your query does not specify the 'mahelle' value. So the best the index can do is lookup on 'ilce'. Now I don't know the heuristics of your data, but the next logical step in debugging this would be:
SELECT * from ilan_gene1 WHERE `ilan_genel`.`ilce` = '703'`
Does this return 3661 rows?
If so, you should be able to see what is happening. The database is using the hepsi index, to the best of it's ability, getting 3661 rows back then sorting those rows in order to eliminate values according to the other criteria (i.e. 'durum', 'kategori', 'tip').
The key point here is that if data is sorted by A, B, C in that order and B is not specified, then the best logical thing that can be done is: first a look up on A then a filter on the remaining values against C. In this case, that filter is performed via a file sort.
Possible solutions
Supply 'mahelle' (B) in your query.
Add a new index on 'ilan_gene1' that doesn't require 'mahelle', i.e. A->C->D...
Another tip
In case I have misdiagnosed your problem (easy to do when I don't have your system to test against), the important thing here is the approach to solving the problem. In particular, how to break a complicated query into a simpler query that produces the same behaviour, until you get to a very simple SELECT statement that demonstrates the problem. At this point, the answer is usually much clearer.
It's been a while since I've written raw SQL, I was hoping someone could help me out in optimizing this SQL query so that it works across, both, MySQL and PostgreSQL.
I would also have to implement this via CodeIgniter (2.x) using ActiveRecord, any help/advice?
SELECT *
FROM notaries, contact_notaries
WHERE notaries.id = contact_notaries.notary_id
AND WHERE ( contact_notaries.city LIKE %$criteria%
OR contact_notaries.state LIKE %$criteria
OR contact_notaries.address LIKE %$criteria%)
Thanks!
Each query can have just one WHERE clause (you don't need the second)
It's much better to put join condition into JOIN rather then WHERE.
Are you sure you really need all the columns from 2 tables (*)?
So I'd refactor it to
SELECT [field_list]
FROM notaries
INNER JOIN contact_notaries ON (notaries.id = contact_notaries.notary_id)
WHERE ( contact_notaries.city LIKE '%$criteria%'
OR contact_notaries.state LIKE '%$criteria'
OR contact_notaries.address LIKE '%$criteria%')
Using a1ex07's query:
SELECT [field_list]
FROM notaries
INNER JOIN contact_notaries ON (notaries.id = contact_notaries.notary_id)
WHERE ( contact_notaries.city LIKE '%$criteria%'
OR contact_notaries.state LIKE '%$criteria'
OR contact_notaries.address LIKE '%$criteria%')
Active record:
$this->db->select(); // Leave empty to select all fields
$this->db->join('contact_notaries', 'notaries.id = contact_notaries.notary_id', 'inner');
$this->db->like('contact_notaries.city', 'criteria');
$this->db->like('contact_notaries.state', 'criteria');
$this->db->like('contact_notaries.address', 'match');
$results = $this->db->get('notaries');
To specify a list of fields you can do $this->db->select('field_1, field_2, ...');.
http://codeigniter.com/user_guide/database/active_record.html
i'm having a little issue with doctrine using symfony 1.4 (I think it's using doctrine 1.2). I have 2 queries, using raw sql in the mysql console, they produce the same resultset. The queries can be generated using this code :
$dates = Doctrine::getTable('Picture')
->createQuery('a')
->select('substr(a.created_at,1,10) as date')
->leftjoin('a.PictureTag pt ON a.id = pt.picture_id')
->leftjoin('pt.Tag t ON t.id = pt.tag_id')
->where('a.created_at <= ?', date('Y-m-d 23:59:59'))
->orderBy('date DESC')
->groupby('date')
->limit(ITEMS_PER_PAGE)
->offset(ITEMS_PER_PAGE * $this->page)
->execute();
If I remove the two joins, it changes the query, but the resultset it's the same.
But using doctrine execute(), one produces only one row.
Somebody have an idea on what's going on here?
PS : Picture table has id, title, file, created_at (format 'Y-m-d h:i:s'), the Tag table is id, name and PictureTag is an relationship table with id and the two foreign keys.
PS 2 : Here are the two sql queries produced (the first without joins)
SELECT substr(l.created_at, 1, 10) AS l__0 FROM lupa_picture l WHERE (l.created_at <= '2010-03-19 23:59:59') GROUP BY l__0 ORDER BY l__0 DESC LIMIT 4
SELECT substr(l.created_at, 1, 10) AS l__0 FROM lupa_picture l LEFT JOIN lupa_picture_tag l2 ON (l.id = l2.picture_id) LEFT JOIN lupa_tag l3 ON (l3.id = l2.tag_id) WHERE (l.created_at <= '2010-03-19 23:59:59') GROUP BY l__0 ORDER BY l__0 DESC LIMIT 4
I had something similar this week. Doctrine's generated SQL (from the Symfony debug toolbar) worked fine in phpMyAdmin, but failed when running the query as in your question. Try adding in the following into your query:
->setHydrationMode(Doctrine::HYDRATE_SCALAR)
and see if it gives you the expected result. If so, it's down to the Doctrine_Collection using the Picture primary key as the index in the collection. If you have more than 1 result with the same index, Doctrine will refuse to add it into the collection, so you only end up with 1 result. I ended up running the query using a different table rather than the one I wanted, which resulted in a unique primary key and then the results I wanted appeared.
Well, the solution was that...besides substr(), it needs another column of the table. Using select(substr(), a.created_at) made it work