Laravel orderBy slowing down response greatly - mysql

Leaving out unnecessary information the table structure is as follows(not listing all the with relations):
products
id
launch_date
name
product_view_history
id
account_id
product_id
timestamps
I have query that is taking and irregularly long amount of time. With all the profiling I've done, the actual time spent in SQL is very small(<50 ms) but the time this code takes to execute is in the 900+ms range:
$this->select('products.*', DB::raw('COUNT(product_view_history.id) as view_count'))
->leftJoin('product_view_history', 'product_view_history.product_id', '=', 'products.id', 'outer')
->groupBy('product_view_history.product_id')
->orderBy('view_count', 'DESC')
->orderBy('products.id', 'DESC')
->whereNotNull('products.launch_date')
->with(['owner.images', 'owner.star', 'owner.follows', 'owner.followers', 'company.products.alphas'])
->take(Config::get('xxxx.limits.small'))
->get();
However the time it takes for this code to execute is reduced the the appropriate <50ms if I comment out ->orderBy('view_count', 'DESC'). If I swap out get() with toSql() and run both those queries manually I'm finding the times to be relatively similar and small. To be clear to measure the time this is taking is not SQL query time; I am just getting the time in milliseconds before and directly after this is done and logging the difference.
Can anyone see any reason why ->orderBy('view_count', 'DESC') would add close to a full second of time to the execution of code, even though the SQL itself is not/minimally slower?

It seems like executing the query raw and hydrating and loading seem to speed up the query. This does not answer WHY that order by would cause such an issue but it does answer how to get around the issue at hand:
$products = self::hydrate(DB::select(
"select `products`.*, COUNT(product_view_history.id) as view_count
from `products` left join `product_view_history`
on `product_view_history`.`product_id` = `products`.`id`
where `products`.`launch_date` is not null
group by `product_view_history`.`product_id`
order by `view_count` desc, `products`.`id` desc limit {$limit}"))
->load(['owner.images', 'owner.star', 'owner.follows', 'owner.followers', 'company.products.alphas']);

For me it was caused by using "wrong" data types in the where query.
For example I filtered by a column called "username" which is a varchar but inserted an Int as value to filter by.
It took very long when using orderBy but when removing the orderBy it was fast again.
The solution was at least for me to cast the username to String and the orderBy was fluent as before. I do not know the real reason but maybe Eloquent casts or sorts differently when using not matching data types.

Related

Simple Eloquent query taking too long time to execute

I have 2 queries. Even though the first one is more complicated and pulls much more data it takes only 154 ms to execute, meanwhile the second one takes 1.76 s to execute.
First (executing fast):
$offers = Offer::select(\DB::raw('tbl_offer.offer_id as sys_id,
tbl_offer.offer_name,
tbl_offer.preview_url,
COALESCE(tbl_offer.is_allow_website_links,
false) as is_allow_website_links,
tbl_offer.is_require_approval,
tbl_relationship.fk_relationship_status_id,
tbl_offer.is_private,
tbl_offer.currency'))
->leftJoin('tbl_relationship', function ($q) use ($affiliateId) {
$q->on('tbl_offer.offer_id', '=', 'tbl_relationship.fk_offer_id')
->where('tbl_relationship.fk_affiliate_id', '=', $affiliateId);})
->whereIn('fk_offer_status_id', [ 18, 19 ])
->where('is_display', 1)
->where('tbl_offer.is_trd_deleted', 0)
->orderBy('offer_name')
->get();
Second (executing slowly):
$currencies = Currency::select(\DB::raw('DISTINCT currency_code_from AS currency'))
->where('sys_name', 'openexchangerates')
->orderBy('currency')
->get();
What could possibly be the issue?
Do you have any ideas on how to decrease loading time?
first of all you are using 2 queries into one.
This is the first query:
$currencies = Currency::where('sys_name', 'openexchangerates')
->orderBy('currency')
->get();
And this is another:
\DB::raw('DISTINCT currency_code_from AS currency')
In order to use both queries into one, you should use this:
$currencies = Currency::selectRaw('DISTINCT currency_code_from AS currency')
->where('sys_name', 'openexchangerates')
->orderBy('currency')
->get();
I hope this way will decrease the executing time.
Just leaving this answer as it might be useful for the people who already tried applying indexing and query optimization, but didn't manage to reduce time significantly.
I have managed to reduce query loading time from 1.76 s to 0.127 s.
I have solved the problem by using some "walkaround". Since the currency rate is changing everyday for every single available currency, I am just getting the biggest currency_rate_batch_id, and getting all the currencies associated with this id (allows me to quickly get all distinct currencies).
However, I will apply indexing (as #Josh suggested) and avoid double queries throughout the project (as suggested by #Nicolas).
As #Nikolas said changing from select(DB::raw.. to selectRaw(... will help with the speed.
The other thing that you will want to check is your indexing on the main columns of your table.
I am assuming that you are using Mysql and so look at the below docs on indexing
https://dev.mysql.com/doc/refman/5.5/en/optimization-indexes.html
Having indexes on the key columns of tables can make a big difference to the speed of queries
The Docs have details about adding indexes through migrations here:
https://laravel.com/docs/5.5/migrations#indexes

MySQL 5.7 RAND() and IF() without LIMIT leads to unexpected results

I have the following query
SELECT t.res, IF(t.res=0, "zero", "more than zero")
FROM (
SELECT table.*, IF (RAND()<=0.2,1, IF (RAND()<=0.4,2, IF (RAND()<=0.6,3,0))) AS res
FROM table LIMIT 20) t
which returns something like this:
That's exactly what you would expect. However, as soon as I remove the LIMIT 20 I receive highly unexpected results (there are more rows returned than 20, I cut it off to make it easier to read):
SELECT t.res, IF(t.res=0, "zero", "more than zero")
FROM (
SELECT table.*, IF (RAND()<=0.2,1, IF (RAND()<=0.4,2, IF (RAND()<=0.6,3,0))) AS res
FROM table) t
Side notes:
I'm using MySQL 5.7.18-15-log and this is a highly abstracted example (real query is much more difficult).
I'm trying to understand what is happening. I do not need answers that offer work arounds without any explanations why the original version is not working. Thank you.
Update:
Instead of using LIMIT, GROUP BY id also works in the first case.
Update 2:
As requested by zerkms, I added t.res = 0 and t.res + 1 to the second example
The problem is caused by a change introduced in MySQL 5.7 on how derived tables in (sub)queries are treated.
Basically, in order to optimize performance, some subqueries are executed at different times and / or multiple times leading to unexpected results when your subquery returns non-deterministic results (like in my case with RAND()).
There are two easy (and likewise ugly) workarounds to get MySQL to "materialize" (aka return deterministic results) these subqueries: Use LIMIT <high number> or GROUP BY id both of which force MySQL to materialize the subquery and return the expected results.
The last option is turn off derived_merge in the optimizer_switch variable: derived_merge=off (make sure to leave all the other parameters as they are).
Further readings:
https://mysqlserverteam.com/derived-tables-in-mysql-5-7/
Subquery's rand() column re-evaluated for every repeated selection in MySQL 5.7/8.0 vs MySQL 5.6

Very slow query when using Eloquent whereNotIn

I have a Question model from very large table of questions (600,000 records), with relation to Customer,Answer and Product models. Relations are irrelevant to this question but I mentioned them to clarify I need to use Eloquent. When I call Question::with('customer')->get(); it runs smoothly and fast.
But there is another table in which I have question_ids of all questions which should not be shown (for specific reasons).
I tried this code:
// omitted product ids, about 95,000 records
$question_ids_in_product = DB::table('question_to_product')
->pluck('product_id')->all();
$questions = Question::with('customer')
->whereNotIn('product_id', $question_ids_in_product)
->paginate($perPage)->get();
It takes so much time and shows this error:
SQLSTATE[HY000]: General error: 1390 Prepared statement contains too many placeholders
and sometimes Fatal error: Maximum execution time of 30 seconds exceeded
When I run it with plain sql query:
SELECT * FROM questions LEFT JOIN customers USING (customer_id)
WHERE question_id NOT IN (SELECT question_id FROM question_to_product)
it takes only 80 milliseconds
How can I use Eloquent in this situation?
You can use whereRaw method:
$questions = Question::with('customer')
->whereRaw('question_id NOT IN (SELECT question_id FROM question_to_product)')
->paginate($perPage)->get();
But ideally as you found out this is a better sollution:
Question::with('customer')->whereNotIn('question_id',
function ($query) {
$query->from('question_to_product') ->select('question_id');
}
);
Difference?
When you will migrate your database to another database the whereRaw might not work as you put in raw statements.
That is why we have Eloquent ORM which handles these transitions and build the appropriate queries to run.
No performance impact because the SQL is the same (for MySQL)
P.S: For better debugging try installing this debug bar
refer from https://laravel.com/docs/5.4/queries#where-clauses
$users = DB::table('questions')
->leftJoin('customers', 'curtomer.id', '=', 'question.user_id')
->whereNotIn('question_id', [1, 2, 3])
->get();
It'll work 100%. When you query getting longer to response like more than 30 seconds when you are using whereNotIn. Use this Query Syntax.
$order = Order::on($databaseCredentials['database'])
->whereRaw('orders_id NOT IN (SELECT orders_id FROM orders)')
->skip($page)
->take(10)
->orderBy('orders.updated_at', 'ASC')
->paginate(10);

Error: MySQL client ran out of memory

Can anyone please advise me on this error...
The database has 40,000 news stories but only the fields 'story' is large,
'old' is a numeric value 0 or 1,
'title' and 'shortstory' are very short or NULL.
any advice appreciated. This is the result of running a search database query.
Error: MySQL client ran out of memory
Statement: SELECT news30_access.usehtml, old, title, story, shortstory, news30_access.name AS accessname, news30_users.user AS authorname, timestamp, news30_story.id AS newsid FROM news30_story LEFT JOIN news30_users ON news30_story.author = news30_users.uid LEFT JOIN news30_access ON news30_users.uid = news30_access.uid WHERE title LIKE ? OR story LIKE ? OR shortstory LIKE ? OR news30_users.user LIKE ? ORDER BY timestamp DESC
The simple answer is: don't use story in the SELECT clause.
If you want the story, then limit the number of results being returned. Start with, say, 100 results by adding:
limit 100
to the end of the query. This will get the 100 most recent stories.
I also note that you are using like with story as well as other string columns. You probably want to be using match with a full text index. This doesn't solve your immediate problem (which is returning too much data to the client). But, it will make your queries run faster.
To learn about full text search, start with the documentation.

Eloquent count distinct returns wrong totals

i'm having an issue with how eloquent is formulation a query that i have no access to. When doing something like
$model->where('something')
->distinct()
->paginate();
eloquent runs a query to get the total count, and the query looks something like
select count(*) as aggregate from .....
The problem is that if you use distinct in the query, you want something like
select count(distinct id) as aggregate from .....
to get the correct total. Eloquent is not doing that though, thus returning wrong totals. The only way to get the distinct in count is to pass an argument through the query builder like so ->count('id') in which case it will add it. Problem is that this query is auto-generated and i have no control over it.
Is there a way to trick it into adding the distinct on the count query?
P.S Digging deep into the builders code we find an IF statement asking for a field on the count() method in order to add the distinct property to the count. Illuminate\Database\Query\Grammars\BaseGrammar#compileAggregate
if ($query->distinct && $column !== '*')
{
$column = 'distinct '.$column;
}
return 'select '.$aggregate['function'].'('.$column.') as aggregate';
P.S.1 I know that in SQL you could do a group by, but since i'm eager loading stuff it is not a good idea cause it will add a IN (number of id's found) to each of the other queries which slows things down significantly.
I faced the exact same problem and found two solutions:
The bad one:
$results = $model->groupBy('foo.id')->paginate();
It works but it will costs too much memory (and time) if you have a high number of rows (it was my case).
The better one:
$ids = $model->distinct()->pluck('foo.id');
$results = $query = $model->whereIn('foo.id', $ids)->paginate();
I tried this with 100k results, and had no problem at all.
This seems to be a wider problem, discussed here:
https://github.com/laravel/framework/issues/3191
https://github.com/laravel/framework/pull/4088
Untill the fixes are shipped with one of the next Laravel releases, you can always try using the raw expressions, like below (I didnt test it, but should work)
$stuff = $model->select(DB::raw('distinct id as did'))
->where('whatever','=','whateverelse')
->paginate();
Reference: http://laravel.com/docs/queries#raw-expressions
$model->where('something')->distinct()->count('id')->paginate();