I have 2 queries. Even though the first one is more complicated and pulls much more data it takes only 154 ms to execute, meanwhile the second one takes 1.76 s to execute.
First (executing fast):
$offers = Offer::select(\DB::raw('tbl_offer.offer_id as sys_id,
tbl_offer.offer_name,
tbl_offer.preview_url,
COALESCE(tbl_offer.is_allow_website_links,
false) as is_allow_website_links,
tbl_offer.is_require_approval,
tbl_relationship.fk_relationship_status_id,
tbl_offer.is_private,
tbl_offer.currency'))
->leftJoin('tbl_relationship', function ($q) use ($affiliateId) {
$q->on('tbl_offer.offer_id', '=', 'tbl_relationship.fk_offer_id')
->where('tbl_relationship.fk_affiliate_id', '=', $affiliateId);})
->whereIn('fk_offer_status_id', [ 18, 19 ])
->where('is_display', 1)
->where('tbl_offer.is_trd_deleted', 0)
->orderBy('offer_name')
->get();
Second (executing slowly):
$currencies = Currency::select(\DB::raw('DISTINCT currency_code_from AS currency'))
->where('sys_name', 'openexchangerates')
->orderBy('currency')
->get();
What could possibly be the issue?
Do you have any ideas on how to decrease loading time?
first of all you are using 2 queries into one.
This is the first query:
$currencies = Currency::where('sys_name', 'openexchangerates')
->orderBy('currency')
->get();
And this is another:
\DB::raw('DISTINCT currency_code_from AS currency')
In order to use both queries into one, you should use this:
$currencies = Currency::selectRaw('DISTINCT currency_code_from AS currency')
->where('sys_name', 'openexchangerates')
->orderBy('currency')
->get();
I hope this way will decrease the executing time.
Just leaving this answer as it might be useful for the people who already tried applying indexing and query optimization, but didn't manage to reduce time significantly.
I have managed to reduce query loading time from 1.76 s to 0.127 s.
I have solved the problem by using some "walkaround". Since the currency rate is changing everyday for every single available currency, I am just getting the biggest currency_rate_batch_id, and getting all the currencies associated with this id (allows me to quickly get all distinct currencies).
However, I will apply indexing (as #Josh suggested) and avoid double queries throughout the project (as suggested by #Nicolas).
As #Nikolas said changing from select(DB::raw.. to selectRaw(... will help with the speed.
The other thing that you will want to check is your indexing on the main columns of your table.
I am assuming that you are using Mysql and so look at the below docs on indexing
https://dev.mysql.com/doc/refman/5.5/en/optimization-indexes.html
Having indexes on the key columns of tables can make a big difference to the speed of queries
The Docs have details about adding indexes through migrations here:
https://laravel.com/docs/5.5/migrations#indexes
Related
I have got an big query that basically just adds a couple of with()'s, whereBetween() and orderBy. At the end i use a simplePaginate(500) because i don't want to call all 30mil records obviously.
Problem is that whenever i use a join than it will take couple minutes to load it, but when i dont use it than it only takes 2 seconds.
my query->join:
$query
->join('balance_accounts', 'balances.balanceable_id','=', 'balance_accounts.id')
->join('currencies', 'currencies.id', "=", 'balance_accounts.currency_id')
->join('balance_types', 'balance_types.id', "=", 'balance_accounts.balance_type_id')
->select("balances.*", "currencies.short_name", "balance_types.name");
i have put indexes on every column. Maybe im putting indexes wrong because it's foreign, i have no idea..
here are my index that i have put on table 'balances':
here are my index that i have put on table 'balance_accounts':
here my index for 'currencies':
I have a Question model from very large table of questions (600,000 records), with relation to Customer,Answer and Product models. Relations are irrelevant to this question but I mentioned them to clarify I need to use Eloquent. When I call Question::with('customer')->get(); it runs smoothly and fast.
But there is another table in which I have question_ids of all questions which should not be shown (for specific reasons).
I tried this code:
// omitted product ids, about 95,000 records
$question_ids_in_product = DB::table('question_to_product')
->pluck('product_id')->all();
$questions = Question::with('customer')
->whereNotIn('product_id', $question_ids_in_product)
->paginate($perPage)->get();
It takes so much time and shows this error:
SQLSTATE[HY000]: General error: 1390 Prepared statement contains too many placeholders
and sometimes Fatal error: Maximum execution time of 30 seconds exceeded
When I run it with plain sql query:
SELECT * FROM questions LEFT JOIN customers USING (customer_id)
WHERE question_id NOT IN (SELECT question_id FROM question_to_product)
it takes only 80 milliseconds
How can I use Eloquent in this situation?
You can use whereRaw method:
$questions = Question::with('customer')
->whereRaw('question_id NOT IN (SELECT question_id FROM question_to_product)')
->paginate($perPage)->get();
But ideally as you found out this is a better sollution:
Question::with('customer')->whereNotIn('question_id',
function ($query) {
$query->from('question_to_product') ->select('question_id');
}
);
Difference?
When you will migrate your database to another database the whereRaw might not work as you put in raw statements.
That is why we have Eloquent ORM which handles these transitions and build the appropriate queries to run.
No performance impact because the SQL is the same (for MySQL)
P.S: For better debugging try installing this debug bar
refer from https://laravel.com/docs/5.4/queries#where-clauses
$users = DB::table('questions')
->leftJoin('customers', 'curtomer.id', '=', 'question.user_id')
->whereNotIn('question_id', [1, 2, 3])
->get();
It'll work 100%. When you query getting longer to response like more than 30 seconds when you are using whereNotIn. Use this Query Syntax.
$order = Order::on($databaseCredentials['database'])
->whereRaw('orders_id NOT IN (SELECT orders_id FROM orders)')
->skip($page)
->take(10)
->orderBy('orders.updated_at', 'ASC')
->paginate(10);
Leaving out unnecessary information the table structure is as follows(not listing all the with relations):
products
id
launch_date
name
product_view_history
id
account_id
product_id
timestamps
I have query that is taking and irregularly long amount of time. With all the profiling I've done, the actual time spent in SQL is very small(<50 ms) but the time this code takes to execute is in the 900+ms range:
$this->select('products.*', DB::raw('COUNT(product_view_history.id) as view_count'))
->leftJoin('product_view_history', 'product_view_history.product_id', '=', 'products.id', 'outer')
->groupBy('product_view_history.product_id')
->orderBy('view_count', 'DESC')
->orderBy('products.id', 'DESC')
->whereNotNull('products.launch_date')
->with(['owner.images', 'owner.star', 'owner.follows', 'owner.followers', 'company.products.alphas'])
->take(Config::get('xxxx.limits.small'))
->get();
However the time it takes for this code to execute is reduced the the appropriate <50ms if I comment out ->orderBy('view_count', 'DESC'). If I swap out get() with toSql() and run both those queries manually I'm finding the times to be relatively similar and small. To be clear to measure the time this is taking is not SQL query time; I am just getting the time in milliseconds before and directly after this is done and logging the difference.
Can anyone see any reason why ->orderBy('view_count', 'DESC') would add close to a full second of time to the execution of code, even though the SQL itself is not/minimally slower?
It seems like executing the query raw and hydrating and loading seem to speed up the query. This does not answer WHY that order by would cause such an issue but it does answer how to get around the issue at hand:
$products = self::hydrate(DB::select(
"select `products`.*, COUNT(product_view_history.id) as view_count
from `products` left join `product_view_history`
on `product_view_history`.`product_id` = `products`.`id`
where `products`.`launch_date` is not null
group by `product_view_history`.`product_id`
order by `view_count` desc, `products`.`id` desc limit {$limit}"))
->load(['owner.images', 'owner.star', 'owner.follows', 'owner.followers', 'company.products.alphas']);
For me it was caused by using "wrong" data types in the where query.
For example I filtered by a column called "username" which is a varchar but inserted an Int as value to filter by.
It took very long when using orderBy but when removing the orderBy it was fast again.
The solution was at least for me to cast the username to String and the orderBy was fluent as before. I do not know the real reason but maybe Eloquent casts or sorts differently when using not matching data types.
I have a query to post last 10 tv show episodes by sorting it by date (from newest to oldest) like this:
return $this->getEntityManager()->createQuery('SELECT t FROM AppBundle:TvShow t JOIN t.episodes e ORDER BY e.date DESC')->setFirstResult(0)->setMaxResults(10)->getResult();
It returns only 9 nine episode. We have similar queries in same page too, they are working fine. When i setMaxResults to (11) just then it returns 10 episodes.
Another issue related with this query is: it takes too long compared to other similar queries. (about 200ms)
What do you suggest for me?
Thanks in advance.
Like in Richard answer - wrong result with setMaxResults and fetch-joined collection is doctrine normal behaviour.
To make it works you can use Doctrine Pagination (from Doctrine 2.2) (docs: http://docs.doctrine-project.org/en/latest/tutorials/pagination.html)
Example usage:
use Doctrine\ORM\Tools\Pagination\Paginator;
$query->setMaxResults($limit);
$query->setFirstResult($offset);
$results = new Paginator($query, $fetchJoin = true);
Long query time looks like a topic for another question.
Straight from the documentation:
If your query contains a fetch-joined collection specifying the result limit methods are not working as you would expect. Set Max Results restricts the number of database result rows, however in the case of fetch-joined collections one root entity might appear in many rows, effectively hydrating less than the specified number of results.
https://doctrine-orm.readthedocs.org/en/latest/
So basicly the problem is in query SELECT COUNT(*) which executed in calculateTotalItemCount function in activedataprovider. As i understood it needed for pagination for $itemcount variable. The problem is this query slow for big tables. For my ~30m table it executes 5 seconds.
So there are 2 ways to solve this problem:
1. Disable pagination ('pagination'=>'false') and write own pagination.
2. Rewrite AR count function.
I dont have enough experience/knowledge to acomplish this.
Maybe some one had same issues before and can share his solution.
Atleast for totalItemCount we can use EXPLAIN SELECT *. Its way more faster.
I appreciate any help. Thank you.
If you have a "cheaper" query in raw SQL than the one that active records create automatically, you can also query manually (e.g. through DAO) and set the totalItemCount on your data provider:
$count = Yii::app()->db->createCommand('SELECT COUNT(*)...')->queryScalar();
$provider = new CActiveDataProvider('SomeModel', array(
'totalItemCount' => $count,
'criteria' => $criteria,
...