So basicly the problem is in query SELECT COUNT(*) which executed in calculateTotalItemCount function in activedataprovider. As i understood it needed for pagination for $itemcount variable. The problem is this query slow for big tables. For my ~30m table it executes 5 seconds.
So there are 2 ways to solve this problem:
1. Disable pagination ('pagination'=>'false') and write own pagination.
2. Rewrite AR count function.
I dont have enough experience/knowledge to acomplish this.
Maybe some one had same issues before and can share his solution.
Atleast for totalItemCount we can use EXPLAIN SELECT *. Its way more faster.
I appreciate any help. Thank you.
If you have a "cheaper" query in raw SQL than the one that active records create automatically, you can also query manually (e.g. through DAO) and set the totalItemCount on your data provider:
$count = Yii::app()->db->createCommand('SELECT COUNT(*)...')->queryScalar();
$provider = new CActiveDataProvider('SomeModel', array(
'totalItemCount' => $count,
'criteria' => $criteria,
...
Related
I have a Question model from very large table of questions (600,000 records), with relation to Customer,Answer and Product models. Relations are irrelevant to this question but I mentioned them to clarify I need to use Eloquent. When I call Question::with('customer')->get(); it runs smoothly and fast.
But there is another table in which I have question_ids of all questions which should not be shown (for specific reasons).
I tried this code:
// omitted product ids, about 95,000 records
$question_ids_in_product = DB::table('question_to_product')
->pluck('product_id')->all();
$questions = Question::with('customer')
->whereNotIn('product_id', $question_ids_in_product)
->paginate($perPage)->get();
It takes so much time and shows this error:
SQLSTATE[HY000]: General error: 1390 Prepared statement contains too many placeholders
and sometimes Fatal error: Maximum execution time of 30 seconds exceeded
When I run it with plain sql query:
SELECT * FROM questions LEFT JOIN customers USING (customer_id)
WHERE question_id NOT IN (SELECT question_id FROM question_to_product)
it takes only 80 milliseconds
How can I use Eloquent in this situation?
You can use whereRaw method:
$questions = Question::with('customer')
->whereRaw('question_id NOT IN (SELECT question_id FROM question_to_product)')
->paginate($perPage)->get();
But ideally as you found out this is a better sollution:
Question::with('customer')->whereNotIn('question_id',
function ($query) {
$query->from('question_to_product') ->select('question_id');
}
);
Difference?
When you will migrate your database to another database the whereRaw might not work as you put in raw statements.
That is why we have Eloquent ORM which handles these transitions and build the appropriate queries to run.
No performance impact because the SQL is the same (for MySQL)
P.S: For better debugging try installing this debug bar
refer from https://laravel.com/docs/5.4/queries#where-clauses
$users = DB::table('questions')
->leftJoin('customers', 'curtomer.id', '=', 'question.user_id')
->whereNotIn('question_id', [1, 2, 3])
->get();
It'll work 100%. When you query getting longer to response like more than 30 seconds when you are using whereNotIn. Use this Query Syntax.
$order = Order::on($databaseCredentials['database'])
->whereRaw('orders_id NOT IN (SELECT orders_id FROM orders)')
->skip($page)
->take(10)
->orderBy('orders.updated_at', 'ASC')
->paginate(10);
the code below is executed but it brings a wrong records(all records in my table) , it's like he doesn't take on consideration the clause where
$result = $this->Posts->query('SELECT * FROM POSTS WHERE id=1');
I know that I can do it easily with find() but for some reasons I want to write the sql statement and to have the right results
Thanks for helping me.
query() method does not take any parameter. you can use it like
$data= $this->Posts->query()
->where(['id'=>1])
->execute()
->fetchAll();
i'm having an issue with how eloquent is formulation a query that i have no access to. When doing something like
$model->where('something')
->distinct()
->paginate();
eloquent runs a query to get the total count, and the query looks something like
select count(*) as aggregate from .....
The problem is that if you use distinct in the query, you want something like
select count(distinct id) as aggregate from .....
to get the correct total. Eloquent is not doing that though, thus returning wrong totals. The only way to get the distinct in count is to pass an argument through the query builder like so ->count('id') in which case it will add it. Problem is that this query is auto-generated and i have no control over it.
Is there a way to trick it into adding the distinct on the count query?
P.S Digging deep into the builders code we find an IF statement asking for a field on the count() method in order to add the distinct property to the count. Illuminate\Database\Query\Grammars\BaseGrammar#compileAggregate
if ($query->distinct && $column !== '*')
{
$column = 'distinct '.$column;
}
return 'select '.$aggregate['function'].'('.$column.') as aggregate';
P.S.1 I know that in SQL you could do a group by, but since i'm eager loading stuff it is not a good idea cause it will add a IN (number of id's found) to each of the other queries which slows things down significantly.
I faced the exact same problem and found two solutions:
The bad one:
$results = $model->groupBy('foo.id')->paginate();
It works but it will costs too much memory (and time) if you have a high number of rows (it was my case).
The better one:
$ids = $model->distinct()->pluck('foo.id');
$results = $query = $model->whereIn('foo.id', $ids)->paginate();
I tried this with 100k results, and had no problem at all.
This seems to be a wider problem, discussed here:
https://github.com/laravel/framework/issues/3191
https://github.com/laravel/framework/pull/4088
Untill the fixes are shipped with one of the next Laravel releases, you can always try using the raw expressions, like below (I didnt test it, but should work)
$stuff = $model->select(DB::raw('distinct id as did'))
->where('whatever','=','whateverelse')
->paginate();
Reference: http://laravel.com/docs/queries#raw-expressions
$model->where('something')->distinct()->count('id')->paginate();
I have a query that's returning a LOT of results and my code is running out of memory trying to parse the results... how can I run a query in CakePHP and just get normal results?
By parsing it I mean....
SELECT table1.*, table2.* FROM table1 INNER JOIN table2 ON table1.id = table2.table1_id
With the above query it'll return....
array(
0 => array(
'table1' => array(
'field1' => value,
'field2' => value
),
'table2' => array(
'field1' => value,
'field2' => value
)
)
)
When it parses those results into nested arrays is when it's running out of memory.... how do I avoid this?
I couldn't hate CakePHP any more than I do right now :-\ If the documentation was decent that would be one thing, but it's not decent and it's functionality is annoying.
you could do:
$list = $this->AnyModel->query("SELECT * FROM big_table");
but i dont think that will solve your problem, because if you have, for exemple, 10millon rows.. php wont be able to manage an array of 10millon values...
but you might want to read this two links to change the execution time and the memory limit.. you could also change them on your php.ini
Good Luck!
EDITED
hmm thanks to your question i've learned something :P First of all, we all agree that you're receiving that error because Cake executes the query and tries to store the results in one array but php doesn't support an array that big so it runs out of memory and crashes.. I have never used the classic mysql_query() (i prefer PDO) but after reading the docs, it seems that mysql_query stores the results inside a resource therefore, it's not loading the results on memory, and that allows you to loop the results (like looping though a big file). So now i see the difference... and your question is actually, this one:
Can I stop CakePHP fetching all rows for a query?
=) i understand your frustration with cake, sometimes i also get frustrated with it (could you believe there's no simple way to execute a query with a HAVING clause?? u_U)
Cheers!
I'd suggest you utilize the Containable behavior on your model. This is the easiest way to control the amount of data that's returned. I've confident that this is precisely what you need to implement.
CakePHP :: Containable :: Core Behaviors
You should limit the rows returned from your query (like 500 rows) and allow the user to fetch more rows when needed (next 500 rows at a time). You could do that nicely with the pagination component and a little AJAX.
Is there a way to speed up the performance of this query.
I have indexes on tswProjectID and tswWeekEdning.
This SQL was generated from my Linq statement which is
what I want to use in my C# code.
Is there a more efficient way to write this?
var qry = (from tsw in TimesheetWeeklies where tsw.TswProjectID == 8263 select tsw).OrderByDescending(x => x.TswWeekEnding).FirstOrDefault();
SELECT TOP (1) [t0].[tswID] AS [TswID]
FROM [TimesheetWeekly] AS [t0]
WHERE [t0].[tswProjectID] = 8263
ORDER BY [t0].[tswWeekEnding] DESC
Try create an index with both columns in it (tswProjectID, tswWeekEnding)
It wont make the query any faster but if you make it a compiled query you could possibly save some time that it takes to build the query if it's done more than once, more info here:
http://msdn.microsoft.com/en-us/library/bb399335.aspx