Limit doesn't work well in yii2 query - yii2

I want to select 20 record in my database so I use this code to do that:
$query = Course::find()
->alias("t")
->select([
't.id', 't.subtitle', 't.title',
't.info', 't.skill_level_id', 't.special',
't.created', 't.modified', 't.price',
't.training_type_id', 't.media_id', 't.instructor_id',
't.extension_type'
])
->where(["t.deleted" => 0])
->joinWith([
'skillLevel', "courseTarget", "requirement", 'categoryCourses', "media",
"instructor", "trainingType"
]);
$query->limit(20);
return $query->all();
but this code will select just 5 records,
When I remove the joinWith part, my code works fine and select 20 records.
Modified code:
$query = Course::find()
->alias("t")
->select([
't.id', 't.subtitle', 't.title',
't.info', 't.skill_level_id', 't.special',
't.created', 't.modified', 't.price',
't.training_type_id', 't.media_id', 't.instructor_id',
't.extension_type'
])
->where(["t.deleted" => 0]);
$query->limit(20);
return $query->all();
The modified code will return 20 records.
UPDATE1:
When I remove the limit(20) it will return 496 records but when I added limit(20) It just returns 5.
$query = Course::find()
->alias("t")
->select([
't.id', 't.subtitle', 't.title',
't.info', 't.skill_level_id', 't.special',
't.created', 't.modified', 't.price',
't.training_type_id', 't.media_id', 't.instructor_id',
't.extension_type'
])
->where(["t.deleted" => 0])
->joinWith([
'skillLevel', "courseTarget", "requirement", 'categoryCourses', "media",
"instructor", "trainingType"
]);
return $query->all()
This code works fine and returns everything but the limit() query will made the response wrong

This is probably because Course have hasMany relation with multiple items. So if you have Course with 2 requirements, such query will return 2 rows with duplicated Course data for each requirement. Duplicated data is merged into one Course model, so 2 rows becomes one model. This is exactly what is happened here - query returns 20 rows, but it has only 5 unique Course models.
If you don't need to access relations in SQL query (for example for filtering), you can replace joinWith() with with() - it will not perform JOIN on SQL query, only register eager loading for relations. You can read more about differences between these two methods here.
If you need this JOIN you should probably group results by Course ID, to avoid duplicated Course rows.
$query->groupBy('t.id');

Related

Add result of subquery to Eloquent query in Laravel 9

In Laravel 9 I am trying to add the result of a subquery to a query(for lack of better wording) and I am stuck. More concretely, I am trying to load all products and at the same time add information about whether the current user has bought that product.
Why do I want to do this?
I am currently loading all products, then loading all bought products, then comparing the 2 to determine if the user has bought a product, but that means extra queries which I would like to avoid. Pretend for the sake of this question that pagination doesn't exist(because when paginating the impact of those multiple queries is far diminished).
There is a many to many relationship between the 2 tables users and products, so these relationships are defined on the models:
public function products()
{
return $this->belongsToMany(Product::class);
}
and
public function users()
{
return $this->belongsToMany(User::class);
}
What I have tried so far:
I created a model for the join table and tried to use selectRaw to add the extra 'column' I want. This throws a SQL syntax error and I couldn't fix it.
$products = Product::query()
->select('id', 'name')
->selectRaw("ProductUser::where('user_id',$user->id)->where('product_id','products.id')->exists() as is_bought_by_auth_user")
->get();
I tried to use addSelect but that also didn't work.
$products = Product::query()
->select('id', 'name')
->addSelect(['is_bought_by_auth_user' => ProductUser::select('product_id')->where('user_id',$user?->id)->where('product_id','product.id')->first()])
->get();
I don't even need a select, I actually just need ProductUser::where('user_id',$user?->id)->where('product_id','product.id')->exists() but I don't know a method like addSelect for that.
The ProductUser table is defined fine btw, tried ProductUser::where('user_id',$user?->id)->where('product_id','product.id')->exists() with hardcoded product id and that worked as expected.
I tried to create a method on the product model hasBeenBoughtByAuthUser in which I wanted to check if Auth::user() bought the product but Auth wasn't recognized for some reason(and I thought it's not really nice to use Auth in the model anyway so didn't dig super deep with this approach).
$products = Product::query()
->select('id', 'name')
->addSelect(\DB::raw("(EXISTS (SELECT * FROM product_user WHERE product_users.product_id = product.id AND product_users.user_id = " . $user->id . ")) as is_bought_by_auth_user"))
->simplePaginate(40);
For all attempts $user=$request->user().
I don't know if I am missing something easy here but any hints in the right direction would be appreciated(would prefer not to use https://laravel.com/docs/9.x/eloquent-resources but if there is no other option I will try that as well).
Thanks for reading!
This should do,
$id = auth()->user()->id;
$products = Product::select(
'id',
'name',
DB::raw(
'(CASE WHEN EXISTS (
SELECT 1
FROM product_users
WHERE product_users.product_id = products.id
AND product_users.user_id = '.$id.'
) THEN "yes" ELSE "no" END) AS purchased'
)
);
return $products->paginate(10);
the collection will have purchased data which either have yes or no value
EDIT
If you want eloquent way you can try using withExists or withCount
i.e.
withExists the purchased field will have boolean value
$products = Product::select('id', 'name')->withExists(['users as purchased' => function($query) {
$query->where('user_id', auth()->user()->id);
}]);
withCount the purchased field will have count of found relationship rows
$products = Product::select('id', 'name')->withCount(['users as purchased' => function($query) {
$query->where('user_id', auth()->user()->id);
}]);

Laravel Query builder where() for when Product has to have multiple tags (with product_tags pivot table many-to-many)

I am new to Laravel and I got a complicated query to build. I managed to sort it except for when a user asks for multiple tags (tags = 1, 2, 3). Product shown has to have all tags that the user asks (not only one or two but all of them).
I have the query in SQL (this example is two tags, I would switch it to different numbers based on how many tags are passed):
SELECT m.*
FROM meals m
WHERE EXISTS (
SELECT 1
FROM meals_tags t
WHERE m.id = t.meals_id AND
t.tags_id IN (227,25)
HAVING COUNT(1) = 2
);
This one works perfectly, but I have an issue when translating it to an Eloquent query builder where method.
I have another where method included in the same query so I want to attach this one as well.
I have tried this:
DB::table('meals')
->select('id')
->where(function ($query) use ($parameters) {
if (isset($parameters['tags'])) {
$array = explode(',', $parameters['tags']);
$query->select(DB::raw(1))
->from('meals_tags')
->where('meals.id', '=', 'meals_tags.meals_id')
->whereIn('meals_tags.tags_id', $array)
->having(DB::raw('COUNT(1)'), '=', count($parameters['tags']));
}
});
But I can't find a way. New to Laravel and PHP.
Let's say I have table meals and tags with meals_tags to connect them (many to many).
$paramaters are comming from GET (...?tags=1,2,3&lang=en&another=something...), they are an array of key-value pairs (['tags' => '1,2,3', 'lang' => 'en'])
$parameters['tags'] is of type string with comma separated numbers (positive integers greater than 0) so that I have the option to explode it if needed somewhere.
Assuming that you have defined belongsToMany (Many-to-Many) meal_tags relationship on the Meal model, you can try
Meal::select('id')
->when(
$request->has('tags'),
function($query) use ($request) {
$requestedTagIds = explode(',', $request->tags);
return $query->whereHas(
'meal_tags',
fn ($query) => $query->whereIn('tags_id', $requestedTagIds),
'=',
count($requestedTagIds)
);
}
)
->get();

Laravel query to get results having at least one element of an array?

I have posts and postWriters tables with one-to-many relationships. (I have also writers table).
Each post has been written by several writers collaboratively.
I want to get first 20 posts which have been written by at least one writer I follow.
For example the writers that I follow:
$arrayOfWriterIds_IFollow = [3, 5, 123, 45, ..., 3456] // total 100 ids
The query:
$posts = Post::where(
array( 'postWriters' => function($l) {
$l->whereIn('writer_id', $arrayOfWriterIds_IFollow); // or array(3, 5) for 2 writers..
})
)
->orderBy('submitTimestamp', 'desc')
->take(20)
->get();
The query does not work.
Is this approach appropriate or should I add a dedicated table to evaluate my results?
If your relationship is defined correctly then the fillowing code should work:
$posts = Post::whereHas('postWriters', function($query) use($arrayOfWriterIds_IFollow) {
$query->whereIn('writer_id', $arrayOfWriterIds_IFollow);
})
->orderBy('submitTimestamp', 'desc')
->take(20)
->get();

one to many query with where Has returns unwanted results

I have a query and don't find the correct solution for this:
$orders = Shipping::with('orderitems')
->whereHas('orderitems', function($query) {
$query->where('status','=',NULL);
})
->paginate(10);
I'm querying a one to many relationship
public function orderitems() {
return $this->hasMany('OrderItems');
}
Above query returns the correct resultset when all entries within 'orderItems' fit the condition, having status 'NULL'. The problem is, when I have an order with 2 items and one of the two items doesn't have status 'NULL', both items are returned though I only want the item which actually has status 'NULL' being returned.
I've tried it with a whereNested query and a query scope, but I think this is not the solution to the problem. Can someone assist?
whereHas will actually filter Shipping based on if they have at least one orderitem that matches the condition. I'm pretty sure what you want is use with with a condition:
$orders = Shipping::with(['orderitems' => function($query){
$query->where('status','=',NULL);
}])->paginate(10);
Edit
Add a whereHas if you also want only order with at least one item with status NULL:
$orders = Shipping::with(['orderitems' => function($query){
$query->where('status','=',NULL);
}])->whereHas('orderitems', function($query){
$query->where('status','=',NULL);
})->paginate(10);

Import of 50K+ Records in MySQL Gives General error: 1390 Prepared statement contains too many placeholders

Has anyone ever come across this error: General error: 1390 Prepared statement contains too many placeholders
I just did an import via SequelPro of over 50,000 records and now when I go to view these records in my view (Laravel 4) I get General error: 1390 Prepared statement contains too many placeholders.
The below index() method in my AdminNotesController.php file is what is generating the query and rendering the view.
public function index()
{
$created_at_value = Input::get('created_at_value');
$note_types_value = Input::get('note_types_value');
$contact_names_value = Input::get('contact_names_value');
$user_names_value = Input::get('user_names_value');
$account_managers_value = Input::get('account_managers_value');
if (is_null($created_at_value)) $created_at_value = DB::table('notes')->lists('created_at');
if (is_null($note_types_value)) $note_types_value = DB::table('note_types')->lists('type');
if (is_null($contact_names_value)) $contact_names_value = DB::table('contacts')->select(DB::raw('CONCAT(first_name," ",last_name) as cname'))->lists('cname');
if (is_null($user_names_value)) $user_names_value = DB::table('users')->select(DB::raw('CONCAT(first_name," ",last_name) as uname'))->lists('uname');
// In the view, there is a dropdown box, that allows the user to select the amount of records to show per page. Retrieve that value or set a default.
$perPage = Input::get('perPage', 10);
// This code retrieves the order from the session that has been selected by the user by clicking on a table column title. The value is placed in the session via the getOrder() method and is used later in the Eloquent query and joins.
$order = Session::get('account.order', 'company_name.asc');
$order = explode('.', $order);
$notes_query = Note::leftJoin('note_types', 'note_types.id', '=', 'notes.note_type_id')
->leftJoin('users', 'users.id', '=', 'notes.user_id')
->leftJoin('contacts', 'contacts.id', '=', 'notes.contact_id')
->orderBy($order[0], $order[1])
->select(array('notes.*', DB::raw('notes.id as nid')));
if (!empty($created_at_value)) $notes_query = $notes_query->whereIn('notes.created_at', $created_at_value);
$notes = $notes_query->whereIn('note_types.type', $note_types_value)
->whereIn(DB::raw('CONCAT(contacts.first_name," ",contacts.last_name)'), $contact_names_value)
->whereIn(DB::raw('CONCAT(users.first_name," ",users.last_name)'), $user_names_value)
->paginate($perPage)->appends(array('created_at_value' => Input::get('created_at_value'), 'note_types_value' => Input::get('note_types_value'), 'contact_names_value' => Input::get('contact_names_value'), 'user_names_value' => Input::get('user_names_value')));
$notes_trash = Note::onlyTrashed()
->leftJoin('note_types', 'note_types.id', '=', 'notes.note_type_id')
->leftJoin('users', 'users.id', '=', 'notes.user_id')
->leftJoin('contacts', 'contacts.id', '=', 'notes.contact_id')
->orderBy($order[0], $order[1])
->select(array('notes.*', DB::raw('notes.id as nid')))
->get();
$this->layout->content = View::make('admin.notes.index', array(
'notes' => $notes,
'created_at' => DB::table('notes')->lists('created_at', 'created_at'),
'note_types' => DB::table('note_types')->lists('type', 'type'),
'contacts' => DB::table('contacts')->select(DB::raw('CONCAT(first_name," ",last_name) as cname'))->lists('cname', 'cname'),
'accounts' => Account::lists('company_name', 'company_name'),
'users' => DB::table('users')->select(DB::raw('CONCAT(first_name," ",last_name) as uname'))->lists('uname', 'uname'),
'notes_trash' => $notes_trash,
'perPage' => $perPage
));
}
Any advice would be appreciated. Thanks.
Solved this issue by using array_chunk function.
Here is the solution below:
foreach (array_chunk($data,1000) as $t)
{
DB::table('table_name')->insert($t);
}
There is limit 65,535 (2^16-1) place holders in MariaDB 5.5 which is supposed to have identical behaviour as MySQL 5.5.
Not sure if relevant, I tested it on PHP 5.5.12 using MySQLi / MySQLND.
This error only happens when both of the following conditions are met:
You are using the MySQL Native Driver (mysqlnd) and not the MySQL client library (libmysqlclient)
You are not emulating prepares.
If you change either one of these factors, this error will not occur. However keep in mind that doing both of these is recommended either for performance or security issues, so I would not recommend this solution for anything but more of a one-time or temporary problem you are having. To prevent this error from occurring, the fix is as simple as:
$dbh->setAttribute(PDO::ATTR_EMULATE_PREPARES, true);
While I think #The Disintegrator is correct about the placeholders being limited. I would not run 1 query per record.
I have a query that worked fine until I added one more column and now I have 72k placeholders and I get this error. However, that 72k is made up of 9000 rows with 8 columns. Running this query 1 record at a time would take days. (I'm trying to import AdWords data into a DB and it would literally take more than 24 hours to import a days worth of data if I did it 1 record at a time. I tried that first.)
What I would recommend is something of a hack. First either dynamically determine the max number of placeholders you want to allow - i.e. 60k to be safe. Use this number to determine, based on the number of columns, how many complete records you can import/return at once. Create the full array of data for you query. Use a array_chunk and a foreach loop to grab everything you want in the minimum number of queries. Like this:
$maxRecords = 1000;
$sql = 'SELECT * FROM ...';
$qMarks = array_fill(0, $maxInsert, '(?, ...)');
$tmp = $sql . $implode(', ', $qMarks);
foreach (array_chunk($data, $maxRecords) AS $junk=>$dataArray) {
if (count($dataArray) < $maxRecords)) { break; }
// Do your PDO stuff here using $tmp as you SQL statement with all those placeholders - the ?s
}
// Now insert all the leftovers with basically the same code as above except accounting for
// the fact that you have fewer than $maxRecords now.
Using Laravel model, copy all 11000 records from sqlite database to mysql database in few seconds. Chunk data array to 500 records:
public function handle(): void
{
$smodel = new Src_model();
$smodel->setTable($this->argument('fromtable'));
$smodel->setConnection('default'); // sqlite database
$src = $smodel::all()->toArray();
$dmodel = new Dst_model();
$dmodel->setTable($this->argument('totable'));
$dmodel->timestamps = false;
$stack = $dmodel->getFields();
$fields = array_shift($stack);
$condb = DB::connection('mysql');
$condb->beginTransaction();
$dmodel::query()->truncate();
$dmodel->fillable($stack);
$srcarr=array_chunk($src,500);
$isOK=true;
foreach($srcarr as $item) {
if (!$dmodel->query()->insert($item)) $isOK=false;
}
if ($isOK) {
$this->notify("Przenieśliśmy tabelę z tabeli : {$this->argument('fromtable')} do tabeli: {$this->argument('totable')}", 'Będzie świeża jak nigdy!');
$condb->commit();
}
else $condb->rollBack();
}
You can do it with array_chunk function, like this:
foreach(array_chunk($data, 1000) as $key => $smallerArray) {
foreach ($smallerArray as $index => $value) {
$temp[$index] = $value
}
DB::table('table_name')->insert(temp);
}
My Fix for above issue:
On my side when i got this error I fixed it by reducing the the bulk insertion chunk size from 1000 to 800 and it worked for me.
Actually there were too many fields in my table and most them contains the details descriptions of size like a complete page text. when i go for there bulk insertion the service caused crashed and through the above error.
I think the number of placeholders is limited to 65536 per query (at least in older mysql versions).
I really can't discern what this piece of code is generating. But if it's a gigantic query, There's your problem.
You should generate one query per record to import and put those into a transaction.