How can fetching huge records using Laravel and MySQL? - mysql

Question

I Need experts Suggestions and Solutions. We are developing job portal website here by handle around 1 million records. We are facing records fetching timeout errors. How can I handle those records using laravel and MySql?
We are trying to follow steps:
Increase the PHP execution time
MySql Indexing
Paginations

Answer 1

You should be chunking results when working with large data sets. This allows you to process smaller loads, reduces memory consumption and allows you to return data to the User while the rest is being fetched/processing. See the laravel documentation on chunking:
https://laravel.com/docs/5.5/eloquent#chunking-results
To further speed things up you can leverage multithreading and spawn concurrent processes that each handle a chunk at a time. Symfony's Symfony\Component\Process\Process class makes this easy to do.
https://symfony.com/doc/current/components/process.html

Answer 2

From the docs:
If you need to work with thousands of database records, consider using the chunk method. This method retrieves a small chunk of the results at a time and feeds each chunk into a Closure for processing. This method is very useful for writing Artisan commands that process thousands of records.
For example, let's work with the entire users table in chunks of 100 records at a time:
DB::table('users')->orderBy('id')->chunk(100, function ($users) {
foreach ($users as $user) {
//
}
});

Answer 3

Hi I think this might help
$users = User::groupBy('id')->orderBy('id', 'asc');
$response = new StreamedResponse(function() use($users){
$handle = fopen('php://output', 'w');
// Add Excel headers
fputcsv($handle, [
'col1', 'Col 2' ]);
$users->chunk(1000, function($filtered_users) use($handle) {
foreach ($filtered_users as $user) {
// Add a new row with user data
fputcsv($handle, [
$user->col1, $user->col2
]);
}
});
// Close the output stream
fclose($handle);
}, 200, [
'Content-Type' => 'text/csv',
'Content-Disposition' => 'attachment; filename="Users'.Carbon::now()->toDateTimeString().'.csv"',
]);
return $response;

Answer 4

Laravel has a lazy feature for this purpose. I tried both chunk and cursor. The cursor makes one query and puts a lot of data in the memory which is not useful if you have millions of records in DB. Chunk also was ok but lazy much cleaner in the way you write your code.
use App\Models\Flight;
foreach (Flight::lazy() as $flight) {
//
}
Source: https://laravel.com/docs/9.x/eloquent#chunking-results