Laravel datatable taking too much of time to load data - mysql

Question

Here loading orders data using below code but it takes too much time to load and I'm also using serverSide and there are only 320 data's something and TTFB are very high and I also tried "pageLength": 30, "stateSave": true, and using Cloudflare and Nginx server but the result is same please let me know how I fix this issue. I tried most of the solutions but still having issues.
Controller.php
$orders = new Order;
$search = $request->search['value'];
$filter_type = $request->filter_type;
if ($filter_type == "custom") {
$from = date('Y-m-d' . ' 00:00:00', strtotime($request->from_dates));
if ($request->has('to_dates')) {
$to = date('Y-m-d' . ' 23:59:59', strtotime($request->to_dates));
$orders = $orders->whereBetween('created_at', array($from, $to));
}
}
elseif ($filter_type == "daily") {
$orders = $orders->where('created_at', '>=', Carbon\Carbon::today());
}
elseif ($filter_type == "weekly") {
$fromDate = Carbon\Carbon::now()->subDay()->startOfWeek()->toDateString();
$tillDate = Carbon\Carbon::now()->subDay()->endOfWeek()->toDateString();
$orders = $orders->whereBetween(DB::raw('date(created_at)'), [$fromDate, $tillDate]);
}
elseif ($filter_type == "monthly") {
$orders = $orders->whereRaw('MONTH(created_at) = ?', [date('m')]);
}
elseif ($filter_type == "yearly") {
$orders = $orders->whereRaw('YEAR(created_at) = ?', [date('Y')]);
}
$orders = $orders->orderByDesc('id')->select();
$orders = $orders->get();
$datatable = DataTables::of($orders)
->addColumn('id', function ($orders) {
return #$orders->id;
})
->addColumn('payment_type', function ($orders) {
return #$orders->payment_type_text;
})
->addColumn('user_name', function ($orders) {
return #$orders->user->name;
})
->addColumn('store_name', function ($orders) {
return #$orders->store->name;
})
->addColumn('service_type', function ($orders) {
return #$orders->store->service_type1->service_name;
})
->addColumn('total', function ($orders) {
return html_entity_decode(currency_symbol() . (#$orders->total_amount+#$orders->wallet_amount));
})
->addColumn('status_text', function ($orders) {
return #$orders->status_text;
})
->addColumn('action', function ($orders) {
return '<a title="' . trans('admin_messages.view') . '" href="' . route('admin.view_order', $orders->id) . '" ><i class="material-icons">edit</i></a>';
});
$columns = ['id', 'payment_type', 'user_name', 'store_name', 'total', 'status_text'];
$base = new DataTableBase($orders, $datatable, $columns, 'Orders');
return $base->render(null);
Blade.php
#extends('admin/template')
#section('main')
<?php flush(); ?>
<div class="content" ng-controller="statements" ng-cloak>
<div class="card">
<div class="card-header card-header-rose card-header-text">
<div class="card-text">
<h4 class="card-title">{{$form_name}}</h4>
</div>
</div>
<div class="card-body ">
<div class="table-responsive">
<table id="statement_table" class="table table-condensed w-100">
</table>
</div>
</div>
</div>
</div>
</div>
#endsection
#push('scripts')
<link rel="stylesheet" href="{{asset('admin_assets/css/buttons.dataTables.css')}}">
<script src="{{asset('admin_assets/js/dataTables.buttons.js')}}">
</script>
<script src={{url('vendor/datatables/buttons.server-side.js')}}></script>
<script>
var column = [
{data: 'id', name: 'id', title: '{{trans("admin_messages.order_id")}}' },
{data: 'payment_type',name: 'payment_type',title: '{{trans("admin_messages.payment_type")}}',searchable: true},
{data: 'user_name',name: 'user_name',title: '{{trans("admin_messages.user_name")}}'},
{data: 'user_address',name: 'user_address',title: '{{trans("admin_messages.address")}}'},
{data: 'mobile_number',name: 'mobile_number',title: '{{trans("admin_messages.mobile_number")}}'},
{data: 'store_name',name: 'store_name',title: '{{trans("admin_messages.store_name")}}'},
{data: 'service_type',name: 'service_type',title: '{{trans("admin_messages.service_type")}}'},
{data: 'total',name: 'total',title: '{{trans("admin_messages.total")}}'},
{data: 'status_text',name: 'status_text',title: '{{trans("admin_messages.order_status")}}'},
{data: 'created_at',name: 'created_at',title: '{{trans("admin_messages.created_at")}}'},
{data: 'action',name: 'action',title: '{{trans("admin_messages.action")}}',orderable: false,searchable: false}
];
var oTable = $('#statement_table').DataTable({
dom:"lBfrtip",
buttons:["csv","excel","print"],
order:[0, 'desc'],
processing: true,
serverSide: true,
ajax: {
url: ajax_url_list['all_orders'],
data: function (d) {
d.filter_type = $('#filter_by').val();
d.from_dates = $('#from_date').val();
d.to_dates = $('#to_date').val();
}
},
columns: column
});
</script>
#endpush
Create Table
CREATE TABLE `order` (
`id` int(10) unsigned NOT NULL AUTO_INCREMENT,
`store_id` int(10) unsigned DEFAULT NULL,
`user_id` int(10) unsigned DEFAULT NULL,
`driver_id` int(11) DEFAULT NULL,
`recipient` tinyint(4) DEFAULT NULL,
`subtotal` decimal(11,2) DEFAULT NULL,
`offer_percentage` decimal(11,2) DEFAULT NULL,
`offer_amount` decimal(11,2) DEFAULT NULL,
`promo_id` int(11) DEFAULT NULL,
`promo_amount` decimal(11,2) DEFAULT NULL,
`delivery_fee` decimal(11,2) DEFAULT NULL,
`booking_fee` decimal(11,2) DEFAULT NULL,
`store_commision_fee` decimal(11,2) DEFAULT NULL,
`driver_commision_fee` decimal(11,2) DEFAULT NULL,
`tax` decimal(11,2) DEFAULT NULL,
`total_amount` decimal(11,2) DEFAULT NULL,
`wallet_amount` decimal(11,2) DEFAULT NULL,
`payment_type` tinyint(4) DEFAULT NULL,
`owe_amount` decimal(11,2) DEFAULT NULL,
`store_owe_amount` decimal(11,2) DEFAULT NULL,
`applied_owe` decimal(11,2) DEFAULT NULL,
`status` tinyint(4) DEFAULT NULL,
`payout_status` tinyint(4) DEFAULT NULL,
`currency_code` varchar(3) COLLATE utf8mb4_unicode_ci DEFAULT NULL,
`est_preparation_time` time DEFAULT NULL,
`est_travel_time` time DEFAULT NULL,
`est_delivery_time` time DEFAULT NULL,
`cancelled_by` tinyint(4) DEFAULT NULL,
`cancelled_reason` int(10) unsigned DEFAULT NULL,
`cancelled_message` text COLLATE utf8mb4_unicode_ci,
`delay_min` time DEFAULT NULL,
`delay_message` text COLLATE utf8mb4_unicode_ci,
`schedule_status` tinyint(4) DEFAULT '0',
`payout_is_create` tinyint(4) DEFAULT '0',
`schedule_time` timestamp NULL DEFAULT NULL,
`notes` varchar(200) COLLATE utf8mb4_unicode_ci DEFAULT NULL,
`user_notes` text COLLATE utf8mb4_unicode_ci,
`store_notes` text COLLATE utf8mb4_unicode_ci,
`driver_notes` text COLLATE utf8mb4_unicode_ci,
`declined_at` timestamp NULL DEFAULT NULL,
`accepted_at` timestamp NULL DEFAULT NULL,
`cancelled_at` timestamp NULL DEFAULT NULL,
`delivery_at` timestamp NULL DEFAULT NULL,
`completed_at` timestamp NULL DEFAULT NULL,
`delivery_type` varchar(200) COLLATE utf8mb4_unicode_ci DEFAULT NULL,
`tips` decimal(11,2) DEFAULT NULL,
`created_at` timestamp NULL DEFAULT NULL,
`updated_at` timestamp NULL DEFAULT NULL,
PRIMARY KEY (`id`),
KEY `order_store_id_foreign` (`store_id`),
KEY `order_user_id_foreign` (`user_id`),
CONSTRAINT `order_store_id_foreign` FOREIGN KEY (`store_id`) REFERENCES `store` (`id`),
CONSTRAINT `order_user_id_foreign` FOREIGN KEY (`user_id`) REFERENCES `user` (`id`)
) ENGINE=InnoDB AUTO_INCREMENT=10268 DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci;
Route
Route::match(array('GET', 'POST'), '/all_orders','OrderController#all_orders')->name('all_orders');
I tried the below code and getting data faster than before 59.0s to 5.0s now its a good result but I don't know the code is an efficient way or not please guide me guys
$orders = Order::latest();
$search = $request->search['value'];
$filter_type = $request->filter_type;
if ($filter_type == "custom") {
$from = date('Y-m-d' . ' 00:00:00', strtotime($request->from_dates));
if ($request->has('to_dates')) {
$to = date('Y-m-d' . ' 23:59:59', strtotime($request->to_dates));
$orders = $orders->whereBetween('created_at', array($from, $to));
}
}
elseif ($filter_type == "daily") {
$orders = $orders->where('created_at', '>=', Carbon\Carbon::today());
}
elseif ($filter_type == "weekly") {
$fromDate = Carbon\Carbon::now()->subDay()->startOfWeek()->toDateString();
$tillDate = Carbon\Carbon::now()->subDay()->endOfWeek()->toDateString();
$orders = $orders->whereBetween(DB::raw('date(created_at)'), [$fromDate, $tillDate]);
}
elseif ($filter_type == "monthly") {
$orders = $orders->whereRaw('MONTH(created_at) = ?', [date('m')]);
}
elseif ($filter_type == "yearly") {
$orders = $orders->whereRaw('YEAR(created_at) = ?', [date('Y')]);
}
return Datatables::of($orders)
->addColumn('id', function ($orders) {
return #$orders->id;
})
->addColumn('payment_type', function ($orders) {
return #$orders->payment_type_text;
})
->addColumn('user_name', function ($orders) {
return #$orders->user->name;
})
->addColumn('user_address', function ($orders) {
return #$orders->user->user_address->first_address;
})
->addColumn('mobile_number', function ($orders) {
return #$orders->user->mobile_number;
})
->addColumn('store_name', function ($orders) {
return #$orders->store->name;
})
->addColumn('service_type', function ($orders) {
return #$orders->store->service_type1->service_name;
})
->addColumn('total', function ($orders) {
return html_entity_decode(currency_symbol() . (#$orders->total_amount+#$orders->wallet_amount));
})
->addColumn('status_text', function ($orders) {
return #$orders->status_text;
})
->addColumn('created_at', function ($orders) {
return #date("d-m-y h:i A", strtotime($orders->created_at));
})
->addColumn('action', function ($orders) {
return '<a title="' . trans('admin_messages.view') . '" href="' . route('admin.view_order', $orders->id) . '" ><i class="material-icons">edit</i></a>';
})->rawColumns(['id', 'payment_type','user_name','store_name','user_address','mobile_number','service_type','total','status_text','action'])
->make(true);

Answer 1

The "table scans" are killing performance.
A bigger cache may help
A better index may help
Look at how much RAM is unused on the server with MySQL. Increase innodb_buffer_pool_size to include most of that RAM.
(128M is an old default; it is pathetically low for most apps today on most hardware.)
If you are running an old version of MySQL, consider upgrading (for many reasons).
MONTH(created_at) = ? is much less efficient than
WHERE created_at >= '2021-02-01'
AND created_at < '2021-02-01' + INTERVAL 1 MONTH
However, that performance boost assumes some index includes created_at. None exists now.
If I can see the generated SQL for a typical SELECT, I can advise further on a better index.

Answer 2

You are firing off several queries for each order that is found. Fix that by using with and it'll only fire 1 query per table:
$orders = $orders->with(['user', 'store.service_type1'])->get();
https://laravel.com/docs/8.x/eloquent-relationships#eager-loading
If you have 500 orders found you'll be firing off over 1000 queries. This will reduce it to 4!
(Pretty minor extra if you want:) And you could make it 3 if you use a through relationship instead of store.service_type1 (which under the hood does 1 query with a join instead of 2 queries)

Answer 3

Enable query logs and check how much time the query is taking, this will you an idea if the table really needs updating.
I am mentioning some points below that can be done on the table regardless.
I see there is a column driver_id but you have not added any indexing or constraint on it.
If this column is not a foreign key then add basic indexing, Do the same for promo_id column.
Add indexing on column delivery_type.
If you can, create bitmap index on payment_type and payout_status
Finally I see you are running queries on timestamp columns, if possible add indexing for that too.
All this will help reduce the time to get the data from Database.
From code side if you are getting the data from different tables and relations, consider using with([]).
This will help by reducing the number of queries running.
If all of this doesn't help then check how much time the code is taking to process the request.
With all of this atleast you should be able to know if the table is an issue or the code.

Answer 4

You have 2 options for fix your issue
Add caching to fronend and backend so its not taking time. Also you can use mysql query cache also
In laravel uses eloquent orm so it takes time so try to use simple mysql query so it takes less time to execute

Answer 5

What you should be doing is paginate the results, as Im not seeing any pagination on your PHP side, and I imagine you have thousands of records, it will take you a long time to get.
Datatable when serverSide is activated, it sends a bunch of GET params on the query to the backend, amongst these is start and length
so on your controller, you should utilize these params

Answer 6

Code readability and performance
Optimizing your code is VERY important when you use loops and conditional branching.
I'd use switch instead of all those elseif conditionals.
Execution locks
Switching back and forth between PHP code execution, and HTML output will always slow things down. Sometimes by a LOT.
Generate your HTML in string variables instead of printing to buffer/output, and then output the entire page at once when finished.

Answer 7

The main issue is your end query : $orders->get();. You do not provide any pagination/limitation parameter, so this query will query the entire order table, and give the results in a Collection. If there is 10.000 rows matching your filter, they'll be returned in a single Collection.
Check the docs, but I think that Datatables accept an Eloquent instances, not only Collections.
So first, try removing the $orders = $orders->get(); line to directly pass to Datatable the Eloquent query instance.
Second, check the docs again, but I think that Datatable has a built-in way to handle the pagination (reload data and send the correct HTTP parameters from the frontend (next/prev or pagination click, sorting, searching...), apply the pagination to the Eloquent query). If so: try to use it (if it does not work directly out of the box, when passing the Eloquent instance instead of the Collection to the Datatable laravel method). If not, you'll need to implement pagination parameters with your HTTP request and apply them to the eloquent query.
Careful as data table will work (and be slow, cause the HTTP response is huge, and the DOM become huge also) with a 10.000 rows collection returned in one request. The sorting will work, the search also but very poorly as JS only sorting (CPU greedy, slow). It's clearly not what you should do.
In the end, all frontend actions: text searching, column sorting, pagination ... should request your Laravel API with custom GET parameters (for search, sort, pagination...), and your laravel controller should "translate" those GET parameters into Eloquent methods/queries. Also take care about eager-loading as #BenGooding mentioned.
UPDATE after OP question update: OP edited it's question and code snippets. In the first version of the question, OP was executing a ->get() without any pagination or limitation. It was so querying the entire table, and Databable methods was given a huge Collection with all the objects. Then, frontend JS datatable would process hundreds of row again, slowing the page even more.
Using my advice, OP changed his code: instead of querying the entire table, just pass the Eloquent query to the Datatable helper, as this libs can handle query pagination and some filtering itself, if you give it something to work with (not a definitive list of object, but a SQL query to adapt). Following this, OP reduced query time from 50 seconds to 5 seconds. And so it was the main issue with his Datatable management.

Answer 8

Get specific select column which do you want?
datatables()->eloquent($query) instance of DataTables::of($orders) (send to object not passed query result.)
Create separate file of datatable handling like datatable service file.
public function dataTable($query)
{
return datatables()
->eloquent($query)
->editColumn('status', function ($query) {
...
})
->addColumn('action', function ($query) {
...
})
->addIndexColumn()
->rawColumns(['status', 'action']);
}
public function query(Category $model)
{
return $model->newQuery();
}