Yii2 MySQL how to prevent duplicate select query - mysql

I saw in my debug menu that I have some duplicate select queries. In particular, this is the one that I got 4 times SELECT * FROM page_adminlang WHERE (language=bg) AND (page_id=1) in 4 different files. Is there a right way to prevent such situations and is this big hit on the site speed performance? Shall I retrieve the row in some variable like $page = PageAdmin::findOne(1) and than call it where I need and is this the right way? I red other articles but they were mainly for duplicate rows on insertion. Not familiar with MySQL performance tuning but want to go deeper in this area.Thank you in advance!

Option 1 - Configure MemCache Component
An Easier method is to enable db caching, then it won't matter, put a 5second duration at first.
In your component db connection settings set the following properties:
'components' => [
'db' => [
'class' => 'yii\db\Connection',
'dsn' => 'mysql:host=localhost;dbname=dbname',
'username' => 'root',
'password' => 'password',
'charset' => 'utf8',
....
'enableQueryCache' => true,
'queryCacheDuration' => 5, // five seconds
....
]
]
See:
https://www.yiiframework.com/doc/guide/2.0/en/caching-data
https://www2.0/yii-db.yiiframework.com/doc/api/-connection
Option 2 - Custom query with cache
public function getResults()
{
// Cache expires every x seconds (60sec/min * 60min/hr * 6hrs)
$duration = 60*60*6;
$sql_query = "SELECT * FROM some_table";
return Yii::$app->db->createCommand($sql_query)->cache($duration)->queryAll();
}

I don't usually face that issue, but if I have to retrieve something from the database multiple times (and I know that won't change), I do something like that in my model
private $_myData = null;
public function getMyData() {
if ($this->_myData !== null) return $this->_myData;
$this->_myData = //query your data;
return $this->getMyData();
}

Related

Laravel returns error max prepared statements after getting many count()'s

I am getting the following error from Laravel.
SQLSTATE[42000]: Syntax error or access violation: 1461 Can't create
more than max_prepared_stmt_count statements (current value: 16382)
(SQL: select count(*) as aggregate from guild)
I have tried removing the guild count statement, but this makes the next SQL query give an error.
$users = User::count();
$crowdfunding = CrowdfundingSettings::find(1);
$guilds = Guild::count();
$data = [
'totalItems' => ItemTemplate::totalCustomItems(),
'totalAbilities' => Ability::totalCustom(),
'totalMobs' => MobTemplate::all()->count(),
'totalQuests' => Quest::all()->count(),
'totalLootTables' => LootTable::all()->count(),
'totalMerchantTables' => MerchantTable::all()->count(),
'totalDialogue' => Dialogue::all()->count(),
'totalCraftingRecipes' => CraftingRecipe::all()->count(),
'totalItemSets' => ItemSetProfile::all()->count(),
'totalSkills' => Skill::totalCustom()
];
return response()->json([
'crowdfunding_settings' => $crowdfunding,
'accounts' => $users,
//'guilds' => $guilds,
'data' => $data,
]);
I am expecting results to from the statement, yet I get an error. I have increased max prepared statements to 32k, but I still get this error displaying it is set to 16k.
As Dimitri mentioned (and also confirmed here) you should not use ->all() when you only need the count and not the data in the model itself.
Replace $data like so:
$data = [
'totalItems' => ItemTemplate::totalCustomItems(),
'totalAbilities' => Ability::totalCustom(),
'totalMobs' => MobTemplate::count(),
'totalQuests' => Quest::count(),
'totalLootTables' => LootTable::count(),
'totalMerchantTables' => MerchantTable::count(),
'totalDialogue' => Dialogue::count(),
'totalCraftingRecipes' => CraftingRecipe::count(),
'totalItemSets' => ItemSetProfile::count(),
'totalSkills' => Skill::totalCustom()
];
Counting the amount of mob templates using MobTemplate::all()->count() will result in the following sql SELECT * FROM mob_template;. The results of that will be loaded into a Eloquent collection, that collection will then count the items it contains in PHP. This is very slow, memory heavy and as it turns out you might also run into issues with prepared statements.
Count the amount of mob templates using MobTemplate::count() will result in the following sql SELECT COUNT(*) FROM mob_template;. This means the database does all the heavy lifting for counting the records and it only returns the result. That way eloquent does not have to load a collection with a bunch of data and it does not have to count all its items in PHP.
Make sure you're disposing PreparedQuery or any other prepared SQL Command after execution. Disposing calls ClosePreparedStatement.

Yii2/PHP: Abstracting Database Access for InfluxDB and MySQL

In my Yii2/PHP project I need to have both databases integrated:
MySQL for meta data, Web-UI, ...
InfluxDB for measurement data (heavy loads of timeserie data)
To reduce complexity I'd like to start with MySQL only and add InfluxDB later.
My idea is to create an abstraction/superclass for both databases (for measurement data only) which allow to do implementation and perform tests with MySQL and enable speedup with InfluxDB at a later stage in the project.
The abstraction should have methods for:
database connection management
writing data
reading data (raw data, aggregations)
Since I am no InfluxDB expert (yet): Does this architecture make sense or are both datamodels and schemes fundamentally different so an abstraction would be worthless? Are there projects out there to learn from?
First, you need to configure your databases like below this example take two mysql db:
return [
'components' => [
'db1' => [
'class' => 'yii\db\Connection',
'dsn' => 'mysql:host=localhost;dbname=db1name', //maybe other dbms such as psql,...
'username' => 'db1username',
'password' => 'db1password',
],
'db2' => [
'class' => 'yii\db\Connection',
'dsn' => 'mysql:host=localhost;dbname=db2name', // Maybe other DBMS such as psql (PostgreSQL),...
'username' => 'db2username',
'password' => 'db2password',
],
],
];
Then you can simply:
// To get from db1
Yii::$app->db1->createCommand((new \yii\db\Query)->select('*')->from('tbl_name'))->queryAll()
// To get from db2
Yii::$app->db2->createCommand((new \yii\db\Query)->select('*')->from('tbl_name'))->queryAll()
If you are using an active record model, in your model you can define:
public static function getDb() {
return Yii::$app->db1;
}
//Or db2
public static function getDb() {
return Yii::$app->db2;
}
Then:
If you have set db1 in the getDb() method, the result will be fetched from db1 and so on.
ModelName::find()->select('*')->all();
I'm not sure trying to fit MySQL and InfluxDB in the same mould would make a lot of sense.
A better approach IMHO, would be to have some sort of helper class for your computations (i.e.: Stats::getViews(), Stats::getVisitors(), ..) first using MySQL, and later rewrite it to use InfluxDB, keeping the same methods signatures and responses formats.

Laravel: Store error messages in database

Any one know how to send error messages to database in laravel which generate from app/exceptions/handler.php ?
I need to send what error massages generated in report() method to database.
If you are interested doing this manually, you can do something as following.
Step 1 -
Create a model to store errors that has a DB structure as following.
class Error extends Model
{
protected $fillable = ['user_id' , 'code' , 'file' , 'line' , 'message' , 'trace' ];
}
Step 2
Locate the App/Exceptions/Handler.php file, include Auth, and the Error model you created. and replace the report function with the following code.
public function report(Exception $exception) {
// Checks if a user has logged in to the system, so the error will be recorded with the user id
$userId = 0;
if (Auth::user()) {
$userId = Auth::user()->id;
}
$data = array(
'user_id' => $userId,
'code' => $exception->getCode(),
'file' => $exception->getFile(),
'line' => $exception->getLine(),
'message' => $exception->getMessage(),
'trace' => $exception->getTraceAsString(),
);
Error::create($data);
parent::report($exception);
}
(I am demonstrating this using laravel 5.6)
Because Laravel uses Monolog for handling logging it seems that writing Monolog Handler would be the cleanest way.
I was able to find something that exists already, please have a look at monolog-mysql package. I did not use it, so I don't know whether it works and if it works well, but it's definitely good starting point.

Yii2 Mirror a database table to redis for high speed active record query

What I am trying to do is to cache all the results in a MySQL table that seldom changes, so as to minimize calls to database and increase query speed. There are about 100k records in there.
Is there a library that can sync changes made in this table, like say when a record is updated or inserted, the redis cache will also be invalidated and updated.
I have seen one for elasticsearch, but nothing for redis.
From this page:
Yii copying data from one model to another
There is this comment:
You can get all models attributes by:
$data = $model->attributes;
and assign them to another model
$anotherModel = new AnotherActiveRecord();
$anotherModel->setAttributes($data);
now another model will extract whatever it can from $data
I'm curious, can a Redis cache also "mirror" the data from a database table in a similar way?
Or is this just a bad idea overall, and its better off caching the query as it comes along, or is there a better way.
You can enable caching based on https://www.yiiframework.com/doc/guide/2.0/en/caching-data
[
'components' => [
'cache' => [
'class' => 'yii\redis\Cache',
'redis' => [
'hostname' => 'localhost',
'port' => 6379,
'database' => 0,
]
],
],
]
and then use Query Caching which natively defined on query builder level
$result = $db->cache(function ($db) {
// the result of the SQL query will be served from the cache
// if query caching is enabled and the query result is found in the cache
// ... perform SQL queries here ...
});
Also you can use Cache Dependencies based on your table (some criteria like if max(updated_at) is changed or not).
// Create a dependency on updated_at field
$dependency = new yii\caching\DbDependency(['sql' => 'select max(updated_at) from my_table']);
$duration = 60; // cache query results for 60 seconds.
$result = $db->cache(function ($db) {
// ... perform SQL queries here ...
return $result;
}, $duration, $dependency);

TYPO3 IRRE open record is extremly slow

We have created a T3 extension where each record can have a couple of related event dates. The event dates are declared as IRRE recordings.
Now as the event dates getting more it takes up to 50 secs. to open a basic record in the Backend, frontend is fast as usual.
Right now there are 600 base records and 17K IRRE records. Things started to slow down at about 8K event dates.
Anyone an idea how to speed things up?
thx for your help
for fronteand enable lazy load in domain model and disable collaps all in tca for backend:
'config' => array(
'type' => 'inline',
'foreign_table' => 'tx_xxx_domain_model_yyy',
'foreign_field' => 'rrrr',
'foreign_sortby' => 'sorting',
'maxitems' => 9999,
'appearance' => array(
'expandSingle' => 1,
'collapseAll' => 1,
'levelLinksPosition' => 'top',
'showSynchronizationLink' => 1,
'showPossibleLocalizationRecords' => 1,
'useSortable' => 1,
'showAllLocalizationLink' => 1
),
IRRE wasn't the bottleneck of that performance issue.
We added a label_userFunc to display a formatted date as label of the IRRE entries but called the userfunction in TCA with
label_userFunc (doesn't work with IRRE)
but it must be called with
formattedLabel_userFunc (works with IRRE)
as only that one works with IRRE.
For other performance problems consider matin his answer:
in TCA set collapseAll = 1 and expandSingle =1