Yii2 Mirror a database table to redis for high speed active record query - mysql

What I am trying to do is to cache all the results in a MySQL table that seldom changes, so as to minimize calls to database and increase query speed. There are about 100k records in there.
Is there a library that can sync changes made in this table, like say when a record is updated or inserted, the redis cache will also be invalidated and updated.
I have seen one for elasticsearch, but nothing for redis.
From this page:
Yii copying data from one model to another
There is this comment:
You can get all models attributes by:
$data = $model->attributes;
and assign them to another model
$anotherModel = new AnotherActiveRecord();
$anotherModel->setAttributes($data);
now another model will extract whatever it can from $data
I'm curious, can a Redis cache also "mirror" the data from a database table in a similar way?
Or is this just a bad idea overall, and its better off caching the query as it comes along, or is there a better way.

You can enable caching based on https://www.yiiframework.com/doc/guide/2.0/en/caching-data
[
'components' => [
'cache' => [
'class' => 'yii\redis\Cache',
'redis' => [
'hostname' => 'localhost',
'port' => 6379,
'database' => 0,
]
],
],
]
and then use Query Caching which natively defined on query builder level
$result = $db->cache(function ($db) {
// the result of the SQL query will be served from the cache
// if query caching is enabled and the query result is found in the cache
// ... perform SQL queries here ...
});
Also you can use Cache Dependencies based on your table (some criteria like if max(updated_at) is changed or not).
// Create a dependency on updated_at field
$dependency = new yii\caching\DbDependency(['sql' => 'select max(updated_at) from my_table']);
$duration = 60; // cache query results for 60 seconds.
$result = $db->cache(function ($db) {
// ... perform SQL queries here ...
return $result;
}, $duration, $dependency);

Related

How to query .tab pages from local wikidata instance using API

I am using the Extension:JsonConfig on my docker instance of wikidata that has some tables loaded onto it. The configuration for the extension in my LocalSettings.php is as follows,
$wgJsonConfigEnableLuaSupport = true;
$wgJsonConfigModels['Tabular.JsonConfig'] = 'JsonConfig\JCTabularContent';
$wgJsonConfigs['Tabular.JsonConfig'] = [
'namespace' => 486,
'nsName' => 'Data',
// page name must end in ".tab", and contain at least one symbol
'pattern' => '/.\.tab$/',
'license' => 'CC0-1.0',
'isLocal' => true,
'store' => true,
];
When i query the local instance using the following url,
http://<DOMAIN_HERE>/w/api.php?action=query&list=search&srsearch=tab contentmodel:Tabular.JsonConfig &srnamespace=486&srlimit=10&format=json
i receive the following response
{"batchcomplete":"","limits":{"search":10},"query":{"searchinfo":{"totalhits":0},"search":[]}}
which means that no matches have been found even though tables that match the query statement do exist.
This same query works with commons database when the following is done
https://commons.wikimedia.org/w/api.php?action=query&list=search&srsearch=tab%20contentmodel:Tabular.JsonConfig%20&srnamespace=486&srlimit=10&format=json
Can anyone point me out as to what i am doing wrong here?

Yii2/PHP: Abstracting Database Access for InfluxDB and MySQL

In my Yii2/PHP project I need to have both databases integrated:
MySQL for meta data, Web-UI, ...
InfluxDB for measurement data (heavy loads of timeserie data)
To reduce complexity I'd like to start with MySQL only and add InfluxDB later.
My idea is to create an abstraction/superclass for both databases (for measurement data only) which allow to do implementation and perform tests with MySQL and enable speedup with InfluxDB at a later stage in the project.
The abstraction should have methods for:
database connection management
writing data
reading data (raw data, aggregations)
Since I am no InfluxDB expert (yet): Does this architecture make sense or are both datamodels and schemes fundamentally different so an abstraction would be worthless? Are there projects out there to learn from?
First, you need to configure your databases like below this example take two mysql db:
return [
'components' => [
'db1' => [
'class' => 'yii\db\Connection',
'dsn' => 'mysql:host=localhost;dbname=db1name', //maybe other dbms such as psql,...
'username' => 'db1username',
'password' => 'db1password',
],
'db2' => [
'class' => 'yii\db\Connection',
'dsn' => 'mysql:host=localhost;dbname=db2name', // Maybe other DBMS such as psql (PostgreSQL),...
'username' => 'db2username',
'password' => 'db2password',
],
],
];
Then you can simply:
// To get from db1
Yii::$app->db1->createCommand((new \yii\db\Query)->select('*')->from('tbl_name'))->queryAll()
// To get from db2
Yii::$app->db2->createCommand((new \yii\db\Query)->select('*')->from('tbl_name'))->queryAll()
If you are using an active record model, in your model you can define:
public static function getDb() {
return Yii::$app->db1;
}
//Or db2
public static function getDb() {
return Yii::$app->db2;
}
Then:
If you have set db1 in the getDb() method, the result will be fetched from db1 and so on.
ModelName::find()->select('*')->all();
I'm not sure trying to fit MySQL and InfluxDB in the same mould would make a lot of sense.
A better approach IMHO, would be to have some sort of helper class for your computations (i.e.: Stats::getViews(), Stats::getVisitors(), ..) first using MySQL, and later rewrite it to use InfluxDB, keeping the same methods signatures and responses formats.

Yii2 MySQL how to prevent duplicate select query

I saw in my debug menu that I have some duplicate select queries. In particular, this is the one that I got 4 times SELECT * FROM page_adminlang WHERE (language=bg) AND (page_id=1) in 4 different files. Is there a right way to prevent such situations and is this big hit on the site speed performance? Shall I retrieve the row in some variable like $page = PageAdmin::findOne(1) and than call it where I need and is this the right way? I red other articles but they were mainly for duplicate rows on insertion. Not familiar with MySQL performance tuning but want to go deeper in this area.Thank you in advance!
Option 1 - Configure MemCache Component
An Easier method is to enable db caching, then it won't matter, put a 5second duration at first.
In your component db connection settings set the following properties:
'components' => [
'db' => [
'class' => 'yii\db\Connection',
'dsn' => 'mysql:host=localhost;dbname=dbname',
'username' => 'root',
'password' => 'password',
'charset' => 'utf8',
....
'enableQueryCache' => true,
'queryCacheDuration' => 5, // five seconds
....
]
]
See:
https://www.yiiframework.com/doc/guide/2.0/en/caching-data
https://www2.0/yii-db.yiiframework.com/doc/api/-connection
Option 2 - Custom query with cache
public function getResults()
{
// Cache expires every x seconds (60sec/min * 60min/hr * 6hrs)
$duration = 60*60*6;
$sql_query = "SELECT * FROM some_table";
return Yii::$app->db->createCommand($sql_query)->cache($duration)->queryAll();
}
I don't usually face that issue, but if I have to retrieve something from the database multiple times (and I know that won't change), I do something like that in my model
private $_myData = null;
public function getMyData() {
if ($this->_myData !== null) return $this->_myData;
$this->_myData = //query your data;
return $this->getMyData();
}

How can fetching huge records using Laravel and MySQL?

I Need experts Suggestions and Solutions. We are developing job portal website here by handle around 1 million records. We are facing records fetching timeout errors. How can I handle those records using laravel and MySql?
We are trying to follow steps:
Increase the PHP execution time
MySql Indexing
Paginations
You should be chunking results when working with large data sets. This allows you to process smaller loads, reduces memory consumption and allows you to return data to the User while the rest is being fetched/processing. See the laravel documentation on chunking:
https://laravel.com/docs/5.5/eloquent#chunking-results
To further speed things up you can leverage multithreading and spawn concurrent processes that each handle a chunk at a time. Symfony's Symfony\Component\Process\Process class makes this easy to do.
https://symfony.com/doc/current/components/process.html
From the docs:
If you need to work with thousands of database records, consider using the chunk method. This method retrieves a small chunk of the results at a time and feeds each chunk into a Closure for processing. This method is very useful for writing Artisan commands that process thousands of records.
For example, let's work with the entire users table in chunks of 100 records at a time:
DB::table('users')->orderBy('id')->chunk(100, function ($users) {
foreach ($users as $user) {
//
}
});
Hi I think this might help
$users = User::groupBy('id')->orderBy('id', 'asc');
$response = new StreamedResponse(function() use($users){
$handle = fopen('php://output', 'w');
// Add Excel headers
fputcsv($handle, [
'col1', 'Col 2' ]);
$users->chunk(1000, function($filtered_users) use($handle) {
foreach ($filtered_users as $user) {
// Add a new row with user data
fputcsv($handle, [
$user->col1, $user->col2
]);
}
});
// Close the output stream
fclose($handle);
}, 200, [
'Content-Type' => 'text/csv',
'Content-Disposition' => 'attachment; filename="Users'.Carbon::now()->toDateTimeString().'.csv"',
]);
return $response;
Laravel has a lazy feature for this purpose. I tried both chunk and cursor. The cursor makes one query and puts a lot of data in the memory which is not useful if you have millions of records in DB. Chunk also was ok but lazy much cleaner in the way you write your code.
use App\Models\Flight;
foreach (Flight::lazy() as $flight) {
//
}
Source: https://laravel.com/docs/9.x/eloquent#chunking-results

WordPress Custom Select Query

I really don't know enough about MySQL queries and it's showing.
I have a custom field set for every post. The custom field stores the posts source URL in a key called "source_url".
I have it working with the below WP_Query parameters, but it's incredibly slow. Keep in mind it's possible to 50+ urls to search for.
So, given an array of source URL's, I want to fetch the matching posts.
For example, here is what I currently have that's slow in WP_Query:
// var_dump of $urls array (this could be 50+ urls)
array(7) {
[0]=>
string(42) "http://www.youtube.com/watch?v=FMghvnqDhT8"
[1]=>
string(42) "http://www.youtube.com/watch?v=RY-yUFpXTnM"
[2]=>
string(58) "http://www.youtube.com/watch?v=nIm2dnyJ1Ps&feature=related"
[3]=>
string(42) "http://www.youtube.com/watch?v=NoCtRQlJAqM"
[4]=>
string(57) "http://holidaycustoms.blogspot.com/2012/08/busy-week.html"
[5]=>
string(42) "http://www.youtube.com/watch?v=DcZvg197Ie4"
[6]=>
string(42) "http://www.youtube.com/watch?v=7P3UEbLmLuo"
}
// Build Media Query
$meta_query = array(
'relation' => 'OR'
);
foreach( $urls as $url ) {
$meta_query[] = array(
'key' => 'source_url',
'value' => $url
);
}
// Get 20 matching posts from a category set by a variable
$args = array(
'post_type' => 'post',
'posts_per_page' => 20,
'orderby' => 'rand',
'cat' => $cat_ID,
'meta_query' => $meta_query
);
$posts = get_posts($args);
What I'm looking to do is replace the above code with a custom query select, which I have read is much faster than WP_Query.
But I don't know enough about MySQL or the WP database to build the custom select query. Can anyone help? Thanks in advance!
In the post you linked yourself, the first reply already states that
[...] the default schema doesn't even have an index on the value column
Which is far more severe a problem than any you would have with a query generator, because without an index the DBMS has to traverse the whole table and compare strings of each field.
Adding an index is fairly easy with an appropriate management tool like PHPMyAdmin. The offending table you will need to add an index to is called wp_postmeta and the field that needs an index is meta_value, and the index type should be INDEX.
Adding an index is transparent and does not affect wordpress other than in performance. It could take some time though since, well MySQL needs to traverse the whole table. Also, because you are indexing string data, the index will be quite big.
You should also try using appropriate structures for your query. You are currently using a big ORed selection with different values but always the same field. There is a construct for just that, and it's called IN.
...
// Build Media Query
$meta_query = array();
$meta_query[] = array(
'key' => 'source_url',
'value' => $urls,
'compare' => 'IN'
);
// Get 20 matching posts from a category set by a variable
..
(Untested. I actually never did this, Reference)
The performance gain would be negligible compared to adding an index I assume, but your code would become a lot simpler.