I'm looking to optimize my SQL queries for a growing website based on CakePHP. I can optimize things using recursive = -1, for example, but before going further, I think it'd be helpful to know which queries are taking the most time.
Is there a simple way to log the time queries are taking on a production site? The idea of adding code around each find() makes me want to quit before I start, and it doesn't look like the beforeFind and afterFind functions carry enough information to track which "after" corresponds to which "before".
Thanks in advance!
Simply use the Debug Kit plugin for CakePHP or use the logging of your DB server? MySQL can be configured to even just log slow queries.
https://github.com/cakephp/debug_kit
https://book.cakephp.org/3.0/en/debug-kit.html
https://dev.mysql.com/doc/refman/8.0/en/query-log.html
Related
I'am a little confused about how to find most frequently used queries in mysql. I'm looking this query because I want use memcache tool in php application. Is really only way to find them is log every query to log file and after work on it?
Using General Query Log is only the solution to do that. I couldn't find any other way. If the purpose is to optimize slow queries then you can use Slow Query Log to find the slowest queries.
I am attempting to count the number of queries that are run against the database in a request against my Golang API server. This is in order to optimize + find bottlenecks in our API.
An example of how to do this in Django is here.
Any idea if there is a simple way to do this in Golang?
In order to find bottlenecks I would suggest going straight to the MySQL database and enabling the slow query log and slowly setting the long_query_time down to a low millisecond value.
Use tools like pt-query-digest to help digest these to get a frequency of like queries. Attack those as slow queries that need fixing and then set a lower value.
The actual count of the queries on each isn't actually that useful.
When attacking the problem from a go point of view measuring the API response time of each interface will help you look holisticly at the service.
No easy solution that I'm aware of.
You could wrap your db in your own struct, and then implement Exec() or whichever function you use directly on that struct. Your function would just call the database one, and count it however you see fit.
A similar example, but with a logger, can be found here:
How to log queries to database drivers?
If I wanted to scale a Rails application by distributing its database on a different machine based on its authorization rules (location and user roles). So any resource attributed to that location would be sitting in a database dedicated to that location.
Should I get down to basic SQL writing, use something like Sequel gem or keep the niceness and magic of ActiveRecord?
It is true that raw SQL execution speed is more than execution of the ActiveRecord's nice magical queries. However, if you talk about scaling then there comes this question of how well would the queries be manageable when the application really grows large.
By far, a lot of complicated database operations can be managed well by caching, and proper indexing and proper eager-loading. In some cases MySQL views also help to improve performance, and Rails treats MySQL views fairly. After this, if one is able to corner out the really slow queries, then it might be worth it to convert them to raw SQL and save some time. Also, Rails offer caching of the database queries. MySQL also has a caching mechanism. Before executing raw SQL directly I would make sure these options (actually many more like avoiding unnecessary joins, as a join operation is resource intensive) are not able to give me what I am looking for. Hope this helps.
It sounds like you are partitioning your database, which Sequel has built-in support for (http://sequel.rubyforge.org/rdoc/files/doc/sharding_rdoc.html). I recommend using Sequel, but as the lead developer, I'm biased.
I am currently looking into writing a plugin for mysql in C. I have been reading and studying MySQL 5.1 Plugin Development (http://www.amazon.com/MySQL-Plugin-Development-Sergei-Golubchik/dp/1849510601) which has been helping me out a lot. However I can't seem to find any examples that execute queries. I tried Googling for one but couldn't find any relevant examples. My goal for the time being is to write just a simple plugin that after some data is entered into a table will just perform some basic select queries. If anyone would be willing to share a link to such an example or provide one, it would be most appreciated.
I haven't used Mysql with C before, however you might want to take a look at these links:
http://dev.mysql.com/doc/refman/5.1/en/c-api-function-overview.html
http://www.mysql.com/downloads/connector/c/
If you know MySql I think those links would be a good starting point at least.
As someone who's written a storage engine plugin, I've found that executing a query from within a MySQL plugin is incredibly difficult. MySQL isn't re-entrant, due to locking within the MySQL process. You could use the MySQL client api (as suggested by chris) to connect to the same server. But it is 99% likely you will simply deadlock the whole server.
I do not know if this might help, but I have created a small mysql plugin written in C. The plugin basically sets up a function which when triggered from mysql will send information about a remove, insert or update query to a running node socket server.
You can find the repository here:
https://github.com/Cyclonecode/mysql-notification
It sounds to me like it's not a good idea to try to execute queries within a storange engine, because this introduces re-entrancy that the server doesn't handle.
However, it is possible that you could do it from a daemon thread, as the event scheduler already does. Likewise, Handlersocket does something similar (but uses the handler API rather than executing queries).
In any case, it all sounds a bit fishy. If you are just learning, try writing a few UDFs (they're easy).
I am about to begin developing a logging system for future implementation in a current PHP application to get load and usage statistics from a MYSQL database.
The statistic will later on be used to get info about database calls per second, query times etc.
Of course, this will only be used when the app is in testing stage, since It will most certainly cause a bit of additional load itself.
However, my biggest questionmark right now is if i should use MYSQL to log the queries, or go for a file-based system. I'll guess that it would be a bit of a headache to create something that would allow writings from multiple locations when using a file based system to handle the logs?
How would you do it?
Use the general log, which will show client activity, including all the queries:
http://dev.mysql.com/doc/refman/5.1/en/query-log.html
If you need very detailed statistics on how long each query is taking, use the slow log with a long_query_time of 0 (or some other sufficiently short time):
http://dev.mysql.com/doc/refman/5.1/en/slow-query-log.html
Then use http://www.maatkit.org/ to analyze the logs as needed.
MySQL already had logging built in- Chapter 5.2 of the manual describes these. You'll probably be interested in The General Query Log (all queries), the Binary Query Log (queries that change data) and the Slow log (queries that take too long, or don't use indexes).
If you insist on using your own solution, you will want to write a database middle layer that all your DB calls go through, which can handle the timing aspects. As to where you write them, if you're in devel, it doesn't matter too much, but the idea of using a second db isn't bad. You don't need to use an entirely separate DB, just as far as using a different instance of MySQL (on a different machine, or just a different instance using a different port). I'd go for using a second MySQL instance instead of the filesystem- you'll get all your good SQL functions like SUM and AVG to parse your data.
If all you are interested in is longer-term, non-real time analysis, turn on MySQL's regular query logging. There are tons of tools for doing analysis on the query-logs (both regular and slow-query), giving you information about the run-times, average rows returned, etc. Seems to be what you are looking for.
If you are doing tests on MySQL you should store the results in a different database such as Postgres, this way you won't increase the load with your operations.
I agree with macabail but would only add that you could couple this with a cron job and a simple script to extract and generate any statistics you might want.