I have tons of queries and wanna count their occurrence.
Problem: the same query can have different parameters.
I thought about using EXPLAIN and then hash the array.
But i guess that wont work well.
Is there any way to get a hash of a query without including the params and "style" (format?)?
F.e. can the mysql (mariadb) server return a prepared statement from a query?
(EDIT: i do not have the prepared statements from code. I got SQL strings only.)
Simple example:
SELECT * FROM foo where bar = 'baz
SELECT * FROM `foo` WHERE `bar` = 'baz';
SELECT * FROM `foo` WHERE `bar` = 3
SELECT * FROM `foo` WHERE `bar` = 3;
...
This is the same query with different "style" and different parameters.
What i would like to count up is the occurrence of SELECT * FROM `foo` WHERE `bar` = {.*};
Note: the queries will contain multiple joins, subqueries, ect.
In this case I got those SQL queries from a general_log table.
No code, just SQL strings.
It's not a question of "how can I do this better".
It's a "how can I get a better result from what I got".
You can get the count of queries by "digest" — a kind of anonymized version of the query with all constant values replaced with ? — from the performance_schema.
SELECT
SCHEMA_NAME,
digest,
digest_text,
round(sum_timer_wait/ 1000000000000, 6),
count_star
FROM performance_schema.events_statements_summary_by_digest
ORDER BY sum_timer_wait DESC LIMIT 10;
See https://www.percona.com/blog/2015/10/13/mysql-query-digest-with-performance-schema/ for a blog on this solution.
Related
I would like a select query that would be able to select a value that may or may not be present in the where clause. Schema:
----------------------------------
studentid|firstname|lastname|major
My select clause would be
select * from students where studentid?={param} AND firstname?={param} AND lastname?={param} AND major?={param};
I put a question mark because I mean to say I could pass a value in the where clause or I might not. It could be
select * from students where studentid?=34344 AND firstname?="john" AND lastname?="smith" AND major?="";
select * from students where studentid?=34344 AND firstname?="john" AND lastname?="smith" AND major?="english";
Is there a way to do this easily without a stored procedure?
You can do that by using variables and checking null like this:
Declare #StudentId nvarchar(100) --can be null or evaluated
select * from students
where (#StudentId is null or studnetId= #StudentId) AND -- for other properties as well
Another option is using dynamic sql and first you have to build your sql query and then execute it (I don't like it)
You can handle it in the application side if possible:
string query= "select * from students where 1=1 /*trick for adding more conditions*/"
if(numberId is not null)
query += "AND studentId= {numberId} ";
//for other conditions ...
I'm using prepared statement for MySQLi PHP which is all working. However, the application that I'm using this in has 108 different possible statements all very similar that can be run either without condition:
select * from table1
or with
select * from table 1 where user_level = 1)
The question that I'm asking: Is there a way that I can create the statement that covers all possibilities such as
select * from table 1 where user_level = {special input}
that will give the same outcome as
select * from table1
Otherwise I'm looking at a lot of repetition.
You would use parameter binding. See https://www.php.net/manual/en/mysqli.quickstart.prepared-statements.php.
Say you have the following query:
SELECT * FROM table1 WHERE table1.id IN (1, 2, 3, 4, 5, ..., 999999)
What is a reasonable maximum for the number of items in the IN clause? I'm using Sphinx to generate full-text search results and inserting the IDs into a MySQL query. Is this an acceptable way to do it?
You can also have the IN clause take the results of a query, such as:
SELECT * FROM table1
WHERE table1.id IN
(
SELECT id from table2
)
That way, you don't need to generate a text string with all the possible values.
In mysql, you should be able to put as many values in the IN clause as you want, only constrained by the value of "max_allowed_packet".
http://dev.mysql.com/doc/refman/5.0/en/comparison-operators.html#function_in
http://dev.mysql.com/doc/refman/5.0/en/server-system-variables.html#sysvar_max_allowed_packet
MariaDB (10.3.22 in my case) has a limit of 999 parameters to IN() before it creates a materialized temporary table, resulting in possibly much longer execution times. Depending on your indices. I haven't found a way to control this behaviour. MySQL 5.6.27 does not have this limit, at least not at ~1000 parameters. MySQL 5.7 might very well have the same "feature".
I ended up doing a series of where id = a or id = b ... but it also works fine by using series of where id in(a, b) or id in(c, d) ....
You have to add laravel row query and then add NOT IN condition into this:
$object->whereRaw('where id NOT IN (' . $array_list . ') ');
This will work for my code.
From my experience the maximum values is 1000 values in clause IN ('1',....,'1000'),
I have 1300 value in my excel sheet,I put them all into IN ,MySQL return only 1000 .
So I am doing a little sql injection challenge because I wanted to learn about it and I have a question. I type 'hi' into the HTML form and I get this back as a response
Error: The following error occurred: [near "hi": syntax error]
Query: SELECT * FROM personnel WHERE name=''hi''
The information we need to get is located in a table called users. I was looking at sql and I see here the union operator which combines the results of 2+ select statements.
So, I try this as input: 1 UNION SELECT * FROM users and I get nothing back so it looks like it searched from that input as a name in table personnel. I thought this would work because the query would look like: SELECT * FROM personnel WHERE name=1 UNION SELECT * FROM users. Am I not understanding how the union operator works or is something else wrong in my input
This one:
set #hi = "'hi'";
select #hi;
SELECT * FROM personnel WHERE name="'hi'"
Simulate:
insert into personnel (`name`) values("'hi'");
insert into personnel (`name`) values("'hello'");
select * from personnel where `name` != "'hi'"
-- you can't use a double '' in sql query
Query: SELECT * FROM personnel WHERE name='hi'
Probably the SQL is invalid because personnel and users have different shape. You need to inject something that is identical to the initial select.
Also your entire problem goes away if you have parameterised queries instead of concatenating into SQL.
Say you have the following query:
SELECT * FROM table1 WHERE table1.id IN (1, 2, 3, 4, 5, ..., 999999)
What is a reasonable maximum for the number of items in the IN clause? I'm using Sphinx to generate full-text search results and inserting the IDs into a MySQL query. Is this an acceptable way to do it?
You can also have the IN clause take the results of a query, such as:
SELECT * FROM table1
WHERE table1.id IN
(
SELECT id from table2
)
That way, you don't need to generate a text string with all the possible values.
In mysql, you should be able to put as many values in the IN clause as you want, only constrained by the value of "max_allowed_packet".
http://dev.mysql.com/doc/refman/5.0/en/comparison-operators.html#function_in
http://dev.mysql.com/doc/refman/5.0/en/server-system-variables.html#sysvar_max_allowed_packet
MariaDB (10.3.22 in my case) has a limit of 999 parameters to IN() before it creates a materialized temporary table, resulting in possibly much longer execution times. Depending on your indices. I haven't found a way to control this behaviour. MySQL 5.6.27 does not have this limit, at least not at ~1000 parameters. MySQL 5.7 might very well have the same "feature".
I ended up doing a series of where id = a or id = b ... but it also works fine by using series of where id in(a, b) or id in(c, d) ....
You have to add laravel row query and then add NOT IN condition into this:
$object->whereRaw('where id NOT IN (' . $array_list . ') ');
This will work for my code.
From my experience the maximum values is 1000 values in clause IN ('1',....,'1000'),
I have 1300 value in my excel sheet,I put them all into IN ,MySQL return only 1000 .