I am following this doc on implementing data shards in mysql cluster using proxysql: https://proxysql.com/documentation/how-to-setup-proxysql-sharding/.
But my application generates prepared statements which are being captured in proxysql like
SELECT * FROM mytable WHERE id=?
I want to shard on value of id like even values goes to node-1 and odd goes to node-2.
How to set query routing rules in proxysql for this case? Is it achievable?
Related
I have a Spring JPA application which uses hibernate and mysql. I would like to log slow running database queries on this application. Hibernate provides an option to configure 'hibernate.session.events.log.LOG_QUERIES_SLOWER_THAN_MS' (with log level org.hibernate.SQL_SHOW: INFO). However using this option will log SQL with parameter values set.
Is there an option in hibernate to replace the parameter values with '?'.
ex: insert into customer (name) values (?)
instead of
insert into customer (name) values ('John')
FYI, Setting the log level on org.hibernate.SQL to DEBUG will log the SQL with ? instead of parameter values. This will print all the SQL in the logs, In my case I only want to log slow running queries
Appreciate any help.
(Maybe Hibernate is doing more than necessary...)
MySQL optionally creates "slowlog" that contains details (in including 'John') on the slow query. However, the next thing that can be done is to "digest" the log. This will replace strings and numbers with ? or S (for string). So, if you can bypass Hibernate and get the log, use pt-query-digest or mysqldumpslow -s t to do the digesting. This gives a much shorter output file (by combining queries with the same 'digest' or 'signature').
The underlying setting is long_query_time; it is a number of "seconds" and can be a fractional value such as "0.5" for half a second. Setting it to 0 would capture all queries, which it sounds like you don't want. (I almost never go much below 1.)
I am working on a high scale application of the order of 35000 Qps, using Hibernate and MySQL.
A large table has AutoIncrement Primary key, and generation defined is IDENTITY at Hibernate. Show Sql is true as well.
Whenever an Insert happens I see only one query being fired in DB, which is an
Insert statement.
Few Questions Follow:
1) I was wondering how does Hibernate get the AutoIncrement Value after insert?
2) If the answer is "SELECT LAST_INSERT_ID()", why does it not show up at VividCortex or in Show Sql Logs...?
3) How does "SELECT LAST_INSERT_ID()" account for multiple autoincrements in different tables?
4) If MySql returns a value on Insert, why aren't the MySql clients built so that we can see what is being returned?
Thanks in Advance for all the help.
You should call SELECT LAST_INSERT_ID().
Practically, you can't do the same thing as the MySQL JDBC driver using another MySQL client. You'd have to write your own client that reads and writes the MySQL protocol.
The MySQL JDBC driver gets the last insert id by parsing packets of the MySQL protocol. The last insert id is returned in this protocol by a MySQL result set.
This is why SELECT LAST_INSERT_ID() doesn't show up in query metrics. It's not calling that SQL statement, it's picking the integer out of the result set at the protocol level.
You asked how it's done internally. A relevant line of code is https://github.com/mysql/mysql-connector-j/blob/release/8.0/src/main/protocol-impl/java/com/mysql/cj/protocol/a/result/OkPacket.java#L55
Basically, it parses an integer from a known position in a packet as it receives a result set.
I'm not going to go into any more detail about parsing the protocol. I don't have experience coding a MySQL protocol client, and it's not something I wish to do.
I think it would not be a good use of your time to implement your own MySQL client.
It probably uses the standard JDBC mechanism to get generated values.
It's not
You execute it imediately after inserting in one table, and you thus get the values that have been generated by that insert. But that's not what is being used, so it's irrelevant
Not sure what you mean by that: the MySQL JDBC driver allows doing that, using the standard JDBC API
(Too long for a comment.)
SELECT LAST_INSERT_ID() uses the value already available in the connection. (This may explain its absence from any log.)
Each table has its own auto_inc value.
(I don't know any details about Hibernate.)
35K qps is possible, but it won't be easy.
Please give us more details on the queries -- SELECTs? writes? 35K INSERTs?
Are you batching the inserts in any way? You will need to do such.
What do you then use the auto_inc value in?
Do you use BEGIN..COMMIT? What value of autocommit?
When I enable server-side prepared statments via useServerPrepStmts jdbc flag, result set update operations fail after the first request for a given query with:
Result Set not updatable.This result set must come from a statement
that was created with a result set type of ResultSet.CONCUR_UPDATABLE,
the query must select only one table, can not use functions and must
select all primary keys from that table
When I disable server-side prepared statements, result set update operations work flawlessly.
Since the query involves only 1 table, has a primary key, returns a single row, and no functions are involved, what must be happening is that the prepared statement is created with ResultSet.CONCUR_READ_ONLY and then cached server-side. Subsequent requests for the same query will draw the prepared statement from the cache and then, even though the client sends ResultSet.CONCUR_UPDATABLE for rs.updateRow(), concurrency is still set to ResultSet.CONCUR_READ_ONLY on the server.
If I am correct in above assumption, how does one override the server-side cache in this case? (everything else is fine with prepared statement caching, just result set row operations are affected).
Linux (CentOS 5.7) with:
mysql-connector-java 5.1.33
mysql 5.6.20
EDIT
not relevant I notice that the first query, which always succeeds, has this in the query log: SET SQL_SELECT_LIMIT=1, and all subsequent queries fail with this: SET SQL_SELECT_LIMIT=DEFAULT. Not sure if this is the cause of the problem, or just a side effect. Guess I'll try to manually setFetchSize on the client and see if that makes a difference...
Workaround is to append FOR UPDATE to ResultSet.CONCUR_READ_ONLY select statement on the client with ResultSet.CONCUR_UPDATABLE concurrency for the new prepared statement. This allows for server statement caching while still being able to modify a JDBC ResultSet.
Side note:
the select ... for update statement itself does not appear to be eligible for caching; i.e. query log shows Prepare and Execute lines on every request.
my query such as having mysql variable declaration
SET #var1=0, #var2=0;
these variables are used in the select query
which works fantastic in phpmyadmin
but then if i write it as query in yii doesnt work
throws exception doesnt not execute but then if i remove
SET #var1=0, #var2=0;
then query executes but with no data fetched from db because it requires the set variables to fetch the result
how do i declare the set values of mysql in yii?is there any way out
As long as you reuse the same CDbCommand, you can issue multiple queries to the DB using the same connection. That will do what you need (and is what phpMyAdmin does).
Your problem is that you're doing two queries on different connections to the DB and your #vars aren't lasting between connections.
If you have set statements, you're probably writing something that is a little more procedural than a single sql statement is designed to deliver.
I would look at write in a stored procedure to the the job (http://forums.mysql.com/read.php?98,358569). Although they are a bit old school - they will probably do what you want quite effectively.
I've setup mysql replication only for a specific database on the master.
If I connect to the master and don't specify a database (e.g. in the connection string or with the 'use database' command) the statement is not sent to the slave. Is this a bug?? Why does this happen?
Example 1
with no db specified up till now: won't replicate
insert into exampledb.mytable values(1,2,3);
Example 2
replicates
use exampeldb;
insert into mytable values(1,2,3);
Not a bug. This behavior is defined in the MySql docs:
The main reason for this “check just
the default database” behavior is that
it is difficult from the statement
alone to know whether it should be
replicated (for example, if you are
using multiple-table DELETE or
multiple-table UPDATE statements that
go across multiple databases). It is
also faster to check only the default
database rather than all databases if
there is no need.