MySQL checksum to all tables in a database - mysql

Question

I am evaluating a PHP/MySQL based software.
I want to look which tables affected when certain operations triggered.
After some googling, I was told that checksum table tbl_name can do the job. I just need to know how to use checksum for all the tables in the db.
To checksum all the tables one by one manually definitely not preferred as the database contains hundreds of tables.

Answer 1

Checksumming all tables seems like a lot of expensive calculation work just to detect which tables changed.
I'd suggest to get this information using the sys.schema_table_statistics table.
mysql> select table_schema, table_name, rows_fetched, rows_inserted, rows_updated, rows_deleted
from sys.schema_table_statistics where table_schema='test'
+--------------+---------------------+--------------+---------------+--------------+--------------+
| table_schema | table_name | rows_fetched | rows_inserted | rows_updated | rows_deleted |
+--------------+---------------------+--------------+---------------+--------------+--------------+
| test | sysbench_results | 870 | 144 | 0 | 0 |
+--------------+---------------------+--------------+---------------+--------------+--------------+
You probably want to reset the counters between your tests. Use sys.ps_truncate_all_tables()
mysql> call sys.ps_truncate_all_tables(FALSE);
+---------------------+
| summary |
+---------------------+
| Truncated 31 tables |
+---------------------+
mysql> select table_schema, table_name, rows_fetched, rows_inserted, rows_updated, rows_deleted
from sys.schema_table_statistics where table_schema='test';
+--------------+---------------------+--------------+---------------+--------------+--------------+
| table_schema | table_name | rows_fetched | rows_inserted | rows_updated | rows_deleted |
+--------------+---------------------+--------------+---------------+--------------+--------------+
| test | sysbench_results | 0 | 0 | 0 | 0 |
+--------------+---------------------+--------------+---------------+--------------+--------------+
The sys schema comes pre-installed in MySQL 5.7.
If you use MySQL 5.6, you may need to install it yourself. It's just an SQL script that creates some views into the performance_schema. Very easy to install.
You can get the sys schema here: https://github.com/mysql/mysql-sys

Answer 2

I want to look which tables affected when certain operations triggered.
What do you mean by this?
Do you know what operations have been triggered, and you're merely attempting to understand what effect they had on your database (e.g. to verify their correctness)? Or do you not know what operations have been triggered (e.g. during some interval) but you nevertheless want to understand how the database has changed, perhaps in an attempt to determine what those operations were?
There are very few situations where I would expect the best approach to be that which you are exploring (inspecting the database for changes). Instead, some form of logging—whether built-in to the RDBMS (such as MySQL's General Query Log or perhaps through triggers as suggested by Sumesh), or more likely at some higher level (e.g. within the accessing application)—would almost always be preferable. This leads me to lean toward thinking you have an XY Problem.
However, on the assumption that you really do want to identify the tables that have been modified since some last known good point in time, you can query the INFORMATION_SCHEMA.TABLES table, which contains not only the CHECKSUM for every table in the RDBMS but also other potentially useful information like UPDATE_TIME. So, for example, to identify all tables changed in the last five minutes one could do:
SELECT TABLE_SCHEMA, TABLE_NAME
FROM INFORMATION_SCHEMA.TABLES
WHERE UPDATE_TIME > NOW() - INTERVAL 5 MINUTE

Answer 3

You could generate the CHECKSUM statements for all tables:
SELECT CONCAT('CHECKSUM TABLE ', table_name, ';') AS statement
FROM information_schema.tables
WHERE table_schema = 'YourDBNameHere'
Then copy this output and paste it into Workbench or whatever tool you need to use. If you need to do this from within application (e.g. PHP) code, then you would probably have to use pure dynamic MySQL.

Answer 4

For those who came here for an answer how to get checksum for all the tables in one query (as it was in my case):
SET group_concat_max_len = CAST(
(
SELECT SUM(LENGTH(TABLE_NAME)) + COUNT(*) * LENGTH(', ')
FROM information_schema.tables WHERE `TABLE_SCHEMA` = 'your_database_name'
) AS UNSIGNED
);
SET #sql_command:= (
SELECT CONCAT(
'CHECKSUM TABLE ',
GROUP_CONCAT( TABLE_NAME ORDER BY `TABLE_NAME` SEPARATOR ', ' )
)
FROM information_schema.tables
WHERE `TABLE_SCHEMA` = 'your_database_name'
ORDER BY `TABLE_NAME`
);
PREPARE statement FROM #sql_command;
EXECUTE statement;
DEALLOCATE PREPARE statement;
The mere idea is to create CHECKSUM TABLE statement which include all tables names in it. So yes, it is some sort of little bit upgraded version of answer given by Tim Biegeleisen.
First we set maximum permitted result lenght for GROUP_CONCAT() function (which is 1024 bytes by default). It is calculated as number of symbols in all table names inculding the separator which will be putted between these names:
SET group_concat_max_len = CAST(
(
SELECT SUM(LENGTH(TABLE_NAME)) + COUNT(*) * LENGTH(', ')
FROM information_schema.tables WHERE `TABLE_SCHEMA` = 'your_database_name'
) AS UNSIGNED
);
Then we put all the tables names together in one CHECKSUM TABLE statement and store it in string variable:
SET #sql_command:= (
SELECT CONCAT(
'CHECKSUM TABLE ',
GROUP_CONCAT( TABLE_NAME ORDER BY `TABLE_NAME` SEPARATOR ', ' )
)
FROM information_schema.tables
WHERE `TABLE_SCHEMA` = 'your_database_name'
ORDER BY `TABLE_NAME`
);
And finally executing the statement to see the results:
PREPARE statement FROM #sql_command;
EXECUTE statement;
DEALLOCATE PREPARE statement;
Unfortunately you can't further manipulate with result set using MySQL statements only (i.e. insert to table or join with other result sets).
So if you require to do some comparisons you will eventually need to use additional code in your favorite programming language (or use capable software) to accomplish the task.

Answer 5

The question does not state using a shell script to accomplish things isn't allowed, so I'll post one such approach here (PHP is able to invoke shell scripts - see http://php.net/manual/en/function.shell-exec.php - if safe mode is not enabled):
If your script has shell access at its disposal and a checksum tool - like md5sum - one can also do something like this to collect checksums for each table:
#!/bin/bash
DATABASEPATH="/var/lib/mysql/yourdatabase"
cd "$DATABASEPATH" &&
for TABLEFILE in `ls -t *.ibd`; do
SUMANDTABLE=`md5sum "$TABLEFILE"`
echo "${SUMANDTABLE//.ibd}"
done
And optionally, if you don't want a checksum calculated for all tables, you could also check if the modification date of the "$TABLEFILE" is within range. If not, you just exit the script (the ls -t orders by modification date, descending).
To access modification date use something like e.g. stat -c %Y "$TABLEFILE". This would give you the modification date in seconds since Epoch.
To access current date, also in seconds since Epoch use: date +%s.
One can then subtract the modification date from the current date to establish how many seconds ago a "$TABLEFILE" has changed.
Another related method, which in some cases could apply, would be to save the ls -t *.ibd listing (without even calculating checksums, just store filenames in order), then start an operation and at the end of that operation check for difference in file listing with another execution of ls -t *.ibd.