I have an old website that uses big database and I do not want to upgrade it now. The issue is the mysql database has some queries takes very long time when high traffic about 4000 online users causes mysql to to reach 600%-800% and I have to manually restart the mysql server from WHM.
I want to use cron job simple shell script to read the mysql process list every 10 seconds and if any process time more than say 10 seconds it kill this process.
This is the query I found for doing such task:
mysql -e 'SELECT * FROM INFORMATION_SCHEMA.PROCESSLIST where time>10 and command<>"Sleep"'
I think to get the process ID to kill I should use:
mysql -e 'SELECT ID FROM INFORMATION_SCHEMA.PROCESSLIST where time>10 and command<>"Sleep"'
The output like that:
+------+
| ID |
+------+
| 1095 |
| 1094 |
| 1081 |
| 1079 |
| 1078 |
| 1074 |
| 1040 |
| 1038 |
+------+
Now I have this output table, I just need to wrap this task in shell script to parse these process ID's and kill them.
You can save the output in an array and use grep to filter only the digits.
mapfile -t array < <(mysql .... | grep -Ewo '[[:digit:]]+')
Another option is to use a while read loop
while read -r digits; do
if [[ $digits =~ .*([[:digit:]]{4}).* ]]; then
array+=("${BASH_REMATCH[1]}")
fi
done < <(mysql ....)
Now "${array[#]}" has all that digits only value.
Kill it check if it is running, loop through it one by one and so on.
Here is what I came up with for specific databases user "db_user" to avoid system long process like backups etc:
#!/bin/bash
for i in $(mysql -Ne 'select id from information_schema.processlist where USER="db_user" and time>20 and command<>"Sleep";'); do
mysql -e "kill ${i}"
done
saved it as mysql_kill_high_processes.sh and added it to the root user cron to run every minute.
I am having issues and not understanding what the underlying cause is.
I have a table that has these three fields:
______________________________________________
| cid | order_id | TxRefNum |
----------------------------------------------
I am making a simple call in my bash script (there is literally no other code to start with)
#!/bin/bash
mysql --login-path=main-data -e "SELECT
cid,
order_id,
TxRefNum
FROM database.orders_temp" |
while read this that other; do
echo "$this || $that || $other"
done
I would expect to see the following:
__________________________________________________________
| 29 | F0VIc - CHATEAU ROOFIN | 5555555 |
----------------------------------------------------------
Instead my script is splitting the string $that into two different strings .. The echo is actually:
___________________________________________________
| 29 | F0VIc | - CHATEAU ROOFIN |
---------------------------------------------------
Do I have to set a delimiter when setting my variables in my while loop?? I am truly stumped!!
Getting output from the mysql command formatted in an intelligent way is problematic. In your case bash is interpreting the as a delimiter. You need to split a different way. I was able to get this working. You'll note the | in the query as well at the IFS line at the tope
#!/bin/bash
IFS='|' # set the delimiter
mysql --login-path=main-data -e "SELECT
29 as cid, '|',
'F0VIc - CHATEAU ROOFIN' as order_id,
'|',
5555555 as TxRefNum
FROM dual" |
while read this that other; do
echo "$this || $that || $other"
done
at first, I am a very new on shell scripting, so please don't shoot me !! :)
What I try to do. I have a multi-site WordPress installation, and I like to write a script that will be able to export specific tables from the schema either by passing the site id as argument in shell script, or by set an option to export all selected the tables of the schema.
The WordPress, in order to recognize which table set is for which site, changes the prefix of each table set. So In example does the following :
wp_options
wp_1_options
...
wp_x_options
In addition, the WordPress store the blog id in a special table called wp_blogs
So, from my shell script I run the following code :
mysql -uUSER -pPASS -e 'SELECT `blog_id` AS `ID`, `path` AS `Slug` FROM `wp`.`wp_blogs`'
and I am getting the following results
+----+---------------------------+
| ID | Slug |
+----+---------------------------+
| 1 | / |
| 2 | /site-2-slug/ |
| 4 | /site-4-slug/ |
| 5 | /site-5-slug/ |
| 6 | /site-6-slug/ |
| 7 | /site-7-slug/ |
| 8 | /site-8-slug/ |
| 9 | /site-9-slug/ |
| 10 | /site-10-slug/ |
+----+---------------------------+
So, now the actual question is, how can I parse the MySql result line by line, in order to get the ID and the Slug information ?
Side note 1 : The whole script has been generated and run's somehow manually. I need now this information in order to automate even farther the exporting script.
*Side note 2 : The MySql executed via the Vagrant ssh like the following line : *
sudo vagrant ssh --command "mysql -uroot -proot -e 'SELECT blog_id FROM wp.wp_blogs'"
You could save the result in a file using INTO like below:
SELECT blog_id, path FROM wp.wp_blogs
INTO OUTFILE '/tmp/blogs.csv'
FIELDS TERMINATED BY ','
ENCLOSED BY '"'
LINES TERMINATED BY '\n'
And then you could process it line by line either usingsed/awk/simple while loop. Say you want to search for site and replace it with mysite, you could do something like:
awk -F',' '{print "Id: " $1 ", path: "$2}' /tmp/blogs.csv ##or simply cat the file.
I have a problem with writing in the variable from table in MySQL database.
DB="--user=$OSD_USERNAME \
--password=$OSD_PASSWORD \
--database=$OSD_DB -N";
mysql --user="$user" \
--password="$password" \
--database="$database" \
--execute="DROP DATABASE $user; CREATE DATABASE $database;"
id_customer = $(echo "SELECT id FROM customers WHERE customers.customer='John'"| mysql $DB)
My problem is that id_customer is still empty.
When I connect to mysql databases and put there this:
SELECT id FROM customers WHERE customers.customer='John'
then it give me table like this
+----+
| id |
+----+
| 1 |
+----+
Only empty, no errors? Try remove spaces between variable name and $
In the footer of my page, I would like to add something like "last updated the xx/xx/200x" with this date being the last time a certain mySQL table has been updated.
What is the best way to do that? Is there a function to retrieve the last updated date? Should I access to the database every time I need this value?
In later versions of MySQL you can use the information_schema database to tell you when another table was updated:
SELECT UPDATE_TIME
FROM information_schema.tables
WHERE TABLE_SCHEMA = 'dbname'
AND TABLE_NAME = 'tabname'
This does of course mean opening a connection to the database.
An alternative option would be to "touch" a particular file whenever the MySQL table is updated:
On database updates:
Open your timestamp file in O_RDRW mode
close it again
or alternatively
use touch(), the PHP equivalent of the utimes() function, to change the file timestamp.
On page display:
use stat() to read back the file modification time.
I'm surprised no one has suggested tracking last update time per row:
mysql> CREATE TABLE foo (
id INT PRIMARY KEY
x INT,
updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
ON UPDATE CURRENT_TIMESTAMP,
KEY (updated_at)
);
mysql> INSERT INTO foo VALUES (1, NOW() - INTERVAL 3 DAY), (2, NOW());
mysql> SELECT * FROM foo;
+----+------+---------------------+
| id | x | updated_at |
+----+------+---------------------+
| 1 | NULL | 2013-08-18 03:26:28 |
| 2 | NULL | 2013-08-21 03:26:28 |
+----+------+---------------------+
mysql> UPDATE foo SET x = 1234 WHERE id = 1;
This updates the timestamp even though we didn't mention it in the UPDATE.
mysql> SELECT * FROM foo;
+----+------+---------------------+
| id | x | updated_at |
+----+------+---------------------+
| 1 | 1235 | 2013-08-21 03:30:20 | <-- this row has been updated
| 2 | NULL | 2013-08-21 03:26:28 |
+----+------+---------------------+
Now you can query for the MAX():
mysql> SELECT MAX(updated_at) FROM foo;
+---------------------+
| MAX(updated_at) |
+---------------------+
| 2013-08-21 03:30:20 |
+---------------------+
Admittedly, this requires more storage (4 bytes per row for TIMESTAMP).
But this works for InnoDB tables before 5.7.15 version of MySQL, which INFORMATION_SCHEMA.TABLES.UPDATE_TIME doesn't.
I don't have information_schema database, using mysql version 4.1.16, so in this case you can query this:
SHOW TABLE STATUS FROM your_database LIKE 'your_table';
It will return these columns:
| Name | Engine | Version | Row_format | Rows | Avg_row_length | Data_length | Max_data_length | Index_length | Data_free | Auto_increment | Create_time | Update_time | Check_time | Collation | Checksum | Create_options | Comment |
As you can see there is a column called: "Update_time" that shows you the last update time for your_table.
The simplest thing would be to check the timestamp of the table files on the disk. For example, You can check under your data directory
cd /var/lib/mysql/<mydatabase>
ls -lhtr *.ibd
This should give you the list of all tables with the table when it was last modified the oldest time, first.
For a list of recent table changes use this:
SELECT UPDATE_TIME, TABLE_SCHEMA, TABLE_NAME
FROM information_schema.tables
ORDER BY UPDATE_TIME DESC, TABLE_SCHEMA, TABLE_NAME
I would create a trigger that catches all updates/inserts/deletes and write timestamp in custom table, something like
tablename | timestamp
Just because I don't like the idea to read internal system tables of db server directly
Although there is an accepted answer I don't feel that it is the right one. It is the simplest way to achieve what is needed, but even if already enabled in InnoDB (actually docs tell you that you still should get NULL ...), if you read MySQL docs, even in current version (8.0) using UPDATE_TIME is not the right option, because:
Timestamps are not persisted when the server is restarted or when the
table is evicted from the InnoDB data dictionary cache.
If I understand correctly (can't verify it on a server right now), timestamp gets reset after server restart.
As for real (and, well, costly) solutions, you have Bill Karwin's solution with CURRENT_TIMESTAMP and I'd like to propose a different one, that is based on triggers (I'm using that one).
You start by creating a separate table (or maybe you have some other table that can be used for this purpose) which will work like a storage for global variables (here timestamps). You need to store two fields - table name (or whatever value you'd like to keep here as table id) and timestamp. After you have it, you should initialize it with this table id + starting date (NOW() is a good choice :) ).
Now, you move to tables you want to observe and add triggers AFTER INSERT/UPDATE/DELETE with this or similar procedure:
CREATE PROCEDURE `timestamp_update` ()
BEGIN
UPDATE `SCHEMA_NAME`.`TIMESTAMPS_TABLE_NAME`
SET `timestamp_column`=DATE_FORMAT(NOW(), '%Y-%m-%d %T')
WHERE `table_name_column`='TABLE_NAME';
END
OS level analysis:
Find where the DB is stored on disk:
grep datadir /etc/my.cnf
datadir=/var/lib/mysql
Check for most recent modifications
cd /var/lib/mysql/{db_name}
ls -lrt
Should work on all database types.
a) It will show you all tables and there last update dates
SHOW TABLE STATUS FROM db_name;
then, you can further ask for specific table:
SHOW TABLE STATUS FROM db_name like 'table_name';
b) As in above examples you cannot use sorting on 'Update_time' but using SELECT you can:
SELECT * FROM information_schema.tables WHERE TABLE_SCHEMA='db_name' ORDER BY UPDATE_TIME DESC;
to further ask about particular table:
SELECT * FROM information_schema.tables WHERE TABLE_SCHEMA='db_name' AND table_name='table_name' ORDER BY UPDATE_TIME DESC';
I got this to work locally, but not on my shared host for my public website (rights issue I think).
SELECT last_update FROM mysql.innodb_table_stats WHERE table_name = 'yourTblName';
'2020-10-09 08:25:10'
MySQL 5.7.20-log on Win 8.1
Just grab the file date modified from file system. In my language that is:
tbl_updated = file.update_time(
"C:\ProgramData\MySQL\MySQL Server 5.5\data\mydb\person.frm")
Output:
1/25/2013 06:04:10 AM
If you are running Linux you can use inotify to look at the table or the database directory. inotify is available from PHP, node.js, perl and I suspect most other languages. Of course you must have installed inotify or had your ISP install it. A lot of ISP will not.
Not sure if this would be of any interest. Using mysqlproxy in between mysql and clients, and making use of a lua script to update a key value in memcached according to interesting table changes UPDATE,DELETE,INSERT was the solution which I did quite recently. If the wrapper supported hooks or triggers in php, this could have been eaiser. None of the wrappers as of now does this.
i made a column by name : update-at in phpMyAdmin and got the current time from Date() method in my code (nodejs) . with every change in table this column hold the time of changes.
Same as others, but with some conditions i've used, to save time:
SELECT
UPDATE_TIME,
TABLE_SCHEMA,
TABLE_NAME
FROM
information_schema.tables
WHERE
1 = 1
AND UPDATE_TIME > '2021-11-09 00:00:00'
AND TABLE_SCHEMA = 'db_name_here'
AND TABLE_NAME not in ('table_name_here',)
ORDER BY
UPDATE_TIME DESC,
TABLE_SCHEMA,
TABLE_NAME;
This is what I did, I hope it helps.
<?php
mysql_connect("localhost", "USER", "PASSWORD") or die(mysql_error());
mysql_select_db("information_schema") or die(mysql_error());
$query1 = "SELECT `UPDATE_TIME` FROM `TABLES` WHERE
`TABLE_SCHEMA` LIKE 'DataBaseName' AND `TABLE_NAME` LIKE 'TableName'";
$result1 = mysql_query($query1) or die(mysql_error());
while($row = mysql_fetch_array($result1)) {
echo "<strong>1r tr.: </strong>".$row['UPDATE_TIME'];
}
?>
Cache the query in a global variable when it is not available.
Create a webpage to force the cache to be reloaded when you update it.
Add a call to the reloading page into your deployment scripts.