I have a MEDIUMTEXT blob in a table, which contains paths, separated by new line characters. I'd like to add a "/" to the begging of each line if it is not already there. Is there a way to write a query to do this with built-in procedures?
I suppose an alternative would be to write a Python script to get the field, convert to a List, process each line and update the record. There aren't that many records in the DB, so I can take the processing delay (if it doesn't lock the entire DB or table). About 8K+ rows.
Either way would be fine. If second option is recommended, do I need to know of specific locking schematics before getting into this -- as this would be run on a live prod DB (of course, I'd take a DB snapshot). But in place updates would be best to not have downtime.
Demo:
mysql> create table mytable (id int primary key, t text );
mysql> insert into mytable values (1, 'path1\npath2\npath3');
mysql> select * from mytable;
+----+-------------------+
| id | t |
+----+-------------------+
| 1 | path1
path2
path3 |
+----+-------------------+
1 row in set (0.00 sec)
mysql> update mytable set t = concat('/', replace(t, '\n', '\n/'));
mysql> select * from mytable;
+----+----------------------+
| id | t |
+----+----------------------+
| 1 | /path1
/path2
/path3 |
+----+----------------------+
However, I would strongly recommend to store each path on its own row, so you don't have to think about this. In SQL, each column should store one value per row, not a set of values.
Is there a reason why there is enormous difference between
1. SELECT * FROM data; -- 45000 rows
2. SELECT data.* FROM data; -- 45000 rows
SHOW PROFILES;
+----------+------------+-------------------------+
| Query_ID | Duration | Query |
+----------+------------+-------------------------+
| 1 | 0.10902800 | SELECT * FROM data |
| 2 | 0.11139200 | SELECT data.* FROM data |
+----------+------------+-------------------------+
2 rows in set, 1 warning (0.00 sec)
As far as I know it, they both return the same number of rows and columns. Why the disparity in duration?
MySQL version 5.6.29
That's not much difference. Neither are optimized. Both do full table scans. Both will parse to the optimizer the same. You are talking about fractions of milliseconds difference.
You can't optimize full table scans. The problem is not "select " or "select data.". The problem is that there is no "where" clause, because that's where optimization starts.
The particular examples specified would return the same result and have the same performance.
[TableName].[column] is usually used to pinpoint the table you wish to use when two tables a present in a join or a complex statement and you want to define which column to use out of the two with the same name.
It's most common use is in a join though, for a basic statement such as the one above there is no difference and the output will be the same.
I use the SQuirreL application from sourceforge to query and extract info from my data sources. Some of the data though are huge and I don't really need to see it, I am only interested in how long it took to run the whole query.
So I am wondering if there is any command in SQL that I can add to a SELECT statement to run the query but not bring back any data just the time that it took to retrieve the data?
Thanks
You can use MySQL Query profiler to do something similar:
mysql> set profiling=1;
select * from table;
mysql> show profiles;
This should show you something like:
+----------+------------+-----------------------------------------------+
| Query_ID | Duration | Query |
+----------+------------+-----------------------------------------------+
| 0 | 0.00007300 | set profiling=1 |
| 1 | 0.00044700 | select * from table; |
+----------+------------+-----------------------------------------------+
2 rows in set (0.00 sec)
Alternately you could do something hacky with microtime(), something like:
// Grab the start time
$start = microtime(true);
// Execute your query
mysql_query('select * from table');
// Do something with the execution time
echo microtime(true) - $start .' seconds';
But obviously this is far from ideal (and you shouldn't be using mysql_*)!
I have a more or less good working query (concerning to the result) but it takes about 45seconds to be processed. That's definitely too long for presenting the data in a GUI.
So my demand is to find a much faster/efficient query (something around a few milliseconds would be nice)
My data table has something around 3000 ~2,619,395 entries and is still growing.
Schema:
num | station | fetchDate | exportValue | error
1 | PS1 | 2010-10-01 07:05:17 | 300 | 0
2 | PS2 | 2010-10-01 07:05:19 | 297 | 0
923 | PS1 | 2011-11-13 14:45:47 | 82771 | 0
Explanation
the exportValue is always incrementing
the exportValue represents the actual absolute value
in my case there are 10 stations
every ~15 minutes 10 new entries are written to the table
error is just an indicator for a proper working station
Working query:
select
YEAR(fetchDate), station, Max(exportValue)-MIN(exportValue)
from
registros
where
exportValue > 0 and error = 0
group
by station, YEAR(fetchDate)
order
by YEAR(fetchDate), station
Output:
Year | station | Max-Min
2008 | PS1 | 24012
2008 | PS2 | 23709
2009 | PS1 | 28102
2009 | PS2 | 25098
My thoughts on it:
writing several queries with between statements like 'between 2008-01-01 and 2008-01-02' to fetch the MIN(exportValue) and between 2008-12-30 and 2008-12-31' to grab the MAX(exportValue) - Problem: a lot of queries and the problem with having no data in a specified time range (it's not guaranteed that there will be data)
limiting the resultset to my 10 stations only with using order by MIN(fetchDate) - problem: takes also a long time to process the query
Additional Info:
I'm using the query in a JAVA Application. That means, it would be possible to do some post-processing on the resultset if necessary. (JPA 2.0)
Any help/approaches/ideas are very appreciated. Thanks in advance.
Adding suitable indexes will help.
2 compound indexes will speed things up significantly:
ALTER TABLE tbl_name ADD INDEX (error, exportValue);
ALTER TABLE tbl_name ADD INDEX (station, fetchDate);
This query running on 3000 records should be extremely fast.
Suggestions:
do You have PK set on this table? station, fetchDate?
add indexes; You should experiment and try with indexes as rich.okelly suggested in his answer
depending on experiments with indexes, try breaking your query into multiple statements - in one stored procedure; this way You will not loose time in network traffic between multiple queries sent from client to mysql
You mentioned that You tried with separate queries and there is a problem when there is no data for particular month; it is regular case in business applications, You should handle it in a "master query" (stored procedure or application code)
guess fetchDate is current date and time at the moment of record insertion; consider keeping previous months data in sort of summary table with fields: year, month, station, max(exportValue), min(exportValue) - this means that You should insert summary records in summary table at the end of each month; deleting, keeping or moving detail records to separate table is your choice
Since your table is rapidly growing (every 15 minutes) You should take the last suggestion into account. Probably, there is no need to keep detailed history at one place. Archiving data is process that should be done as part of maintenance.
In the footer of my page, I would like to add something like "last updated the xx/xx/200x" with this date being the last time a certain mySQL table has been updated.
What is the best way to do that? Is there a function to retrieve the last updated date? Should I access to the database every time I need this value?
In later versions of MySQL you can use the information_schema database to tell you when another table was updated:
SELECT UPDATE_TIME
FROM information_schema.tables
WHERE TABLE_SCHEMA = 'dbname'
AND TABLE_NAME = 'tabname'
This does of course mean opening a connection to the database.
An alternative option would be to "touch" a particular file whenever the MySQL table is updated:
On database updates:
Open your timestamp file in O_RDRW mode
close it again
or alternatively
use touch(), the PHP equivalent of the utimes() function, to change the file timestamp.
On page display:
use stat() to read back the file modification time.
I'm surprised no one has suggested tracking last update time per row:
mysql> CREATE TABLE foo (
id INT PRIMARY KEY
x INT,
updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
ON UPDATE CURRENT_TIMESTAMP,
KEY (updated_at)
);
mysql> INSERT INTO foo VALUES (1, NOW() - INTERVAL 3 DAY), (2, NOW());
mysql> SELECT * FROM foo;
+----+------+---------------------+
| id | x | updated_at |
+----+------+---------------------+
| 1 | NULL | 2013-08-18 03:26:28 |
| 2 | NULL | 2013-08-21 03:26:28 |
+----+------+---------------------+
mysql> UPDATE foo SET x = 1234 WHERE id = 1;
This updates the timestamp even though we didn't mention it in the UPDATE.
mysql> SELECT * FROM foo;
+----+------+---------------------+
| id | x | updated_at |
+----+------+---------------------+
| 1 | 1235 | 2013-08-21 03:30:20 | <-- this row has been updated
| 2 | NULL | 2013-08-21 03:26:28 |
+----+------+---------------------+
Now you can query for the MAX():
mysql> SELECT MAX(updated_at) FROM foo;
+---------------------+
| MAX(updated_at) |
+---------------------+
| 2013-08-21 03:30:20 |
+---------------------+
Admittedly, this requires more storage (4 bytes per row for TIMESTAMP).
But this works for InnoDB tables before 5.7.15 version of MySQL, which INFORMATION_SCHEMA.TABLES.UPDATE_TIME doesn't.
I don't have information_schema database, using mysql version 4.1.16, so in this case you can query this:
SHOW TABLE STATUS FROM your_database LIKE 'your_table';
It will return these columns:
| Name | Engine | Version | Row_format | Rows | Avg_row_length | Data_length | Max_data_length | Index_length | Data_free | Auto_increment | Create_time | Update_time | Check_time | Collation | Checksum | Create_options | Comment |
As you can see there is a column called: "Update_time" that shows you the last update time for your_table.
The simplest thing would be to check the timestamp of the table files on the disk. For example, You can check under your data directory
cd /var/lib/mysql/<mydatabase>
ls -lhtr *.ibd
This should give you the list of all tables with the table when it was last modified the oldest time, first.
For a list of recent table changes use this:
SELECT UPDATE_TIME, TABLE_SCHEMA, TABLE_NAME
FROM information_schema.tables
ORDER BY UPDATE_TIME DESC, TABLE_SCHEMA, TABLE_NAME
I would create a trigger that catches all updates/inserts/deletes and write timestamp in custom table, something like
tablename | timestamp
Just because I don't like the idea to read internal system tables of db server directly
Although there is an accepted answer I don't feel that it is the right one. It is the simplest way to achieve what is needed, but even if already enabled in InnoDB (actually docs tell you that you still should get NULL ...), if you read MySQL docs, even in current version (8.0) using UPDATE_TIME is not the right option, because:
Timestamps are not persisted when the server is restarted or when the
table is evicted from the InnoDB data dictionary cache.
If I understand correctly (can't verify it on a server right now), timestamp gets reset after server restart.
As for real (and, well, costly) solutions, you have Bill Karwin's solution with CURRENT_TIMESTAMP and I'd like to propose a different one, that is based on triggers (I'm using that one).
You start by creating a separate table (or maybe you have some other table that can be used for this purpose) which will work like a storage for global variables (here timestamps). You need to store two fields - table name (or whatever value you'd like to keep here as table id) and timestamp. After you have it, you should initialize it with this table id + starting date (NOW() is a good choice :) ).
Now, you move to tables you want to observe and add triggers AFTER INSERT/UPDATE/DELETE with this or similar procedure:
CREATE PROCEDURE `timestamp_update` ()
BEGIN
UPDATE `SCHEMA_NAME`.`TIMESTAMPS_TABLE_NAME`
SET `timestamp_column`=DATE_FORMAT(NOW(), '%Y-%m-%d %T')
WHERE `table_name_column`='TABLE_NAME';
END
OS level analysis:
Find where the DB is stored on disk:
grep datadir /etc/my.cnf
datadir=/var/lib/mysql
Check for most recent modifications
cd /var/lib/mysql/{db_name}
ls -lrt
Should work on all database types.
a) It will show you all tables and there last update dates
SHOW TABLE STATUS FROM db_name;
then, you can further ask for specific table:
SHOW TABLE STATUS FROM db_name like 'table_name';
b) As in above examples you cannot use sorting on 'Update_time' but using SELECT you can:
SELECT * FROM information_schema.tables WHERE TABLE_SCHEMA='db_name' ORDER BY UPDATE_TIME DESC;
to further ask about particular table:
SELECT * FROM information_schema.tables WHERE TABLE_SCHEMA='db_name' AND table_name='table_name' ORDER BY UPDATE_TIME DESC';
I got this to work locally, but not on my shared host for my public website (rights issue I think).
SELECT last_update FROM mysql.innodb_table_stats WHERE table_name = 'yourTblName';
'2020-10-09 08:25:10'
MySQL 5.7.20-log on Win 8.1
Just grab the file date modified from file system. In my language that is:
tbl_updated = file.update_time(
"C:\ProgramData\MySQL\MySQL Server 5.5\data\mydb\person.frm")
Output:
1/25/2013 06:04:10 AM
If you are running Linux you can use inotify to look at the table or the database directory. inotify is available from PHP, node.js, perl and I suspect most other languages. Of course you must have installed inotify or had your ISP install it. A lot of ISP will not.
Not sure if this would be of any interest. Using mysqlproxy in between mysql and clients, and making use of a lua script to update a key value in memcached according to interesting table changes UPDATE,DELETE,INSERT was the solution which I did quite recently. If the wrapper supported hooks or triggers in php, this could have been eaiser. None of the wrappers as of now does this.
i made a column by name : update-at in phpMyAdmin and got the current time from Date() method in my code (nodejs) . with every change in table this column hold the time of changes.
Same as others, but with some conditions i've used, to save time:
SELECT
UPDATE_TIME,
TABLE_SCHEMA,
TABLE_NAME
FROM
information_schema.tables
WHERE
1 = 1
AND UPDATE_TIME > '2021-11-09 00:00:00'
AND TABLE_SCHEMA = 'db_name_here'
AND TABLE_NAME not in ('table_name_here',)
ORDER BY
UPDATE_TIME DESC,
TABLE_SCHEMA,
TABLE_NAME;
This is what I did, I hope it helps.
<?php
mysql_connect("localhost", "USER", "PASSWORD") or die(mysql_error());
mysql_select_db("information_schema") or die(mysql_error());
$query1 = "SELECT `UPDATE_TIME` FROM `TABLES` WHERE
`TABLE_SCHEMA` LIKE 'DataBaseName' AND `TABLE_NAME` LIKE 'TableName'";
$result1 = mysql_query($query1) or die(mysql_error());
while($row = mysql_fetch_array($result1)) {
echo "<strong>1r tr.: </strong>".$row['UPDATE_TIME'];
}
?>
Cache the query in a global variable when it is not available.
Create a webpage to force the cache to be reloaded when you update it.
Add a call to the reloading page into your deployment scripts.