Having a simple mysql table with id (primary key) and hash (index). Some other columns (varchar / int) but no queries on them needed.
My total table size is around 350MB with 2.5M rows.
SELECT COUNT(*) FROM table LIMIT 1;
Is taking about 0.5 - 1s. My innodb buffer is set at 1GB. I've also tried variations (without improvements) like:
SELECT COUNT(id) FROM table LIMIT 1;
SELECT COUNT(*) FROM table WHERE id > 0 LIMIT 1;
A single
SELECT * FROM table WHERE id = 'x' LIMIT 1;
would return within 1 ms (localhost mysql). Any tips on improving the slow count (0.5 - 1s) would be greatly appreciated.
You can find a bried explanation here. In short, innodb has to make a full table scan in order to count all rows (without a where clause, which would utilize an index).
See also this answer.
BTW, I can't see any point in using LIMIT 1 in your query. Since there is no group by clause, it will always return one record.
Some time ago I have found for me, that MyISAM tables make these operations faster. But not all tables and architectures can be MyISAM. Check your schema, maybe you can switch this table to MyISAM.
Also use COUNT(1) instead of COUNT(*)
And another technique for you. Create trigger and save count in separated place. Create counter_table and folowing trigger:
DELIMITER //
CREATE TRIGGER update_counter AFTER INSERT ON table_name
FOR EACH ROW
BEGIN
UPDATE counter_table
SET counter = counter + 1
END;
Related
I've a big innodb table which contain 10.000.000 rows
this query
SELECT count(id) FROM table_name takes 4-6 seconds to execute
I need to decrease the query execution time
1) can someone advice how to achieve this without changing the table to MyIsam
2) In case we need to use a MySQL cache how can we turn that ON on the server?
Lie about the count in the application. Really, in rare cases you need exact number.
Approximate row count (but as fast as select count(*) in MyISAM) you can get from
SELECT MAX(id) - MIN(id) AS count FROM table
if you still need the exact number you can create a table with the count number and update it with a trigger ON INSERT and ON DELETE
If your table does not change often, using the MySQL query cache is a good solution.
[mysqld]
query_cache_type = 1
query_cache_size = 10M
Also add index on field 'id' if it doesnt exist. Also if you are writing an application store count in a seperate table as you insert or update. Its handy.
Can someone tell me a good method for automatically placing a unique random number in a mysql database table when a new record is created.
I would create a table with a pool of numbers:
Create Table pool (number int PRIMARY KEY AUTO_INCREMENT);
Insert Into pool (),(),(),(),(),(),(),(),…;
And then define a trigger which picks one random number from that pool:
CREATE TRIGGER pickrand BEFORE INSERT ON mytable
FOR EACH ROW BEGIN
DECLARE nr int;
SET nr = (SELECT number FROM pool order by rand() limit 1);
DELETE FROM pool WHERE number = nr;
SET NEW.nr = nr;
END
In order to avoid concurrency issues you have to run queries in transactions. If performance becomes an issue (because of the slow order by rand()) you can change the way to select a random record.
Your criteria of unique and random are generally conflicting. You can easily accomplish one or the other, but both is difficult, and would require evaluating every row when testing a new potential number to insert.
The best method that meets your criteria is to generate a UUID with the UUID function.
The better choice would be to re-evaluate your design and remove one of the (unique, random) criteria.
Your best option is Autoincrement Column
see here for syntax
perhaps for Random number
try this!
select FLOOR(RAND() * 401) + 100
edit
SELECT FLOOR(RAND() * 99999) AS sup_rand
FROM table
WHERE "sup_rand" NOT IN (SELECT sup_rand FROM table)
LIMIT 1
Steps:
1.Generate a random number
2.Check if its already present
3.if not continue use the number
4.Repeat
I have a column
id
-----
1
32
3
6
5
22
54
21
Ques1: How can I select all records from the column excepting the first 3 records ?
Ques2: How can select last 3 records ?
-Thanks.
You basically need to put such queries into stored procedures due to some limitations on LIMIT. You can't use sub selects or variables in plain sql. In stored procedures you can use variables.
This works, unfortunately I can't show it in sqlfiddle cause they seem to have limited support for stored procedures.
drop procedure if exists all_but_3;
delimiter //
create procedure all_but_3()
begin
declare v_max bigint unsigned default ~0;
select * from your_table limit 3, v_max;
end//
delimiter ;
drop procedure if exists last_3;
delimiter //
create procedure last_3()
begin
declare v_max bigint;
declare v_mid bigint;
select count(*) from your_table into v_max;
set v_mid := v_max - 3;
select * from your_table limit v_mid, v_max;
end//
delimiter ;
call all_but_3();
call last_3();
Elaboration on InnoDB Clustered Indexes
After discussions in one of the other answers with #fthiella I've decided to elaborate some on how this can work.
A table using InnoDB as engine will always have a clustered index. Always. It's they way data is stored in InnoDB and it's not in any way possible to create a table without a clustered index.
InnoDB will choose the primary key if there is one or the first unique index with all columns set to not null. If no such index exists InnoDB will create a hidden column with a row id. This row id works similar to auto increment and if it helps to think about it as a invisible column with auto increment I think that is fine.
Further will InnoDB return rows according to the index used. It will always use some index (only way to retrieve data is to use either a secondary index, the clustered index or a combination) so in the case where there are no explicitly created indexes rows are returned by the hidden clustered index.
This means that a query against a table with no primary key and no unique indexes with all columns set to not null and no ORDER BY will return rows in the order they where inserted.
This is the case for this question and the base for mine and many other answers.
I don't mean to say this is a good way to work with the data. Here are some things that you should think about before using this solution:
If a index that can be used as a clustered index is ever created the table will be rewritten to use that index and by doing so order the data on disk. If the index is later dropped the original insert order is lost and cannot be retrieved.
If a index is created, even if it's not unique, it could be chosen by the optimizer to be used and the rows will be ordered by that index instead.
All this is documented and for 5.5 it's the 3rd bullet points on this page
select all except the first 3 records:
select id from table1 limit 3, 18446744073709551615;
select the last 3 records:
select a.id from (select #row_num:=#row_num+1 as RowNumber,id from table1, (select #row_num:=0) x order by RowNumber desc) as a limit 3;
Use LIMIT
For #1, you theoretically need to know the number of rows, but you can just throw a huge number in instead:
SELECT bleh FROM blah LIMIT 3, 18446744073709551615
(18446744073709551615 is the max number possible -- thanks to alex vasi for pointing me to this)
For #2, you do actually need to know the number of records:
SELECT bleh FROM blah LIMIT NUM-3, 3
You'll have to get the number of rows from a separate query (which should be where NUM is).
If there's a criteria you can sort on, you can actually do the second query without doing a full count:
SELECT bleh FROM blah ORDER BY field DESC LIMIT 3
I'm assuming that's not the case though.
(Another option if you don't want to do a count on the table is to just pull all of the data and ignore the first or last 3 rows in code-land [PHP, C, etc].)
Try this:
totalCounts = SELECT COUNT(*) FROM tbl;
SELECT * FROM tbl LIMIT 3,totalCounts
# Retrieve rows 4 to number of table rows
I currently have a MySQL table of about 20 million rows, and I need to prune it. I'd like to remove every row whose updateTime (timestamp of insertion) was more than one month ago. I have not personally performed any alterations of the table's order, so the data should be in the order in which it was inserted, and there is a UNIQUE key on two fields, id and updateTime. How would I go about doing this in a short amount of time?
How much down time can you incur? How big are the rows? How many are you deleting?
Simply put, deleting rows is one of the most expensive things you can do to a table. It's just a horrible thing overall.
If you don't have to do it, and you have the disk space for it, and your queries aren't affected by the table size (well indexed queries typically ignore table size), then you may just leave well enough alone.
If you have the opportunity and can take the table offline (and you're removing a good percentage of the table), then your best bet would be to copy the rows you want to keep to a new table, drop the old one, rename the new one to the old name, and THEN recreate your indexes.
Otherwise, you're pretty much stuck with good 'ol delete.
There are two ways to remove a large number of rows. First there is the obvious way:
DELETE FROM table1 WHERE updateTime < NOW() - interval 1 month;
The second (slightly more complicated) way is to create a new table and copy the data that you want to keep, truncate your old table, then copy the rows back.
CREATE TABLE table2 AS
SELECT * FROM table1 WHERE updateTime >= NOW() - interval 1 month;
TRUNCATE table1;
INSERT INTO table1
SELECT * FROM table2;
Using TRUNCATE is much faster than a DELETE with a WHERE clause when you have a large number of rows to delete and a relatively small number that you wish to keep.
Spliting the deletes with limit might speed up the process;
I had to delete 10M rows and i issued the command. It never responded for hours.
I killed the query ( which took couple of hours)
then Split the deletes.
DELETE from table where id > XXXX limit 10000;
DELETE from table where id > XXXX limit 10000;
DELETE from table where id > XXXX limit 10000;
DELETE from table where id > XXXX limit 10000;
Then i duplicated this statement in a file and used the command.
mysql> source /tmp/delete.sql
This was much faster.
You can also try to use tools like pt-tools. and pt-archiver.
Actually even if you can't take the table offline for long, you can still use the 'rename table' technique to get rid of old data.
Stop processes writting to table.
rename table tableName to tmpTableName;
create table tableName like tmpTableName;
set #currentId=(select max(id) from tmpTableName);
set #currentId=#currentId+1;
set #indexQuery = CONCAT("alter table test auto_increment = ", #currentId);
prepare stmt from #indexQuery;
execute stmt;
deallocate prepare stmt;
Start processes writting to table.
insert into tableName
select * from tmpTableName;
drop table;
New inserts to tableName will begin at the correct index; The old data will be inserted in correct indexes.
MySQL 4.0 doesn't have information_schema and 'show table status from db' only gives approximate row count for innodb tables.
So whats the quickest way to get the count of innodb tables, of course other than count(*), which could be slower with big tables.
Updated
When using InnoDB the only accurate count of rows in your entire table is COUNT(*). Since your upgrade from 4.0 to 5.0 will only occur once, you'll just have to deal with the speed.
This applies to all versions of MySQL. As other commenters have pointed out - there is no fast SELECT COUNT(*) in InnoDB. Part of the reason for this is that InnoDB is multi-versional, and it will depend on the context of your transaction how many rows there are supposed to be in a table.
There are some workarounds:
1) If you never delete, SELECT MAX(id) should return the right number of rows.
2) Instead of deleting rows, you can archive them to a 'deleted rows table' (a lot of people seem to want to keep everything these days). Assuming that the delete is a much smaller subset of still current, you may be able to subtract count(*) from deleted_rows from SELECT max(id) from not_deleted.
3) Use triggers. This sucks for performance.
There's quite a technical discussion on this problem here:
http://mysqlha.blogspot.com/2009/08/fast-count-for-innodb.html
Wait! There is a fast way! Use a trigger and a meta table..
CREATE TABLE meta (
`name` char(32) NOT NULL ,
`value_int` int ,
unique (name)
) ENGINE = INNODB;
insert into meta (name, value_int) values ('mytable.count', 0);
then
set delimiter |
CREATE TRIGGER mytablecountinsert AFTER INSERT ON mytable
FOR EACH ROW BEGIN
update meta set value_int=value_int+1 where name='mytable.count';
END;
|
CREATE TRIGGER mytablecountdelete AFTER DELETE ON mytable
FOR EACH ROW BEGIN
update meta set value_int=value_int-1 where name='mytable.count';
END;
|
Well, using * is definitely not optimal but how about just 1 column. I usually use the id column to count # of rows.