I have table ip_per_nat_vlan, innodb format. When I give truncate table, table is empty.
Then I have php script, which fill data into this table.
When is this script finished without errors (simple insert statemets) situation is following:
select * from ip_per_nat_vlan;
Empty set (0.00 sec)
.
select count(*) from ip_per_nat_vlan;
+----------+
| count(*) |
+----------+
| 0 |
+----------+
1 row in set (0.00 sec)
.
show table status;
+----------------------------------+--------+---------+------------+----------+----------------+-------------+-----------------+--------------+------------+----------------+---------------------+---------------------+------------+-----------------+----------+----------------+---------+
| Name | Engine | Version | Row_format | Rows | Avg_row_length | Data_length | Max_data_length | Index_length | Data_free | Auto_increment | Create_time | Update_time | Check_time | Collation | Checksum | Create_options | Comment |
+----------------------------------+--------+---------+------------+----------+----------------+-------------+-----------------+--------------+------------+----------------+---------------------+---------------------+------------+-----------------+----------+----------------+---------+
| ip_per_nat_vlan | InnoDB | 10 | Dynamic | 141291 | 100 | 14172160 | 0 | 6832128 | 25165824 | 143563 | 2017-12-24 16:26:40 | 2018-06-13 09:01:33 | NULL | utf8_unicode_ci | NULL |
MySQL says, that there should be 14172160 rows, but I dont see any. Where could be a problem? Transactions? But I dont see any running thread and no any fault.
Thank you. D
Structure of table is:
+------------+--------------+------+-----+---------+----------------+
| Field | Type | Null | Key | Default | Extra |
+------------+--------------+------+-----+---------+----------------+
| id | int(11) | NO | PRI | NULL | auto_increment |
| ipAddress | varchar(255) | NO | UNI | NULL | |
| nat | int(11) | NO | | NULL | |
| vlan | int(11) | NO | | NULL | |
| district | varchar(255) | YES | | NULL | |
| idOblasti | int(11) | YES | | NULL | |
| type | varchar(255) | NO | | NULL | |
| macAddress | varchar(255) | NO | | NULL | |
+------------+--------------+------+-----+---------+----------------+
There are various ways to "count" rows in a table.
The normal way. Just count them.
select count(*) as table_rows from table_name ;
Accuracy: 100% accurate count at the time of the query is run.
using the information_schema tables
select table_rows
from information_schema.tables
where table_schema = 'database_name'
and table_name = 'table_name' ;
Accuracy: Only an approximation. If the table is the target of frequent inserts and deletes, the result can be way off the actual count. This can be improved by running ANALYZE TABLE more often.
Efficiency: Very good, it doesn't touch the table at all.
As count option is 100% accurate, your table doesn't contain any data.
Check your code and default commit option of MySQL.
Looks like you are inserting rows, but not committing them, check your index length.
Check more details here
https://dba.stackexchange.com/questions/151769/mysql-difference-between-using-count-and-information-schema-tables-for-coun
First thing, I am not sure how mysql run this line and produce the result
select count() from ip_per_nat_vlan
count() will return [Err] 1064.
count(*) or else a field name should be mentioned inside.
Related
i have a performance problem in MySQL.
I have a table with 215000 rows (InnoDB Engine) inserted in it and to execute the function SUM() on one column for only 1254 rows is taking 500ms.
The version i am using is : MySQL 5.7.32
The computer specs are the following:
Core I5 3.0 Ghz
8 Gb Ram
Solid State Drive
Here i leave information about the structure of the database:
mysql> select count(*) from cuenta_corriente;
+----------+
| count(*) |
+----------+
| 214514 |
+----------+
mysql> describe cuenta_corriente;
+-----------------------+-------------+------+-----+---------+----------------+
| Field | Type | Null | Key | Default | Extra |
+-----------------------+-------------+------+-----+---------+----------------+
| ID_CUENTA_CORRIENTE | int(11) | NO | PRI | NULL | auto_increment |
| ID_CLIENTE | int(11) | YES | MUL | NULL | |
| ID_PROVEEDOR | int(11) | YES | MUL | NULL | |
| FECHA | varchar(50) | YES | MUL | NULL | |
| FECHA_FISCAL | varchar(50) | YES | | NULL | |
| ID_OPERACION | int(11) | YES | MUL | NULL | |
| ID_TIPO_OPERACION | int(11) | YES | MUL | NULL | |
| DEBE | double | YES | | 0 | |
| HABER | double | YES | | 0 | |
| TOTAL | double | YES | | 0 | |
| SALDO_ANTERIOR | double | YES | | 0 | |
| SALDO_ACTUAL | double | YES | | 0 | |
| ID_OPERACION_ASOCIADA | int(11) | YES | | NULL | |
| ELIMINADO | int(11) | YES | | 0 | |
| ID_EMPLEADO | int(11) | NO | | 0 | |
+-----------------------+-------------+------+-----+---------+----------------+
show indexes from cuenta_corriente;
+------------+------------------------------------------------------------+
| Non_unique | Key_name Column_name |
+------------+------------------------------------------------------------+
| 0 | PRIMARY ID_CUENTA_CORRIENTE |
| 1 | IDX_CUENTA_CORRIENTE ID_CLIENTE |
| 1 | IX_cuenta_corriente_FECHA FECHA |
| 1 | IX_cuenta_corriente_ID_CLIENTE ID_CLIENTE |
| 1 | IX_cuenta_corriente_ID_PROVEEDOR ID_PROVEEDOR |
| 1 | IX_cuenta_corriente_ID_TIPO_OPERACION ID_TIPO_OPERACION |
| 1 | IX_cuenta_corriente_ID_OPERACION ID_OPERACION |
| 1 | IDX_cuenta_corriente_ID_OPERACION ID_OPERACION |
+------------+------------------------------------------------------------+
8 rows in set (0.00 sec)
The problem is with the folowing queries, in my opinion they are taking too long, considering that i have an index for the column ID_CLIENTE and that there are only 1254 rows with the ID_CLIENTE column = 19. Here are the query results:
mysql> SELECT COUNT(*) FROM CUENTA_CORRIENTE WHERE ID_CLIENTE = 19;
1254 rows
mysql> SELECT DEBE FROM CUENTA_CORRIENTE WHERE ID_CLIENTE = 19;
1254 rows - 0.513 sec
mysql> SELECT SUM(DEBE) FROM CUENTA_CORRIENTE WHERE ID_CLIENTE = 19;
0.582 sec
The strange thing is if i select all the columns instead selecting only the "DEBE" column, it takes less time:
mysql> SELECT * FROM CUENTA_CORRIENTE WHERE ID_CLIENTE = 19;
0.095 sec
Can anyone help me to improve the performance?
You can make just that query fast by creating a composite index to support it.
ie:
CREATE INDEX IDX_QUERY_FAST ON cuenta_corriente (ID_CLIENTE, DEBE)
But don't forget, each index has to be maintained, so it slows down any inserts into the table, so you don't want 200 indexes supporting every possible query.
With the existing index, the engine should be smart enough to identify the 1200 rows you care about using the index, but then it has to go read all the table records (across however many pages) that have the individual rows to get the DEBE column.
Add this index to help most of the queries you have shown:
INDEX(ID_CLIENTE, DEBE)
and drop INDEX(ID_CLIENTE) if you have such.
Are all of your secondary indexes starting with the PRIMARY KEY?? (The names imply such; please provide SHOW CREATE TABLE to definitively say what columns are in each index.) Don't start an index with the PK; it is likely to be useless.
Run EXPLAIN SELECT ... to see which index a query uses.
When timing a query, run it twice. The first run may spend extra time loading index or data rows into cache (in RAM in the buffer_pool); the second run may be significantly faster because of the caching.
I am new to MySQL, and trying to using MySQL on the project, basically was tracking players performance.
Below is the table fields.
+-------------------+----------------------+-------------------+------+-----+---------+----------------+---------------------------------+---------+
| Field | Type | Collation | Null | Key | Default | Extra | Privileges | Comment |
+-------------------+----------------------+-------------------+------+-----+---------+----------------+---------------------------------+---------+
| unique_id | int(11) | NULL | NO | PRI | NULL | auto_increment | select,insert,update,references | |
| record_time | datetime | NULL | NO | | NULL | | select,insert,update,references | |
| game_sourceid | char(20) | latin1_swedish_ci | NO | MUL | NULL | | select,insert,update,references | |
| game_number | smallint(6) | NULL | NO | | NULL | | select,insert,update,references | |
| game_difficulty | char(12) | latin1_swedish_ci | NO | MUL | NULL | | select,insert,update,references | |
| cost_time | smallint(5) unsigned | NULL | NO | MUL | NULL | | select,insert,update,references | |
| country | char(3) | latin1_swedish_ci | NO | | NULL | | select,insert,update,references | |
| source | char(7) | latin1_swedish_ci | NO | | NULL | | select,insert,update,references | |
+-------------------+----------------------+-------------------+------+-----+---------+----------------+---------------------------------+---------+
and I have adding game_sourceid and game_difficulty as index and the engine is innodb.
I have insert about 11m rows of test data into this table, which is generated randomly but resembles the real data.
Basically the mostly query was like this, to get the average time and best time for a specific game_sourceid
SELECT avg(cost_time) AS avgtime
, min(cost_time) AS mintime
, count(*) AS count
FROM statistics_work_table
WHERE game_sourceid = 'standard_easy_1';
+-----------+---------+--------+
| avgtime | mintime | count |
+-----------+---------+--------+
| 1681.2851 | 420 | 138034 |
+-----------+---------+--------+
1 row in set (4.97 sec)
and the query took about 5s
I have googled about this and someone said that may caused by the amout of query count, so I am trying to narrow down the scope like this
SELECT avg(cost_time) AS avgtime
, min(cost_time) AS mintime
, count(*) AS count
FROM statistics_work_table
WHERE game_sourceid = 'standard_easy_1'
AND record_time > '2015-11-19 04:40:00';
+-----------+---------+-------+
| avgtime | mintime | count |
+-----------+---------+-------+
| 1275.2222 | 214 | 9 |
+-----------+---------+-------+
1 row in set (4.46 sec)
As you can see the 9 rows data also took about 5s, so i think it's not the problem about the query count.
The test data was generated randomly to simulate the real user's activity, so the data was discontinuous, so i added more continuous data(about 250k) with the same game_sourceid='standard_easy_9' but keep all others randomly, in other words the last 250k rows in this table has the same game_sourceid. And i'm trying to query like this:
SELECT avg(cost_time) AS avgtime
, min(cost_time) AS mintime
, count(*) AS count
FROM statistics_work_table
WHERE game_sourceid = 'standard_easy_9';
+-----------+---------+--------+
| avgtime | mintime | count |
+-----------+---------+--------+
| 1271.4806 | 70 | 259379 |
+-----------+---------+--------+
1 row in set (0.40 sec)
This time the query magically took only 0.4s, that's totally beyond my expectations.
So here's the question, the data was retrived from the player at real time, so it must be randomly and discontinuous.
I am thinking of separating the data into multiple tables by the game_sourceid, but it will take another 80 tables for that, maybe more in the future.
Since I am new to MySQL, I am wondering if there are any other solutions for this, or just my query was too bad.
Update: Here's the index of my table
mysql> show index from statistics_work_table;
+-----------------------+------------+-------------------------+--------------+-----------------+-----------+-------------+----------+--------+------+------------+---------+---------------+
| Table | Non_unique | Key_name | Seq_in_index | Column_name | Collation | Cardinality | Sub_part | Packed | Null | Index_type | Comment | Index_comment |
+-----------------------+------------+-------------------------+--------------+-----------------+-----------+-------------+----------+--------+------+------------+---------+---------------+
| statistics_work_table | 0 | PRIMARY | 1 | unique_id | A | 11362113 | NULL | NULL | | BTREE | | |
| statistics_work_table | 1 | GameSourceId_CostTime | 1 | game_sourceid | A | 18 | NULL | NULL | | BTREE | | |
| statistics_work_table | 1 | GameSourceId_CostTime | 2 | cost_time | A | 344306 | NULL | NULL | | BTREE | | |
+-----------------------+------------+-------------------------+--------------+-----------------+-----------+-------------+----------+--------+------+------------+---------+---------------+
ALTER TABLE `statistics_work_table`
ADD INDEX `GameSourceId_CostTime` (`game_sourceid`,`cost_time`)
This index should make your queries super fast. Also, after you run the above statement, you should drop the single column index you have on game_sourceid, as the above will make the single column one redundant. (Which will hurt insert speed.)
The reason your queries are slow is because the database is using your single column index on game_sourceid, finding the rows, and then, for each row, using the primary key that is stored along with the index to find the main clustered index (aka primary key in this, and most cases), and then looking up the cost_time value. This is referred to as a double lookup, and it is something you want to avoid.
The index I provided above is called a "covering index". It allows your query to use ONLY the index, and so you only need a single lookup per row, greatly improving performance.
The table is in InnoDB table. Here is some information that might be helpful.
EXPLAIN SELECT COUNT(*) AS y0_ FROM db.table this_ WHERE this_.id IS NOT NULL;
+----+-------------+-------+-------+---------------+---------+---------+------+---------+--------------------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+-------+-------+---------------+---------+---------+------+---------+--------------------------+
| 1 | SIMPLE | this_ | index | PRIMARY | PRIMARY | 8 | NULL | 4711235 | Using where; Using index |
+----+-------------+-------+-------+---------------+---------+---------+------+---------+--------------------------+
1 row in set (0.00 sec)
mysql> DESCRIBE db.table;
+--------------+--------------+------+-----+---------+-------+
| Field | Type | Null | Key | Default | Extra |
+--------------+--------------+------+-----+---------+-------+
| id | bigint(20) | NO | PRI | NULL | |
| id2 | varchar(28) | YES | | NULL | |
| photo | longblob | YES | | NULL | |
| source | varchar(10) | YES | | NULL | |
| file_name | varchar(120) | YES | | NULL | |
| file_type | char(1) | YES | | NULL | |
| created_date | datetime | YES | | NULL | |
| updated_date | datetime | YES | | NULL | |
| createdby | varchar(50) | YES | | NULL | |
| updatedby | varchar(50) | YES | | NULL | |
+--------------+--------------+------+-----+---------+-------+
10 rows in set (0.05 sec)
The explain query gives me the result right there. But the actual query has been running for quite a while. How can I fix this? What am I doing wrong?
I basically need to figure out how many photos there are in this table. Initially the original coder had a query which checked WHERE photo IS NOT NULL (which took 3hours+) but I changed this query to check the id column as it is a primary key. I expected a huge performance gain there and was expecting an answer in under a second but that seems to not be the case.
What sort of optimizations on the database do I need to do? I think the query is fine but feel free to correct me if I am wrong.
Edit: mysql Ver 14.14 Distrib 5.1.52, for redhat-linux-gnu (x86_64) using readline 5.1
P.S: I renamed the tables for some crazy reason. I don't actually have the database named db and the table in question named table.
How long is 'long'? How many rows are there in this table?
A MyISAM table keeps track of how many rows it has, so a simple COUNT(*) will always return almost instantly.
InnoDB, on the other hand works differently: an InnoDB table doesn't keep track of how many rows it has, and so when you COUNT(*), it literally has to go and count each row. If you have a large table, this can take a number of seconds.
EDIT: Try COUNT(ID) instead of COUNT(*), where ID is an indexed column that has no NULLs in it. That may run faster.
EDIT2: If you're storing the binary data of the files in the longblob, your table will be massive, which will slow things down.
Possible solutions:
Use MyISAM instead of InnoDB.
Maintain your own count, perhaps using triggers on inserts and deletes.
Strip out the binary data into another table, or preferably regular files.
I have a MySQL 5.0 database with a few tables containing over 50M rows. But how do I know this? By running "SELECT COUNT(1) FROM foo", of course. This query on one table containing 58.8M rows took 10 minutes to complete!
mysql> SELECT COUNT(1) FROM large_table;
+----------+
| count(1) |
+----------+
| 58778494 |
+----------+
1 row in set (10 min 23.88 sec)
mysql> EXPLAIN SELECT COUNT(1) FROM large_table;
+----+-------------+-------------------+-------+---------------+----------------------------------------+---------+------+-----------+-------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+-------------------+-------+---------------+----------------------------------------+---------+------+-----------+-------------+
| 1 | SIMPLE | large_table | index | NULL | fk_large_table_other_table_id | 5 | NULL | 167567567 | Using index |
+----+-------------+-------------------+-------+---------------+----------------------------------------+---------+------+-----------+-------------+
1 row in set (0.00 sec)
mysql> DESC large_table;
+-------------------+---------------------+------+-----+---------+----------------+
| Field | Type | Null | Key | Default | Extra |
+-------------------+---------------------+------+-----+---------+----------------+
| id | bigint(20) unsigned | NO | PRI | NULL | auto_increment |
| created_on | datetime | YES | | NULL | |
| updated_on | datetime | YES | | NULL | |
| other_table_id | int(11) | YES | MUL | NULL | |
| parent_id | bigint(20) unsigned | YES | MUL | NULL | |
| name | varchar(255) | YES | | NULL | |
| property_type | varchar(64) | YES | | NULL | |
+-------------------+---------------------+------+-----+---------+----------------+
7 rows in set (0.00 sec)
All of the tables in question are InnoDB.
Any ideas why this is so slow, and how I can speed it up?
Counting all the rows in a table is a very slow operation; you can't really speed it up, unless you are prepared to keep a count somewhere else (and of course, that can become out of sync).
People who are used to MyISAM tend to think that they get count(*) "for free", but it's not really. MyISAM cheats by not having MVCC, which makes it fairly easy.
The query you're showing is doing a full index scan of a not-null index, which is generally the fastest way of counting the rows in an innodb table.
It is difficult to guess from the information you've given, what your application is, but in general, it's ok for users (etc) to see close approximations of the number of rows in large tables.
If you need to have the result instantly and you don't care if it's 58.8M or 51.7M, you can find out the approximate number of rows by calling
show table status like 'large_table';
See the column rows
For more information about the result take a look at the manual at http://dev.mysql.com/doc/refman/5.1/en/show-table-status.html
select count(id) from large_table will surely run faster
I have a table I want to bea ble to do fulltext searches on, but I can't seem to get any hits on my search terms.
Here's what my table looks like (I took out a few columns that I am confident are not important):
mysql> SHOW TABLE STATUS LIKE 'transcripts';
+-------------+--------+---------+------------+-------------------+----------+----------------+
| Name | Engine | Version | Row_format | Collation | Checksum | Create_options |
+-------------+--------+---------+------------+-------------------+----------+----------------+
| transcripts | MyISAM | 10 | Dynamic | latin1_swedish_ci | NULL | |
+-------------+--------+---------+------------+-------------------+----------+----------------+
mysql> DESCRIBE transcripts;
+-------------+----------+------+-----+---------+----------------+
| Field | Type | Null | Key | Default | Extra |
+-------------+----------+------+-----+---------+----------------+
| id | int(11) | NO | PRI | NULL | auto_increment |
| content | text | YES | | NULL | |
| raw_content | text | YES | MUL | NULL | |
| tape_id | int(11) | YES | | NULL | |
| state | int(11) | YES | | 0 | |
| created_at | datetime | YES | | NULL | |
| updated_at | datetime | YES | | NULL | |
+-------------+----------+------+-----+---------+----------------+
mysql> SHOW INDEXES FROM transcripts;
+-------------+------------+-------------------+--------------+-------------+-----------+-------------+----------+--------+------+------------+
| Table | Non_unique | Key_name | Seq_in_index | Column_name | Collation | Cardinality | Sub_part | Packed | Null | Index_type |
+-------------+------------+-------------------+--------------+-------------+-----------+-------------+----------+--------+------+------------+
| transcripts | 0 | PRIMARY | 1 | id | A | 2 | NULL | NULL | | BTREE |
| transcripts | 1 | raw_content_index | 1 | raw_content | NULL | NULL | NULL | NULL | YES | FULLTEXT |
+-------------+------------+-------------------+--------------+-------------+-----------+-------------+----------+--------+------+------------+
Proof that there is a row to find:
mysql> SELECT id, raw_content FROM transcripts;
+----+-------------+
| id | raw_content |
+----+-------------+
| 1 | foo |
+----+-------------+
And here are my flailing attempts to get the search to work:
mysql> SELECT * FROM transcripts WHERE MATCH(raw_content) AGAINST ('foo');
Empty set (0.00 sec)
mysql> SELECT * FROM transcripts WHERE MATCH(raw_content) AGAINST ('foo' IN BOOLEAN MODE);
Empty set (0.00 sec)
mysql> SELECT id FROM transcripts WHERE MATCH(raw_content) AGAINST ('foo' IN NATURAL LANGUAGE MODE);
Empty set (0.00 sec)
mysql> SELECT * FROM transcripts WHERE MATCH(raw_content) AGAINST ('+foo*' IN BOOLEAN MODE);
Empty set (0.00 sec)
What am I doing wrong?
by default fulltext ignores words with 3 or less characters
(now posted as an answer by request :) )
Here is something else to consider:
You can not only change the default length, but you can also change the stopword list. You may want to change the stop words because MySQL will not index this list of 543 words.
Try creating your own stopword list and changing the min word length.
Step 1) Create a stop word list of your own. You could add 'a','an', and 'the'.
echo "a" > /var/lib/mysql/custom_stopwords.txt
echo "an" >> /var/lib/mysql/custom_stopwords.txt
echo "the" >> /var/lib/mysql/custom_stopwords.txt
Step 2) Add these options to /etc/my.cnf
ft_min_word_len=2
ft_stopword_file=/var/lib/mysql/custom_stopwords.txt
Step 3) service mysql restart
Step 4) Create new FULLTEXT indexes. Any existing FULLTEXT indexes before restart of mysql should be reindexed.
Give it a Try !!!