ALTER TABLE increase INDEX LENGTH - mysql

I have a mysql table named t_media_items. I have index on 3 cols (parent_id, type, weight). The index size is 2.52MB.
mysql> show indexes from t_media_items;
+---------------+------------+--------------+--------------+-----------------+-----------+-------------+----------+--------+------+------------+---------+---------------+
| Table | Non_unique | Key_name | Seq_in_index | Column_name | Collation | Cardinality | Sub_part | Packed | Null | Index_type | Comment | Index_comment |
+---------------+------------+--------------+--------------+-----------------+-----------+-------------+----------+--------+------+------------+---------+---------------+
| t_media_items | 0 | PRIMARY | 1 | id | A | 113779 | NULL | NULL | | BTREE | | |
| t_media_items | 1 | idx_ptw | 1 | parent_id | A | 16254 | NULL | NULL | | BTREE | | |
| t_media_items | 1 | idx_ptw | 2 | type | A | 16254 | NULL | NULL | | BTREE | | |
| t_media_items | 1 | idx_ptw | 3 | weight | A | 113779 | NULL | NULL | | BTREE | | |
+---------------+------------+--------------+--------------+-----------------+-----------+-------------+----------+--------+------+------------+---------+---------------+
4 rows in set (0.01 sec)
mysql> SELECT table_name AS "Tables", round(((index_length) / 1024 / 1024), 2) SIB
FROM information_schema.TABLES
WHERE table_schema = "XXXX" and table_name='t_media_items'
ORDER BY (index_length ) DESC;
+---------------+------+
| Tables | SIB |
+---------------+------+
| t_media_items | 2.52 |
+---------------+------+
1 row in set (0.00 sec)
I tried to alter length on another column, name "rand_key". The strange issue is after columned alter, the INDEX size suddenly increase to 5.52MB, even the "rand_key" is not part of the index.
mysql> ALTER TABLE `t_media_items` CHANGE `rand_key` `rand_key` VARCHAR( 255 ) CHARACTER SET utf8 COLLATE utf8_general_ci NULL DEFAULT NULL;
Query OK, 108503 rows affected (7.24 sec)
Records: 108503 Duplicates: 0 Warnings: 0
Here is INDEX_LENGTH after ALTER
mysql> SELECT table_name AS "Tables", round(((index_length) / 1024 / 1024), 2) SIB
FROM information_schema.TABLES
WHERE table_schema = "tallcat" and table_name='t_media_items'
ORDER BY (index_length ) DESC;
+---------------+------+
| Tables | SIB |
+---------------+------+
| t_media_items | 5.52 |
+---------------+------+
1 row in set (0.00 sec)
Could anyone help me explain the issue? Thank you

ALTER TABLE performs a table restructure, which is make an empty clone of the table, apply the alter, and copy data to it. Indexes are rebuild as a by-product of this process.
But as it builds the index incrementally while filling the table with data, it doesn't take advantage of fast index creation. The indexes are stored less compactly.
I can't tell if this explains the more than 2x increase in size, but it's possible.
I'm assuming that you're using MySQL 5.5 or later, or 5.1 with the InnoDB plugin. Earlier versions do not support fast index creation anyway.

Related

MySQL - i dont see data, but mysql says, that there are rows

I have table ip_per_nat_vlan, innodb format. When I give truncate table, table is empty.
Then I have php script, which fill data into this table.
When is this script finished without errors (simple insert statemets) situation is following:
select * from ip_per_nat_vlan;
Empty set (0.00 sec)
.
select count(*) from ip_per_nat_vlan;
+----------+
| count(*) |
+----------+
| 0 |
+----------+
1 row in set (0.00 sec)
.
show table status;
+----------------------------------+--------+---------+------------+----------+----------------+-------------+-----------------+--------------+------------+----------------+---------------------+---------------------+------------+-----------------+----------+----------------+---------+
| Name | Engine | Version | Row_format | Rows | Avg_row_length | Data_length | Max_data_length | Index_length | Data_free | Auto_increment | Create_time | Update_time | Check_time | Collation | Checksum | Create_options | Comment |
+----------------------------------+--------+---------+------------+----------+----------------+-------------+-----------------+--------------+------------+----------------+---------------------+---------------------+------------+-----------------+----------+----------------+---------+
| ip_per_nat_vlan | InnoDB | 10 | Dynamic | 141291 | 100 | 14172160 | 0 | 6832128 | 25165824 | 143563 | 2017-12-24 16:26:40 | 2018-06-13 09:01:33 | NULL | utf8_unicode_ci | NULL |
MySQL says, that there should be 14172160 rows, but I dont see any. Where could be a problem? Transactions? But I dont see any running thread and no any fault.
Thank you. D
Structure of table is:
+------------+--------------+------+-----+---------+----------------+
| Field | Type | Null | Key | Default | Extra |
+------------+--------------+------+-----+---------+----------------+
| id | int(11) | NO | PRI | NULL | auto_increment |
| ipAddress | varchar(255) | NO | UNI | NULL | |
| nat | int(11) | NO | | NULL | |
| vlan | int(11) | NO | | NULL | |
| district | varchar(255) | YES | | NULL | |
| idOblasti | int(11) | YES | | NULL | |
| type | varchar(255) | NO | | NULL | |
| macAddress | varchar(255) | NO | | NULL | |
+------------+--------------+------+-----+---------+----------------+
There are various ways to "count" rows in a table.
The normal way. Just count them.
select count(*) as table_rows from table_name ;
Accuracy: 100% accurate count at the time of the query is run.
using the information_schema tables
select table_rows
from information_schema.tables
where table_schema = 'database_name'
and table_name = 'table_name' ;
Accuracy: Only an approximation. If the table is the target of frequent inserts and deletes, the result can be way off the actual count. This can be improved by running ANALYZE TABLE more often.
Efficiency: Very good, it doesn't touch the table at all.
As count option is 100% accurate, your table doesn't contain any data.
Check your code and default commit option of MySQL.
Looks like you are inserting rows, but not committing them, check your index length.
Check more details here
https://dba.stackexchange.com/questions/151769/mysql-difference-between-using-count-and-information-schema-tables-for-coun
First thing, I am not sure how mysql run this line and produce the result
select count() from ip_per_nat_vlan
count() will return [Err] 1064.
count(*) or else a field name should be mentioned inside.

MySQL `INSERT INTO SELECT` clause generate Duplicate entry error on unique field

I use INSERT INTO SELECT to migrate users' data across databases, but it generates
Duplicate entry '  ' for key 'users_name_unique'
although the data source is another unique index and should not contain any duplicate data.('users_name_unique' is the index name on db2.users)
Here is the query, where name field from db2.users is varchar(50) unique and not null index, and name field from db1.users is varchar(60) unique and not null index. I have already checked the length of the field in every record, and the lengths are all much smaller than 50.
INSERT INTO db2.users (name, email, uid) SELECT
name,
IF (mail = '', NULL, mail) AS email,
uid
FROM
db1.users;
There are non-printable or white spaces in the name field from db1.users.
What might be the problem?
update
I created multi test tables, and as following, there are two tables with very similar structure and no data (I changed the length on purpose since the source data is varchar(60)), but different results.
mysql> desc ttt3;
+-------+-------------+------+-----+---------+----------------+
| Field | Type | Null | Key | Default | Extra |
+-------+-------------+------+-----+---------+----------------+
| id | int(11) | NO | PRI | NULL | auto_increment |
| name | varchar(50) | NO | UNI | NULL | |
+-------+-------------+------+-----+---------+----------------+
2 rows in set (0.01 sec)
mysql> desc users;
+-------+------------------+------+-----+---------+----------------+
| Field | Type | Null | Key | Default | Extra |
+-------+------------------+------+-----+---------+----------------+
| id | int(10) unsigned | NO | PRI | NULL | auto_increment |
| name | varchar(60) | NO | UNI | NULL | |
+-------+------------------+------+-----+---------+----------------+
2 rows in set (0.01 sec)
mysql> show index from ttt3;
+-------+------------+----------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+---------------+
| Table | Non_unique | Key_name | Seq_in_index | Column_name | Collation | Cardinality | Sub_part | Packed | Null | Index_type | Comment | Index_comment |
+-------+------------+----------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+---------------+
| ttt3 | 0 | PRIMARY | 1 | id | A | 0 | NULL | NULL | | BTREE | | |
| ttt3 | 0 | name | 1 | name | A | 0 | NULL | NULL | | BTREE | | |
+-------+------------+----------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+---------------+
2 rows in set (0.00 sec)
mysql> show index from users;
+-------+------------+-------------------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+---------------+
| Table | Non_unique | Key_name | Seq_in_index | Column_name | Collation | Cardinality | Sub_part | Packed | Null | Index_type | Comment | Index_comment |
+-------+------------+-------------------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+---------------+
| users | 0 | PRIMARY | 1 | id | A | 0 | NULL | NULL | | BTREE | | |
| users | 0 | users_name_unique | 1 | name | A | 0 | NULL | NULL | | BTREE | | |
+-------+------------+-------------------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+---------------+
2 rows in set (0.00 sec)
mysql> insert into ttt3(name) select name from scratch.users where scratch.users.uid != 0;
Query OK, 1556 rows affected (0.24 sec)
Records: 1556 Duplicates: 0 Warnings: 0
mysql> insert into users(name) select name from scratch.users where scratch.users.uid != 0;
ERROR 1062 (23000): Duplicate entry '  ' for key 'users_name_unique'
update
It turns out that the destination field's collation is set to 'utf8_unicode_ci' and the original field is 'utf8_general_ci', changing this option solve the problem.
Here is the reason:
The destination field's collation is set to 'utf8_unicode_ci' (laravel's default collation) and the original field is 'utf8_general_ci'.
These collations have different rules of "sort" or "equal". Changing this option solved the problem.

Simple mySQL query does not use a index

I have very strange problem with mySQL and simple query with simple index. I am trying to use this query:
SELECT * FROM `counter_links` WHERE `link_id`=1544;
as you can see there is also index named link_id:
mysql> show indexes from counter_links;
+---------------+------------+-------------+--------------+-------------+-----------+-------------+----------+--------+------+------------+----------+---------------+
| Table | Non_unique | Key_name | Seq_in_index | Column_name | Collation | Cardinality | Sub_part | Packed | Null | Index_type | Comment | Index_comment |
+---------------+------------+-------------+--------------+-------------+-----------+-------------+----------+--------+------+------------+----------+---------------+
| counter_links | 1 | link_id | 1 | link_id | A | NULL | NULL | NULL | | BTREE | disabled | |
+---------------+------------+-------------+--------------+-------------+-----------+-------------+----------+--------+------+------------+----------+---------------+
1 row in set (0.12 sec)
But look what EXPLAIN returned:
mysql> explain SELECT * FROM `counter_links` WHERE `link_id`=1544;
+----+-------------+---------------+------+---------------+------+---------+------+----------+-------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+---------------+------+---------------+------+---------+------+----------+-------------+
| 1 | SIMPLE | counter_links | ALL | NULL | NULL | NULL | NULL | 22103687 | Using where |
+----+-------------+---------------+------+---------------+------+---------+------+----------+-------------+
1 row in set (0.03 sec)
Why mySQL don't use index there? I see that the same is working good on smaller version of the table, but I am not able to recognize where the problem is. You do?
It is more stranger that it seems, because lower IDs sometimes works with the index.
Thanks you for all comments in advance!
Cheers!
Jakub
The comment field for th eindex says that the index is disabled, not sure why.
ALTER TABLE ... ENABLE KEYS may work

Why Are These Tables the Same Size?

I was trying to measure the difference between TINYINT and INT when I came across something interesting. For tables with small numbers of columns, the choice of data type does not seem to affect the size of the table.
Server version: 5.1.41-3ubuntu12.10 (Ubuntu)
Example:
mysql> describe tinyint_test;
+----------+------------+------+-----+---------+-------+
| Field | Type | Null | Key | Default | Extra |
+----------+------------+------+-----+---------+-------+
| id | int(11) | YES | | NULL | |
| test_int | tinyint(4) | YES | | NULL | |
+----------+------------+------+-----+---------+-------+
2 rows in set (0.00 sec)
mysql> describe tinyint_id_test;
+-------+------------+------+-----+---------+-------+
| Field | Type | Null | Key | Default | Extra |
+-------+------------+------+-----+---------+-------+
| id | tinyint(4) | YES | | NULL | |
+-------+------------+------+-----+---------+-------+
1 row in set (0.00 sec)
mysql> describe int_test;
+--------+---------+------+-----+---------+-------+
| Field | Type | Null | Key | Default | Extra |
+--------+---------+------+-----+---------+-------+
| not_id | int(11) | YES | | NULL | |
+--------+---------+------+-----+---------+-------+
1 row in set (0.00 sec)
mysql> select * from tinyint_test;
+------+----------+
| id | test_int |
+------+----------+
| 1 | 1 |
| 2 | 2 |
| 3 | 127 |
| 10 | 50 |
+------+----------+
4 rows in set (0.00 sec)
mysql> select * from tinyint_id_test;
+------+
| id |
+------+
| 1 |
| 2 |
| 127 |
| 50 |
+------+
4 rows in set (0.00 sec)
mysql> select * from int_test;
+--------+
| not_id |
+--------+
| 1 |
| 2 |
| 127 |
| 50 |
+--------+
4 rows in set (0.00 sec)
mysql> SELECT TABLE_NAME, DATA_LENGTH FROM INFORMATION_SCHEMA.TABLES where TABLE_SCHEMA like '%test%';
+-----------------+-------------+
| TABLE_NAME | DATA_LENGTH |
+-----------------+-------------+
| int_test | 28 |
| tinyint_id_test | 28 |
| tinyint_test | 28 |
+-----------------+-------------+
3 rows in set (0.00 sec)
I vaguely suspect that there might be an internal column in each row, or that the minimum data size for a given row must be at least the size of a full INT, but neither of these suspicions really account for what's happening here. What could be the case is my choice of DATA_LENGTH is the incorrect tool for measuring the true size of the tables, in which case an acceptable answer would point me in the right direction for actually measuring these tables.
EDIT:
I can generate a table of a different size by using two INTs:
mysql> describe int_id_test;
+----------+---------+------+-----+---------+-------+
| Field | Type | Null | Key | Default | Extra |
+----------+---------+------+-----+---------+-------+
| id | int(11) | YES | | NULL | |
| test_int | int(11) | YES | | NULL | |
+----------+---------+------+-----+---------+-------+
2 rows in set (0.01 sec)
mysql> select * from int_id_test;
+------+----------+
| id | test_int |
+------+----------+
| 1 | 1 |
| 2 | 2 |
| 3 | 127 |
| 10 | 50 |
+------+----------+
4 rows in set (0.00 sec)
mysql> SELECT TABLE_NAME, DATA_LENGTH FROM INFORMATION_SCHEMA.TABLES where TABLE_SCHEMA like '%test%';
+-----------------+-------------+
| TABLE_NAME | DATA_LENGTH |
+-----------------+-------------+
| int_id_test | 36 |
| int_test | 28 |
| tinyint_id_test | 28 |
| tinyint_test | 28 |
+-----------------+-------------+
4 rows in set (0.01 sec)
the data_length column is how much hard drive space the operating system allocates
for a table.
mysql database page sizes configurable default is 16KB, the three table's data may used same pages, so the data_length are same!!
edit:
innodb engine default page size is 16KB, i don't know this size for other engines
I have found a work around for this problem as well as something of an explanation.
After looking at the table structure in a hex editor (on my linux machines these were located in /var/lib/mysql/[DATABASE NAME]/[TABLE NAME].MYD), I found that in all cases the records were created using a minimum of 7 bytes for a row, regardless of the actual data types involved. Any extra bytes that were not used by the table were zeroed out.
Here is an example with a smaller data set to illustrate:
mysql> describe int_test_2;
+-------+---------+------+-----+---------+-------+
| Field | Type | Null | Key | Default | Extra |
+-------+---------+------+-----+---------+-------+
| id | int(11) | YES | | NULL | |
+-------+---------+------+-----+---------+-------+
1 row in set (0.00 sec)
mysql> select * from int_test_2;
+------+
| id |
+------+
| 1 |
| 2 |
+------+
2 rows in set (0.00 sec)
Looking at this guy in a hex editor, we see:
fd01 0000 0000 00fd 0200 0000 0000
Using information from Neo's link, I was able to decode this row:
fd Record header bits.
01000000 Integer value "1" (little endian)
0000 Wasted Space!
fd Record header bits.
02000000 Integer value "2" (little endian)
0000 Wasted Space!
However, notice the following:
mysql> alter table int_test_2 MAX_ROWS=50000000, AVG_ROW_LENGTH=4;
Query OK, 2 rows affected (0.01 sec)
Records: 2 Duplicates: 0 Warnings: 0
Now, the MYD file looks like this:
fd01 0000 00fd 0200 0000
That is, it uses the correct sizes.
One thing to note is that the number in brackets does not effect the size of that column, i.e an INT(4) is the same size as an INT(11) in terms of storage, all the number in brackets does is pad the returned value with spaces so that it fills 11 or 4 characters.
I suspect if you trully want to work out the size of the tables, you will need to look in the MySQL file itself and see how they are stored. All the data is stored in /var/lib/mysql/ - ibdata & ib_logfile are the main files. Open this in a text editor (Caution - this file may be HUGE depending on the sizes of your databases.. also DO NOT modify this file!!)
All the tables and cells are stored in here, however they are not delimeted, so its very difficult to see where one column ends and the next begins - it is all based on the data size which you are trying to establish. If you know the data in the table you should be able to work out the structure.
Edit: I think some of the data in these files may be stored in hex, so if it doesnt immediately make sense, try a hex editor.

Can't get MySQL fulltext search to work

I have a table I want to bea ble to do fulltext searches on, but I can't seem to get any hits on my search terms.
Here's what my table looks like (I took out a few columns that I am confident are not important):
mysql> SHOW TABLE STATUS LIKE 'transcripts';
+-------------+--------+---------+------------+-------------------+----------+----------------+
| Name | Engine | Version | Row_format | Collation | Checksum | Create_options |
+-------------+--------+---------+------------+-------------------+----------+----------------+
| transcripts | MyISAM | 10 | Dynamic | latin1_swedish_ci | NULL | |
+-------------+--------+---------+------------+-------------------+----------+----------------+
mysql> DESCRIBE transcripts;
+-------------+----------+------+-----+---------+----------------+
| Field | Type | Null | Key | Default | Extra |
+-------------+----------+------+-----+---------+----------------+
| id | int(11) | NO | PRI | NULL | auto_increment |
| content | text | YES | | NULL | |
| raw_content | text | YES | MUL | NULL | |
| tape_id | int(11) | YES | | NULL | |
| state | int(11) | YES | | 0 | |
| created_at | datetime | YES | | NULL | |
| updated_at | datetime | YES | | NULL | |
+-------------+----------+------+-----+---------+----------------+
mysql> SHOW INDEXES FROM transcripts;
+-------------+------------+-------------------+--------------+-------------+-----------+-------------+----------+--------+------+------------+
| Table | Non_unique | Key_name | Seq_in_index | Column_name | Collation | Cardinality | Sub_part | Packed | Null | Index_type |
+-------------+------------+-------------------+--------------+-------------+-----------+-------------+----------+--------+------+------------+
| transcripts | 0 | PRIMARY | 1 | id | A | 2 | NULL | NULL | | BTREE |
| transcripts | 1 | raw_content_index | 1 | raw_content | NULL | NULL | NULL | NULL | YES | FULLTEXT |
+-------------+------------+-------------------+--------------+-------------+-----------+-------------+----------+--------+------+------------+
Proof that there is a row to find:
mysql> SELECT id, raw_content FROM transcripts;
+----+-------------+
| id | raw_content |
+----+-------------+
| 1 | foo |
+----+-------------+
And here are my flailing attempts to get the search to work:
mysql> SELECT * FROM transcripts WHERE MATCH(raw_content) AGAINST ('foo');
Empty set (0.00 sec)
mysql> SELECT * FROM transcripts WHERE MATCH(raw_content) AGAINST ('foo' IN BOOLEAN MODE);
Empty set (0.00 sec)
mysql> SELECT id FROM transcripts WHERE MATCH(raw_content) AGAINST ('foo' IN NATURAL LANGUAGE MODE);
Empty set (0.00 sec)
mysql> SELECT * FROM transcripts WHERE MATCH(raw_content) AGAINST ('+foo*' IN BOOLEAN MODE);
Empty set (0.00 sec)
What am I doing wrong?
by default fulltext ignores words with 3 or less characters
(now posted as an answer by request :) )
Here is something else to consider:
You can not only change the default length, but you can also change the stopword list. You may want to change the stop words because MySQL will not index this list of 543 words.
Try creating your own stopword list and changing the min word length.
Step 1) Create a stop word list of your own. You could add 'a','an', and 'the'.
echo "a" > /var/lib/mysql/custom_stopwords.txt
echo "an" >> /var/lib/mysql/custom_stopwords.txt
echo "the" >> /var/lib/mysql/custom_stopwords.txt
Step 2) Add these options to /etc/my.cnf
ft_min_word_len=2
ft_stopword_file=/var/lib/mysql/custom_stopwords.txt
Step 3) service mysql restart
Step 4) Create new FULLTEXT indexes. Any existing FULLTEXT indexes before restart of mysql should be reindexed.
Give it a Try !!!