I'm trying to create a partitioned table on another drive with DATA DIRECTORY:
mysql> CREATE TABLE t1 (c1 INT PRIMARY KEY) DATA DIRECTORY = '/stats_tables/test/db/mysql/';
Query OK, 0 rows affected (0.03 sec)
mysql> \! ls /stats_tables/test/db/mysql/db_test/
t1.ibd
mysql> drop table t1;
Query OK, 0 rows affected (0.01 sec)
So we see that an unpartitioned table is created successfully on the drive. Now let's try the same thing with a partitioned table:
mysql> CREATE TABLE t1 (c1 INT PRIMARY KEY) DATA DIRECTORY = '/stats_tables/test/db/mysql/' PARTITION BY HASH ( c1 ) PARTITIONS 10;
Query OK, 0 rows affected (0.34 sec)
mysql> \! ls /stats_tables/test/db/mysql/db_test/
mysql> \! ls /defa_storage/test/db/mysql/db_test/t1* | xargs -n1 basename
t1.frm
t1.par
t1#P#p0.ibd
t1#P#p1.ibd
t1#P#p2.ibd
t1#P#p3.ibd
t1#P#p4.ibd
t1#P#p5.ibd
t1#P#p6.ibd
t1#P#p7.ibd
t1#P#p8.ibd
t1#P#p9.ibd
We see that in this case it's created under the regular data directory.
This has perfect explanation in the MySQL 5.5 documentation:
The DATA DIRECTORY and INDEX DIRECTORY options have no effect when defining partitions for tables using the InnoDB storage engine.
Although I use version 5.6 and there's no such a restriction mentioned in its documentation.
Anyway, is there a way to set a common DATA DIRECTORY for all my BY HASH partitions?
In the end I've found out I can actually control the created partitions in this manner:
CREATE TABLE ...
PARTITION BY HASH( ... )
(
PARTITION p0 DATA DIRECTORY = '/another/directory',
PARTITION p1 DATA DIRECTORY = '/another/directory',
PARTITION p2 DATA DIRECTORY = '/another/directory',
PARTITION p3 DATA DIRECTORY = '/another/directory',
...
);
The last part can be easily generated by any scripting language, say, Python:
max_partitions = 100
for i in range(0, max_partitions):
print("PARTITION p" + str(i) + " DATA DIRECTORY = '/another/directory',")
Related
I am new in MySQL and I do not understand, why if I use index on JSON column, result set is different as without index.
I have a simple table:
CREATE TABLE jsontest (
jsondata JSON
);
Table is filled up with 50000 JSONs and one filed in json is also:
"allowedNfTypes": ["aaa", "bbb", "ccc"]}
This field in some cases have 1 or 2 or 3 values in array (but there are about 10 string options - let's say from "aaa" to "iii").
In some cases this filed does not exist at all.
Then if I execute:
mysql> SELECT * FROM jsontest WHERE "AMF" MEMBER OF(jsondata->'$.allowedNfTypes')
I get:
10045 rows in set (0.21 sec)
Then I created an index:
CREATE INDEX allowedNfTypes_index ON jsontest((CAST(jsondata->'$.allowedNfTypes' AS CHAR(128) ARRAY)))
And the same query
SELECT * FROM jsontest WHERE "AMF" MEMBER OF(jsondata->'$.allowedNfTypes');
return much less hits:
1402 rows in set (0.03 sec)
Any idea why?
I have downloaded your data file.
CLI output copy:
Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 16
Server version: 8.0.23 MySQL Community Server - GPL
Copyright (c) 2000, 2021, Oracle and/or its affiliates.
Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
mysql> CREATE TABLE jsontest (
-> jsondata JSON
-> );
Query OK, 0 rows affected (0.07 sec)
mysql> LOAD DATA INFILE 'C:\\ProgramData\\MySQL\\MySQL Server 8.0\\Uploads\\test_data.json' INTO TABLE jsontest;
Query OK, 50000 rows affected (15.50 sec)
Records: 50000 Deleted: 0 Skipped: 0 Warnings: 0
mysql> SELECT COUNT(*) FROM jsontest WHERE "AMF" MEMBER OF(jsondata->'$.allowedNfTypes');
+----------+
| COUNT(*) |
+----------+
| 10045 |
+----------+
1 row in set (0.32 sec)
mysql> CREATE INDEX allowedNfTypes_index ON jsontest((CAST(jsondata->'$.allowedNfTypes' AS CHAR(128) ARRAY)));
Query OK, 0 rows affected (0.88 sec)
Records: 0 Duplicates: 0 Warnings: 0
mysql> SELECT COUNT(*) FROM jsontest WHERE "AMF" MEMBER OF(jsondata->'$.allowedNfTypes');
+----------+
| COUNT(*) |
+----------+
| 10045 |
+----------+
1 row in set (0.28 sec)
Also original query (not SELECT COUNT(*) FROM .. but SELECT * FROM ..) was tested - the result is the same.
The issue was not reproduced.
After version updating till actual 8.0.28 the issue is reproduced. The amount of rows selected with the index is the same - 1402.
After adding autoincremented primary key ALTER TABLE jsontest ADD COLUMN id INT AUTO_INCREMENT PRIMARY KEY; the amount of rows selected with index alters to 1448.
Comparing the rows selected with and without index I have found that:
There is a lot of duplicates in jsondata->'$.allowedNfTypes'.
For each unique jsondata->'$.allowedNfTypes' value only the rows which have created unique id less than approximately 6800 are selected with the index by jsondata->'$.allowedNfTypes', all another rows with duplicated value are not selected.
Deleting the rows which have id value above some N decreases the amount of rows returned by the query without the index and does not alter the amount of rows returned by the query with index (tested N = 40000, 25000, 10000, 7500, 7000, 6800). When N is 6797 or less the amount of rows selected become equal.
I have tried to add generated column which extracts this property as JSON array with aLTER TABLE jsontest ADD COLUMN allowedNfTypes JSON AS (CAST(jsondata->'$.allowedNfTypes' AS JSON));, create multivalued index by this column and use this column insrtead of the whole JSON. The amount of rows selected with index increases till 1498. The value of N is 5097.
Then I add the index described by OP (the table contains 2 indices - by original column and by generated one) - and the query returns 10045! I drops the generated column and the index by it - 10045. I drop the index by original column and re-create it - again 1498.
There was a lot of additional experiments, but their results are less interesting.
Looks like a bug. Reproduceable bug. I think that you may repeat my expriment (and/or perform your own) and report this to MySQL bugtrack.
I have a weird scenario where a technician deleted the .frm and .idb files for a table from one of my servers, fortunately it was a non critical table. I'm trying to recreate the table however when trying to use the CREATE TABLE, DROP TABLE or ALTER TABLE I get errors cos the tablespace still exists. I'm not really able to move the good tables to a new schema, drop the old schema then recreate the table that way as there are critical tables still in the schema which cannot be taken offline. Any thoughts on how I get rid of this "broken" table?
I created a test table:
mysql> use test;
mysql> create table test ( id serial primary key );
mysql> insert into test () values ();
Check that the files exist in my datadir:
$ cd /usr/local/var/mysql/test
$ ls
db.opt test.frm test.ibd
Do the unthinkable!
$ rm test.*
Now I have a catch-22 situation:
mysql> show tables;
Empty set (0.00 sec)
mysql> create table test (id serial primary key);
ERROR 1050 (42S01): Table 'test' already exists
mysql> drop table test;
ERROR 1051 (42S02): Unknown table 'test.test'
What to do?
mysql> create schema recovery;
mysql> create table recovery.test ( id serial primary key );
mysql> system cp /usr/local/var/mysql/recovery/test.frm /usr/local/var/mysql/test/
The .frm file stores table metadata in MySQL 5.x. Now that I have a .frm file, MySQL lets me query the table.
mysql> select * from test;
+----+
| id |
+----+
| 1 |
+----+
That's funny — how does it know about that row of data? Didn't I rm the .ibd file?
Answer: Files are not truly deleted while a process still has a filehandle open to it. Since I have not restarted the MySQL Server process, the .ibd tablespace file still exists, even though I unlinked it from the datadir!
Anyway, I can now drop the table. This removes the .frm and .ibd AND closes the filehandles to them, so they are truly deleted from the filesystem.
mysql> drop table test;
Query OK, 0 rows affected (0.01 sec)
And finally I can now create a new table of the same name, even with different columns.
mysql> create table test ( bar int primary key );
Query OK, 0 rows affected (0.02 sec)
I am attempting use MySQL to create two tables on separate drives so I can do some bench marking. I found this documentation on the Data Directory command which seems to do exactly what I need, but it is not working for me. Here is what I get:
mysql> create database test;
Query OK, 1 row affected (0.00 sec)
mysql> use test;
Database changed
mysql> show variables like 'innodb_file_per_table';
+-----------------------+-------+
| Variable_name | Value |
+-----------------------+-------+
| innodb_file_per_table | ON |
+-----------------------+-------+
1 row in set (0.00 sec)
mysql> create table t1 (id int primary key, name text) data directory = '/media/ssd';
Query OK, 0 rows affected (0.03 sec)
However the table file is made in the default /var/lib/mysql/test and /media/ssd remains empty.
$ sudo ls /var/lib/mysql/test
db.opt t1.frm t1.ibd
$ ls -l /media/ssd
total 0
I have also tried other locations like /home and I get the same result so I don't believe it is a drive mounting issue. I also set the /media/ssd permissions to 777.
Any ideas why it is creating the table in the default location?
If need be can I just move the .idb file to my desired location and put a .isl in it's place?
When I try to create a database which already exists,
CREATE DATABASE IF NOT EXISTS test;
Query OK, 1 row affected (0.00 sec)
CREATE DATABASE IF NOT EXISTS test;
Query OK, 1 row affected, 1 warning (0.00 sec)
Why does it show 1 row affected message second time , even though it is not creating a new database with the same name?
Although the CREATE DATABASE IF NOT EXISTS test; command won't directly modify rows in an exiting instance of the test database, it will affect the actual details stored internally in the mysql database, or possibly in one of the derived meta views, like the information_schema or performance_schema etc.
The reported Query OK, 1 row affected (0.00 sec) is referring to a row in one of these internal data constructs. When you reissue the CREATE DATABASE command, and it fails gracefully thanks to the IF NOT EXISTS clause, it is still likely to store meta-data internally, maybe an accumulating field that counts warnings or similar, or even just a 'last acted on' timestamp against this database's row. In any case the stored data in this record is changed, and is reflected as an 'affected' row.
Can someone explain me, what is happening here,
I created one table in MyISAM.
show tables -- shows it.
select command gives empty table from the table.
I delete the myisam files from location.
show tables -- does not show it.
Ok till now.
Now, still I am able to insert into table, and able to select from the table [can it happen?? how??]
Again, drop table says table does not exist.
and after this, select command says table does not exist.
Question is "where did it get inserted, and from where it is selecting the data; and after drop table call, where did it vanish?"
Server version: 5.5.38-log MySQL Community Server (GPL)
The overall sequence ..
MySQL [test]> CREATE TABLE test (id int(11) DEFAULT NULL) ENGINE=MyISAM DEFAULT CHARSET=latin1;
Query OK, 0 rows affected (0.05 sec)
MySQL [test]> show tables;
+----------------+
| Tables_in_test |
+----------------+
| test |
+----------------+
1 row in set (0.00 sec)
MySQL [test]> select * from test;
Empty set (0.00 sec)
~~~~~~~~~~~~~~~~~~~~~~~~~
[root#localhost test]# pwd
/var/lib/mysql/test
[root#localhost test]# ll
total 16
-rw-rw---- 1 mysql mysql 8556 Dec 23 14:08 test.frm
-rw-rw---- 1 mysql mysql 0 Dec 23 14:08 test.MYD
-rw-rw---- 1 mysql mysql 1024 Dec 23 14:08 test.MYI
[root#localhost test]# rm -f *
[root#localhost test]# ll
total 0
[root#localhost test]#
~~~~~~~~~~~~~~~~~~~~~~~~~
MySQL [test]> show tables;
Empty set (0.00 sec)
MySQL [test]> insert into test values (1234);
Query OK, 1 row affected (0.00 sec)
MySQL [test]> select * from test;
+------+
| id |
+------+
| 1234 |
+------+
1 row in set (0.00 sec)
MySQL [test]> drop table test;
ERROR 1051 (42S02): Unknown table 'test'
MySQL [test]> select * from test;
ERROR 1146 (42S02): Table 'test.test' doesn't exist
On Unix, if you remove a file that another process has opened, it just removes the name from the directory, but the file still exists on disk, and the other process can continue to access it. It doesn't really go away until all processes close the file.
The MySQL daemon presumably opens the files related to a table the first time it's accessed, and then keeps the file open for as long as it can. So once it has accessed a table, removing the filename doesn't affect its operations on the contents of the table.
However, DML operations like show tables and drop tables work by accessing the directory, not the open files, so they notice the change. Then all the internal data structures are sync'ed with the filesystem.