I encountered the following problem:
I have temporary table
CREATE temporary TABLE IF NOT EXISTS tmp_GL_VAR
(
G_TABLE_NAME VARCHAR(100) DEFAULT '',
G_DATE DATETIME,
G_ERROR_CODE INT DEFAULT 0
);
And I need to use it several times in single queries,for instance,
update t1 set c1 = (select G_TABLE_NAME from tmp_GL_VAR), c2 = (select G_ERROR_CODE from tmp_GL_VAR);
in functions with cursors, etc. However, in all these cases MySQL throws error:
SQL Error(1137): Can't reopen table 'tmp_GL_VAR'.
Then I tried to create permanent table with STORAGE MEMORY clause (also tried ENGINE MEMORY clause), hoping that table will be cleaned when the session ends
CREATE TABLE GL_VAR
(
G_TABLE_NAME VARCHAR(100) DEFAULT '',
G_DATE DATETIME,
G_ERROR_CODE INT DEFAULT 0
) STORAGE MEMORY;
But unfortunately this option had no effect. The data was available across different sessions (connections).
Please advise how I can bypass 'Can't reopen table' without rewriting all queries, stored functions, etc. (there're far too many LOC).
The requirement is: table should be either dropped or at least truncated as session ends and the data from one session shouldn't be available in another session (each user can see only its own data in this table).
Any help is appreciated.
Yes. there are problems http://dev.mysql.com/doc/refman/5.1/en/temporary-table-problems.html
You cannot refer to a TEMPORARY table more than once in the same query. For example, the following does not work:
mysql> SELECT * FROM temp_table, temp_table AS t2;
ERROR 1137: Can't reopen table: 'temp_table'
This error also occurs if you refer to a temporary table multiple times in a stored function under different aliases, even if the references occur in different statements within the function.
I think u should use usual table for this purpose.
And add some session identifier to make it work in your multi-user system. (Yes, u will have to add this identifier to ALL other queries using this table)
Delete rows by this identifier (or old timestamps if you want) any time u wish
Related
I'm trying to execute an ALTER TABLE in MySQL. MySQL only lets me execute it with the ALGORITHM=COPY (because I need to change the type of a column).
There aren't queries using that table (to write neither to read).
But, I don't know why, when I execute the ALTER there are queries (UPDATES) which are not using this table (they are in a transaction) locked. MySQL says "mysql waiting for metadata lock".
So the question is, why the query is waiting for metadata lock if the UPDATE is not using the table altered?
I read some doc:
https://dev.mysql.com/doc/refman/8.0/en/alter-table.html#alter-table-performance
https://dev.mysql.com/doc/refman/8.0/en/innodb-online-ddl-performance.html#innodb-online-ddl-locking-options
But I don't understand why the queries are locked for metadata.
Reproduction of the problem in dev environment:
First, do the alter:
ALTER TABLE API.SEARCHES_ELEMENTS
MODIFY COLUMN TYPE ENUM('A', 'B') NOT NULL,
ALGORITHM=COPY;
Second, change values in other tables (there isn't transaction):
UPDATE CLIENTS
SET NAME = CONCAT('test-', RAND())
WHERE ID_CLIENT = 1;
The locks:
SELECT *
FROM performance_schema.metadata_locks
INNER JOIN performance_schema.threads ON THREAD_ID = OWNER_THREAD_ID
WHERE
PROCESSLIST_ID <> CONNECTION_ID();
Maybe the problem is due the lock over the SCHEMA?
Seems like the idea is to avoid the ALGORITHM=COPY (rebuilds without in-place mode)
So instead of modify the column type
ALTER TABLE API.SEARCHES_ELEMENTS
MODIFY COLUMN TYPE ENUM('A', 'B') NOT NULL,
ALGORITHM=COPY;
is better to create a new column, copy the data and remove the old one:
ALTER TABLE API.SEARCHES_ELEMENTS
ADD COLUMN TYPE_NEW ENUM('A', 'B') NOT NULL AFTER TYPE,
ALGORITHM=INSTANT;
LOCK TABLES API.SEARCHES_ELEMENTS WRITE;
UPDATE API.SEARCHES_ELEMENTS SET TYPE_NEW = TYPE;
ALTER TABLE API.SEARCHES_ELEMENTS
RENAME COLUMN TYPE TO TYPE_OLD,
RENAME COLUMN TYPE_NEW TO TYPE,
ALGORITHM=INSTANT;
UNLOCK TABLES;
ALTER TABLE API.SEARCHES_ELEMENTS
DROP COLUMN TYPE_OLD,
ALGORITHM=INPLACE;
Note: adding a value in ENUM might be use the algorithm=instant
Modifying the definition of an ENUM or SET column by adding new
enumeration or set members to the end of the list of valid member
values may be performed instantly or in place, as long as the storage
size of the data type does not change. For example, adding a member to
a SET column that has 8 members changes the required storage per value
from 1 byte to 2 bytes; this requires a table copy. Adding members in
the middle of the list causes renumbering of existing members, which
requires a table copy.
A tables metadata is not only locked when a running query is using it, but also if it has previously been used in an active transaction until that transaction commits or rolls back, to prevent the table from changing while still being referenced by the transaction.
If you are running at least MySQL 5.7 and have the performance_schema enabled you can check for current metadata locks via the performance_schema.metadata_locks table.
See also:
https://dev.mysql.com/doc/refman/5.7/en/performance-schema-metadata-locks-table.html
https://dev.mysql.com/doc/refman/5.7/en/metadata-locking.html
Joining multiple times to a temporary table. Getting this error:
Error Code: 1137. Can't reopen table: 'a'
I googled it and found that there is a restriction of some kind on referencing the same temporary table multiple times in a query. Can anyone explain why this restriction exists?
Here is an exmaple of simple query that will cause this error:
CREATE TEMPORARY TABLE foo
SELECT * FROM shopify_us limit 10;
SELECT *
FROM (
SELECT *
FROM shopify_us
LIMIT 10
) boo
LEFT JOIN foo a ON a.customer_id = boo.customer_id
LEFT JOIN foo b on b.customer_id = boo.customer_id
However, if I simply remove the 2nd join, I no longer encounter the error.
The restriction is that you can't reference a temporary table multiple times in the same query. For example, doing a self-join on a temporary table cannot be done. But you can use a temporary table by multiple subsequent queries.
Here's the comment in the MySQL Server code that explains the reason, right before it raises this error:
/*
We're trying to use the same temporary table twice in a query.
Right now we don't support this because a temporary table is always
represented by only one TABLE object in THD, and it can not be
cloned. Emit an error for an unsupported behaviour.
*/
https://github.com/mysql/mysql-server/blob/8.0/sql/sql_base.cc#L7284-L7289
THD refers to the data structure for session-scoped information. Temporary tables are scoped to the session in which they are created. They aren't references to tables that can be shared by multiple sessions.
In other words, it's just a limitation in the code with respect to the data structure that tracks temporary tables.
See also the history of this as a bug report dating back to 2005: https://bugs.mysql.com/bug.php?id=10327
There is a workaround in MySQL 8.0: you can us a common table expression instead, for some kinds of uses for which you would have used a temp table.
I created a table, master-domain. This table should only have 3 records.
How can I limit mysql database to only allow NO MORE than that number of records?
Is there a specific sql command to acomplish this?
This is my current SQL:
CREATE TABLE `mydatabase`.`master-domain`
(
`domain` VARCHAR( 50 ) NOT NULL COMMENT 'Domain Name',
PRIMARY KEY ( `domain` )
)
PS. I have godaddy and it includes phpMyAdmin, in addition to MySQL databases.
You can make a table's primary key a field of type ENUM. For example:
CREATE TABLE test (
id enum('1','2') NOT NULL,
domain varchar(50) NOT NULL,
primary key (id));
When you update it you have to explicitly set the ID to "", "1", or "2".* It can't be null and there can only be one record with each ID. The domain is still stored in the domain field, so hopefully whatever external system is querying this database won't have any problems getting the results it wants.
If you want to replicate the current restriction that domain names have to be unique, you could also add unique key (domain).
* note that the empty string is allowed (and not the same as NULL) because enum is actually a type of string. So you'd specify two permitted ID values in order to have three total.
Alternately: What are you trying to achieve / prevent here? Is there some automated process that might add records to the table? Are you trying to make sure that you don't accidentally do it, or that someone who hijacks your account can't do it?
If the process that might insert records is running on your user, you could put your three records into the table and then take away INSERT privileges from yourself. You'd still be able to alter the existing records but you wouldn't be able to add any more unless you explicitly re-granted the ability.
You can have a look here at the MAX_ROWS parameter. However, I believe this is normally used to make the table size larger than the disk size and I don't think you would get the restriction you are looking for using it. Alternatively, you could just select the top 3 rows.
I would question the point of using a database to only store 3 rows - it seems a total waste.
I think there is no such inbuilt functionality provided by MySQL. One solution is that you can create trigger.
CREATE TRIGGER your_trigger_name
BEFORE INSERT ON master-domain
FOR EACH ROW
BEGIN
DECLARE cnt INT;
SELECT count(*) INTO cnt FROM master-domain;
IF cnt = 10 THEN
SIGNAL SQLSTATE '45000' SET MESSAGE_TEXT = 'You can store only 3 records.';
END IF;
END;
Try above trigger on your table. Hope this will help you.
I am trying to implement simple program in Java that will be used to populate a MySQL database from a CSV source file. For each row in the CSV file, I need to execute following sequence of SQL statements (example in pseudo code):
execute("INSERT INTO table_1 VALUES(?, ?)");
String id = execute("SELECT LAST_INSERT_ID()");
execute("INSERT INTO table_2 VALUES(?, ?)");
String id2 = execute("SELECT LAST_INSERT_ID()");
execute("INSERT INTO table_3 values("some value", id1, id2)");
execute("INSERT INTO table_3 values("some value2", id1, id2)");
...
There are three basic problems:
1. Database is not on localhost so each single INSERT/SELECT has latency and this is the basic problem
2. CSV file contains millions of rows (like 15 000 000) so it takes too long.
3. I cannot modify the database structure (add extra tables, disable keys etc).
I was wondering how can I speed up the INSERT/SELECT process? Currently 80% of the execution time is consumed by communication.
I already tried to group the above statements and execute them as batch but because of LAST_INSERT_ID it does not work. In any other cases it takes too long (see point 1).
Fastest way is to let MySQL parse the CSV and load records into the table. For that, you can use "LOAD DATA INFILE":
http://dev.mysql.com/doc/refman/5.1/en/load-data.html
It works even better if you can transfer the file to server or keep it on a shared directory that is accessible to server.
Once that is done, you can have a column that indicates whether the records has been processed or not. Its value should be false by default.
Once data is loaded, you can pick up all records where processed=false.
For all such records you can populate table 2 and 3.
Since all these operation would happen on server, server <> client latency would not come into the picture.
Feed the data into a blackhole
CREATE TABLE `test`.`blackhole` (
`t1_f1` int(10) unsigned NOT NULL,
`t1_f2` int(10) unsigned NOT NULL,
`t2_f1` ... and so on for all the tables and all the fields.
) ENGINE=BLACKHOLE DEFAULT CHARSET=latin1;
Note that this is a blackhole table, so the data is going nowhere.
However you can create a trigger on the blackhole table, something like this.
And pass it on using a trigger
delimiter $$
create trigger ai_blackhole_each after insert on blackhole for each row
begin
declare lastid_t1 integer;
declare lastid_t2 integer;
insert into table1 values(new.t1_f1, new.t1_f2);
select last_insert_id() into lastid_t1;
insert into table2 values(new.t2_f1, new.t2_f1, lastid_t1);
etc....
end$$
delimiter ;
Now you can feed the blackhole table with a single insert statement at full speed and even insert multiple rows in one go.
insert into blackhole values(a,b,c,d,e,f,g,h),(....),(...)...
Disable index updates to speed things up
ALTER TABLE $tbl_name DISABLE KEYS;
....Lot of inserts
ALTER TABLE $tbl_name ENABLE KEYS;
Will disable all non-unique key updates and speed up the insert. (an autoincrement key is unique, so that's not affected)
If you have any unique keys and you don't want MySQL to check for them during the mass-insert, make sure you do an alter table to eliminate the unique key and enable it afterwards.
Note that the alter table to put the unique key back in will take a long time.
We're using MySQL with InnoDB storage engine and transactions a lot, and we've run into a problem: we need a nice way to emulate Oracle's SEQUENCEs in MySQL. The requirements are:
- concurrency support
- transaction safety
- max performance (meaning minimizing locks and deadlocks)
We don't care if some of the values won't be used, i.e. gaps in sequence are ok. There is an easy way to archieve that by creating a separate InnoDB table with a counter, however this means it will take part in transaction and will introduce locks and waiting. I am thinking to try a MyISAM table with manual locks, any other ideas or best practices?
If auto-increment isn't good enough for your needs, you can create a atomic sequence mechanism with n named sequences like this:
Create a table to store your sequences:
CREATE TABLE sequence (
seq_name varchar(20) unique not null,
seq_current int unsigned not null
);
Assuming you have a row for 'foo' in the table you can atomically get the next sequence id like this:
UPDATE sequence SET seq_current = (#next := seq_current + 1) WHERE seq_name = 'foo';
SELECT #next;
No locks required. Both statements need to be executed in the same session, so that the local variable #next is actually defined when the select happens.
The right way to do this is given in the MySQL manual:
UPDATE child_codes SET counter_field = LAST_INSERT_ID(counter_field + 1);
SELECT LAST_INSERT_ID();
We are a high transaction gaming company and need these sort of solutions for our needs. One of the features of Oracle sequences was also the increment value that could also be set.
The solution uses DUPLICATE KEY.
CREATE TABLE sequences (
id BIGINT DEFAULT 1,
name CHAR(20),
increment TINYINT,
UNIQUE KEY(name)
);
To get the next index:
Abstract the following with a stored procedure or a function sp_seq_next_val(VARCHAR):
INSERT INTO sequences (name) VALUES ("user_id") ON DUPLICATE KEY UPDATE id = id + increment;<br/>
SELECT id FROM sequences WHERE name = "user_id";
Won't the MySQL Identity column on the table handle this?
CREATE TABLE table_name
(
id INTEGER AUTO_INCREMENT PRIMARY KEY
)
Or are you looking to use it for something other than just inserting into another table?
If you're writing using a procedural language as well (instead of just SQL) then the other option would be to create a table containing a single integer (or long integer) value and a stored procedure which locked it, selected from it, incremented it and unlocked it before returning the value.
(Note - always increment before you return the value - it maximise the chance of not getting duplicates if there are errors - or wrap the whole thing in a transaction.)
You would then call this independently of your main insert / update (so it doesn't get caught in any transactions automatically created by the calling mechanism) and then pass it as a parameter to wherever you want to use it.
Because it's independent of the rest of the stuff you're doing it should be quick and avoid locking issues. Even if you did see an error caused by locking (unlikely unless you're overloading the database) you could just call it a second / third time.