Is it possible to execute stored procedures on temporary tables in MySQL? I'm trying to create a system for data import that could, theoretically, run concurrently for different users. If one user is importing files into a temporary table while another user is doing the same thing, is it possible for the same shared procedure to be called by both users since the tables referenced in the procedure will match the temporary tables?
The workflow for an individual user would look like this...
Load data into temporary table newdata
Stored procedure is called where munging and updates are done to table newdata
Stored procedure moves data from newdata to the live/permanent tables.
...while another user could, possibly, be doing the same thing.
Yes, you can reference temp tables in a stored procedure:
mysql> create procedure p() select * from t;
Query OK, 0 rows affected (0.03 sec)
mysql> create temporary table t as select 123 union select 456;
Query OK, 2 rows affected (0.02 sec)
Records: 2 Duplicates: 0 Warnings: 0
mysql> call p;
+-----+
| 123 |
+-----+
| 123 |
| 456 |
+-----+
(Tested on MySQL 5.6.31)
However, most experienced MySQL users try to avoid doing what you're planning, if they use replication. The reason is that when a slave restarts for any reason, it drops any temp tables. So any subsequent UPDATE and INSERT...SELECT referencing the temp table get an error because the temp table no longer exists. An error in the replication stream stops replication.
It might seem like this should be an uncommon occurrence for a slave to restart suddenly, but if your app creates temp tables frequently, there's a high chance for there to be a temp table going at the moment a slave restarts.
The best use of temp tables is to fill them with temp data in one statement, and then use the temp table only by SELECT queries (which are not replicated).
Related
I am running 10.1.26-MariaDB-0+deb9u1 Debian 9.1 in multiple locations.
Just got a call today that some scripts are no longer running at one of the locations. I've diagnosed that whenever a script tries to execute TRUNCATE <table name> it just hangs.
I've tried it from the CLI and Workbench as well with the same results. I have also tried TRUNCATE TABLE <table name> with the same results.
I cannot figure out A) why this all of a sudden stopped working. and B) what's different between this location and other three, where it does work.
I expect you see something like this:
mysql> show processlist;
+----+-----------------+-----------+------+---------+------+---------------------------------+------------------------+
| Id | User | Host | db | Command | Time | State | Info |
+----+-----------------+-----------+------+---------+------+---------------------------------+------------------------+
| 8 | msandbox | localhost | test | Query | 435 | Waiting for table metadata lock | truncate table mytable |
Try this experiment in a test instance of MySQL (like on your local development environment): open two shell windows and run the mysql client. Create a test table.
mysql> create table test.mytable ( answer int );
mysql> insert into test.mytable set answer = 42;
Now start a transaction and query the table, but do not commit the transaction yet.
mysql> begin;
mysql> select * from test.mytable;
+--------+
| answer |
+--------+
| 42 |
+--------+
In the second window, try to truncate that table.
mysql> truncate table mytable;
<hangs>
What it's waiting for is a metadata lock. It will wait for a number of seconds equal to the lock_wait_timeout configuration option.
Now go back to the first window, and commit.
mysql> commit;
Now see in your second window, the TRUNCATE TABLE stops waiting, and it finally does its work, truncating the table.
Any DDL statement like ALTER TABLE, TRUNCATE TABLE, DROP TABLE needs to acquire an exclusive metadata lock on the table. But any transaction that has been reading or writing that table holds a shared metadata lock. This means many concurrent sessions can do their work, like SELECT/UPDATE/INSERT/DELETE without blocking each other (because their locks are shared). But a DDL statement requires an exclusive metadata lock, meaning no other metadata lock, either shared or exclusive, can exist.
So I'd guess there's some transaction hanging around that has done some read or write against your table, without committing. Either the query itself is very long-running, or else the query has finished but the transaction hasn't.
You have to figure out where you have an outstanding transaction. If you are using MySQL 5.7 or later, you can read the sys.schema_lock_waits table while one of your truncate table statements is waiting.
select * from sys.schema_table_lock_waits\G
*************************** 1. row ***************************
object_schema: test
object_name: mytable
waiting_thread_id: 47
waiting_pid: 8
waiting_account: msandbox#localhost
waiting_lock_type: EXCLUSIVE
waiting_lock_duration: TRANSACTION
waiting_query: truncate table mytable
waiting_query_secs: 625
waiting_query_rows_affected: 0
waiting_query_rows_examined: 0
blocking_thread_id: 48
blocking_pid: 9
blocking_account: msandbox#localhost
blocking_lock_type: SHARED_READ
blocking_lock_duration: TRANSACTION
sql_kill_blocking_query: KILL QUERY 9
sql_kill_blocking_connection: KILL 9
This tells us which session is blocked, waiting for a metadata lock. The waiting_pid (8 in the above example) corresponds to the Id in the processlist of the blocked session.
The blocking_pid (9 in the above example) corresponds to the Id in the processlist of the session that currently holds the lock, and which is blocking the truncate table.
It even tells you exactly how to kill the session that's holding the lock:
mysql> KILL 9;
Once the session is killed, it must release its locks, and the truncate table finally finishes.
mysql> truncate table mytable;
Query OK, 0 rows affected (13 min 34.50 sec)
Unfortunately, you're using MariaDB 10.1. This doesn't support the sys schema or the performance_schema.metadata_locks table that it needs to report those locks. MariaDB is a fork from MySQL 5.5, which is nearly ten years old now, and they didn't have the metadata_locks table at that time.
I don't use MariaDB, but I googled and found that they have their own proprietary implementation for querying metadata locks: https://mariadb.com/kb/en/library/metadata_lock_info/ I haven't used it, so I'll leave it to you to read the docs about that.
I try to create a table as a select result of query as the following:
CREATE TABLE IF NOT EXISTS x AS (SELECT ...)
That works fine and a new table x is created with adequate columns and content.
But when I re-execute the ddl query the existing table is populated again with the same content (so double...)
After first execution, table x:
1 a
2 b
3 c
After secondexecution, table x:
1 a
2 b
3 c
1 a
2 b
3 c
Why? And how to resolva that? because this executed by a user in my app so it can be executed many times and that's why I put IF NOT EXISTS to not re create the table each time...
Best Regards,
This is version dependent. As described in the documentation:
For CREATE TABLE ... SELECT, if IF NOT EXISTS is given and the
destination table already exists, the result is version dependent.
Before MySQL 5.5.6, MySQL handles the statement as follows:
The table definition given in the CREATE TABLE part is ignored. No error occurs, even if the definition does not match that of the
existing table. MySQL attempts to insert the rows from the SELECT part
anyway.
MySQL applies these rules when various CREATE ... IF NOT EXISTS statements are replicated:
Every CREATE DATABASE IF NOT EXISTS statement is replicated, whether
or not the database already exists on the master.
Similarly, every CREATE TABLE IF NOT EXISTS statement without a SELECT
is replicated, whether or not the table already exists on the master.
This includes CREATE TABLE IF NOT EXISTS ... LIKE. Replication of
CREATE TABLE IF NOT EXISTS ... SELECT follows somewhat different
rules; see Section 17.4.1.6, “Replication of CREATE TABLE ... SELECT
Statements”, for more information.
For more information- CREATE EVENT IF NOT EXISTS is always replicated in MySQL 5.5, whether
or not the event named in the statement already exists on the master.
I have this largish table with three columns as such:
+-----+-----+----------+
| id1 | id2 | associd |
+-----+-----+----------+
| 1 | 38 | 73157604 |
| 1 | 112 | 73157605 |
| 1 | 113 | 73157606 |
| 1 | 198 | 31936810 |
| 1 | 391 | 73157607 |
+-----+-----+----------+
This continues for 38m rows. The problem is I want to remove the 'associd' column but running ALTER TABLE table_name DROP COLUMN associd; simply takes too long. I wanted to do something like: ALTER TABLE table_name SET UNUSED associd; and ALTER TABLE table_name DROP UNUSED COLUMNS CHECKPOINT 250; then which apparently speeds up the process but it isn't possible in MySQL?
Is there an alternative to remove this column-- maybe creating a new table with only the two columns or getting a drop with checkpoints?
Anything that you do is going to require reading and writing 38m rows, so nothing is going to be real fast. Probably the fastest method is probably to put the data into a new table:
create table newTable as
select id1, id2
from oldTable;
Or, if you want to be sure that you preserve types and indexes:
create table newTable like oldTable;
alter table newTable drop column assocId;
insert into newTable(id1, id2)
select id1, id2
from oldTable;
However, it is usually faster to drop all index on a table before loading a bunch of data and then recreate the indexes afterwards.
Disclaimer: this answer is MySQL oriented and might not work for other databases.
I think in the accepted answer there are some things missing, I have tried to expose here a generic sequence I use to do this kind of operations in a production environment, not only for adding/removing columns but also to add indexes for example.
We call it the Indiana Jones' movement.
Create a new table
A new table using the old one as template:
create table my_table_new like my_table;
Remove the column in the new table
In the new table:
alter table my_table_new drop column column_to_delete;
Add the foreign keys to the new table
The are not generate automatically in the create table like command.
You can check the actual foreign keys:
mysql> show create table my_table;
Then apply them to the new table:
alter table my_table_new
add constraint my_table_fk_1 foreign key (field_1) references other_table_1 (id),
add constraint my_table_fk_2 foreign key (field_2) references other_table_2 (id)
Clone the table
Copy all fields but the one you want to delete.
I use a where sentence to be able to run this command many times if necessary.
As I suppose this is a production environment the my_table will have new records continuously so we have to keep synchronizing until we are capable to do the name changing.
Also I have added a limit because if the table is too big and the indexes are too heavy making a one-shot clone can shut down the performance of your database. Plus, if in the middle of the process you want to cancel the operation it will must to rollback all the already done insertions which means your database won't be recovered instantly (https://dba.stackexchange.com/questions/5654/internal-reason-for-killing-process-taking-up-long-time-in-mysql)
insert my_table_new select field_1, field_2, field_3 from my_table
where id > ifnull((select max(id) from my_table_new), 0)
limit 100000;
As I was doing this several times I created a procedure: https://gist.github.com/fguillen/5abe87f922912709cd8b8a8a44553fe7
Do the name changing
Be sure you run this commands inmediately after you have replicate the last records from your table. Idealy run all commands at once.
rename table my_table to my_table_3;
rename table my_table_new to my_table;
Delete the old table
Be sure you have a back up before you do this ;)
drop table my_table_3
Disclaimer: I am not sure what will happen with foreign keys that were pointing to the old table.
The best solution in this case in MySQL is to:
1) change the table Engine to MyISAM
2) change whatever you want to do (Drop column, alter data type,etc..)
3) change it back to InnoDB
In this case the DBMS will not be locking/unlocking at each record iteration.
However note that this solution would be good if you have several things you want to change in your table/database, because once you revert it back to InnoDB, this will take the same amount of time to drop one column.
So only consider this solution if you have multiple things to change in your database.
You can speed up the process by temporarily turning off unique checks and foreign key checks. You can also change the algorithm that gets used.
SET unique_checks=0;
SET foreign_key_checks=0;
ALTER TABLE table_name DROP COLUMN column_name, algorithm=inplace;
SET unique_checks=1;
SET foreign_key_checks=1;
Using the above code, it took my PC about 2 minutes to drop a column from a 20 million row table.
If you're using a program like Workbench, then you may want to increase the default timeout period in your settings before starting the operation.
If you find that the operation is hanging indefinitely, then you may need to look through the list of processes and kill whatever process has a lock on the table. You can do that using these commands:
SHOW FULL PROCESSLIST;
KILL PROCESS_NUMBER_GOES_HERE;
I have a service which produce formatted text files with information from certain tables in a MYSQL db.
Currently I programmatically query the DB and dump it in a TEXT file.
I wonder if i can do the same from within the MYSQL db.
This means that when a new record is added to a specific table, MYSQL will run a piece of code that will generate that formatted text file.
You didn't tell much about your use cases, but maybe the CSV engine might be what you're looking for?
From the manual:
When you create a CSV table, the server creates a table format file in the database directory. The file begins with the table name and has an .frm extension. The storage engine also creates a data file. Its name begins with the table name and has a .CSV extension. The data file is a plain text file. When you store data into the table, the storage engine saves it into the data file in comma-separated values format.
mysql> CREATE TABLE test (i INT NOT NULL, c CHAR(10) NOT NULL)
-> ENGINE = CSV;
Query OK, 0 rows affected (0.12 sec)
mysql> INSERT INTO test VALUES(1,'record one'),(2,'record two');
Query OK, 2 rows affected (0.00 sec)
Records: 2 Duplicates: 0 Warnings: 0
mysql> SELECT * FROM test;
+------+------------+
| i | c |
+------+------------+
| 1 | record one |
| 2 | record two |
+------+------------+
2 rows in set (0.00 sec)
Starting with MySQL 5.1.9, creating a CSV table also creates a corresponding Metafile that stores the state of the table and the number of rows that exist in the table. The name of this file is the same as the name of the table with the extension CSM.
If you examine the test.CSV file in the database directory created by executing the preceding statements, its contents should look like this:
"1","record one"
"2","record two"
This format can be read, and even written, by spreadsheet applications such as Microsoft Excel or StarOffice Calc.
What's is better to use in a stored procedure: a temporary table or a memory table?
The table is used to stored summary data for reports.
Are there any trade offs that developers should be aware off?
CREATE TEMPORARY TABLE t (avg (double));
or
CREATE TABLE t (avg (double)) ENGINE=MEMORY;
Why is this restricted to just the two options? You can do:
CREATE TEMPORARY TABLE t (avg double) ENGINE=MEMORY;
Which works, although I'm not sure how to check if the memory engine is actually being used here.
Of the two, I'd use a temporary table for report.
A memory table holds data across user sessions & connections, so you'd have to truncate it every time to make sure you wouldn't be using data from someone else. Assuming you put in whats necessary to maintain a memory table depending on your needs, it's fine - the temp table is a little safer from a maintenance perspective.
A temporary table will only exist for the duration of your session. A table declared with Engine=Memory will persist across user sessions / connections but will only exist in the lifetime of the MySQL instance. So if MySQL gets restarted the table goes away.
In MySQL, temporary tables are seriously crippled:
http://dev.mysql.com/doc/refman/5.6/en/temporary-table-problems.html
You cannot refer to a TEMPORARY table more than once in the same query.
For example, the following does not work:
mysql> SELECT * FROM temp_table, temp_table AS t2;
ERROR 1137: Can't reopen table: 'temp_table'
I Just wanted to point out that, in 2021 using MariaDB-10.3.27, the code #biziclop said doesn't work, is not the case any more, this is possible:
CREATE TEMPORARY TABLE tmp1 AS
SELECT * FROM products LIMIT 10;
SELECT * FROM tmp1, tmp1 AS t2;
(I just tested it)