We started an ALTER TABLE which dropped one index and added another. Although we were only expecting the table to be locked for writes, reads started queueing up, so we killed the ALTER process. But when the KILL finished, the old index was gone, and the new index was there in its place, with a much lower cardinality than expected.
Searching on the table seems to be faster now, so it seems like the ALTER went through fine, but we're not sure. Is it possible that our KILL has left the index in a partially-built stage?
If the index is there you may assume that it is complete.
You can use SHOW CREATE TABLE or SHOW INDEXES to see the indexes on the table.
As noted in the comments, the cardinality listed by SHOW INDEXES is just an estimate.
One test you can try is to run SHOW INDEXES, then run ANALYZE TABLE, then run SHOW INDEXES again and see how the estimated cardinality value changes.
By your description (both reads/writes locked) you are most likely using an older version of InnoDB, or adding an index to a column in utf8 character set.
Here is how it works in your version:
Empty table is created with the new table definition.
Rows are copied 1 after the other from old table to new table (new indexes are also created).
Once copy is complete, old table is deleted, new table is renamed.
(If you cancel between steps 2&3 the new table is just safely removed.)
For full disclosure - here is how it works in InnoDB plugin (default for MySQL 5.5, available from 5.1+):
Table is read through to find data for the index, and written to a temporary file.
The temporary file is sorted.
The index is created by inserting the data in order.
(This method is more optimized. InnoDB calls is "fast-index creation".)
Related
Is there a way to perform ALTER TABLE in MySQL, telling the server to skip creating a backup of the table first? I have a backup of the table already and I'm doing some tests on it (adding indexes), so I don't care if the table gets corrupted in the process. I'll just restore it from the backup. But what I do care about is for the ALTER TABLE to finish quickly, so I can see the test results.
Given that I have a big MyISAM table (700 GB) it really isn't an option to wait for couple of hours so that MySQL can first finish creating a backup of the original table, before actually adding an index to it.
It's not doing a backup; it is building the new version. (The existing table serves as a backup in case of a crash.)
With InnoDB, there are many flavors of ALTER TABLE -- some of which take essentially zero time, regardless of the size of the table. MyISAM (mostly) does the brute force way: Create an empty table with the new schema; copy all the data and build all the indexes; swap tables. For some alters, InnoDB must also do the brute force way: Example changing the PRIMARY KEY.
Running OPTIMIZE TABLE results in "Waiting for table metadata lock". Checking SHOW PROCESSLIST confirms optimizing is the only active query.
I have a table that is 750GB, and 69GB left on the drive. To free up space I decided to cleanup that table. I've turned all access to that server off, and started by deleting old records, which would have ended up taking forever. It has been decided that the table can just be truncated but a small chunk of the data needed to be extracted first. Problem, even a simple SELECT * FROM my_table LIMIT 1 takes hours before it is manually killed. Is this an indexing issue? And if so is, 69GB enough for the index process.
If you have something else you can delete to free up disk space, that might release the lock.
Kill the process, make sure there are no tmp tables left around clogging disk.
Then do the cleanup a different way...
CREATE TABLE new LIKE real;
DROP any indexes you don't immediately need from `new`.
INSERT INTO new
SELECT ... FROM real
WHERE ...;
RENAME TABLE real TO old, new TO real;
DROP TABLE old;
If you make it this far, ADD back the indexes you should have.
Potential problem: If the table is Engine=InnoDB and was created with innodb_file_per_table=OFF, then this is not sufficient to free up any disk space.
If you don't delete more than 90% of the table, and you have only 69GB of free space, the process will eventually fail.
"For the index process" -- This phrase "does not compute".
OPTIMIZE TABLE does:
Create a new table like the old one, but perhaps without any indexes other than the PRIMARY KEY.
Copy all the rows (other than deleted ones) over.
Build the indexes (assuming they were not incrementally built in step 2).
RENAME and DROP (as above)
How should I convert MyISAM table to InnoDB on live working site in real time?
Should I just change in phpmyadmin in operations menu "Storage Engine" to InnoDB?
Would it lock all table during conversion?
You will indeed get a write lock on the table. You will still be able to read from the table while the new table is being created and copied over.
From the documentation on ALTER TABLE:
While ALTER TABLE is executing, the original table is readable by
other sessions (with the exception noted shortly). Updates and writes
to the table that begin after the ALTER TABLE operation begins are
stalled until the new table is ready, then are automatically
redirected to the new table without any failed updates.
...
The exception referred to earlier is that ALTER TABLE blocks reads
(not just writes) at the point where it is ready to install a new
version of the table .frm file, discard the old file, and clear
outdated table structures from the table and table definition caches.
At this point, it must acquire an exclusive lock. To do so, it waits
for current readers to finish, and blocks new reads (and writes).
Check out pt-online-schema-change. See example in this answer https://dba.stackexchange.com/questions/60570/best-way-to-add-a-new-column-to-a-large-table-mysql-myisam/60576#60576
It will block a table for a second or so to rename two tables.
While it might work you'd better make a copy of your table and try the storage engine change operation on the copy.
That would also give you an estimate of time needed for the operation.
Even better would be if you do that on a copy of your application and make extensive checks whether it would work at all with the new storage engine.
I just ran a query to drop an index, and it was taking so long I killed the query. I'm curious now, did it partially delete the index, or is the index unaffected?
I used ALTER TABLE the first time, and DROP INDEX the second time, to remove the index, but ended up killing both queries. My understanding of MySQL's approach is that it copies to a temporary table, applies the changes (removing the index) on the temporary table, deletes the original and renames the temporary table. Is this correct? Is it true for both the ALTER TABLE and DROP INDEX commands, or is it only true for one of them?
Yes you are correct, if the default settings are used.
It is possible to configure the database so it will not make (use of) temp table.
You also can look at the result off course to double check
This will rollback everything. There is no such thing as a half completed transaction in a ACID complaint database. It is a all or nothing situation.
Now something your have to remember especially as far as InnoDB is concerned a index is not stored with a table. It is a separate B-Tree structure on disk with pointer to the data in the tables. Thus MySQL will never need to remove the index from the table. It just needs to destroy the index structure.
I was indexing a huge table today containing 2 billion records. I thought MySQL would eat away my 2TB drive... The disk consumption kept increasing to 400GB and then 500GB and then finally drops to 180GB and MySQL says successfully added the index. Why the space increase and what happened in the end? Can someone please give me some pointers?
Incidentally yesterday I answered a question on how to make index creation faster in MySQL, and the following came out from my research:
The CREATE INDEX and DROP INDEX commands work by creating a new, empty table defined with the requested set of indexes. It then copies the existing rows to the new table one-by-one, updating the indexes as it goes. Inserting entries into the indexes in this fashion, where the key values are not sorted, requires random access to the index nodes, and is far from optimal. After all rows from the original table are copied, the old table is dropped and the copy is renamed with the name of the original table.
Source: Overview of Fast Index Creation