I am using mysql
I have a table called address and the table has a column called zip5 which is of type varchar(6) .
I am using query
alter table address change zip5 zip5 varchar(14);
but the query execution is taking too long I am waiting from almost 15 minutes and waiting for query to execute, the address table has 9.7 million records. Does it take this long for this amount of data or am I doing something wrong here?
Hmm, I dont know why but
ALTER TABLE address MODIFY zip5 varchar(14)
seems to be a bit faster. At least on my system with a comparable table-structure.
ALTER TABLE makes a copy of you table. Perhaps the bottle-neck is your HD? Do you use SSDs? or is your temporary table storage not set on a fast disk? Is the disk full?
Related
I've been running a website, with a large amount of data in the process.
A user's save data like ip , id , date to the server and it is stored in a MySQL database. Each entry is stored as a single row in a table.
Right now there are approximately 24 million rows in the table
Problem 1:
Things are getting slow now, as a full table scan can take too many minutes but I already indexed the table.
Problem 2:
If a user is pulling a select data from table it could potentially block all other users (as the table is locked) access to the site until the query is complete.
Our server
32 Gb Ram
12 core with 24 thread cpu
table use MyISAM engine
EXPLAIN SELECT SUM(impresn), SUM(rae), SUM(reve), `date` FROM `publisher_ads_hits` WHERE date between '2015-05-01' AND '2016-04-02' AND userid='168' GROUP BY date ORDER BY date DESC
Lock to comment from #Max P. If you write to MyIsam Tables ALL SELECTs are blocked. There is only a Table lock. If you use InnoDB there is a ROW Lock that only locks the ROWs they need. Aslo show us the EXPLAIN of your Queries. So it is possible that you must create some new one. MySQL can only handle one Index per Query. So if you use more fields in the Where Condition it can be useful to have a COMPOSITE INDEX over this fields
According to explain, query doesn't use index. Try to add composite index (userid, date).
If you have many update and delete operations, try to change engine to INNODB.
Basic problem is full table scan. Some suggestion are:
Partition the table based on date and dont keep more than 6-12months data in live system
Add an index on user_id
I have been switching my tables one by one to InnoDB on phpMyAdmin. Each table took a max of 30 seconds.
One table is stuck and has taken over 15 minutes (still going).
In the mysql process list, it shows:
status:
copy to tmp table
info:
ALTER TABLE `table` auto_increment = 2446976 ROW_FORMAT = DYNAMIC
Why is this process taking so long?
Can I kill this process? Or should I just let it go? The table is hot so some rows are waiting to be inserted.
The table does have a unique index on a varchar(30) column. Could this be the problem?
It takes long time because MySQL needs to create new table with the new structure , then copy data from old table (MyISAM) to the new table (InnoDB). When all records are copied it will replace the tables.
I don't recommend to kill it, because rollback process (the new table is InnoDB) will take longer. Just wait until it finishes. After ALTER is done the table will be InnoDB in good condition.
I've got a large (1 million+ rows) MySQL InnoDB table with the following columns:
id int(11)
name varchar(255)
If I perform the following query it takes a very long time (Well over 3 minutes [I got bored whilst waiting and stopped it]).
SELECT name FROM large_table WHERE name LIKE '%peter%'
I have added an index on the name column which has made it perform much better 0.009 seconds
I just want to confirm that this is this a suitable work-around for the issue? Or are you reading this cringing?
I was just trying to add a column called "location" to a table (main_table) in a database. The command I run was
ALTER TABLE main_table ADD COLUMN location varchar (256);
The main_table contains > 2,000,000 rows. It keeps running for more than 2 hours and still not completed.
I tried to use mytop
to monitor the activity of this database to make sure that the query is not locked by other querying process, but it seems not. Is it supposed to take that long time? Actually, I just rebooted the machine before running this command. Now this command is still running. I am not sure what to do.
Your ALTER TABLE statement implies mysql will have to re-write every single row of the table including the new column. Since you have more than 2 million rows, I would definitely expect it takes a significant amount of time, during which your server will likely be mostly IO-bound. You'd usually find it's more performant to do the following:
CREATE TABLE main_table_new LIKE main_table;
ALTER TABLE main_table_new ADD COLUMN location VARCHAR(256);
INSERT INTO main_table_new SELECT *, NULL FROM main_table;
RENAME TABLE main_table TO main_table_old, main_table_new TO main_table;
DROP TABLE main_table_old;
This way you add the column on the empty table, and basically write the data in that new table that you are sure no-one else will be looking at without locking as much resources.
I think the appropriate answer for this is using a feature like pt-online-schema-change or gh-ost.
We have done migration of over 4 billion rows with this, though it can take upto 10 days, with less than a minute of downtime.
Percona works in a very similar fashion as above
Create a temp table
Creates triggers on the first table (for inserts, updates, deletes) so that they are replicated to the temp table
In small batches, migrate data
When done, rename table to new table, and drop the other table
You can speed up the process by temporarily turning off unique checks and foreign key checks. You can also change the algorithm that gets used.
If you want the new column to be at the end of the table, use algorithm=instant:
SET unique_checks = 0;
SET foreign_key_checks = 0;
ALTER TABLE main_table ADD location varchar(256), algorithm=instant;
SET unique_checks = 1;
SET foreign_key_checks = 1;
Otherwise, if you need the column to be in a specific location, use algorithm=inplace:
SET unique_checks = 0;
SET foreign_key_checks = 0;
ALTER TABLE main_table ADD location varchar(256) AFTER othercolumn, algorithm=inplace;
SET unique_checks = 1;
SET foreign_key_checks = 1;
For reference, it took my PC about 2 minutes to alter a table with 20 million rows using the inplace algorithm. If you're using a program like Workbench, then you may want to increase the default timeout period in your settings before starting the operation.
If you find that the operation is hanging indefinitely, then you may need to look through the list of processes and kill whatever process has a lock on the table. You can do that using these commands:
SHOW FULL PROCESSLIST;
KILL PROCESS_NUMBER_GOES_HERE;
Alter table takes a long time with a big data like in your case, so avoid to use it in such situations, and use some code like this one:
select main_table.*,
cast(null as varchar(256)) as null_location, -- any column you want accepts null
cast('' as varchar(256)) as not_null_location, --any column doesn't accept null
cast(0 as int) as not_null_int, -- int column doesn't accept null
into new_table
from main_table;
drop table main_table;
rename table new_table TO main_table;
DB2 z/OS does a virtual add of the column instantly. And puts the table into Advisory-Reorg status. Anything that runs before the reorg gets the default value or null if no default. When updates are done, they expand the rows updated. Inserts are done expanded. The next reorg expands every unexpanded row and assigns the default value to anything it expands.
Only a real database handles this well. DB2 z/OS.
Fro clarity all other tables in the DB work as expected, and load ~2million rows in a fraction of a second. The one table of just ~600 rows is taking 10+minutes to load in navcat.
I can't think of any possible reason for this. There are just 4 columns. One of them is a large text field, but I've worked with large text fields before and they've never been this slow.
running explain select * from parser_queue I get
id setect type table type possible keys key key len ref rows extra
1 SIMPLE parser_queue ALL - - - - 658 -
The profile tells me that 453 seconds are spent 'sending data'
I've also got this in the 'Status' tab. I don't understand most of it, but these numbers are much higher than my other tables.
Bytes_received 31
Bytes_sent 32265951
Com_select 1
Created_tmp_files 16
Handler_read_rnd_next 659
Key_read_requests 9018487
Key_reads 3928
Key_write_requests 310431
Key_writes 4290
Qcache_hits 135077
Qcache_inserts 14289
Qcache_lowmem_prunes 4133
Qcache_queries_in_cache 983
Questions 1
Select_scan 1
Table_locks_immediate 31514
The data stored in the text field is about 12000 chars on average.
There is a primary, auto increment int id field, a tinyint status field, a text field, and a timestamp field with on update current timestamp.
OK I will try out both answers, but I can answer the questions quickly first:
Primary key on the ID field is the only key. This table is used for queuing, with ~50 records added/deleted per hour, but I only created it yesterday. Could it become corrupted in such a short time?
It is MyISAM
More work trying to isolate the problem:
repair table did nothing
optimize table did nothing
created a temp table. queries were about 50% slower on the temp table.
Deleted the table and rebuilt it. SELECT * takes 18 seconds with just 4 rows.
Here is the SQL I used to create the table:
CREATE TABLE IF NOT EXISTS `parser_queue` (
`id` int(10) unsigned NOT NULL AUTO_INCREMENT,
`status` tinyint(4) NOT NULL DEFAULT '1',
`data` text NOT NULL,
`last_updated` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
PRIMARY KEY (`id`)
) ENGINE=MyISAM;
Stranger still, everything seems fine on my local box. The slowness only happens on the dev site.
For clarity: there are more than 100 tables on the dev site and this is the only one that is funky.
OK I have disabled all cron jobs which use this table. SHOW PROCESSLIST does not reveal any locks on the table.
Changing the engine to InnoDB did not produce any significant improvement (86 seconds vs 94 for MyISAM)
any other ideas? . . .
Running SHOW PROCESSLIST during the query reveals it spends most of its time writing to net
If you suspect corruption somewhere, you can try either (or both) of the following:
CREATE TABLE temp SELECT * FROM parser_queue;
This will create a new table "identical" to the previous one, except it will be recreated. Alternatively (or maybe after you've made a copy), you can try:
REPAIR TABLE parser_queue;
You may also want to try optimizing the table; it might have gotten fragmented since you are using it as a queue.
OPTIMIZE TABLE parser_queue;
You can determine if the table is fragmented by running SHOW TABLE STATUS LIKE 'Data_Free' and see if this produces a high number.
Update
You say you are storing gzcompressed data in the TEXT columns. Try changing the TEXT column to BLOB instead, which is meant to handle binary data, such as compressed text.
The name gives away that you are using the table for queueing (lots of inserts and delets, maybe?). Maybe you have had the table a while and it's heavily fragmented. If my assumptions are correct, try OPTIMIZE TABLE parser_queue;
You can read more about this in the manual:
http://dev.mysql.com/doc/refman/5.1/en/optimize-table.html
Right, the problem seems to have been only this: the text fields where too huge.
Running
SELECT id, status, last_updated FROM parser_queue
takes less time than
SELECT data FROM parser_queue WHERE id = 6
Since all the queries I will be running return only one row, the slowdown will not affect me so much. I'm already using gzcompress on the data stored, so I don't think there is much more I could do anyway.