I have been researching this problem for quite some time but have not been able to find any helpful results.
I have a table:
CREATE TABLE `jobs` (
`jb_id` MEDIUMINT(7) UNSIGNED NOT NULL AUTO_INCREMENT,
`wo_id` MEDIUMINT(7) UNSIGNED NOT NULL,
`file_name` VARCHAR(140) NOT NULL COLLATE 'latin1_bin',
`jb_status` TINYINT(1) UNSIGNED NOT NULL DEFAULT '0',
`descr` TEXT NULL COLLATE 'latin1_bin',
`syncronized` TINYINT(2) UNSIGNED NOT NULL,
`failedcnt` TINYINT(3) UNSIGNED NOT NULL,
`clip_title` TINYTEXT NULL COLLATE 'latin1_bin',
`clip_description` TEXT NULL COLLATE 'latin1_bin',
`clip_tags` TINYTEXT NULL COLLATE 'latin1_bin',
PRIMARY KEY (`jb_id`),
INDEX `woid` (`wo_id`),
INDEX `job_stat` (`jb_status`),
INDEX `synced` (`syncronized`),
INDEX `failedcnt` (`failedcnt`),
INDEX `file_name` (`file_name`)
)
COLLATE='latin1_bin'
ENGINE=MyISAM;
When i run SELECT or UPDATE commands everything works ok.
select jobs.clip_description from jobs Limit 1;
/* 0 rows affected, 1 rows found. Duration for 1 query: 0.768 sec. */
UPDATE `jobs` SET `clip_description`='test' WHERE `jb_id`=2 LIMIT 1;
But as I try to run
INSERT INTO `jobs` (`clip_description`) VALUES ('test');
/* SQL Error (1054): Unknown column 'clip_description' in 'field list' */
This also happened yesterday, but as i did not have much time to deal with the issue then, i created new table with different name but same structure, copied over all the data and then renamed both tables and it worked again. That is until about two hours ago when the issue returned. It is not really an option to start coping the table every 12h.
For creating a copy i used:
CREATE TABLE jobs_new LIKE jobs; INSERT jobs_new SELECT * FROM jobs;
After which previously mentioned insert would work.
Any help will be greatly appreciated.
EDIT:
If it makes any difference I'm running
Server version: 5.5.28-0ubuntu0.12.04.2-log (Ubuntu)
On ubuntu server 12.04 LTS 64bit
It looks like you have other constrains related to table, may be a trigger or some calculated column depending upon clip_description column. Is not it?
Please check the dependencies and triggers with this table.
Related
I have a mysql table with 2 million rows, when I'm running any select query on the table it's taking long time to execute and ultimately it does not return any result.
I have tried running select query from both Mysql Workbench and terminal, it's the same issue happening.
Below is the table:
`object_master`
`key` varchar(300) CHARACTER SET utf8 COLLATE utf8_bin NOT NULL,
`bucket` varchar(255) DEFAULT NULL,
`object_name` varchar(300) DEFAULT NULL,
`object_type` varchar(50) DEFAULT NULL,
`last_modified_date` datetime DEFAULT NULL,
`last_accessed_date` datetime DEFAULT NULL,
`is_deleted` tinyint(1) DEFAULT '0',
`p_object` varchar(300) DEFAULT NULL,
`record_insert_time` datetime DEFAULT CURRENT_TIMESTAMP,
`record_update_time` datetime DEFAULT CURRENT_TIMESTAMP,
PRIMARY KEY (`key`)
ENGINE=InnoDB DEFAULT CHARSET=latin1
And below is the select query i'm running :
select `key` from object_master;
even with a limit 1 is also taking long time and not returning a result, its getting timed out :
select `key` from object_master limit 1;
Could anyone tell me what can be the real reason here?
Also I would like to mention: before I was running these select queries, there was an alter table statement executed on this table which got timed out after 10 minutes and table remained un-altered.
Following is the alter statement:
alter table object_master
modify column object_name varchar(1096) CHARACTER SET utf8 COLLATE utf8_bin DEFAULT NULL;
Note: Using MYSQL version 5.7.24 and Mysql running on Linux Docker container.
So I got this resolved:
There was Java batch program which was executing a query on the same table for long time and was holding a lock on the table. I found this through "processlist" table of information_schema.
Had to kill the long running query through terminal:
mysql> kill <processlist_id> ;
Then it released the lock on that table and all got resolved.
Got help from below SO answers:
Unlocking tables if thread is lost
How do I find which transaction is causing a "Waiting for table metadata lock" state?
I'm trying to denormalize a few MySQL tables I have into a new table that I can use to speed up some complex queries with lots of business logic. The problem that I'm having is that there are 2.3 million records I need to add to the new table and to do that I need to pull data from several tables and do a few conversions too. Here's my query (with names changed)
INSERT INTO database_name.log_set_logs
(offload_date, vehicle, jurisdiction, baselog_path, path,
baselog_index_guid, new_location, log_set_name, index_guid)
(
select STR_TO_DATE(logset_logs.offload_date, '%Y.%m.%d') as offload_date,
logset_logs.vehicle, jurisdiction, baselog_path, path,
baselog_trees.baselog_index_guid, new_location, logset_logs.log_set_name,
logset_logs.index_guid
from
(
SELECT SUBSTRING_INDEX(SUBSTRING_INDEX(path, '/', 7), '/', -1) as offload_date,
SUBSTRING_INDEX(SUBSTRING_INDEX(path, '/', 8), '/', -1) as vehicle,
SUBSTRING_INDEX(path, '/', 9) as baselog_path, index_guid,
path, log_set_name
FROM database_name.baselog_and_amendment_guid_to_path_mappings
) logset_logs
left join database_name.log_trees baselog_trees
ON baselog_trees.original_location = logset_logs.baselog_path
left join database_name.baselog_offload_location location
ON location.baselog_index_guid = baselog_trees.baselog_index_guid);
The query itself works because I was able to run it using a filter on log_set_name however that filter's condition will only work for less than 1% of the total records because one of the values for log_set_name has 2.2 million records in it which is the majority of the records. So there is nothing else I can use to break this query up into smaller chunks from what I can see. The problem is that the query is taking too long to run on the rest of the 2.2 million records and it ends up timing out after a few hours and then the transaction is rolled back and nothing is added to the new table for the 2.2 million records; only the 0.1 million records were able to be processed and that was because I could add a filter that said where log_set_name != 'value with the 2.2 million records'.
Is there a way to make this query more performant? Am I trying to do too many joins at once and perhaps I should populate the row's columns in their own individual queries? Or is there some way I can page this type of query so that MySQL executes it in batches? I already got rid of all my indexes on the log_set_logs table because I read that those will slow down inserts. I also jacked my RDS instance up to a db.r4.4xlarge write node. I am also using MySQL Workbench so I increased all of it's timeout values to their maximums giving them all nines. All three of these steps helped and were necessary in order for me to get the 1% of the records into the new table but it still wasn't enough to get the 2.2 million records without timing out. Appreciate any insights as I'm not adept to this type of bulk insert from a select.
'CREATE TABLE `log_set_logs` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`purged` tinyint(1) NOT NULL DEFAUL,
`baselog_path` text,
`baselog_index_guid` varchar(36) DEFAULT NULL,
`new_location` text,
`offload_date` date NOT NULL,
`jurisdiction` varchar(20) DEFAULT NULL,
`vehicle` varchar(20) DEFAULT NULL,
`index_guid` varchar(36) NOT NULL,
`path` text NOT NULL,
`log_set_name` varchar(60) NOT NULL,
`protected_by_retention_condition_1` tinyint(1) NOT NULL DEFAULT ''1'',
`protected_by_retention_condition_2` tinyint(1) NOT NULL DEFAULT ''1'',
`protected_by_retention_condition_3` tinyint(1) NOT NULL DEFAULT ''1'',
`protected_by_retention_condition_4` tinyint(1) NOT NULL DEFAULT ''1'',
`general_comments_about_this_log` text,
PRIMARY KEY (`id`)
) ENGINE=InnoDB AUTO_INCREMENT=1736707 DEFAULT CHARSET=latin1'
'CREATE TABLE `baselog_and_amendment_guid_to_path_mappings` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`path` text NOT NULL,
`index_guid` varchar(36) NOT NULL,
`log_set_name` varchar(60) NOT NULL,
PRIMARY KEY (`id`),
KEY `log_set_name_index` (`log_set_name`),
KEY `path_index` (`path`(42))
) ENGINE=InnoDB AUTO_INCREMENT=2387821 DEFAULT CHARSET=latin1'
...
'CREATE TABLE `baselog_offload_location` (
`baselog_index_guid` varchar(36) NOT NULL,
`jurisdiction` varchar(20) NOT NULL,
KEY `baselog_index` (`baselog_index_guid`),
KEY `jurisdiction` (`jurisdiction`)
) ENGINE=InnoDB DEFAULT CHARSET=latin1'
'CREATE TABLE `log_trees` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`baselog_index_guid` varchar(36) DEFAULT NULL,
`original_location` text NOT NULL, -- This is what I have to join everything on and since it's text I cannot index it and the largest value is above 255 characters so I cannot change it to a vachar then index it either.
`new_location` text,
`distcp_returncode` int(11) DEFAULT NULL,
`distcp_job_id` text,
`distcp_stdout` text,
`distcp_stderr` text,
`validation_attempt` int(11) NOT NULL DEFAULT ''0'',
`validation_result` tinyint(1) NOT NULL DEFAULT ''0'',
`archived` tinyint(1) NOT NULL DEFAULT ''0'',
`archived_at` timestamp NULL DEFAULT NULL,
`created_at` timestamp NULL DEFAULT CURRENT_TIMESTAMP,
`updated_at` timestamp NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
`dir_exists` tinyint(1) NOT NULL DEFAULT ''0'',
`random_guid` tinyint(1) NOT NULL DEFAULT ''0'',
`offload_date` date NOT NULL,
`vehicle` varchar(20) DEFAULT NULL,
PRIMARY KEY (`id`),
UNIQUE KEY `baselog_index_guid` (`baselog_index_guid`)
) ENGINE=InnoDB AUTO_INCREMENT=1028617 DEFAULT CHARSET=latin1'
baselog_offload_location has not PRIMARY KEY; what's up?
GUIDs/UUIDs can be terribly inefficient. A partial solution is to convert them to BINARY(16) to shrink them. More details here: http://localhost/rjweb/mysql/doc.php/uuid ; (MySQL 8.0 has similar functions.)
It would probably be more efficient if you have a separate (optionally redundant) column for vehicle rather than needing to do
SUBSTRING_INDEX(SUBSTRING_INDEX(path, '/', 8), '/', -1) as vehicle
Why JOIN baselog_offload_location? Three seems to be no reference to columns in that table. If there, be sure to qualify them so we know what is where. Preferably use short aliases.
The lack of an index on baselog_index_guid may be critical to performance.
Please provide EXPLAIN SELECT ... for the SELECT in your INSERT and for the original (slow) query.
SELECT MAX(LENGTH(original_location)) FROM .. -- to see if it really is too big to index. What version of MySQL are you using? The limit increased recently.
For the above item, we can talk about having a 'hash'.
"paging the query". I call it "chunking". See http://mysql.rjweb.org/doc.php/deletebig#deleting_in_chunks . That talks about deleting, but it can be adapted to INSERT .. SELECT since you want to "chunk" the select. If you go with chunking, Javier's comment becomes moot. Your code would be chunking the selects, hence batching the inserts:
Loop:
INSERT .. SELECT .. -- of up to 1000 rows (see link)
End loop
I have a table with a virtual generated column that concatenates five other columns (int and char) using CONCAT_WS(). This table contains 200-odd records and is never updated - it's just used as a lookup table. Recently, after months of untroubled processing, when I update records in a child table during which a SELECT is performed on this table, I sometimes see this error (ignore the "ITEM UPDATE FAILED" - that's me):
I am in development with a many changes every day, so it is impossible for me to determine if there is a correlating change. I have recently added "created" and "lastmodified" datetime fields to several tables with CURRENT_TIMESTAMP for DEFAULT or ON UPDATE, but not to this table.
Here's the table:
{EDIT} --- adding table definition:
CREATE TABLE `cpct_fixedfield` (
`id` int(11) UNSIGNED NOT NULL,
`name` varchar(256) NOT NULL,
`label` varchar(50) NOT NULL,
`field` int(11) NOT NULL,
`start` int(11) NOT NULL,
`rectype` int(11) NOT NULL ,
`mediatype` char(1) NOT NULL DEFAULT '' ,
`length` int(11) NOT NULL,
`userdefined` tinyint(1) NOT NULL,
`defaultval` varchar(5) NOT NULL,
`helpcode` varchar(10) NOT NULL,
`mandatory` varchar(2) NOT NULL ,
`idx` varchar(20) GENERATED ALWAYS AS (concat_ws('.',`field`,`rectype`,`mediatype`,`start`,`length`)) VIRTUAL NOT NULL)
ENGINE=InnoDB DEFAULT CHARSET=utf8mb4;
The length of the data in field never exceeds 11chars. I can view the entire table in pma or Mysql Workbench and the virtual field materialises in all records without complaint, which suggests to me that there is nothing wrong with either the expression for the virtual column or the data in the columns that expression draws on.
The error occurs in several contexts when I am updating a child table. All the updates occur in Stored Procedures/Functions. One section of code that seems to trigger the error is this:
SET idxvar = CONCAT_WS(".", SUBSTRING(tmpfldkey,3,1), rectype, ptype, position, "%") COLLATE utf8mb4_general_ci;
SELECT id INTO ffid FROM cpct_fixedfield WHERE idx LIKE idxvar AND idx != "0.0..6.2";
All the variables involved are varchars or ints. utf8mb4_general_ci is used throughout the database.
I cannot find any reference in MYSQL documentation to CONCAT or CONCAT_WS being unsafe, and none of the columns referenced has a default using a non-deterministic function. All the other questions I can find in this forum and elsewhere about this error have arisen because of the use of non-deterministic functions like CURRENT_TIMESTAMP() in the virtual field, or a component of the field.
I replaced the SELECT on the table with a (large) CASE statement and all was well, and in fact, after I did this then reverted to the SELECT I had no errors for many hours. But it just happened again (so I'm back to the case statement).
I have run out of ideas - I'm hoping someone has some knowledge/experience that can help.
Thanks
FINAL: I'm closing this question because there is no way for me to reproduce the condition and trying to find a similar occurrence has proven fruitless.
A work-around using perl is being implemented to circumvent the failure that only some mysql installs are experiencing.
UPDATE: I've had another case and verified that this query is returning 'OFF'
SELECT `VARIABLE_VALUE` FROM `information_schema`.`GLOBAL_VARIABLES` WHERE `VARIABLE_NAME` LIKE 'INNODB_STRICT_MODE'
I'm having a hard time tracking down a query issue...
This is part of an update script that I made to install new tables and import data from old ones - if they exist. This is from a procedure that is installed, and then called, by the update script:
In this snippet, the table bot_inventories is created, then a check is performed to see if the old schema table exists. If it does, the old 'base' data is then copied to the new table.
The suspect query is the 'inner' if near the bottom:
CREATE TABLE `bot_inventories` (
`inventories_index` INTEGER UNSIGNED NOT NULL AUTO_INCREMENT,
`bot_id` INT(11) UNSIGNED NOT NULL DEFAULT '0',
`slot_id` MEDIUMINT(7) UNSIGNED NOT NULL DEFAULT '0',
`item_id` INT(11) UNSIGNED NULL DEFAULT '0',
`inst_charges` TINYINT(3) UNSIGNED DEFAULT 0,
`inst_color` INT(11) UNSIGNED NOT NULL DEFAULT '0',
`inst_no_drop` TINYINT(1) UNSIGNED NOT NULL DEFAULT '0',
`inst_custom_data` TEXT NULL,
`ornament_icon` INT(11) UNSIGNED NOT NULL DEFAULT '0',
`ornament_id_file` INT(11) UNSIGNED NOT NULL DEFAULT '0',
`ornament_hero_model` INT(11) NOT NULL DEFAULT '0',
`augment_1` MEDIUMINT(7) UNSIGNED NOT NULL DEFAULT '0',
`augment_2` MEDIUMINT(7) UNSIGNED NOT NULL DEFAULT '0',
`augment_3` MEDIUMINT(7) UNSIGNED NOT NULL DEFAULT '0',
`augment_4` MEDIUMINT(7) UNSIGNED NOT NULL DEFAULT '0',
`augment_5` MEDIUMINT(7) UNSIGNED NOT NULL DEFAULT '0',
`augment_6` MEDIUMINT(7) UNSIGNED NOT NULL DEFAULT '0',
PRIMARY KEY (`inventories_index`),
KEY `FK_bot_inventories_1` (`bot_id`),
CONSTRAINT `FK_bot_inventories_1` FOREIGN KEY (`bot_id`) REFERENCES `bot_data` (`bot_id`)
) ENGINE=InnoDB;
IF ((SELECT COUNT(*) FROM `information_schema`.`TABLES` WHERE `TABLE_SCHEMA` = DATABASE() AND `TABLE_NAME` = 'botinventory') > 0) THEN
INSERT INTO `bot_inventories` (
`inventories_index`,
`bot_id`,
`slot_id`,
`item_id`,
`inst_charges`,
`inst_color`,
`inst_no_drop`,
`augment_1`,
`augment_2`,
`augment_3`,
`augment_4`,
`augment_5`
)
SELECT
bi.`BotInventoryID`,
bi.`BotID`,
bi.`SlotID`,
bi.`ItemID`,
bi.`charges`,
bi.`color`,
bi.`instnodrop`,
bi.`augslot1`,
bi.`augslot2`,
bi.`augslot3`,
bi.`augslot4`,
bi.`augslot5`
FROM `botinventory` bi
INNER JOIN `bot_data` bd
ON bi.`BotID` = bd.`bot_id`;
IF ((SELECT COUNT(*) FROM `information_schema`.`COLUMNS` WHERE `TABLE_SCHEMA` = DATABASE() AND `TABLE_NAME` = 'botinventory' AND `COLUMN_NAME` = 'augslot6') > 0) THEN
UPDATE `bot_inventories` bi
INNER JOIN `botinventory` bio
ON bi.`inventories_index` = bio.`BotInventoryID`
SET bi.`augment_6` = bio.`augslot6`
WHERE bi.`bot_id` = bio.`BotID`;
END IF;
RENAME TABLE `botinventory` TO `botinventory_old`;
END IF;
I have run this script myself dozens of times on both 'clean' and convertible (existing and populated botinventory table) databases with no problems.
The script executes properly on both MySQL v5.1.73 and MariaDB v10.0.22 (both win x86)
However, I've had two reported cases of failure surrounding the 'inner' if statement.
That check is in place to catch any 'customization' that may have been done by an admin and import their data.
The failed cases are reporting: "ERROR 1136 (21S01): Column count doesn't match value count at row 1"
I walked one of the admins through the manual installation of this update and found the problem to be with the 'if' query.
EDIT: It's acting like the query inside of the con check is firing off - defeating the purpose of the check itself (NULL is greater than '0'?)
(The walk-through admin is running MySQL v5.1.x as x86 on windows server 2012..the other is running MySQL v5.1.x as win (7?) x86.)
It seems to report that error when 'augslot6' does not exist - opposite behavior of what the check is in place to prevent.
I'm just not sure whether this is a syntax/methodology issue or an actual database code bug.
Searching for similar cases almost exclusively turns up results for column-entry count mis-matches, not why an if statement proves true with a false/null result...
Any suggestions?
EDIT: I am aware of what ANSI SQL code '21S01' indicates and how it could be interpreted to mean the missing augslot6 column. But, if so, why is the 'if' query proving 'true' when it should be 'false' on some systems?
EDIT: I wasn't able to perform the failing test myself as I don't have access to these systems.
Why do I get an error of the form:
Error in query: Duplicate entry '10' for key 1
...when doing an INSERT statement like:
INSERT INTO wp_abk_period (pricing_id, apartment_id) VALUES (13, 27)
...with 13 and 27 being valid id-s for existing pricing and apartment rows, and the table is defined as:
CREATE TABLE `wp_abk_period` (
`id` int(11) NOT NULL auto_increment,
`apartment_id` int(11) NOT NULL,
`pricing_id` int(11) NOT NULL,
`type` enum('available','booked','unavailable') collate utf8_unicode_ci default NULL,
`starts` datetime default NULL,
`ends` datetime default NULL,
`recur_type` enum('daily','weekly','monthly','yearly') collate utf8_unicode_ci default NULL,
`recur_every` char(3) collate utf8_unicode_ci default NULL,
`timedate_significance` char(4) collate utf8_unicode_ci default NULL,
`check_in_times` varchar(255) collate utf8_unicode_ci default NULL,
`check_out_times` varchar(255) collate utf8_unicode_ci default NULL,
PRIMARY KEY (`id`),
KEY `fk_period_apartment1_idx` (`apartment_id`),
KEY `fk_period_pricing1_idx` (`pricing_id`),
CONSTRAINT `fk_period_apartment1` FOREIGN KEY (`apartment_id`) REFERENCES `wp_abk_apartment` (`id`) ON DELETE NO ACTION ON UPDATE NO ACTION,
CONSTRAINT `fk_period_pricing1` FOREIGN KEY (`pricing_id`) REFERENCES `wp_abk_pricing` (`id`) ON DELETE NO ACTION ON UPDATE NO ACTION
) ENGINE=InnoDB AUTO_INCREMENT=10 DEFAULT CHARSET=utf8 COLLATE=utf8_unicode_ci
Isn't key 1 id in this case and having it on auto_increment sufficient for being able to not specify it?
Note: If I just provide an unused value for id, like INSERT INTO wp_abk_period (id, pricing_id, apartment_id) VALUES (3333333, 13, 27) it works fine, but then again, it is set as auto_increment so I shouldn't need to do this!
Note 2: OK, this is a complete "twilight zone" moment: so after running the query above with the huge number for id, things started working normally, no more duplicate entry errors. Can someone explain me WTF was MySQL doing to produce this weird behavior?
It could be that your AUTO_INCREMENT value for the table and the actual values in id column have got out of whack.
This might help:
Step 1 - Get Max id from table
select max(id) from wp_abk_period
Step 2 - Align the AUTO_INCREMENT counter on table
ALTER TABLE wp_abk_period AUTO_INCREMENT = <value from step 1 + 100>;
Step 3 - Retry the insert
As for why the AUTO_INCREMENT has got out of whack I don't know. Added auto_increment after data was in the table? Altered the auto_increment value after data was inserted into the table?
Hope it helps.
I had the same problem and here is my solution :
My ID column had a bad parameter. It was Tinyint, and MySql want to write a 128th line.
Sometimes, your problem you think the bigger you have is only a tiny parameter...
Late to the party, but I just ran into this tonight - duplicate key '472817' and the provided answers didn't help.
On a whim I ran:
repair table wp_abk_period
which output
Number of rows changed from 472816 to 472817
Seems like mysql had the row count wrong, and the issue went away.
My environment:
mysql Ver 14.14 Distrib 5.1.73, for Win64 (unknown)
Create table syntax:
CREATE TABLE `env_events` (
`tableId` int(11) NOT NULL AUTO_INCREMENT,
`deviceId` varchar(50) DEFAULT NULL,
`timestamp` int(11) DEFAULT NULL,
`temperature` float DEFAULT NULL,
`humidity` float DEFAULT NULL,
`pressure` float DEFAULT NULL,
`motion` int(11) DEFAULT NULL,
PRIMARY KEY (`tableId`)
) ENGINE=MyISAM AUTO_INCREMENT=528521 DEFAULT CHARSET=latin1
You can check the current value of the auto_increment with the following command:
show table status
Then check the max value of the id and see if it looks right. If not change the auto_increment value of your table.
When debugging this problem check the table name case sensitivity (especially if you run MySql not on Windows).
E.g. if one script uses upper case to 'CREATE TABLE my_table' and another script tries to 'INSERT INTO MY_TABLE'. These 2 tables might have different contents and different file system locations which might lead to the described problem.