I have a table with 8 columns, as shown below in the create statement.
Rows have to be unique, that is, no two rows can have the exact same value in each column. To this end I defined each column to be a Primary Key.
However, performing a select as show below takes extremely long as, i suppose, MySQL will have to scan each row to find results. As the table is pretty large, this takes a lot of time.
Do you have any suggestion how I could increase performance?
EDIT create statement:
CREATE TABLE `volatilities` (
`instrument` varchar(45) NOT NULL DEFAULT '',
`baseCurrencyId` int(11) NOT NULL,
`tenor` varchar(45) NOT NULL DEFAULT '',
`tenorUnderlying` varchar(45) NOT NULL DEFAULT '',
`strike` double NOT NULL DEFAULT '0',
`evalDate` date NOT NULL DEFAULT '0000-00-00',
`volatility` double NOT NULL DEFAULT '0',
`underlying` varchar(45) NOT NULL DEFAULT '',
PRIMARY KEY (`instrument`,`baseCurrencyId`,`tenor`,`tenorUnderlying`,`strike`,`evalDate`,`volatility`,`underlying`)) ENGINE=InnoDB DEFAULT CHARSET=utf8
Select statement:
SELECT evalDate,
max(case when strike = 0.25 then volatility end) as '0.25'
FROM niels_testdb.volatilities
WHERE
instrument = 'Swaption' and tenor = '3M'
and tenorUnderlying = '3M' and strike = 0.25
GROUP BY
evalDate
One of your requirements is that all the rows need to have unique values. So that is why you created the table with composite primary keys for all columns. But your table WOULD allow duplicated values for every column, as long as the rows themselves were unique.
Take a look at this sql fiddler post: http://sqlfiddle.com/#!2/85ae6
In there you'll see that the column instrument and tenor do have duplicate values.
I'd say you need to investigate more how unique keys work and what primary keys are.
My suggestion is to re-think your requirements and investigate what needs to be unique and why and have a different structure to support your decision. Composite primary keys, in this case, is not the way to go.
Related
I have a sharded database that is a a result from multi able sources, but some do not have data for all of the common columns. I shard base on source, so the whole resulting table current gets a column char(1) and all values set to ''. Once data is imported, it's read only with rare exceptions.
Is there a better way performance wise to tell mysql to always return a null or '' result for the given column? I tested char(0), but it forces a table scan. Prefer not to have to look up table layout to static select the column (SELECT '' as ip).
With results (ip)
CREATE TABLE `shard1` (
`id` int(11) NOT NULL,
`type` varchar(128) NOT NULL DEFAULT '',
`message` TEXT NOT NULL DEFAULT '',
`ip` varchar(39) NOT NULL
PRIMARY KEY (`id`),
KEY `type` (`type`),
KEY `ip` (`ip`,`type`)
) ENGINE=MyISAM DEFAULT CHARSET=utf8
Without results (ip)
CREATE TABLE `shard2` (
`id` int(11) NOT NULL,
`type` varchar(128) NOT NULL DEFAULT '',
`message` TEXT NOT NULL DEFAULT '',
`ip` char(1) NOT NULL
PRIMARY KEY (`id`),
KEY `type` (`type`),
KEY `ip` (`ip`,`type`)
) ENGINE=MyISAM DEFAULT CHARSET=utf8
Selects are typically like this
SELECT type,message,ip FROM shard1 WHERE id = 123;
SELECT message,ip FROM shard1 WHERE ip = '127.0.0.1';
SELECT message,ip FROM shard1 WHERE type = 'error' and ip = '127.0.0.1';
This is an overly simplified representation of the system. Smallest shard table is as little as 27 rows, and the biggest one is over 500m rows. Performance currently is acceptable, around 0.05 secs but I'd always love to make things more efficient.
Seems I figured out a possible solution using a view with a static empty string as ip. Never really used views before but the optimizer is smart enough in a view, but not in the main table.
View test was created on this statement
SELECT id,type,message,'' as ip FROM shard2
Optimizer is smart enough to know any search but '' will result in no results even WHERE ip LIKE '%anything%' is instant (0.001 sec) even without an index for ip on a 10m row table I'm testing with.
SELECT message,ip FROM test WHERE ip LIKE '%1';
Also to explain the statement the optimizer sees it's an "impossible WHERE" so the software effectively gets no results from the table.
id select_type table type possible_keys key key_len ref rows Extra
1 SIMPLE NULL NULL NULL NULL NULL NULL NULL Impossible WHERE
Learn something new everyday I guess.
This is a pretty basic question, but I'm confused by what I'm reading in various places. I have a simple table that doesn't contain a huge amount of data (less than 500 rows for any given db is typical) A typical query against this table looks like :
select system_fields.name from system_fields where system_fields.form_id=? and system_fields.field_id=?
My question is, should I have a separate index for form_id and one for field_id, or should I be creating an index on a combination of those two fields? I've never really done anything with multi-column indexes before.
CREATE TABLE IF NOT EXISTS `system_fields` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`field_id` int(11) NOT NULL,
`form_id` int(11) NOT NULL,
`name` varchar(50) NOT NULL,
`reference_field_id` varchar(1000) DEFAULT NULL,
PRIMARY KEY (`id`),
KEY `field_id` (`field_id`)
) ENGINE=MyISAM DEFAULT CHARSET=latin1 AUTO_INCREMENT=293 ;
If you are always going to query by these two fields, then add a multi-column index.
I'll also point out that if you're going to have < 500 rows in the table, your index may not even get used. Any performance difference with or without an index on a 500-row table will be negligible.
Here's a bit more (good) reading:
https://www.percona.com/blog/2014/01/03/multiple-column-index-vs-multiple-indexes-with-mysql-56/
I have simple categories table. Category can have parent category (par_cat column) or null if it is main category and with the same parent category there shouldn't be 2 or more categories with the same name or url.
Code for this table:
CREATE TABLE IF NOT EXISTS `categories` (
`id` int(10) unsigned NOT NULL,
`par_cat` int(10) unsigned DEFAULT NULL,
`lang` varchar(2) COLLATE utf8_unicode_ci NOT NULL DEFAULT 'pl',
`name` varchar(100) COLLATE utf8_unicode_ci NOT NULL,
`url` varchar(120) COLLATE utf8_unicode_ci NOT NULL,
`active` tinyint(3) unsigned NOT NULL DEFAULT '1',
`accepted` tinyint(3) unsigned NOT NULL DEFAULT '1',
`priority` int(10) unsigned NOT NULL DEFAULT '1000',
`entries` int(10) unsigned NOT NULL DEFAULT '0',
`created_at` timestamp NOT NULL DEFAULT '0000-00-00 00:00:00',
`updated_at` timestamp NOT NULL DEFAULT '0000-00-00 00:00:00'
) ENGINE=InnoDB DEFAULT CHARSET=utf8 COLLATE=utf8_unicode_ci AUTO_INCREMENT=3 ;
ALTER TABLE `categories`
ADD PRIMARY KEY (`id`),
ADD UNIQUE KEY `categories_name_par_cat_unique` (`name`,`par_cat`),
ADD UNIQUE KEY `categories_url_par_cat_unique` (`url`,`par_cat`),
ADD KEY `categories_par_cat_foreign` (`par_cat`);
ALTER TABLE `categories`
MODIFY `id` int(10) unsigned NOT NULL AUTO_INCREMENT,AUTO_INCREMENT=3;
ALTER TABLE `categories`ADD CONSTRAINT `categories_par_cat_foreign`
FOREIGN KEY (`par_cat`) REFERENCES `categories` (`id`);
The problem is that even if I have unique keys it doesn't work. If I try to insert into database 2 categories that have par_cat set to null and same name and url, those 2 categories can be inserted into database without a problem (and they shouldn't). However if I select for those categories other par_cat (for example 1 assuming category with id 1 exists), only first record will be inserted (and that's desired behaviour).
Question - how to handle this case? I read that:
A UNIQUE index creates a constraint such that all values in the index
must be distinct. An error occurs if you try to add a new row with a
key value that matches an existing row. This constraint does not apply
to NULL values except for the BDB storage engine. For other engines, a
UNIQUE index permits multiple NULL values for columns that can contain
NULL. If you specify a prefix value for a column in a UNIQUE index,
the column values must be unique within the prefix.
however if I have unique on multiple columns I expected it's not the case (only par_cat can be null, name and url cannot be null). Because par_cat references to id of the same table but some categories don't have parent category it should allow null values.
This works as defined by the SQL standard. NULL means unknown. If you have two records of par_cat = NULL and name = 'X', then the two NULLs are not regarded to hold the same value. Thus they don't violate the unique key constraint. (Well, one could argue that the NULLs still might mean the same value, but applying this rule would make working with unique indexes and nullable fields almost impossible, for NULL could as well mean 1, 2 or whatever other value. So they did well to define it such as they did in my opinion.)
As MySQL does not support functional indexes where you could have an index on ISNULL(par_cat,-1), name, your only option is to make par_cat a NOT NULL column with 0 or -1 or whatever for "no parent", if you want your constraints to work.
I see that this was asked in 2014.
However it is often requested from MySQL: https://bugs.mysql.com/bug.php?id=8173 and https://bugs.mysql.com/bug.php?id=17825 for example.
People can click on affects me to try and get attention from MySQL.
Since MySQL 5.7 we can now use the following workaround:
ALTER TABLE categories
ADD generated_par_cat INT UNSIGNED AS (ifNull(par_cat, 0)) NOT NULL,
ADD UNIQUE INDEX categories_name_generated_par_cat (name, generated_par_cat),
ADD UNIQUE INDEX categories_url_generated_par_cat (url, generated_par_cat);
The generated_par_cat is a virtual generated column, so it has no storage space. When a user inserts (or updates) then the unique indexes cause the value of generated_par_cat to be generated on the fly which is a very quick operation.
Just in case you come from Laravel...
This is Laravel's Migration version for Virtual Column to workaround the UNIQUE issue when one of the columns is NULL in value
$table->integer('generated_par_cat')->virtualAs('ifNull(par_cat, 0)');
$table->unique(['name', 'generated_par_cat'], 'name_par_cat_unique');
I have two tables. One is called map_life, and the second one is called scripts. The map_life table has a lot of rows, that are identified by a column called lifeid. The rows at the table scripts are identified by the column objectid. I want to create a query that gets all the rows from the table map_life and also adds the column scriptfrom scripts table if lifeidmatches objectid, and that the objecttype is npc.
I created the following query.
SELECT id
,lifeid
,x_pos
,y_pos
,foothold
,min_click_pos
,max_click_pos
,respawn_time
,flags
,script.script
FROM map_life life
LEFT JOIN scripts script
ON script.objectid = life.lifeid
AND script.script_type = 'npc'
However, that query takes a lot of time. Any way I can tune it? Thanks.
EDIT: I have ran EXPLAIN command, there are the results.
"id","select_type","table","type","possible_keys","key","key_len","ref","rows","Extra"
1,"SIMPLE","life","ALL","","","","",47600,""
1,"SIMPLE","script","ref","PRIMARY","PRIMARY","1","const",1834,"Using where"
EDIT 2: Here are the create statmenets of each table.
map_life
DROP TABLE IF EXISTS `mcdb`.`map_life`;
CREATE TABLE `mcdb`.`map_life` (
`id` bigint(21) unsigned NOT NULL AUTO_INCREMENT,
`mapid` int(11) NOT NULL,
`life_type` enum('npc','mob','reactor') NOT NULL,
`lifeid` int(11) NOT NULL,
`life_name` varchar(50) DEFAULT NULL COMMENT 'For reactors, specifies a handle so scripts may interact with them; for NPC/mob, this field is useless',
`x_pos` smallint(6) NOT NULL DEFAULT '0',
`y_pos` smallint(6) NOT NULL DEFAULT '0',
`foothold` smallint(6) NOT NULL DEFAULT '0',
`min_click_pos` smallint(6) NOT NULL DEFAULT '0',
`max_click_pos` smallint(6) NOT NULL DEFAULT '0',
`respawn_time` int(11) NOT NULL DEFAULT '0',
`flags` set('faces_left') NOT NULL DEFAULT '',
PRIMARY KEY (`id`,`lifeid`) USING BTREE,
KEY `lifetype` (`mapid`,`life_type`)
) ENGINE=InnoDB AUTO_INCREMENT=47557 DEFAULT CHARSET=latin1;
scripts
DROP TABLE IF EXISTS `mcdb`.`scripts`;
CREATE TABLE `mcdb`.`scripts` (
`script_type` enum('npc','reactor','quest','item','map','map_enter','map_first_enter') NOT NULL,
`helper` tinyint(3) NOT NULL DEFAULT '-1' COMMENT 'Represents the quest state for quests, and the index of the script for NPCs (NPCs may have multiple scripts).',
`objectid` int(11) NOT NULL DEFAULT '0',
`script` varchar(40) NOT NULL DEFAULT '',
PRIMARY KEY (`script_type`,`helper`,`objectid`)
) ENGINE=InnoDB DEFAULT CHARSET=latin1 COMMENT='Lists all the scripts that belong to NPCs/reactors/etc. ';
You should probably add an index to the 'script_type' field depending on the type. If it's not using a type that can be indexed, you should change the type if possible and index
Here is a link that discusses more about indexes with MySQL, http://dev.mysql.com/doc/refman/5.0/en/mysql-indexes.html
Your primary key on scripts is:
PRIMARY KEY (`script_type`,`helper`,`objectid`)
The order of multi-column keys is important.
From the docs:
Any index that does not span all AND levels in the WHERE clause is not
used to optimize the query. In other words, to be able to use an
index, a prefix of the index must be used in every AND group.
Your primary key on scripts does include both the script_type and objectid columns, which are both used in the join's ON clause:
ON script.objectid = life.lifeid
AND script.script_type = 'npc'
but the primary key also includes the helper column between those two, so MySQL can only use the primary key index for searching using the first column (script_type).
So, for every join, MySQL must search through all scripts records where script_type is 'npc' to find the particular objectid record to join on.
MySQL could full utilize the primary key index if your ON clause included all three columns like this:
ON script.objectid = life.lifeid
AND script.helper = 1
AND script.script_type = 'npc'
If you often query the scripts table without specifying the helper column, consider changing the order of the columns in the primary key to put the helper column last:
PRIMARY KEY (`script_type`,`objectid`,`helper`)
Then, your original ON clause is appropriate for the index because the index prefix includes all of the columns in your predicate (script_type,objectid):
ON script.objectid = life.lifeid
AND script.script_type = 'npc'
Alternatively, add an additional index with just the two columns mentioned in the ON clause:
KEY `scrypt_type_objectid` (`script_type`,`objectid`)
Simplifying the database/table structure i have a situation with two tables where we store 'items' and item properties (the relation between the two is 1-N)
I'm trying to optimize the following query, which fetches latest items being in the hotdeals section. To do that we have item_property table which stores items sections along with many other item metadata
NOTE: table structure can't be changed to optimize the query, ie: we can't simply add the section as a column in the item table as we can have unlimited amount of sections for each item.
Here's the Structure of both tables:
CREATE TABLE `item` (
`iditem` int(11) unsigned NOT NULL AUTO_INCREMENT,
`itemname` varchar(200) NOT NULL,
`desc` text NOT NULL,
`ok` int(11) NOT NULL DEFAULT '10',
`date_created` datetime NOT NULL,
PRIMARY KEY (`iditem`)
) ENGINE=MyISAM DEFAULT CHARSET=utf8;
CREATE TABLE `item_property` (
`iditem` int(11) unsigned NOT NULL,
`proptype` varchar(64) NOT NULL DEFAULT '',
`propvalue` varchar(200) NOT NULL DEFAULT '',
KEY `iditem` (`iditem`,`proptype`)
) ENGINE=MyISAM DEFAULT CHARSET=utf8;
And here's the query:
SELECT *
FROM item
JOIN item_property ON item.iditem=item_property.iditem
WHERE
item.ok > 70
AND item_property.proptype='section'
AND item_property.propvalue = 'hotdeals'
ORDER BY item.date_created desc
LIMIT 20
Which would be the best indexes to optimize this query?
Right now the optimizer (Explain) will use temporary and filesort, processing a Ton of rows (the size of the join)
Tables are both MyIsam at the moment, but can be changed to InnoDB if its really necessary to optimize the queries
Thanks
What is the type of item_property.idOption and item_property.type columns?
If they contain a limited number of options - make them ENUM (if they are not already). Enum values are indexed automatically.
And (of course) you should have item_property.iditem and item.date_created columns indexed also. This will increase the size of the tables, but will considerably fasten the queries that join and sort by these fields.
A note about data correctness:
One of the big benefits of a NOT NULL is to prevent your program from creating a row that doesn't have all columns properly specified. Having a DEFAULT renders that useless.
Is it ever OK to have a blank proptype or propvalue? What does a blank in those fields mean? If it's OK to not have a proptype set, then remove the NOT NULL constraint. If you must always have a proptype set, then having DEFAULT '' will not save you from the case of inserting into the row but forgetting to set proptype.
In most cases, you want either NOT NULL or DEFAULT 'something' on your columns, but not both.