I have a table
CREATE TABLE `partially_allowed` (
`service_area_code` int(2) DEFAULT NULL,
`number` varchar(20) DEFAULT NULL,
`phone_type` int(1) DEFAULT NULL,
`date_created` datetime DEFAULT NULL,
`preference` varchar(10) DEFAULT NULL,
KEY `idx_number` (`number`(4))
) ENGINE=InnoDB DEFAULT CHARSET=latin1
having 29,731,330 rows. I need to join this with another table
CREATE TABLE `data_from_files` (
`mobile` varchar(10) DEFAULT NULL,
`city` varchar(50) DEFAULT NULL,
`pin` varchar(10) DEFAULT NULL
) ENGINE=InnoDB DEFAULT CHARSET=latin1
having over 10,000,000 rows.
What is the best possible index for a fast select query?
Related
I need to find an open date time in the future and allocate that time in the calendar for that employee based on his/her worktimes Thanks in advance
I have a calendar table, employee(recourse) table ,employee working times and jobs (with Duration)
CREATE TABLE `calendar` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`Subject` varchar(200) DEFAULT NULL,
`DateField` date DEFAULT NULL,
`EndDate` date DEFAULT NULL,
`TimeField` time DEFAULT NULL,
`EndTime` time DEFAULT NULL,
`job_id` int(11) DEFAULT NULL,
`recourse_employee_id` int(11) DEFAULT NULL,
PRIMARY KEY (`id`)
) ENGINE=InnoDB AUTO_INCREMENT=1902 DEFAULT CHARSET=utf8;
CREATE TABLE `rc_resources_employee` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`name` varchar(255) DEFAULT NULL,
`categoryid` int(11) DEFAULT NULL,
`login_user_id` int(11) DEFAULT NULL,
PRIMARY KEY (`id`)
) ENGINE=InnoDB AUTO_INCREMENT=6 DEFAULT CHARSET=utf8;
CREATE TABLE `jobs` (
`Job_ID` int(11) NOT NULL AUTO_INCREMENT,
`Job_Subject` varchar(150) DEFAULT NULL,
`Job_Description` mediumtext,
`Job_Start_Date` date DEFAULT NULL,
`Job_End_Date` date DEFAULT NULL,
`Job_Duration` float DEFAULT NULL,
`Job_Start_Time` time DEFAULT NULL,
`Job_End_Time` time DEFAULT NULL,
`Job_employee` int(11) DEFAULT NULL,
PRIMARY KEY (`Job_ID`)
) ENGINE=InnoDB AUTO_INCREMENT=1875 DEFAULT CHARSET=utf8;
CREATE TABLE `work_hours` (
`wh_id` int(11) NOT NULL AUTO_INCREMENT,
`recourse_id` int(11) unsigned NOT NULL,
`wh_day_of_week` int(11) unsigned NOT NULL,
`wh_start_time` time NOT NULL,
`wh_end_time` time NOT NULL,
PRIMARY KEY (`wh_id`)
) ENGINE=InnoDB AUTO_INCREMENT=8 DEFAULT CHARSET=utf8;
I have following structure on mysql database:
sqlfiddle
What I want to do is:
To select DISTINCT industry from Company table
To insert into Industry table first and get auto incremented ID
With this ID to insert again into IndustryTranslation table and set "language"="en"
To insert Company's id and newly generated Industry's id into MapCompanyIndustry table
I know that it's not possible with one statement. But definitely it's possible with transaction. Can't figure out how to achieve this result with one transaction.
Any suggestions?
Schema
CREATE TABLE `Industry` (
`id` int(4) unsigned NOT NULL AUTO_INCREMENT,
PRIMARY KEY (`id`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8;
CREATE TABLE `IndustryTranslation` (
`industryID` int(4) unsigned NOT NULL,
`language` varchar(5) NOT NULL,
`name` varchar(255) NOT NULL,
`confirmed` tinyint(1) DEFAULT '0',
PRIMARY KEY (`industryID`,`language`),
KEY `language` (`language`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8;
CREATE TABLE `Company` (
`id` int(10) unsigned NOT NULL AUTO_INCREMENT,
`name` varchar(255) DEFAULT NULL,
`imageUri` varchar(255) DEFAULT NULL,
`countryID` int(3) unsigned DEFAULT NULL,
`phone` varchar(255) DEFAULT NULL,
`email` varchar(255) DEFAULT NULL,
`verified` tinyint(1) DEFAULT NULL,
`industry` varchar(255) DEFAULT NULL,
`headquarters` varchar(255) DEFAULT NULL,
`uri` varchar(255) DEFAULT NULL,
`createdAt` timestamp NULL DEFAULT CURRENT_TIMESTAMP,
`updatedAt` timestamp NULL DEFAULT NULL ON UPDATE CURRENT_TIMESTAMP,
PRIMARY KEY (`id`),
KEY `countryID` (`countryID`)
) ENGINE=InnoDB AUTO_INCREMENT=4004 DEFAULT CHARSET=utf8;
CREATE TABLE `MapCompanyIndustry` (
`companyID` int(10) unsigned NOT NULL,
`industryID` int(4) unsigned NOT NULL,
PRIMARY KEY (`companyID`,`industryID`),
KEY `industryID` (`industryID`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8;
I have the following simple join query
SELECT
count(*)
FROM
DBx.caseview p2015
INNER JOIN DBy.caseview p2014 ON p2015.casenumber=p2014.casenumber;
For some reason it just leaves MySQL hanging there for a lot of time until I get tired and cancel it. On the contrary, if run exactly the same code on MSSQL with the same data set the query takes a few seconds at most.
Is there a parameter that needs to be changed on MySQL to speed up this type of queries?
Here's my table in MySQL
CREATE TABLE `caseview` (
`ID` bigint(20) NOT NULL AUTO_INCREMENT,
`CASEID` varchar(18) DEFAULT NULL,
`CASENUMBER` int(10) DEFAULT NULL,
`ACCOUNTID` varchar(18) DEFAULT NULL,
`ACCOUNT` varchar(256) DEFAULT NULL,
`ASSETID` varchar(18) DEFAULT NULL,
`SAPPRODUCTGROUP` varchar(10) DEFAULT NULL,
`PRODUCT` varchar(128) DEFAULT NULL,
`FAMILY` varchar(128) DEFAULT NULL,
`CONTACTID` varchar(18) DEFAULT NULL,
`OWNERID` varchar(18) DEFAULT NULL,
`TYPE` varchar(128) DEFAULT NULL,
`PRIORITY` varchar(24) DEFAULT NULL,
`ORIGIN` varchar(24) DEFAULT NULL,
`SUBJECT` varchar(256) DEFAULT NULL,
`STATUS` varchar(24) DEFAULT NULL,
`LASTACTIVITY` varchar(1024) DEFAULT NULL,
`INITALDESCRIPTION` varchar(1024) DEFAULT NULL,
`CLOSEDDATE` datetime DEFAULT NULL,
`CREATEDDATE` datetime DEFAULT NULL,
`LASTMODIFIEDDATE` datetime DEFAULT NULL,
PRIMARY KEY (`ID`),
UNIQUE KEY `ID_UNIQUE` (`ID`)
) ENGINE=InnoDB AUTO_INCREMENT=122393 DEFAULT CHARSET=utf8;
There's two tables with exactly the same configuration as above, just different data
DBx.caseview p2015 has 197647 rows
DBy.caseview p2014 has 122392 rows
Create an Index for CASENUMBER field.
ALTER TABLE `caseview` ADD INDEX ( `CASENUMBER` ) ;
Give it a few seconds to create the index and run the query again.
I have the following SELECT query with table joins and it taking about a minute to return 6 records:
SELECT * FROM specimen, topography_index, morphology, specimen_image_lookup, image
WHERE
SUBSTRING(specimen.topography_index, 2, 2) = topography_index.topography_index_code
AND
morphology.morphology_code = specimen.snop_code
AND
specimen_image_lookup.specimen_fk = specimen.specimen_pk
AND
image.image_pk = specimen_image_lookup.image_fk
AND
specimen.topography_index, 2, 2) IN('".implode("','",$system)."')
Any ideas what I should here?
Table structures are:
CREATE TABLE `specimen` (
`specimen_pk` int(4) NOT NULL AUTO_INCREMENT,
`number` varchar(20) NOT NULL,
`unit_number` varchar(10) NOT NULL,
`topography_index` varchar(5) NOT NULL DEFAULT '',
`snop_axis` char(1) NOT NULL,
`snop_code` varchar(4) NOT NULL,
`example` int(2) NOT NULL,
`gender` char(1) NOT NULL,
`age` varchar(3) NOT NULL DEFAULT 'NA',
`clinical_history` text NOT NULL,
`specimen` text NOT NULL,
`macroscopic` text NOT NULL,
`microscopic` text NOT NULL,
`conclusion` text NOT NULL,
`comment` text NOT NULL,
`room` char(1) NOT NULL,
`position` varchar(8) NOT NULL,
`created` datetime NOT NULL,
`created_by` int(3) NOT NULL,
`updated` datetime NOT NULL,
`updated_by` int(3) NOT NULL,
PRIMARY KEY (`specimen_pk`),
FULLTEXT KEY `clinical_history` (`clinical_history`),
FULLTEXT KEY `specimen` (`specimen`),
FULLTEXT KEY `macroscopic` (`macroscopic`),
FULLTEXT KEY `microscopic` (`microscopic`),
FULLTEXT KEY `conclusion` (`conclusion`),
FULLTEXT KEY `comment` (`comment`)
) ENGINE=MyISAM DEFAULT CHARSET=utf8 AUTO_INCREMENT=500 ;
CREATE TABLE `topography_index` (
`topography_index_pk` int(3) NOT NULL AUTO_INCREMENT,
`topography_index_code` varchar(2) DEFAULT NULL,
`topography_index_nomen` text,
PRIMARY KEY (`topography_index_pk`)
) ENGINE=MyISAM DEFAULT CHARSET=utf8 AUTO_INCREMENT=138 ;
CREATE TABLE `specimen_image_lookup` (
`specimen_image_lookup_pk` int(8) NOT NULL AUTO_INCREMENT,
`specimen_fk` int(4) NOT NULL,
`image_fk` int(4) NOT NULL,
PRIMARY KEY (`specimen_image_lookup_pk`)
) ENGINE=MyISAM DEFAULT CHARSET=utf8 AUTO_INCREMENT=141 ;
CREATE TABLE `morphology` (
`morphology_pk` int(6) NOT NULL AUTO_INCREMENT,
`morphology_code` varchar(4) NOT NULL,
`morphology_nomen` varchar(120) NOT NULL,
PRIMARY KEY (`morphology_pk`)
) ENGINE=MyISAM DEFAULT CHARSET=utf8 AUTO_INCREMENT=2295 ;
CREATE TABLE `image` (
`image_pk` int(4) NOT NULL AUTO_INCREMENT,
`image_title` varchar(80) NOT NULL,
`image_description` text NOT NULL,
`image_thumbnail` varchar(100) NOT NULL,
`image_small` varchar(100) NOT NULL,
`image_large` varchar(100) NOT NULL,
`created` datetime NOT NULL,
`created_by` int(3) NOT NULL,
`updated` datetime NOT NULL,
`updated_by` int(3) NOT NULL,
PRIMARY KEY (`image_pk`)
) ENGINE=MyISAM DEFAULT CHARSET=utf8 AUTO_INCREMENT=138 ;
By performing a substring on specimen.topography_index, you're asking the database to perform that calculation on every row in the specimen table before finding if the value exists in topography_index. One way to address this is to store the actual integer value that will match with topography_index, rather than a string with that value embedded.
I have 3 tables in which I'm trying to preform joins on, and inserting the resulting data into another table. The query is taking anywhere between 15-30 mins depending on the dataset. The tables I'm selecting from and joining on are at least 25k records each but will quickly grow to be 500k+.
I tried adding indexes on the fields but still isn't helping that much. Are there any other things I can try or are joins on this scale just going to take this long?
Here is the query I'm trying to perform:
INSERT INTO audience.topitem
(runs_id, total_training_count, item, standard_index_value, significance, seed_count, nonseed_count, prod, model_type, level_1, level_2, level_3, level_4, level_5)
SELECT 5, seed_count + nonseed_count AS total_training_count,
ii.item, standard_index_value, NULL, seed_count, nonseed_count,
standard_index_value * seed_count AS prod, 'site', topic_L1, topic_L2, topic_L3, topic_L4, topic_L5
FROM audience.item_indexes ii
LEFT JOIN audience.usercounts uc ON ii.item = uc.item AND ii.runs_id = uc.runs_id
LEFT JOIN categorization.categorization at on ii.item = at.url
WHERE ii.runs_id = 5
Table: audience.item_indexes
CREATE TABLE `item_indexes` (
`item` varchar(1024) DEFAULT NULL,
`standard_index_value` float DEFAULT NULL,
`runs_id` int(11) DEFAULT NULL,
`model_type` enum('site','term','combo') DEFAULT NULL,
KEY `item_idx` (`item`(333))
) ENGINE=MyISAM DEFAULT CHARSET=utf8;
Table: audience.usercounts
CREATE TABLE `usercounts` (
`item` varchar(1024) DEFAULT NULL,
`seed_count` int(11) DEFAULT NULL,
`nonseed_count` int(11) DEFAULT NULL,
`significance` float(19,6) DEFAULT NULL,
`runs_id` int(11) DEFAULT NULL,
`model_type` enum('site','term','combo') DEFAULT NULL,
KEY `item_idx` (`item`(333))
) ENGINE=MyISAM DEFAULT CHARSET=utf8;
Table: audience.topitem
CREATE TABLE `topitem` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`total_training_count` int(11) DEFAULT NULL,
`item` varchar(1024) DEFAULT NULL,
`standard_index_value` float(19,6) DEFAULT NULL,
`significance` float(19,6) DEFAULT NULL,
`seed_count` int(11) DEFAULT NULL,
`nonseed_count` int(11) DEFAULT NULL,
`prod` float(19,6) DEFAULT NULL,
`cat_type` varchar(32) DEFAULT NULL,
`cat_level` int(11) DEFAULT NULL,
`conf` decimal(19,9) DEFAULT NULL,
`level_1` varchar(64) DEFAULT NULL,
`level_2` varchar(64) DEFAULT NULL,
`level_3` varchar(64) DEFAULT NULL,
`level_4` varchar(64) DEFAULT NULL,
`level_5` varchar(64) DEFAULT NULL,
`runs_id` int(11) DEFAULT NULL,
`model_type` enum('site','term','combo') DEFAULT NULL,
PRIMARY KEY (`id`)
) ENGINE=MyISAM AUTO_INCREMENT=825 DEFAULT CHARSET=utf8;
Table: categorization.categorization
CREATE TABLE `AT_categorization` (
`url` varchar(760) NOT NULL ,
`language` varchar(10) DEFAULT NULL,
`category` text,
`entity` text,
`source` varchar(255) DEFAULT NULL,
`topic_L1` varchar(45) NOT NULL DEFAULT '',
`topic_L2` varchar(45) NOT NULL DEFAULT '',
`topic_L3` varchar(45) NOT NULL DEFAULT '',
`topic_L4` varchar(45) NOT NULL DEFAULT '',
`topic_L5` varchar(45) NOT NULL DEFAULT '',
`last_refreshed` datetime DEFAULT NULL,
PRIMARY KEY (`url`,`topic_L1`,`topic_L2`,`topic_L3`,`topic_L4`,`topic_L5`),
UNIQUE KEY `inx_url` (`url`)
) ENGINE=MyISAM DEFAULT CHARSET=latin1;
If you add the following indexes, your query will run faster:
CREATE INDEX runs_idx ON audience.item_indexes (runs_id);
ALTER TABLE audience.usercounts
DROP INDEX item_idx,
ADD INDEX item_idx (runs_id, item(333));
Also, item_indexes is utf8, but AT_categorization is latin1, which keeps any indexes from being used. To address this issue, change AT_categorization to utf8:
ALTER TABLE AT_categorization CHARSET=utf8;
Lastly, for the AT_categorization table, the two indexes
PRIMARY KEY (`url`,`topic_L1`,`topic_L2`,`topic_L3`,`topic_L4`,`topic_L5`),
UNIQUE KEY `inx_url` (`url`)
are redundant. So you could DROP these, and simply have the url field be the primary key:
ALTER TABLE AT_categorization
DROP PRIMARY KEY,
DROP KEY `inx_url`,
ADD PRIMARY KEY (url);