Related
I am trying to select one property per state from a table with 1,000,000 properties. I am trying
select * from properties
where latitude is not null and longitude is not null
group by property_state;
But the query takes 3 seconds. I have an index on latitude and longitude, and an index on state. I tried adding a third index on all 3 columns, but that did not help.
Any ideas?
Here is the create table code, if that helps (I removed new index that did not help)
CREATE TABLE `t_national_comps` (
`deal_Id` INT(11) NULL DEFAULT NULL,
`nc_id` INT(11) NOT NULL AUTO_INCREMENT,
`property_id` INT(15) NULL DEFAULT NULL,
`reonomy_property_id` VARCHAR(50) NULL DEFAULT NULL,
`reonomy_url` VARCHAR(80) NULL DEFAULT NULL,
`confidence` FLOAT NULL DEFAULT NULL,
`latitude` DECIMAL(11,8) NULL DEFAULT NULL,
`longitude` DECIMAL(11,8) NULL DEFAULT NULL,
`prop_key` VARCHAR(255) NULL DEFAULT NULL,
`fmt_address` VARCHAR(255) NULL DEFAULT NULL,
`property_street_number` VARCHAR(20) NULL DEFAULT NULL,
`property_street_name` VARCHAR(40) NULL DEFAULT NULL,
`property_street_mode` VARCHAR(20) NULL DEFAULT NULL,
`property_city` VARCHAR(40) NULL DEFAULT NULL,
`property_state` VARCHAR(10) NULL DEFAULT NULL,
`property_zip` VARCHAR(10) NULL DEFAULT NULL,
`property_zip4` VARCHAR(10) NULL DEFAULT NULL,
`municipality` VARCHAR(40) NULL DEFAULT NULL,
`property_class_id` VARCHAR(15) NULL DEFAULT NULL,
`std_land_use_code` VARCHAR(15) NULL DEFAULT NULL,
`sale_doc_num` VARCHAR(30) NULL DEFAULT NULL,
`mortgage_doc_num` VARCHAR(30) NULL DEFAULT NULL,
`mortgage_date` DATE NULL DEFAULT NULL,
`lender` VARCHAR(100) NULL DEFAULT NULL,
`bank_id` INT(11) NULL DEFAULT NULL,
`loan_amount` BIGINT(15) NULL DEFAULT NULL,
`maturity_date` DATE NULL DEFAULT NULL,
`rate` VARCHAR(20) NULL DEFAULT NULL,
`sale_date` DATE NULL DEFAULT NULL,
`curr_sale_contract_date` DATE NULL DEFAULT NULL,
`curr_sale_document_type` VARCHAR(20) NULL DEFAULT NULL,
`sale_price` BIGINT(22) NULL DEFAULT NULL,
`curr_sale_buyer1_full_name` VARCHAR(60) NULL DEFAULT NULL,
`curr_sale_buyer2_full_name` VARCHAR(60) NULL DEFAULT NULL,
`reported_owner` VARCHAR(60) NULL DEFAULT NULL,
`mailing_address` VARCHAR(500) NULL DEFAULT NULL,
`curr_sale_seller1_full_name` VARCHAR(60) NULL DEFAULT NULL,
`curr_sale_seller2_full_name` VARCHAR(60) NULL DEFAULT NULL,
`sq_footage` VARCHAR(10) NULL DEFAULT NULL,
`resi_units` VARCHAR(10) NULL DEFAULT NULL,
`commercial_units` VARCHAR(10) NULL DEFAULT NULL,
`num_floors` VARCHAR(10) NULL DEFAULT NULL,
`num_buildings` VARCHAR(10) NULL DEFAULT NULL,
`price_per_sq_ft` INT(11) NULL DEFAULT NULL,
`price_per_unit` INT(11) NULL DEFAULT NULL,
`property_type_id` INT(11) NULL DEFAULT NULL,
`property_type` VARCHAR(60) NULL DEFAULT NULL,
`long_lat_point` POINT NULL DEFAULT NULL,
PRIMARY KEY (`nc_id`),
INDEX `t_national_comps_latitude_longitude_index` (`latitude`, `longitude`),
INDEX `t_national_comps_property_city_index` (`property_city`),
INDEX `t_national_comps_property_state_index` (`property_state`),
INDEX `t_national_comps_sale_date_index` (`sale_date`),
INDEX `t_national_comps_point_index` (`long_lat_point`(25)),
INDEX `t_national_comps_reonomy_id_index` (`reonomy_property_id`),
INDEX `mailing_address_index` (`mailing_address`),
INDEX `mortgage_date_index` (`mortgage_date`),
INDEX `t_national_comps_lender_index` (`lender`),
INDEX `bank_id_index` (`bank_id`),
INDEX `street_num_and_zip` (`property_street_number`, `property_zip`)
);
EDIT
The reason I did not aggregate anything in the query, is because I have nothing to aggregate. I know that is not the primary use of group by, but it is commonly used as such, just to get one of each record.
I was able to speed up the query by forcing the use of an index on all 3 columns, like
select latitude, longitude, property_street_number, property_street_name,
property_city, property_state, property_zip from properties
USE INDEX (lat_long_state_index)
where latitude is not null and longitude is not null
group by property_state;
but I am still looking for more optimization.
Thanks to all for your help.
Group By
I am not convinced Group By should be used in such a way, although internally MySQL could be smart enough ( I am not sure) to use a Distinct when it sees a group by with no aggregation, but i don't think it's a correct way to use Group By.
Index
MySQL uses one index per table per query and it will only pick one, so before you have the three columns, it's correct to pick the index with property_state because MySQL generally won't use index for not equal condition.
You can do a EXPLAIN compare the query before and after force index. MySQL optimizer believe the single column index is better.
Too many indexes will also add overhead to insert. After you have the three columns index, you can actually delete the property_state index because it's covered by the three columns index(leftmost). Your future query will surely use the new index you created.
My suggestion is do not use select *. Instead of using select * use select id, .... This will definitely reduce your execution time.
Here is the sql that I am trying to insert and I get a error.
insert into instruments (symbol,exchange,FullName,IPOYear,Sector,Industry)
values('PIH','Nasdaq','1347 Property Insurance Holdings, Inc.','Finance','Property-Casualty Insurers','http://www.nasdaq.com/symbol/pih')
here is my ddl and I don't find anything wrong with my sql.
CREATE TABLE `instruments` (
`id` INT(10) NOT NULL AUTO_INCREMENT,
`symbol` VARCHAR(100) NOT NULL,
`exchange` VARCHAR(50) NOT NULL,
`FullName` VARCHAR(100) NULL DEFAULT NULL,
`IPOYear` VARCHAR(10) NULL DEFAULT NULL,
`Sector` VARCHAR(20) NULL DEFAULT NULL,
`Industry` VARCHAR(100) NULL DEFAULT NULL,
`LastUpdated` DATETIME NULL DEFAULT NULL,
PRIMARY KEY (`id`)
Sector VARCHAR(20) NULL DEFAULT NULL
You limited Sector to 20 characters and inserted 26.
Make it like this Sector VARCHAR(30) NULL DEFAULT NULL and it should work
I need to search records in the table which contains millions of records. I have recently updated Mysql version from 5.1 to 5.6.
I was using like in query which was taking around 15 sec to 30 sec.
Currently I have modified query to use MATCH... AGAINST feature of mysql 5.6 INNODB. My query goes like this.
SELECT JOB.IDJOB, JOB.IDEMPLOYER .....
FROM JOB
WHERE ( ( JOB.ITJOBSTARTTYPE=2 AND JOB.DTPLANNEDEND >= '2015-07-06' )
OR ( JOB.ITJOBSTARTTYPE=1
AND ( ( JOB.ITJOBENDTYPE=2 ) OR ( JOB.DTJOBEND>='2015-07-06') )
)
OR ( JOB.ITJOBSTARTTYPE=3
AND ( (JOB.DTJOBEND >='2015-07-06') OR (JOB.ITJOBENDTYPE=2) )
)
)
AND MATCH(VCJOBTITLE, LVJOBCOMPANYDESCRIPTION, VCCOMPANYNAME,
VCSALARYDESC, VCCITY, VCQUALIFICATIONREQUIRED,VCJOBREFERENCE,
LVJOBKEYWORDS )
AGAINST ('test' IN NATURAL LANGUAGE MODE)
ORDER BY JOB.ITJOBBAND ASC,
JOB.VCRANKING DESC,
JOB.IDJOB DESC,
JOB.FJOBWEIGHT ASC
LIMIT 50;
Which takes 80- 120 seconds (3X slower )
If I remove ORDER BY it will load in 6 to 10 sec.
Is their any setting Or performance tunning we can apply here?
Edit: using show create table JOB;
CREATE TABLE `JOB` (
`idJob` int(11) NOT NULL AUTO_INCREMENT,
`idEmployer` int(10) DEFAULT NULL,
`vcJobTitle` varchar(255) NOT NULL,
`lvJobCompanyDescription` text,
`idBasket` int(10) DEFAULT NULL,
`dtActualgolive` datetime DEFAULT NULL COMMENT 'whenever job status become online',
`boImmediatelygolive` int(11) NOT NULL,
`dtRequestedgolive` date DEFAULT NULL,
`dtActualend` datetime DEFAULT NULL COMMENT 'when job is set to be archieved',
`dtPlannedend` date NOT NULL COMMENT 'when posting a job this is calculated. batch will look at this date. date on which regular jobs will be archievd, or date on which internship jobs will be changed to listing',
`dtRequestedend` date DEFAULT NULL COMMENT 'exact date on which employer wants jobs to be archieved',
`dtApplicationdeadline` date DEFAULT NULL COMMENT 'job will be converted to listing on this date',
`boDurationinweeks` int(11) NOT NULL COMMENT 'whether employer gave duration in weeks or not',
`itDurationweeks` int(2) DEFAULT NULL,
`boIsoncredit` int(11) NOT NULL,
`dtCreditend` date DEFAULT NULL,
`fJobbaseprice` float NOT NULL,
`fJobpriceafteradminitemdiscount` float NOT NULL COMMENT 'price after admin item discount',
`fJobpriceafterpromodiscount` float NOT NULL COMMENT 'price after promo discount',
`vcPromotioncode` varchar(50) DEFAULT NULL,
`vcCompanyname` varchar(100) NOT NULL,
`vcSalarydesc` varchar(255) NOT NULL,
`vcCity` varchar(100) NOT NULL,
`vcCounty` varchar(120) DEFAULT NULL,
`vcQualificationrequired` text,
`boWorkfromhome` int(11) NOT NULL,
`boResidential` int(11) NOT NULL,
`boIndoor` int(11) NOT NULL,
`boOutdoor` int(11) NOT NULL,
`boIndoorandoutdoor` int(11) NOT NULL,
`itJobstarttype` int(11) DEFAULT NULL COMMENT 'date,immidiate,always recruiting',
`dtJobstart` date DEFAULT NULL,
`itJobendtype` int(11) DEFAULT NULL COMMENT 'date,ongoing',
`dtJobend` date DEFAULT NULL,
`dtFeaturedstart` date DEFAULT NULL,
`dtFeaturedend` date DEFAULT NULL,
`itFulltimeparttime` int(11) NOT NULL COMMENT 'fulltime,parttime,both',
`boEveningtime` int(11) NOT NULL,
`boDaytime` int(11) NOT NULL,
`boWeekend` int(11) NOT NULL,
`boNewsletter` int(11) NOT NULL,
`vcApplyemail` varchar(500) DEFAULT NULL,
`vcApplyphone` varchar(200) DEFAULT NULL,
`vcApplyaddress` varchar(100) DEFAULT NULL,
`vcApplyURL` varchar(500) DEFAULT NULL,
`boRequirephone` int(11) DEFAULT NULL,
`boRequireaddress` int(11) DEFAULT NULL,
`boRequirecv` int(11) DEFAULT NULL,
`boFeatured` int(11) NOT NULL,
`blLogo` longblob,
`itJobstatus` int(1) DEFAULT NULL COMMENT 'online,offline',
`itEmailedtofriendcount` int(5) DEFAULT NULL COMMENT 'keeps count of how many times users clicked on link email this to friend',
`itDeadlinetype` int(1) DEFAULT NULL,
`idInternshiptypemaster` int(11) DEFAULT NULL,
`vcLengthofscheme` varchar(100) DEFAULT NULL,
`dtInternshiplistingend` date DEFAULT NULL COMMENT 'date when internship job will archieve',
`itJobband` int(2) DEFAULT NULL,
`boGraduate` int(11) NOT NULL,
`boInternship` int(11) NOT NULL,
`boGaptemp` int(11) NOT NULL,
`boParttimeholiday` int(11) NOT NULL,
`boEntrylevel` int(11) NOT NULL DEFAULT '0',
`boHundredPercentDiscountApplicable` int(11) NOT NULL,
`vcContactdetails` varchar(100) DEFAULT NULL,
`vcJobreference` varchar(100) DEFAULT NULL COMMENT 'for importing totaljobs feeds',
`vcJoburlparam` varchar(100) DEFAULT NULL COMMENT 'for importing totaljobs url params',
`dtPutInCurrentBand` datetime DEFAULT NULL COMMENT 'date when job is online and put in band 3',
`itInitcountycount` int(2) NOT NULL,
`created_at` datetime DEFAULT NULL,
`updated_at` datetime DEFAULT NULL,
`lvJobkeywords` text,
`fJobweight` decimal(10,2) DEFAULT '1.00' COMMENT 'Holds the job weight ranging between 0 to 1 (1 being the highest weight)',
`dtReinstatedOn` datetime DEFAULT NULL,
`boIncludeiniframe` int(11) DEFAULT NULL,
`boIsAdvertFeatured` int(11) DEFAULT NULL,
`itincludexml` int(1) DEFAULT '1',
`boAddtojobalert` int(11) DEFAULT '1',
`vcRanking` int(11) DEFAULT '999999',
`itIncludeUkptj` int(11) DEFAULT '1',
PRIMARY KEY (`idJob`),
KEY `JOB_I_1` (`idEmployer`,`itJobstatus`),
KEY `JOB_I_2` (`itJobstatus`,`boGaptemp`,`dtJobstart`),
KEY `JOB_I_3` (`itJobstatus`,`boGraduate`,`dtJobstart`),
KEY `JOB_I_4` (`itJobstatus`,`boInternship`,`dtJobstart`),
KEY `JOB_I_5` (`itJobstatus`,`boParttimeholiday`,`dtJobstart`),
KEY `FI_JOB_idOrder_ORDER_idOrder` (`idBasket`),
KEY `FI_JOB_vcPromotioncode_PROMOTION_vcPromotioncode` (`vcPromotioncode`),
KEY `FI_JOB_idInternshiptype_ITTM_idInternshiptypemaster` (`idInternshiptypemaster`),
KEY `JOB_I_6` (`created_at`),
KEY `boEntrylevel` (`boEntrylevel`),
KEY `itJobband` (`itJobband`),
FULLTEXT KEY `index_ft_search` (`vcJobTitle`,`lvJobCompanyDescription`,`vcCompanyname`,`vcSalarydesc`,`vcCity`,`vcQualificationrequired`,`vcJobreference`,`lvJobkeywords`),
CONSTRAINT `fk_JOB_idEmployer_EMPLOYER_idEmployer` FOREIGN KEY (`idEmployer`) REFERENCES `EMPLOYER` (`idEmployer`),
CONSTRAINT `fk_JOB_idInternshiptype_ITTM_idInternshiptypemaster` FOREIGN KEY (`idInternshiptypemaster`) REFERENCES `INTERNSHIPTYPEMASTER` (`idInternshiptypemaster`),
CONSTRAINT `fk_JOB_idOrder_ORDER_idOrder` FOREIGN KEY (`idBasket`) REFERENCES `BASKET` (`idBasket`),
CONSTRAINT `fk_JOB_vcPromotioncode_PROMOTION_vcPromotioncode` FOREIGN KEY (`vcPromotioncode`) REFERENCES `PROMOTION` (`vcPromotioncode`)
) ENGINE=InnoDB AUTO_INCREMENT=1739324 DEFAULT CHARSET=latin1 |
Edit:
Without sort:
With Sort:
I have an Users Table with a staff column and a remove column.
I don't want to show Users that have the remove column with the value of 1
SELECT *
FROM Users
WHERE
Users.staff = 1
AND Users.remove != 1
I don't get an error but my code doesn't work.
this is the schema
CREATE TABLE `Users` (
`userId` int(11) unsigned NOT NULL AUTO_INCREMENT,
`fullName` varchar(50) DEFAULT NULL,
`firstName` varchar(25) NOT NULL DEFAULT '',
`lastName` varchar(25) NOT NULL DEFAULT '',
`address` varchar(50) NOT NULL DEFAULT '',
`city` varchar(25) DEFAULT NULL,
`state` char(2) DEFAULT NULL,
`zipCode` varchar(25) DEFAULT NULL,
`email` varchar(50) NOT NULL DEFAULT '',
`cellPhone` varchar(15) DEFAULT NULL,
`birthDate` date NOT NULL,
`creditCard` varchar(250) NOT NULL DEFAULT '',
`subscriptionStarted` date NOT NULL,
`subscriptionEnded` date NOT NULL,
`basicPlan` tinyint(1) DEFAULT NULL,
`standardPlan` tinyint(1) DEFAULT NULL,
`premiumPlan` tinyint(1) DEFAULT NULL,
`remove` tinyint(1) DEFAULT NULL,
`staff` tinyint(1) DEFAULT NULL,
`admin` tinyint(1) DEFAULT NULL,
`systemAdmin` tinyint(1) DEFAULT NULL,
`edited` datetime DEFAULT NULL,
`created` datetime NOT NULL,
PRIMARY KEY (`userId`)
) ENGINE=InnoDB AUTO_INCREMENT=20 DEFAULT CHARSET=latin1;
both work for me Boss.
create table users
( staff int not null,
remove int not null
);
insert users (staff,remove) values (1,0);
insert users (staff,remove) values (1,1);
insert users (staff,remove) values (2,0);
insert users (staff,remove) values (2,1);
SELECT *
FROM Users
WHERE
Users.staff = 1
AND Users.remove != 1
SELECT *
FROM Users
WHERE
Users.staff = 1
AND Users.remove <> 1
Edit due to nullability:
SELECT * FROM Users
WHERE staff=1
and remove <> 1 or remove is null
SELECT * FROM Users
WHERE staff=1
and remove is not null
I have a sql-statement which correctly fetches data from one table. However, I need to fetch in the same way from n number of tables named table_n. All of these tables contain a 3-field primary key of projid, docid and revnr. I need to return n as docType as well, to differentiate the tables. The result will be sorted by projid and/or docid.
I tried sorting all the outputs from the different queries in PHP but it was way too slow (at least a few seconds on a 3MB database). I'm convinced MySQL/MSSQL will do it faster.
This is my current query:
SELECT a.* FROM `table_1` a
INNER JOIN (SELECT docid,
Max(revnr) max_val
FROM `table_1`
WHERE ( projid = something )
GROUP BY docid) b
ON a.docid = b.docid
AND a.revnr = b.max_val ORDER BY docid DESC
My current query gets the rows with highest revnr for each docid and projid.
I'm developing on MySQL but I need it to work on MSSQL as well. A general SQL solution would be great.
Thanks!
EDIT: Table schemas of the tables i currently have:
CREATE TABLE IF NOT EXISTS `table_1` (
`projid` int(11) NOT NULL,
`docid` int(11) NOT NULL,
`revnr` int(11) NOT NULL,
`revname` varchar(64) NOT NULL,
`signedOn` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
`sign` int(11) NOT NULL,
`ritningsnr` varchar(128) NOT NULL,
`moment` varchar(256) NOT NULL,
`omrade` varchar(256) NOT NULL,
`start` datetime NOT NULL,
`stop` datetime NOT NULL,
`extTodo` int(11) NOT NULL,
PRIMARY KEY (`projid`,`docid`,`revnr`)
) ENGINE=MyISAM DEFAULT CHARSET=latin1 COMMENT='egenkontroll';
CREATE TABLE IF NOT EXISTS `table_2` (
`projid` int(11) NOT NULL,
`docid` int(11) NOT NULL,
`revnr` int(11) NOT NULL,
`revname` varchar(64) NOT NULL,
`signedOn` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
`sign` int(11) NOT NULL,
`extWater` int(11) NOT NULL,
`extRisk` int(11) NOT NULL,
`extSystem` int(11) NOT NULL,
`extHelp` int(11) NOT NULL,
`extProvtryck` int(11) NOT NULL,
`extDoc` int(11) NOT NULL,
`extEgenkontroll` int(11) NOT NULL COMMENT 'exttabell',
`extOther` int(11) NOT NULL,
`extMontorer` int(11) NOT NULL,
PRIMARY KEY (`projid`,`docid`,`revnr`)
) ENGINE=MyISAM DEFAULT CHARSET=latin1 COMMENT='arbetsberedning';
CREATE TABLE IF NOT EXISTS `table_3` (
`projid` int(11) NOT NULL,
`docid` int(11) NOT NULL,
`revnr` int(11) NOT NULL,
`revname` varchar(64) NOT NULL,
`adress` varchar(256) NOT NULL,
`pipesMark` tinyint(1) NOT NULL,
`pipesKulvert` tinyint(1) NOT NULL,
`pipesBasement` tinyint(1) NOT NULL,
`pipesVaning` tinyint(1) NOT NULL,
`pipesIngjutna` tinyint(1) NOT NULL,
`ledningTappvatten` tinyint(1) NOT NULL,
`ledningVarmevatten` tinyint(1) NOT NULL,
`ledningHetvatten` tinyint(1) NOT NULL,
`ledningKylaPrim` tinyint(1) NOT NULL,
`ledningKylaSek` tinyint(1) NOT NULL,
`ledningGas` tinyint(1) NOT NULL,
`ledningLuft` tinyint(1) NOT NULL,
`ledningAvlopp` tinyint(1) NOT NULL,
`ledningOther` varchar(512) NOT NULL,
`materialGjutjarn` tinyint(1) NOT NULL,
`materialSteel` tinyint(1) NOT NULL,
`materialKoppar` tinyint(1) NOT NULL,
`materialPlast` tinyint(1) NOT NULL,
`materialRostfritt` tinyint(1) NOT NULL,
`materialOther` varchar(512) NOT NULL,
`omfattningLength` int(11) NOT NULL COMMENT 'meter',
`omfattningDimension` varchar(16) NOT NULL,
`omfattningRitningnr` varchar(128) NOT NULL,
`doneWithPump` tinyint(1) NOT NULL,
`doneWithVattenledning` tinyint(1) NOT NULL,
`doneWithKompressor` tinyint(1) NOT NULL,
`doneWithTathetsprovare` tinyint(1) NOT NULL,
`tryckmedieVatten` tinyint(1) NOT NULL,
`tryckmedieLuft` tinyint(1) NOT NULL,
`tryckmedieOther` varchar(128) NOT NULL,
`manometerDiameter` int(11) NOT NULL COMMENT 'mm',
`manometerGradering` int(11) NOT NULL COMMENT 'kPa',
`manometerReadPressure` int(11) NOT NULL,
`manometerTid` int(11) NOT NULL COMMENT 'sekunder',
`testedOn` datetime NOT NULL,
`testedBy` varchar(128) NOT NULL COMMENT '"id_" + (userid)',
`comments` varchar(1024) NOT NULL,
`commentsBy` varchar(128) NOT NULL COMMENT '"id_" + (userid)',
`signedOn` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
`sign` int(11) NOT NULL,
PRIMARY KEY (`projid`,`docid`,`revnr`)
) ENGINE=MyISAM DEFAULT CHARSET=latin1;
The fields I need are projid, docid, revnr, revname, signedOn, sign and are present in all current and future tables.