This surely seems like poor design of the college_major table.
CREATE TABLE `college_majors` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`date_time` datetime DEFAULT NULL,
`UNITID` varchar(255) DEFAULT NULL,
`CIPCODE` varchar(255) DEFAULT NULL,
`AWLEVEL` varchar(255) DEFAULT NULL,
`CTOTALT` varchar(255) DEFAULT NULL,
`CTOTALM` varchar(255) DEFAULT NULL,
`CTOTALW` varchar(255) DEFAULT NULL,
`CAIANT` varchar(255) DEFAULT NULL,
`CASIAT` varchar(255) DEFAULT NULL,
`CBKAAT` varchar(255) DEFAULT NULL,
`CHISPT` varchar(255) DEFAULT NULL,
`CNHPIT` varchar(255) DEFAULT NULL,
`CWHITT` varchar(255) DEFAULT NULL,
`C2MORT` varchar(255) DEFAULT NULL,
`CUNKNT` varchar(255) DEFAULT NULL,
`CNRALT` varchar(255) DEFAULT NULL,
PRIMARY KEY (`id`)
) ENGINE=MyISAM AUTO_INCREMENT=270167 DEFAULT CHARSET=utf8;
I can reduce this table to three columns - id, CIPCODE and UNITID. But the problem is even simple queries like *select * FROM college_majors* is taking too long to execute and sometime not even executing.
I increased the query execution to 6000.00 sec, but still the query won't run.
Any suggestion on how to improve the design, create a new table and insert the data from this table (college_majors).
Thanks,
A
If field 'codevalue' in college_majors_mapping is unique you can indexed it and increase join performance.
Related
Trying to set up a user profile page for a job site. The database I plan to use is the MySQL database.
Looking into a few database design I came up with this schema.
First, the user management tables
CREATE TABLE `user` (
`user_id` int(11) NOT NULL,
`firstname` varchar(32) NOT NULL,
`lastname` varchar(32) NOT NULL,
`email` varchar(96) NOT NULL,
`mobile_number` varchar(32) NOT NULL,
`password` varchar(40) NOT NULL,
`salt` varchar(9) NOT NULL,
`address_id` int(11) NOT NULL DEFAULT '0',
`ip` varchar(40) NOT NULL,
`status` tinyint(1) NOT NULL,
`approved` tinyint(1) NOT NULL,
`registration_date` datetime NOT NULL
) ENGINE=MyISAM DEFAULT CHARSET=utf8;
CREATE TABLE `user_address` (
`user_id` int(11) NOT NULL,
`city` varchar(128) NOT NULL
`work_city` varchar(128) NOT NULL,
`postal_code` varchar(10) NOT NULL,
`country_id` int(11) NOT NULL DEFAULT '0'
) ENGINE=MyISAM DEFAULT CHARSET=utf8;
CREATE TABLE `user_description` (
`user_id` int(11) NOT NULL,
`description` text NOT NULL
) ENGINE=MyISAM DEFAULT CHARSET=utf8;
and then the education table and work experience
CREATE TABLE `education_detail` (
`user_id` int(11) NOT NULL,
`certificate_degree_name` varchar(255) DEFAULT NULL,
`major` varchar(255) DEFAULT NULL,
`institute_university_name` varchar(255) DEFAULT NULL,
`start_date` date NOT NULL DEFAULT '0000-00-00',
`completion_date` date NOT NULL DEFAULT '0000-00-00'
)
CREATE TABLE `experience_detail` (
`user_id` int(11) NOT NULL,
`is_current_job` int(2) DEFAULT NULL,
`start_date` date NOT NULL DEFAULT '0000-00-00',
`end_date` date NOT NULL DEFAULT '0000-00-00',
`job_title` varchar(255) DEFAULT NULL,
`company_name` varchar(255) DEFAULT NULL,
`job_location_city` varchar(255) DEFAULT NULL,
`job_location_state` varchar(255) DEFAULT NULL,
`job_location_country` varchar(255) DEFAULT NULL,
`job_description` varchar(255) DEFAULT NULL
)
Note that user_id in table user_address, user_description, education_detail and experience_detail is a foreign key referencing it to the table user.
There are a few more table like skills, certification etc to which I plan on using user_id as a FK.
My question, is this database design good enough? Can you suggest me what should be done more to make the design much better?
Keep in mind not all will have work experience, some may be freshers.
Use InnoDB, not MyISAM. (There are many Q&A explaining 'why'.)
NULL or an empty string is perfectly fine for a missing description. Do you have any further argument for disliking such? (Meanwhile, InnoDB is more efficient at handling optional big strings.)
Every table should have a PRIMARY KEY; you don't seem to have any. The first table probably needs user_id as the PK. Read about AUTO_INCREMENT.
As with description, why is address in a separate table?
May I suggest this for country name/code/id:
country_code CHAR(2) CHARACTER SET ascii
Education is 1:many from users, so user_id cannot be the PK. Ditto for jobs.
query is simple, as below:
select count(1) from ec_account a join ec_card b on a.id = b.AccountId
there are 2.5 million rows in either ec_account and ec_card.(InnoDB)
here is the execution plan:
execution plan
as you see,
it already added index and used it, but the query still costed almost 60 seconds, is there any way could optimize it except changing database(mariadb has no such choke point as far as i know).
here is table DDL,ec_ccount:
CREATE TABLE `ec_account` (
`Id` varchar(64) NOT NULL,
`AccountType` varchar(32) NOT NULL,
`Name` varchar(32) NOT NULL,
`Status` tinyint(3) unsigned NOT NULL,
`IDCardType` varchar(32) DEFAULT NULL,
`IDCardNo` varchar(64) DEFAULT NULL,
`Password` varchar(256) DEFAULT NULL,
`PasswordHalt` varchar(128) DEFAULT NULL,
`Sex` varchar(8) DEFAULT NULL,
`BirthDay` datetime NOT NULL,
`Mobile` varchar(16) DEFAULT NULL,
`Address` varchar(64) DEFAULT NULL,
`Linkman` varchar(32) DEFAULT NULL,
`LinkmanRelation` varchar(16) DEFAULT NULL,
`LinkmanTel` varchar(16) DEFAULT NULL,
`Remark` varchar(128) DEFAULT NULL,
`Nationality` varchar(32) DEFAULT NULL,
`Nation` varchar(32) DEFAULT NULL,
`MaritalStatus` varchar(8) DEFAULT NULL,
`NativePlace` varchar(64) DEFAULT NULL,
`Occupation` varchar(32) DEFAULT NULL,
`BloodType` varchar(8) DEFAULT NULL,
`Education` varchar(8) DEFAULT NULL,
`LinkmanAddress` varchar(64) DEFAULT NULL,
`HomeAddress` varchar(128) DEFAULT NULL,
`Email` varchar(64) DEFAULT NULL,
`CompanyName` varchar(64) DEFAULT NULL,
`CompanyAddress` varchar(128) DEFAULT NULL,
`CompanyTel` varchar(16) DEFAULT NULL,
`Creator` char(36) CHARACTER SET utf8 COLLATE utf8_bin DEFAULT NULL,
`CreateTime` datetime NOT NULL,
`LastModifier` char(36) CHARACTER SET utf8 COLLATE utf8_bin DEFAULT NULL,
`LastModifyTime` datetime DEFAULT NULL,
`Avatar` longblob,
PRIMARY KEY (`Id`),
KEY `IX_Name` (`Name`) USING HASH,
KEY `Idx_IDCard_Account` (`IDCardType`,`IDCardNo`) USING HASH,
KEY `Idx_Mobile` (`Mobile`) USING HASH,
KEY `Idx_CreateTime` (`CreateTime`) USING BTREE
) ENGINE=InnoDB DEFAULT CHARSET=utf8;
and ec_card :
CREATE TABLE `ec_card` (
`Id` char(36) CHARACTER SET utf8 COLLATE utf8_bin NOT NULL DEFAULT '',
`AccountId` varchar(64) NOT NULL,
`CardType` varchar(32) NOT NULL,
`CardNo` varchar(32) NOT NULL,
`Status` tinyint(3) unsigned NOT NULL,
`IsPasswordAuth` tinyint(1) NOT NULL,
PRIMARY KEY (`Id`),
UNIQUE KEY `Idx_Unique_AccountId_CardType` (`AccountId`,`CardType`) USING HASH,
UNIQUE KEY `Idx_Unique_CardType_CardNo` (`CardType`,`CardNo`) USING HASH,
KEY `Idx_Uniques_AccountId` (`AccountId`) USING BTREE,
CONSTRAINT `FK_ec_card_ec_account_AccountId` FOREIGN KEY (`AccountId`) REFERENCES `ec_account` (`Id`) ON DELETE CASCADE ON UPDATE CASCADE
) ENGINE=InnoDB DEFAULT CHARSET=utf8;
Not without fundamentally changing the query.
There are no conditions on your query! It selects all 2.5 million rows from ec_card, as well as every matching row from ec_account. Reading all this data from disk and sending it over the network is the bottleneck; there is no way to change that without changing what the query does.
Here is a workaround for you. I think it would run much faster, and get the same result.
Calculate the total count of ec_account:
SELECT count(1) AS total_count FROM ec_account;
Calculate the amount of records those existed in ec_account but not existed in ec_card:
SELECT count(1) AS missing_count
FROM ec_account a LEFT JOIN ec_card b on a.id = b.AccountId
WHERE b.AccountId IS NULL;
Matched count = total_count - missing_count
The core problem here is that you combined two large table together, it requires a lot of memory and it apparently needs a lot of time to finish.
try it using correlated subquery. This might help:
select count(1) from ec_account a where exists (select * from ec_card b
where b.AccountId=a.id)
Also, other than indexing following strategies generally help:
- Denormalization
- Caching results
- Using a NoSQL database
I have the following simple join query
SELECT
count(*)
FROM
DBx.caseview p2015
INNER JOIN DBy.caseview p2014 ON p2015.casenumber=p2014.casenumber;
For some reason it just leaves MySQL hanging there for a lot of time until I get tired and cancel it. On the contrary, if run exactly the same code on MSSQL with the same data set the query takes a few seconds at most.
Is there a parameter that needs to be changed on MySQL to speed up this type of queries?
Here's my table in MySQL
CREATE TABLE `caseview` (
`ID` bigint(20) NOT NULL AUTO_INCREMENT,
`CASEID` varchar(18) DEFAULT NULL,
`CASENUMBER` int(10) DEFAULT NULL,
`ACCOUNTID` varchar(18) DEFAULT NULL,
`ACCOUNT` varchar(256) DEFAULT NULL,
`ASSETID` varchar(18) DEFAULT NULL,
`SAPPRODUCTGROUP` varchar(10) DEFAULT NULL,
`PRODUCT` varchar(128) DEFAULT NULL,
`FAMILY` varchar(128) DEFAULT NULL,
`CONTACTID` varchar(18) DEFAULT NULL,
`OWNERID` varchar(18) DEFAULT NULL,
`TYPE` varchar(128) DEFAULT NULL,
`PRIORITY` varchar(24) DEFAULT NULL,
`ORIGIN` varchar(24) DEFAULT NULL,
`SUBJECT` varchar(256) DEFAULT NULL,
`STATUS` varchar(24) DEFAULT NULL,
`LASTACTIVITY` varchar(1024) DEFAULT NULL,
`INITALDESCRIPTION` varchar(1024) DEFAULT NULL,
`CLOSEDDATE` datetime DEFAULT NULL,
`CREATEDDATE` datetime DEFAULT NULL,
`LASTMODIFIEDDATE` datetime DEFAULT NULL,
PRIMARY KEY (`ID`),
UNIQUE KEY `ID_UNIQUE` (`ID`)
) ENGINE=InnoDB AUTO_INCREMENT=122393 DEFAULT CHARSET=utf8;
There's two tables with exactly the same configuration as above, just different data
DBx.caseview p2015 has 197647 rows
DBy.caseview p2014 has 122392 rows
Create an Index for CASENUMBER field.
ALTER TABLE `caseview` ADD INDEX ( `CASENUMBER` ) ;
Give it a few seconds to create the index and run the query again.
I have a client that is describing an occurrence when they are adding new data from a web form. If the submitted data matches any existing database entries, the system deletes the existing entry.
I've checked the scripts and I don't see a DELETE query that would affect the related table. There are UPDATE queries, none of which affect a field in the DB indicating a deletion or marked as deleted.
Are there ways data could be lost without the execution of a DELETE query?
As asked for, here's the "create table" statement:
CREATE TABLE `msp_zip_codes`
(
`zip_code` varchar(5) DEFAULT NULL,
`city` varchar(35) DEFAULT NULL,
`state_prefix` varchar(2) DEFAULT NULL,
`county` varchar(45) DEFAULT NULL,
`area_code` varchar(45) DEFAULT NULL,
`CityType` varchar(1) DEFAULT NULL,
`CityAliasAbbreviation` varchar(13) DEFAULT NULL,
`CityAliasName` varchar(35) DEFAULT NULL,
`lat` decimal(18,6) DEFAULT NULL,
`lon` decimal(18,6) DEFAULT NULL,
`time_zone` varchar(7) DEFAULT NULL,
`Elevation` int(10) DEFAULT NULL,
`CountyFIPS` varchar(5) DEFAULT NULL,
`DayLightSaving` varchar(1) DEFAULT NULL,
`PreferredLastLineKey` varchar(25) DEFAULT NULL,
`ClassificationCode` varchar(2) DEFAULT NULL,
`MultiCounty` varchar(1) DEFAULT NULL,
`StateFIPS` varchar(2) DEFAULT NULL,
`CityStateKey` varchar(6) DEFAULT NULL,
`CityAliasCode` varchar(5) DEFAULT NULL,
`PrimaryRecord` varchar(50) DEFAULT NULL,
`CityMixedCase` varchar(50) DEFAULT NULL,
`CityAliasMixedCase` varchar(50) DEFAULT NULL,
KEY `AreaCode` (`area_code`),
KEY `CityAliasCode` (`CityAliasCode`),
KEY `CityStateKey` (`CityStateKey`),
KEY `ClassificationCode` (`ClassificationCode`),
KEY `PreferredLastLineKey` (`PreferredLastLineKey`),
KEY `ZipCode` (`zip_code`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8
One of the possibilities is that the table has an UPDATE TRIGGER set which fires every time there's an update. Or the other one, there's an Event Scheduler running on the server.
Code to show all triggers on the database:
SELECT *
FROM INFORMATION_SCHEMA.TRIGGERS a
WHERE a.TRIGGER_SCHEMA LIKE CONCAT('%', 'databaseNameHERE', '%')
Code to show all events on the database:
SHOW EVENTS FROM databaseNameHERE;
The following query is timing out after 600 seconds.
update placed p
,Results r
set p.position = r.position
where p.competitor = r.competitor
AND p.date = r.date
AND REPLACE(p.time,":","") = r.time;
The structure is as follows:
'CREATE TABLE `placed` (
`idplaced` varchar(50) DEFAULT NULL,
`date` decimal(8,0) DEFAULT NULL,
`time` varchar(45) DEFAULT NULL,
`field1` varchar(45) DEFAULT NULL,
`competitor` varchar(45) DEFAULT NULL,
`field2` int(2) DEFAULT NULL,
`field3` varchar(45) DEFAULT NULL,
`field4` varchar(45) DEFAULT NULL,
`field5` decimal(6,2) DEFAULT NULL,
`field6` decimal(10,2) DEFAULT NULL,
`field7` decimal(6,2) DEFAULT NULL,
`field8` char(1) DEFAULT NULL,
`field9` varchar(45) DEFAULT NULL,
`position` char(4) DEFAULT NULL,
`field10` decimal(6,2) DEFAULT NULL,
`field11` char(1) DEFAULT NULL,
`field12` char(1) DEFAULT NULL,
`field13` decimal(6,2) DEFAULT NULL,
`field14` decimal(6,2) DEFAULT NULL,
`field15` decimal(6,2) DEFAULT NULL,
`field16` decimal(6,2) DEFAULT NULL,
`field17` decimal(6,2) DEFAULT NULL,
`field18` char(1) DEFAULT NULL,
`field19` char(20) DEFAULT NULL,
`field20` char(1) DEFAULT NULL,
`field21` char(5) DEFAULT NULL,
`field22` char(5) DEFAULT NULL,
`field23` int(11) DEFAULT NULL
PRIMARY KEY (`idplaced`),
UNIQUE KEY `date_time_competitor_field18_combo` (`date`,`time`,`competitor`,`field18`)
) ENGINE=InnoDB AUTO_INCREMENT=100688607 DEFAULT CHARSET=latin1;
CREATE TABLE `results` (
`idresults` int(11) NOT NULL AUTO_INCREMENT,
`date` char(8) DEFAULT NULL,
`time` char(4) DEFAULT NULL,
`field1` varchar(45) DEFAULT NULL,
`competitor` varchar(45) DEFAULT NULL,
`position` char(4) DEFAULT NULL,
`field2` varchar(45) DEFAULT NULL,
`field3` decimal(2,0) DEFAULT NULL,
PRIMARY KEY (`idresults`)
) ENGINE=InnoDB AUTO_INCREMENT=6644 DEFAULT CHARSET=latin1;
The PLACED table has 65,000 records, the RESULTS table has 9,000 records.
I am assuming the solution involves a JOIN statement of some descript, and I have tried taking several suggestions from this site, but am simply not finding the answer I am looking for. Simply put, I would be grateful for suggestions on this. I can put up example tables / create table code if requried.
The index cannot be used efficiently to perform the join because of your REPLACE operation.
I'd suggest creating an index with the columns in the following slightly different order:
(date, competitor, time, position)
It may also help to add this index on both tables.
It would be even better if you could modify the data in the database so that the data in the time column was stored in the same format in both tables.
First of all, you'd better send us your full tables description, using
show create table
Second, you'd better use join syntax :
update placed p
join Results r on r.competitor = p.competitor
set p.position = r.position
where p.date = r.date
AND REPLACE(p.time,":","") = r.time;
Hope this will help.