MySQL Temporary Table with Group By and Group Concat Extremely Slow - mysql

I'm trying to build out a fairly simple temporary table. The table will end up being 2 columns:
1 A product ID
2 A string of concatenated compliance data in a format that can be consumed later
CREATE TEMPORARY TABLE products.compliances_data
(INDEX product_id_idx (product_id))
SELECT
products.product_id,
GROUP_CONCAT(JSON_OBJECT('compliance_code', cc.compliance_code, 'compliance_full_name', pc.full_name, 'compliance_web_description_short', pc.web_description_short) SEPARATOR ' - ') as compliances
FROM products.products products
LEFT OUTER JOIN products.material_compliance_map cc on products.material_id = cc.material_id
LEFT OUTER JOIN products.compliances pc on cc.compliance_id = pc.compliance_id
GROUP BY products.part_number
The table gathers and joins information from 3 tables.
The products table is the main source of information. The products table has a column for material_id.
The material_compiance_map table. This is a many-to-many table that maps material_id's to compliance_id's
The compliances table. This is the table where the actual data is stored that I need to pull from to build out the concatenated object.
The problem is the creation of the temporary table takes 3-4 minutes to run yet only yields about 1.2 million entries which seems incredibly slow.
Running an explain on the select portion of the query yields:
Explain on select image
There are only 642 entries in the material_comliance_map so there's not an awful lot of data here that needs to be traversed.
I've tried removing the group_concat and that seems to speed up the query about 33%. The problem seems to revolve around the group by statement.
How can I improve the speed when building this temp table?
EDIT:
Schemas:
material_compliance_map schema
'material_compliance_map'
CREATE TABLE `material_compliance_map` (
`material_id` int(11) NOT NULL,
`material_code` varchar(20) COLLATE utf8_unicode_ci NOT NULL,
`compliance_id` int(11) NOT NULL,
`compliance_code` varchar(20) COLLATE utf8_unicode_ci NOT NULL,
PRIMARY KEY (`material_id`,`compliance_id`),
KEY `fk_compliance_id_material_compliances_compliances` (`compliance_id`),
KEY `fk_material_id_material_compliances_materials` (`material_id`),
CONSTRAINT `fk_compliance_id_material_compliances_compliances` FOREIGN KEY (`compliance_id`) REFERENCES `compliances` (`compliance_id`) ON UPDATE CASCADE,
CONSTRAINT `fk_material_id_material_compliances_materials` FOREIGN KEY (`material_id`) REFERENCES `materials` (`material_id`) ON UPDATE CASCADE
) ENGINE=InnoDB DEFAULT CHARSET=utf8 COLLATE=utf8_unicode_ci
compliances schema:
'compliances'
CREATE TABLE `compliances` (
`compliance_id` int(11) NOT NULL AUTO_INCREMENT,
`compliance_code` varchar(20) NOT NULL,
`full_name` varchar(155) DEFAULT NULL,
`web_description_short` varchar(45) DEFAULT NULL,
PRIMARY KEY (`compliance_id`),
UNIQUE KEY `compliance_id_UNIQUE` (`compliance_id`),
UNIQUE KEY `compliance_code_UNIQUE` (`compliance_code`)
) ENGINE=InnoDB AUTO_INCREMENT=15 DEFAULT CHARSET=latin1
products schema:
'products'
CREATE TABLE `products` (
`part_number` varchar(27) NOT NULL,
`material_code` varchar(30) DEFAULT NULL,
`material_id` int(11) DEFAULT NULL,
`size_code` varchar(15) DEFAULT NULL,
`size_id` int(11) DEFAULT NULL,
`erp_description_1` varchar(31) DEFAULT NULL,
`erp_description_2` varchar(31) DEFAULT NULL,
`search_description` varchar(250) DEFAULT NULL,
`weight_lbs` decimal(8,4) DEFAULT NULL,
`part_number_prefix` varchar(15) DEFAULT NULL,
`tight_tolerance` tinyint(4) DEFAULT NULL,
`product_id` int(11) NOT NULL AUTO_INCREMENT,
PRIMARY KEY (`product_id`),
UNIQUE KEY `part_number_UNIQUE` (`part_number`),
UNIQUE KEY `product_id_UNIQUE` (`product_id`),
KEY `fk_material_id_products_materials` (`material_id`),
KEY `fk_size_id_products_sizes` (`size_id`),
KEY `product_id` (`product_id`),
KEY `size_id_idx` (`size_id`),
KEY `size_id_productsidx` (`size_id`),
KEY `material_id_idx` (`material_id`),
CONSTRAINT `fk_material_id_products_materials` FOREIGN KEY (`material_id`) REFERENCES `materials` (`material_id`),
CONSTRAINT `fk_size_id_products_sizes` FOREIGN KEY (`size_id`) REFERENCES `sizes` (`size_id`)
) ENGINE=InnoDB AUTO_INCREMENT=1140987 DEFAULT CHARSET=latin1

Related

Should I use Inner Join in this case?

I am working on a student attendance mini-project, and I don't know how to proceed for my database. I'm new to SQL and databases in general so this might seem dumb to you.
So, I want to do a database containing the table student, which contains : student_id (primary key) , name (string) and attendance(boolean) (that's the bare minimum, i'll add more afterwards) and I want to register the daily attendance of the students. So I want to have all the students tied to every date of the week.
I created a date table in phpMyadmin but I don't know how to proceed to link them, i've tried an Inner Join and it was successful.
The problem is : If i want to add another line to the student table my table won't update, so is there a way to "automatically" tie all the students to the date table ?
Sorry if this seems confused I've tried my best to summarize it !
Lets have some idea about tables should be there to implement a proper Student Attendance system in place. I have copied create script for some of my tables that used for maintaining Students record per course. I hope following sample Table scripts with relation will help you understanding regarding table structure and also to solve your issue.
Please be noted, That this table structures for your your reference only. You can add/remove tables/columns as per your requirement once you get an overall idea from this post.
CREATE TABLE `staff` (
`id` int(11) NOT NULL,
`type` varchar(200) DEFAULT NULL,
`first_name` varchar(200) DEFAULT NULL,
`last_name` varchar(200) DEFAULT NULL,
`emal` varchar(200) DEFAULT NULL,
`contact` varchar(200) DEFAULT NULL,
PRIMARY KEY (`id`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8;
CREATE TABLE `batch` (
`id` int(11) NOT NULL,
`department` varchar(200) DEFAULT NULL,
`details` varchar(200) DEFAULT NULL,
`staff_id` int(11) DEFAULT NULL,
PRIMARY KEY (`id`),
KEY `batch_staff_idx` (`staff_id`),
CONSTRAINT `batch_staff` FOREIGN KEY (`staff_id`) REFERENCES `staff` (`id`) ON DELETE NO ACTION ON UPDATE NO ACTION
) ENGINE=InnoDB DEFAULT CHARSET=utf8;
CREATE TABLE `student` (
`id` int(11) NOT NULL,
`batch_id` int(11) DEFAULT NULL,
`first_name` varchar(200) DEFAULT NULL,
`last_name` varchar(200) DEFAULT NULL,
`email` varchar(200) DEFAULT NULL,
`contact` varchar(200) DEFAULT NULL,
PRIMARY KEY (`id`),
KEY `batch_student_idx` (`batch_id`),
CONSTRAINT `batch_student` FOREIGN KEY (`batch_id`) REFERENCES `batch` (`batch_id`) ON DELETE NO ACTION ON UPDATE NO ACTION
) ENGINE=InnoDB DEFAULT CHARSET=utf8;
CREATE TABLE `course` (
`id` int(11) NOT NULL,
`name` varchar(200) DEFAULT NULL,
`details` varchar(200) DEFAULT NULL,
`staff_id` int(11) DEFAULT NULL,
PRIMARY KEY (`id`),
KEY `course_staff_idx` (`staff_id`),
CONSTRAINT `course_staff` FOREIGN KEY (`staff_id`) REFERENCES `staff` (`id`) ON DELETE NO ACTION ON UPDATE NO ACTION
) ENGINE=InnoDB DEFAULT CHARSET=utf8;
CREATE TABLE `attendence` (
`course_id` int(11) NOT NULL,
`student_id` int(11) DEFAULT NULL,
`class_date` date DEFAULT NULL,
KEY `att_course_idx` (`course_id`),
KEY `att_student_idx` (`student_id`),
CONSTRAINT `att_course` FOREIGN KEY (`course_id`) REFERENCES `course` (`id`) ON DELETE NO ACTION ON UPDATE NO ACTION,
CONSTRAINT `att_student` FOREIGN KEY (`student_id`) REFERENCES `student` (`id`) ON DELETE NO ACTION ON UPDATE NO ACTION
) ENGINE=InnoDB DEFAULT CHARSET=utf8;
Your required data will finally store into table Attendance. From this table data, you will be able to find list of students absent/present per date and per course. Remember, the attendance table should enrich daily from a automated OR a manual process.

MySQL UNIQUE KEY and FOREIGN KEY same column not getting created

I have following tables/CREATE sintaxis:
CREATE TABLE `users` (
`id` int(11) unsigned NOT NULL AUTO_INCREMENT,
`parentId` int(10) unsigned DEFAULT NULL,
`fullName` varchar(50) COLLATE utf8_unicode_ci NOT NULL DEFAULT '',
`alias` varchar(35) COLLATE utf8_unicode_ci DEFAULT NULL,
`username` varchar(50) COLLATE utf8_unicode_ci NOT NULL DEFAULT '',
`password` varchar(255) COLLATE utf8_unicode_ci NOT NULL DEFAULT '',
PRIMARY KEY (`id`),
UNIQUE KEY `uk_username` (`username`),
UNIQUE KEY `uk_parentId_fullName_alias` (`parentId`,`fullName`,`alias`),
KEY `fk_users_parentId` (`parentId`),
CONSTRAINT `fk_users_parentId` FOREIGN KEY (`parentId`) REFERENCES `users` (`id`) ON DELETE CASCADE
) ENGINE=InnoDB AUTO_INCREMENT=2 DEFAULT CHARSET=utf8 COLLATE=utf8_unicode_ci;
CREATE TABLE `userSettings` (
`id` int(11) unsigned NOT NULL AUTO_INCREMENT,
`userId` int(11) unsigned NOT NULL,
`settingsArray` text NOT NULL,
PRIMARY KEY (`id`),
UNIQUE KEY `uk_userId` (`userId`),
KEY `fk_userSettings_userId` (`userId`),
CONSTRAINT `fk_userSettings_userId` FOREIGN KEY (`userId`) REFERENCES `users` (`id`) ON DELETE CASCADE
) ENGINE=MyISAM AUTO_INCREMENT=6 DEFAULT CHARSET=latin1
im trying to create one table with user data and another one with the user settings, when i create the userSettings table it doesnt create the foreign key, is there something wrong with the create sintaxis? It is related with creating two indexes for same column?
Here what i get after creating the userSettings table:
CREATE TABLE `userSettings` (
`id` int(11) unsigned NOT NULL AUTO_INCREMENT,
`userId` int(11) unsigned NOT NULL,
`settingsArray` text NOT NULL,
PRIMARY KEY (`id`),
UNIQUE KEY `uk_userId` (`userId`),
KEY `fk_userSettings_userId` (`userId`)
) ENGINE=MyISAM AUTO_INCREMENT=6 DEFAULT CHARSET=latin1;
As you discovered, MyISAM doesn't support foreign keys. Both users and userSettings must be InnoDB.
[I'm] just curious if having a UNIQUE_KEY and FOREIGN_KEY in same column is a good practice
This means the userSettings table can have at most one row for each userId. I guess you need only one row per userId because you store an "array" of settings encoded somehow in your settingsArray TEXT column. This is not a good practice.
You should either store each setting in its own column:
CREATE TABLE `userSettings` (
`id` int(11) unsigned NOT NULL AUTO_INCREMENT,
`userId` int(11) unsigned NOT NULL,
`isAdmin` bool NOT NULL,
`timezone` varchar(10) NOT NULL,
`theme` varchar(10) NOT NULL,
...other settings...
PRIMARY KEY (`id`),
UNIQUE KEY `uk_userId` (`userId`),
KEY `fk_userSettings_userId` (`userId`)
)
Or else store multiple rows per userId, with one setting name and value per row.
CREATE TABLE `userSettings` (
`id` int(11) unsigned NOT NULL AUTO_INCREMENT,
`userId` int(11) unsigned NOT NULL,
`setting` varchar(20) NOT NULL,
`value` varchar(255) NOT NULL,
PRIMARY KEY (`id`),
UNIQUE KEY `uk_userId` (`userId`,`setting`),
KEY `fk_userSettings_userId` (`userId`)
)
It's also puzzling why you need an id column for the primary key, if the userId is already NOT NULL and UNIQUE, and that's probably the key you'll use to look up rows anyway. You can make the userId the PRIMARY KEY as well (or userId, setting in the second example), and omit the id column.
Just realized for the table users the ENGINE was InnoDB and for the userSettings table ENGINE was MyISAM, changed that and worked, im just curious if having a UNIQUE_KEY and FOREIGN_KEY in same column is a good practice

Foreign key constraint fails with Insert Into Select in MySQL

I'm quite new to SQL, and I'm trying to upload data to my tables. For this I have special tables where I upload the data from a CSV file, and then, from this table, I am trying to copy the data to the final table.
But now I have a problem with an intermediate table where I have uploaded my data. The table is:
CREATE TABLE `_work_has_person` (
`work_id` int(11) NOT NULL,
`person_id` int(11) NOT NULL,
`primary_contribution_id` int(11) DEFAULT NULL,
PRIMARY KEY (`work_id`,`person_id`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8;
I want to copy the data in
CREATE TABLE `work_has_person` (
`work_id` int(11) NOT NULL,
`person_id` int(11) NOT NULL,
`primary_contribution_id` int(11) NOT NULL,
PRIMARY KEY (`work_id`,`person_id`),
KEY `fk_work_has_person_person1_idx` (`person_id`),
KEY `fk_work_has_person_work1_idx` (`work_id`),
KEY `fk_work_has_person_primary_contribution1_idx` (`primary_contribution_id`),
CONSTRAINT `fk_work_has_person_person1` FOREIGN KEY (`person_id`) REFERENCES `person` (`id`) ON DELETE NO ACTION ON UPDATE NO ACTION,
CONSTRAINT `fk_work_has_person_primary_contribution1` FOREIGN KEY (`primary_contribution_id`) REFERENCES `primary_contribution` (`id`) ON DELETE NO ACTION ON UPDATE NO ACTION,
CONSTRAINT `fk_work_has_person_work1` FOREIGN KEY (`work_id`) REFERENCES `work` (`id`) ON DELETE NO ACTION ON UPDATE NO ACTION
) ENGINE=InnoDB DEFAULT CHARSET=utf8;
Which is an intermediate table between:
CREATE TABLE `work` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`title_work` varchar(250) DEFAULT NULL,
`subtitle_work` varchar(250) DEFAULT NULL,
`date_work` varchar(45) DEFAULT NULL,
`unix_date_work` varchar(100) DEFAULT NULL,
`sinopsis` text,
`ref_bne` varchar(100) DEFAULT NULL,
`ref_alt` longtext,
`language_id` int(11) NOT NULL,
PRIMARY KEY (`id`),
KEY `fk_work_language1_idx` (`language_id`),
KEY `title_work` (`title_work`),
CONSTRAINT `fk_work_language1` FOREIGN KEY (`language_id`) REFERENCES `language` (`id`) ON DELETE NO ACTION ON UPDATE NO ACTION
) ENGINE=InnoDB AUTO_INCREMENT=24610 DEFAULT CHARSET=utf8;
and
CREATE TABLE `person` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`name` varchar(100) NOT NULL,
`img_person` varchar(250) DEFAULT NULL,
`born_date` varchar(45) DEFAULT NULL,
`unix_born_date` varchar(100) DEFAULT NULL,
`city_born_date` varchar(100) DEFAULT NULL,
`country_born_date` varchar(100) DEFAULT NULL,
`death_date` varchar(45) DEFAULT NULL,
`unix_death_date` varchar(100) DEFAULT NULL,
`city_death_date` varchar(100) DEFAULT NULL,
`country_death_date` varchar(45) DEFAULT NULL,
`biography` longtext,
`ref_bne` varchar(100) DEFAULT NULL,
`ref_alt` longtext,
PRIMARY KEY (`id`),
UNIQUE KEY `name_UNIQUE` (`name`),
KEY `name` (`name`)
) ENGINE=InnoDB AUTO_INCREMENT=9159 DEFAULT CHARSET=utf8;
But everytime I try to run
INSERT INTO work_has_person (work_id, person_id, primary_contribution_id)
SELECT work_id, person_id, primary_contribution_id
FROM _work_has_person;
It says
Cannot add or update a child row: a foreign key constraint fails (`cdu93hfg93r`.
`work_has_person`, CONSTRAINT `fk_work_has_person_person1` FOREIGN KEY (`person_id`)
REFERENCES `person` (`id`) ON DELETE NO ACTION ON UPDATE NO ACTION)
I am pretty sure that the tables has the neccesary data, but, ¿is there a way to know which data fails? I have seen Mysql error 1452 - Cannot add or update a child row: a foreign key constraint fails: but don't understand exactly how to use it here.
A.
It is relatively easy to find out what data is causing the conflict: get all person_ids from _work_has_person that are not in the persons table. You can achieve this via an outer join and filtering for person.id is null in the where clause.
select * from `_work_has_person` whp
left join person p on whp.person_id=p.id
where p.id is null
You can actually remove such data from the results being inserted by including the reverse criterion into the select part of your insert query (an inner join):
INSERT INTO work_has_person (work_id, person_id, primary_contribution_id)
SELECT whp.work_id, whp.person_id, whp.primary_contribution_id
FROM _work_has_person whp
INNER join person p on whp.person_id=p.id

Only allow one unique foreign key for primary

I have 3 tables in mysql, One is a Company table, the other is a license table and the last is a joining table between both primary keys, When a person adds a company id to the license id in the joining table, it allows multiple companies to exist for one license, this cannot happen, so I need to do something that will only allow one company id for one license id
heres the tables
Table license
CREATE TABLE `License` (
`license_id` int(11) NOT NULL AUTO_INCREMENT,
`license_number` varchar(45) NOT NULL,
`start_date` date NOT NULL,
`end_date` date NOT NULL,
`duration` int(11) NOT NULL,
`expiry_date` date NOT NULL,
`product_id` int(11) DEFAULT NULL,
PRIMARY KEY (`license_id`)
) ENGINE=InnoDB AUTO_INCREMENT=21 DEFAULT CHARSET=latin1;
;
Company Table
CREATE TABLE `Company` (
`company_id` int(11) NOT NULL AUTO_INCREMENT,
`name` varchar(255) NOT NULL,
`physical_address` varchar(255) DEFAULT NULL,
`postal_address` varchar(255) DEFAULT NULL,
`reseller_id` int(11) DEFAULT NULL,
PRIMARY KEY (`company_id`)
) ENGINE=InnoDB AUTO_INCREMENT=18 DEFAULT CHARSET=latin1;
and Joining table
CREATE TABLE `CompanyLicense` (
`license_id` int(11) NOT NULL,
`company_id` int(11) NOT NULL,
PRIMARY KEY (`license_id`,`company_id`),
KEY `companlicence_company_fk_idx` (`company_id`),
CONSTRAINT `companylicense_company_fk` FOREIGN KEY (`company_id`) REFERENCES `Company` (`company_id`) ON DELETE CASCADE ON UPDATE CASCADE,
CONSTRAINT `companylicense_license_fk` FOREIGN KEY (`license_id`) REFERENCES `License` (`license_id`) ON DELETE CASCADE ON UPDATE CASCADE
) ENGINE=InnoDB DEFAULT CHARSET=latin1;
So far i have this
INSERT INTO CompanyLicense (license_id, company_id) VALUES
('2','6') on duplicate key update license_id = '2';
doesnt seem to do the job
You need to make company unique in companylicense:
ALTER TABLE companylicense ADD UNIQUE KEY (company)
or better yet, make company a field in license instead of having a link table.

MariaDB: Multiple joins and where clause searching for 1 in TINYINT Column not working

I have tables called person and book and image and bookhit.
Person has id, name and Book has id, owner_id, info and Image has a column for id, owner_id, url and thumbnail which is a TINYINT (In the database half the rows are 0s and 1s.) By the way, the image column stores images of the cover of the book in two version: big-one and thumbnail. The table bookhit stores the times the book has been retrieved from the database and has a column hits.
So I tried multiple INNER JOIN to retrieve all the thumbnails for the most popular books. The SQL Query is the following:
SELECT `imagehit`.`hits`, `person`.`name`, `book`.`info`, `image`.`url`, `image`.`thumbnail` FROM `imagehit`
INNER JOIN `person` ON `person`.`id`=`book`.`owner_id`
INNER JOIN `image` ON `image`.`owner_id`=`book`.`id`
ORDER BY `imagehit`.`hits` DESC
WHERE `image`.`thumbnail`=1
LIMIT 10;
And that doesn't work, even though half rows has 1s in image.thumbnail . If I change the following line:
WHERE `image`.`thumbnail`=1
To
WHERE `image`.`thumbnail`=0
It does work. Well, I went to the image table and did a simple query like the following:
SELECT * FROM `image` WHERE `image`.`thumbnail`=0;
And gave me total rows stored in the table. But when I browse image table in phpMyAdmin I see there are 1s stored in the table. :(
Any ideas why this happens? thank you in advance.
Table definitions:
CREATE TABLE `image` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`owner_id` int(11) NOT NULL,
`thumbnail` tinyint(1) NOT NULL,
`url` varchar(255) NOT NULL,
PRIMARY KEY (`id`),
UNIQUE KEY `image_url` (`url`),
KEY `image_owner_id` (`owner_id`),
CONSTRAINT `image_ibfk_1` FOREIGN KEY (`owner_id`) REFERENCES `book` (`id`)
) ENGINE=InnoDB AUTO_INCREMENT=1450 DEFAULT CHARSET=utf8
CREATE TABLE `person` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`name` varchar(255) NOT NULL,
`url` varchar(60) NOT NULL,
PRIMARY KEY (`id`),
UNIQUE KEY `person_url` (`url`),
) ENGINE=InnoDB AUTO_INCREMENT=6287 DEFAULT CHARSET=utf8
CREATE TABLE `book` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`owner_id` int(11) NOT NULL,
`book` varchar(3000) NOT NULL,
`info` varchar(3000) NOT NULL,
`url` varchar(60) NOT NULL,
PRIMARY KEY (`id`),
UNIQUE KEY `book_url` (`url`),
KEY `book_owner_id` (`owner_id`),
CONSTRAINT `book_ibfk_1` FOREIGN KEY (`owner_id`) REFERENCES `person` (`id`)
) ENGINE=InnoDB AUTO_INCREMENT=725 DEFAULT CHARSET=utf8
CREATE TABLE `imagehit` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`owner_id` int(11) NOT NULL,
`person_id` int(11) NOT NULL,
`hits` int(11) NOT NULL,
PRIMARY KEY (`id`),
KEY `imagehit_person_id` (`person_id`),
KEY `imagehit_owner_id` (`owner_id`),
KEY `hits` (`hits`),
CONSTRAINT `imagehit_ibfk_1` FOREIGN KEY (`owner_id`) REFERENCES `image` (`id`),
CONSTRAINT `imagehit_ibfk_2` FOREIGN KEY (`person_id`) REFERENCES `person` (`id`)
) ENGINE=InnoDB AUTO_INCREMENT=725 DEFAULT CHARSET=utf8
Proof I'm not crazy:
I inserted the data using Peewee, when I created the row I set thumbnail=True if the image was a thumbnail and as thumbnail=False if it wasn't. The column thumbnail is the field BooleanField in Peewee.