Related
I have a query which looks something like this:
SELECT
STRAIGHT_JOIN `reviewApp_review`.`id`, `reviewApp_review`.`reviewTitle`,
`reviewApp_review`.`reviewContent`, `reviewApp_review`.`translatedEnTitle`,
`reviewApp_review`.`translatedEnContent`, `reviewApp_review`.`translatedEnDate`,
`reviewApp_review`.`reviewLink`, `reviewApp_review`.`reviewUser`,
`reviewApp_review`.`reviewUserProfile`, `reviewApp_review`.`reviewDataCreated`,
`reviewApp_review`.`reviewDataDiscovered`, `reviewApp_review`.`reviewData`,
`reviewApp_review`.`reviewRating`, `reviewApp_review`.`reviewSignature`,
`reviewApp_review`.`reviewSignature2`, `reviewApp_review`.`reviewExternalId`,
`reviewApp_review`.`reviewStatus`, `reviewApp_review`.`reviewWebsite`,
`reviewApp_review`.`language`, `reviewApp_review`.`helpfulVotes`,
`reviewApp_review`.`verified`, `reviewApp_review`.`color`,
`reviewApp_review`.`style`, `reviewApp_review`.`size`,
`reviewApp_review`.`lastUpdated`, `reviewApp_review`.`lastSeen`,
`reviewApp_review`.`deleted`, `reviewApp_review`.`alerted`,
`reviewApp_review`.`analyzed`, `reviewApp_review`.`productLink_id`
FROM `reviewApp_review`
INNER JOIN `reviewApp_productlink` ON (`reviewApp_review`.`productLink_id` = `reviewApp_productlink`.`id`)
INNER JOIN `reviewApp_product` ON (`reviewApp_productlink`.`product_id` = `reviewApp_product`.`id`)
WHERE (`reviewApp_product`.`owner` = 'my product'
AND `reviewApp_productlink`.`customer_id` = '1241'
AND (`reviewApp_review`.`translatedEnContent` LIKE '%%urban%%'
OR (`reviewApp_review`.`reviewContent` LIKE '%%urban%%'
AND `reviewApp_review`.`translatedEnDate` = '0'))
)
ORDER BY `reviewApp_review`.`reviewRating` ASC
LIMIT 10
I want to select reviews from the database and when a result is a number of reviews under 10, it takes a lot of time, more than a minute. And I'm wondering if there is a solution to optimize this query.
I tried to make different operations in the SQL query, as you can see I tried to use STRAIGHT_JOIN, and INNER_JOIN but I didn't have a better time.
The result of the query should be a querySet that contains a list of reviews, matching all those conditions from the query.
From Django I'm using something like that to run this query:
querySet.model.objects.raw(rawQuery)
where rawQuery is the query described above.
Here I have a python function and I want to optimize it because it takes to much time.
def fixQS(querySet):
"""
Optimisation for SQL QUERY using order by
https://dba.stackexchange.com/a/40195/246455
https://docs.djangoproject.com/en/2.1/ref/models/querysets/#extra
STRAIGHT_JOIN
"""
# complete the SQL with params encapsulated in quotes
sql, params = querySet.query.sql_with_params()
newParams = ()
for param in params:
if not str(param).startswith("'"):
if isinstance(param, str):
param = re.sub("'", "\\'", param)
newParams = newParams + ("'{}'".format(param),)
else:
newParams = newParams + (param,)
rawQuery = sql % newParams
# escape the percent used in SQL LIKE statements
rawQuery = re.sub('%', '%%', rawQuery)
# replace SELECT with SELECT STRAIGHT_JOIN
rawQuery = rawQuery.replace('SELECT', 'SELECT STRAIGHT_JOIN')
return querySet.model.objects.raw(rawQuery)
I'm calling it like this :
currentReviews = fixQS(currentReviews)
And when the debugger evaluates this it takes a lot of time.
Here is the result of the command SHOW CREATE TABLE:
CREATE TABLE `reviewApp_review` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`reviewTitle` varchar(500) COLLATE utf8mb4_unicode_ci NOT NULL,
`reviewContent` longtext COLLATE utf8mb4_unicode_ci NOT NULL,
`reviewLink` varchar(500) COLLATE utf8mb4_unicode_ci NOT NULL,
`reviewDataCreated` int(11) NOT NULL,
`reviewDataDiscovered` int(11) NOT NULL,
`reviewRating` varchar(30) COLLATE utf8mb4_unicode_ci NOT NULL,
`reviewSignature` varchar(255) COLLATE utf8mb4_unicode_ci NOT NULL,
`reviewStatus` varchar(200) COLLATE utf8mb4_unicode_ci NOT NULL,
`productLink_id` int(11) NOT NULL,
`reviewWebsite` varchar(200) COLLATE utf8mb4_unicode_ci NOT NULL,
`language` varchar(200) COLLATE utf8mb4_unicode_ci NOT NULL,
`verified` tinyint(1) NOT NULL,
`color` varchar(500) COLLATE utf8mb4_unicode_ci NOT NULL,
`size` varchar(500) COLLATE utf8mb4_unicode_ci NOT NULL,
`style` varchar(500) COLLATE utf8mb4_unicode_ci NOT NULL,
`alerted` tinyint(1) NOT NULL,
`reviewUser` varchar(500) COLLATE utf8mb4_unicode_ci NOT NULL,
`reviewUserProfile` varchar(500) COLLATE utf8mb4_unicode_ci NOT NULL,
`reviewSignature2` varchar(255) COLLATE utf8mb4_unicode_ci NOT NULL,
`reviewData` int(11) NOT NULL,
`reviewExternalId` varchar(500) COLLATE utf8mb4_unicode_ci NOT NULL,
`helpfulVotes` int(11) NOT NULL,
`lastSeen` int(11) NOT NULL,
`lastUpdated` int(11) NOT NULL,
`translatedEnContent` longtext COLLATE utf8mb4_unicode_ci NOT NULL,
`translatedEnDate` int(11) NOT NULL,
`translatedEnTitle` varchar(500) COLLATE utf8mb4_unicode_ci NOT NULL,
`deleted` tinyint(1) NOT NULL,
`analyzed` datetime(6) DEFAULT NULL,
PRIMARY KEY (`id`),
KEY `reviewApp_re_productLink_id_084fa2ec_fk_reviewApp_productlink_id` (`productLink_id`),
KEY `reviewApp_review_reviewSignature_2c31b21d_uniq` (`reviewSignature`),
KEY `reviewApp_review_reviewSignature_224ab822_idx` (`reviewSignature`,`productLink_id`),
KEY `reviewApp_review_fa0816ae` (`reviewSignature2`),
KEY `reviewApp_review_alerted_57344ef1_uniq` (`alerted`),
KEY `reviewApp_review_reviewData_5dd52e46_uniq` (`reviewData`),
KEY `reviewApp_review_reviewRating_d418fcfc_uniq` (`reviewRating`),
KEY `reviewApp_review_reviewWebsite_25d7fd04_uniq` (`reviewWebsite`),
KEY `reviewApp_review_reviewExternalId_32dfb169_uniq` (`reviewExternalId`),
KEY `reviewApp_reviewContent` (`reviewContent`(100)),
KEY `reviewApp_translatedEnContent` (`translatedEnContent`(100)),
KEY `reviewApp_translatedEnDate` (`translatedEnDate`),
KEY `reviewApp_review_da602f0b` (`deleted`),
KEY `reviewApp_review_language_3de815cd_uniq` (`language`),
KEY `reviewApp_review_order_index` (`reviewData`),
KEY `reviewApp_review_rating_order_index` (`reviewRating`),
KEY `reviewApp_review_rating_order_index_desc` (`reviewRating`),
CONSTRAINT `reviewApp_re_productLink_id_084fa2ec_fk_reviewApp_productlink_id` FOREIGN KEY (`productLink_id`) REFERENCES `reviewApp_productlink` (`id`)
) ENGINE=InnoDB AUTO_INCREMENT=24274768 DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci
And this one is for the product:
CREATE TABLE `reviewApp_product` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`name` varchar(500) COLLATE utf8mb4_unicode_ci NOT NULL,
`owner` varchar(500) COLLATE utf8mb4_unicode_ci NOT NULL,
`customer_id` int(11) DEFAULT NULL,
`ean` varchar(500) COLLATE utf8mb4_unicode_ci NOT NULL,
`internalCode` varchar(500) COLLATE utf8mb4_unicode_ci NOT NULL,
`sku` varchar(500) COLLATE utf8mb4_unicode_ci NOT NULL,
`asin` varchar(500) COLLATE utf8mb4_unicode_ci NOT NULL,
PRIMARY KEY (`id`),
UNIQUE KEY `reviewApp_product_name_37c04b41_uniq` (`name`,`customer_id`),
KEY `reviewApp_product_cb24373b` (`customer_id`),
KEY `reviewApp_product_name_23da262c_uniq` (`name`),
KEY `reviewApp_product_owner_680ac2b4_uniq` (`owner`),
KEY `reviewApp_product_index` (`owner`,`id`),
CONSTRAINT `reviewApp_product_customer_id_7663d434_fk_reviewApp_customer_id` FOREIGN KEY (`customer_id`) REFERENCES `reviewApp_customer` (`id`)
) ENGINE=InnoDB AUTO_INCREMENT=31556 DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci
And also for productLink I have this:
CREATE TABLE `reviewApp_productlink` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`productLink` varchar(2000) COLLATE utf8mb4_unicode_ci NOT NULL,
`customer_id` int(11) NOT NULL,
`productDataCreated` int(11) NOT NULL,
`product_id` int(11) DEFAULT NULL,
`domain` varchar(255) COLLATE utf8mb4_unicode_ci NOT NULL,
`httpStatus` int(11) NOT NULL,
`externalProductId` varchar(1000) COLLATE utf8mb4_unicode_ci NOT NULL,
`ratingsNumber` int(11) NOT NULL,
`rating` double DEFAULT NULL,
`fb_data` longtext COLLATE utf8mb4_unicode_ci NOT NULL,
PRIMARY KEY (`id`),
KEY `reviewApp_productl_customer_id_f0141212_fk_reviewApp_customer_id` (`customer_id`),
KEY `reviewApp_productlink_9bea82de` (`product_id`),
CONSTRAINT `reviewApp_productl_customer_id_f0141212_fk_reviewApp_customer_id` FOREIGN KEY (`customer_id`) REFERENCES `reviewApp_customer` (`id`),
CONSTRAINT `reviewApp_productlin_product_id_6123214e_fk_reviewApp_product_id` FOREIGN KEY (`product_id`) REFERENCES `reviewApp_product` (`id`)
) ENGINE=InnoDB AUTO_INCREMENT=127823 DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci
Remove STRAIGHT_JOIN so that the Optimizer is allowed to try looking at the tables in different orders. The following composite indexes may give the Optimizer better options:
reviewApp_review: INDEX(translatedEnContent, reviewContent,
translatedEnDate, productLink_id, reviewRating)
reviewApp_productlink: INDEX(customer_id, product_id, id)
reviewApp_product: INDEX(owner, id)
translatedEnDate is a DATE? Yet you are testing for "0"??
More
(repeating original, but with aliases to shorten the text)
SELECT
STRAIGHT_JOIN rr.((lots of stuff))
FROM `reviewApp_review` AS rr
INNER JOIN `reviewApp_productlink` AS pl ON (rr.`productLink_id` = pl.`id`)
INNER JOIN `reviewApp_product` AS p ON (pl.`product_id` = p.`id`)
WHERE ( p.`owner` = 'my product'
AND pl.`customer_id` = '1241'
AND (rr.`translatedEnContent` LIKE '%%urban%%'
OR ( rr.`reviewContent` LIKE '%%urban%%'
AND rr.`translatedEnDate` = '0')
) )
ORDER BY rr.`reviewRating` ASC
LIMIT 10
You seem to be gathering all those columns only to throw away all but 10 rows. An optimization is to first find the 10 ids, then fetch the rest of the columns:
SELECT rr.((lots of stuff))
FROM ( SELECT rr2.id
FROM reviewApp_review` AS rr2
INNER JOIN `reviewApp_productlink` AS pl
ON (rr2.`productLink_id` = pl.`id`)
INNER JOIN `reviewApp_product` AS p
ON (pl.`product_id` = p.`id`)
WHERE ( p.`owner` = 'my product'
AND pl.`customer_id` = '1241'
AND ( rr2.`translatedEnContent` LIKE '%%urban%%'
OR ( rr2.`reviewContent` LIKE '%%urban%%'
AND rr2.`translatedEnDate` = '0')
) )
ORDER BY rr2.`reviewRating` ASC
LIMIT 10
) AS ids
INNER JOIN reviewApp_review` AS rr ON ids.id = rr.id
ORDER BY rr.`reviewRating` ASC -- Yes, this needs repeating
p: INDEX(owner)
pl: INDEX(customer_id, product_id)
reviewApp_review: INDEX(productLink_id, reviewRating)
reviewApp_review: INDEX(reviewRating)
These are useless (at least for the current query):
KEY `reviewApp_reviewContent` (`reviewContent`(100)),
KEY `reviewApp_translatedEnContent` (`translatedEnContent`(100)),
Better might be
FULLTEXT(translatedEnContent),
FULLTEXT(reviewContent)
together with
MATCH(reviewContent) AGAINST ('+urban' IN BOOLEAN MODE)
MATCH(translatedEnContent) AGAINST ('+urban' IN BOOLEAN MODE)
But, unfortunately, it won't be cheap to do the subquery. The hope is that 'my product' and '1241' do enough filtering to minimize the usage of LIKE or MATCH.
A minor side note: When you have INDEX(a,b) or UNIQUE(a,b), it is unnecessary to also have INDEX(a). (I see several instances of this.)
Even better...
If possible, get rid of the test on translatedEnDate and combine the two text columns:
FULLTEXT(reviewContent, translatedEnContent)
MATCH(reviewContent, translatedEnContent)
AGAINST ('+urban' IN BOOLEAN MODE)
This will make it possible to start the search with an efficient FT lookup, followed by checking the owner and customer_id. This would avoid a full table scan of the multi-GB reviewApp_review.
I'm building a twitter-like app using node.js for fun and I have multiple tables :
users: to store users' data.
tweets: to store tweets.
likes: to store what users liked what tweet.
retweets: to store what users retweeted what tweet.
following: to store what user is following other users.
CREATE TABLE IF NOT EXISTS `following` (
`user_id` varchar(50) NOT NULL,
`followed_id` varchar(50) NOT NULL,
`date_time` datetime NOT NULL DEFAULT CURRENT_TIMESTAMP,
PRIMARY KEY (`user_id`,`followed_id`)
)
CREATE TABLE IF NOT EXISTS `likes` (
`user_id` varchar(50) NOT NULL,
`tweet_id` varchar(50) NOT NULL,
`date_time` datetime NOT NULL DEFAULT CURRENT_TIMESTAMP,
PRIMARY KEY (`user_id`,`tweet_id`)
)
CREATE TABLE IF NOT EXISTS `retweets` (
`user_id` varchar(50) NOT NULL,
`tweet_id` varchar(50) NOT NULL,
`date_time` datetime NOT NULL DEFAULT CURRENT_TIMESTAMP,
PRIMARY KEY (`user_id`,`tweet_id`)
)
CREATE TABLE IF NOT EXISTS `tweets` (
`tweet_id` varchar(50) CHARACTER SET utf8mb4 COLLATE utf8mb4_0900_ai_ci NOT NULL,
`created_at` datetime NOT NULL DEFAULT CURRENT_TIMESTAMP,
`reply_to_tweet_id` varchar(50) CHARACTER SET utf8mb4 COLLATE utf8mb4_0900_ai_ci NOT NULL,
`reply_to_user_id` varchar(50) CHARACTER SET utf8mb4 COLLATE utf8mb4_0900_ai_ci NOT NULL,
`truncated` tinyint(1) NOT NULL,
`author` varchar(30) NOT NULL,
`text` varchar(255) NOT NULL,
`media` varchar(255) NOT NULL,
`entities` json NOT NULL,
PRIMARY KEY (`tweet_id`)
)
CREATE TABLE IF NOT EXISTS `users` (
`user_id` varchar(30) CHARACTER SET utf8mb4 COLLATE utf8mb4_0900_ai_ci NOT NULL,
`email` varchar(30) NOT NULL,
`username` varchar(30) NOT NULL,
`password` varchar(255) NOT NULL,
`handelname` varchar(255) CHARACTER SET utf8mb4 COLLATE utf8mb4_0900_ai_ci NOT NULL,
`account_url` varchar(255) NOT NULL,
`bio` varchar(255) NOT NULL,
`profile_pic` varchar(255) NOT NULL,
`cover_pic` varchar(255) NOT NULL,
`protected` tinyint(1) NOT NULL,
`created_at` datetime NOT NULL DEFAULT CURRENT_TIMESTAMP,
PRIMARY KEY (`user_id`),
UNIQUE KEY `email` (`email`),
UNIQUE KEY `username` (`username`)
)
and I want to create a new view for the users containing there information along with how many tweets
they did like, the number of users they follow, the number of users follow them and how many tweets they have created.
I want this information in one table but I tried many ways and none of them worked fine for me!
what is the proper solution for such a problem?
You may try below query -
SELECT `user_id`
,LI.LI_CNT no_of_tweets_liked
,FL.FL_CNT no_of_followers
,FL2.FL2_CNT no_of_followee
,RT.RT_CNT no_of_retweets
FROM `users` U
JOIN (SELECT `user_id`, COUNT(`tweet_id`) LI_CNT
FROM `likes`
GROUP BY `user_id`) LI ON U.`user_id` = LI.`user_id`
JOIN (SELECT `user_id`, COUNT(`followed_id`) FL_CNT
FROM `following`
GROUP BY `user_id`) FL ON U.`user_id` = FL.`user_id`
JOIN (SELECT `followed_id`, COUNT(`followed_id`) FL2_CNT
FROM `following`
GROUP BY `followed_id`) FL2 ON U.`user_id` = FL2.`followed_id`
JOIN (SELECT `user_id`, COUNT(`tweet_id`) RT_CNT
FROM `retweets`
GROUP BY `user_id`) RT ON U.`user_id` = RT.`user_id`;
I am working with three SQL tables and I am using INNER JOIN to join these tables. Here is an overview of what the tables look like.
CREATE TABLE `User` (
`id` varchar(32) CHARACTER SET utf8 COLLATE utf8_unicode_ci NOT NULL,
`firstName` varchar(64) CHARACTER SET utf8 COLLATE utf8_unicode_ci NOT NULL,
`lastName` varchar(64) CHARACTER SET utf8 COLLATE utf8_unicode_ci NOT NULL,
`email` varchar(64) CHARACTER SET utf8 COLLATE utf8_unicode_ci NOT NULL,
PRIMARY KEY (`id`) ) ENGINE=InnoDB DEFAULT CHARSET=utf8 |
CREATE TABLE `Ownership` (
`id` int(10) unsigned NOT NULL AUTO_INCREMENT,
`user` varchar(32) COLLATE utf8_unicode_ci NOT NULL,
`certificate` int(10) unsigned NOT NULL,
PRIMARY KEY (`id`),
UNIQUE KEY `unicity` (`user`,`certificate`) ) ENGINE=InnoDB AUTO_INCREMENT=5 DEFAULT CHARSET=utf8 COLLATE=utf8_unicode_ci
CREATE TABLE `Certificate` (
`id` int(10) unsigned NOT NULL AUTO_INCREMENT,
`domain` varchar(64) CHARACTER SET utf8 COLLATE utf8_unicode_ci NOT NULL,
`creationDate` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
`expirationDate` timestamp NOT NULL DEFAULT '0000-00-00 00:00:00',
`type` varchar(32) CHARACTER SET utf8 COLLATE utf8_unicode_ci NOT NULL,
PRIMARY KEY (`id`) ) ENGINE=InnoDB AUTO_INCREMENT=4 DEFAULT CHARSET=utf8
The request to join the tables is the following :
select user, lastName, domain
from Ownership
INNER JOIN User ON Ownership.user = User.id
INNER JOIN Certificate ON Ownership.certificate = Certificate.id;`
I have this type of result for example :
What I would like is to retrieve only one user for one domain name and create an exception to do the join only if the certificate.domain value does not already exists.
Is this something possible with MYSQL ?
Thank you
You need a pre-query to at least qualify some uniqueness component associated with a given domain. To get that, lets consider this.
select
C.domain,
min( O.user ) JustOneUser
from
Certificate C
JOIN Ownership O
on C.ID = O.Certificate
group by
C.Domain
So this will get one user for a given domain. Now, use THAT result to get the user associated to the domain.
select
PQ.Domain,
U.LastName,
U.FirstName
from
(select
C.domain,
min( O.user ) JustOneUser
from
Certificate C
JOIN Ownership O
on C.ID = O.Certificate
group by
C.Domain ) PQ
JOIN User U
on PQ.JustOneUser = U.ID
I am trying to insert values into my comments table and I am getting a error. Its saying that I can not add or update child row and I have no idea what that means.
My schema looks something like this:
--
-- Baza danych: `koxu1996_test`
--
-- --------------------------------------------------------
--
-- Struktura tabeli dla tabeli `user`
--
CREATE TABLE IF NOT EXISTS `user` (
`id` int(8) NOT NULL AUTO_INCREMENT,
`username` varchar(32) COLLATE utf8_bin NOT NULL,
`password` varchar(64) COLLATE utf8_bin NOT NULL,
`password_real` char(32) COLLATE utf8_bin NOT NULL,
`email` varchar(32) COLLATE utf8_bin NOT NULL,
`code` char(8) COLLATE utf8_bin NOT NULL,
`activated` enum('0','1') COLLATE utf8_bin NOT NULL DEFAULT '0',
`activation_key` char(32) COLLATE utf8_bin NOT NULL,
`reset_key` varchar(32) COLLATE utf8_bin NOT NULL,
`name` varchar(32) COLLATE utf8_bin NOT NULL,
`street` varchar(32) COLLATE utf8_bin NOT NULL,
`house_number` varchar(32) COLLATE utf8_bin NOT NULL,
`apartment_number` varchar(32) COLLATE utf8_bin NOT NULL,
`city` varchar(32) COLLATE utf8_bin NOT NULL,
`zip_code` varchar(32) COLLATE utf8_bin NOT NULL,
`phone_number` varchar(16) COLLATE utf8_bin NOT NULL,
`country` int(8) NOT NULL,
`province` int(8) NOT NULL,
`pesel` varchar(32) COLLATE utf8_bin NOT NULL,
`register_time` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP,
`authorised_time` datetime NOT NULL,
`edit_time` datetime NOT NULL,
`saldo` decimal(9,2) NOT NULL,
`referer_id` int(8) NOT NULL,
`level` int(8) NOT NULL,
PRIMARY KEY (`id`),
KEY `country` (`country`),
KEY `province` (`province`),
KEY `referer_id` (`referer_id`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8 COLLATE=utf8_bin AUTO_INCREMENT=83 ;
and the mysql statement I am trying to do looks something like this:
INSERT INTO `user` (`password`, `code`, `activation_key`, `reset_key`, `register_time`, `edit_time`, `saldo`, `referer_id`, `level`) VALUES (:yp0, :yp1, :yp2, :yp3, NOW(), NOW(), :yp4, :yp5, :yp6). Bound with :yp0='fa1269ea0d8c8723b5734305e48f7d46', :yp1='F154', :yp2='adc53c85bb2982e4b719470d3c247973', :yp3='', :yp4='0', :yp5=0, :yp6=1
the error I get looks like this:
SQLSTATE[23000]: Integrity constraint violation: 1452 Cannot add or
update a child row: a foreign key constraint fails
(koxu1996_test.user, CONSTRAINT user_ibfk_1 FOREIGN KEY
(country) REFERENCES country_type (id) ON DELETE NO ACTION ON
UPDATE NO ACTION)
It just simply means that the value for column country on table comments you are inserting doesn't exist on table country_type or you are not inserting value for country on table user.
Bear in mind that the values of column country on table comments is dependent on the values of ID on table country_type.
You have foreign keys between this table and another table and that new row would violate that constraint.
You should be able to see the constraint if you run show create table user, it shows up as CONSTRAINT... and it shows what columns reference what tables/columns.
In this case country references country_type (id) and you are not specifying the value of country. You need to put a value that exists in country_type.
Just to throw in my own issue in case it can help someone else, I was copy/pasting entries in my migration files and messed up by putting quotes around an integer. Since the value I was trying to enter was considered a string going into an integer field that was referencing another integer, this error came up.
Another option could be thath your primary key in source table IS NOT unsigned, so I solved same insert with (notice id int(8) unsigned):
CREATE TABLE IF NOT EXISTS user ( id int(8) unsigned NOT NULL
AUTO_INCREMENT, username varchar(32) COLLATE utf8_bin NOT NULL,
password varchar(64) COLLATE utf8_bin NOT NULL, password_real
char(32) COLLATE utf8_bin NOT NULL, email varchar(32) COLLATE
utf8_bin NOT NULL, code char(8) COLLATE utf8_bin NOT NULL,
activated enum('0','1') COLLATE utf8_bin NOT NULL DEFAULT '0',
activation_key char(32) COLLATE utf8_bin NOT NULL, reset_key
varchar(32) COLLATE utf8_bin NOT NULL, name varchar(32) COLLATE
utf8_bin NOT NULL, street varchar(32) COLLATE utf8_bin NOT NULL,
house_number varchar(32) COLLATE utf8_bin NOT NULL,
apartment_number varchar(32) COLLATE utf8_bin NOT NULL, city
varchar(32) COLLATE utf8_bin NOT NULL, zip_code varchar(32)
COLLATE utf8_bin NOT NULL, phone_number varchar(16) COLLATE
utf8_bin NOT NULL, country int(8) NOT NULL, province int(8)
NOT NULL, pesel varchar(32) COLLATE utf8_bin NOT NULL,
register_time timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP,
authorised_time datetime NOT NULL, edit_time datetime NOT NULL,
saldo decimal(9,2) NOT NULL, referer_id int(8) NOT NULL,
level int(8) NOT NULL, PRIMARY KEY (id), KEY country
(country), KEY province (province), KEY referer_id
(referer_id) ) ENGINE=InnoDB DEFAULT CHARSET=utf8 COLLATE=utf8_bin
AUTO_INCREMENT=83 ;
In my case, the value was empty for target table column while the reference table column has it. hence was throwing this error.
In my case the references values does not corresponding on the related table. Just be sure the values exist on reference table and currents rows has corresponding valid values.
In my case , its no problem in SQL commands , but the inputs was not sending as $request , so in php.ini file :
max_input_vars was 1000 by default and I changed it to :
max_input_vars = 2000
then you have to restart web server .
You should pass NULL for empty values. Not empty string such as ''
in my case this was the error:
Illuminate\Database\QueryException
SQLSTATE[23000]: Integrity constraint violation: 1048 Column 'brand_name_ur' cannot be null (SQL: insert into brands (brand_name_en, brand_name_ur, brand_slug_en, brand_slug_ur, brand_image) values (SAMI, ?, sami, سامی, upload/brand/1742772943246486.png))
and fixed by
'brand_name_ur' => $request -> brand_image_ur,
< changing brand_image_ur to brand_name_ur. Noob mistak. took my hour.
thanks to #Tsimtsum
By running a request like :
SELECT user.id as user_id, country as user_country_type_id, country_type.id as country_type_id
FROM `user`
LEFT JOIN country_type ON country_type.id = user.country
WHERE country_type.id is null;
it probably returns some rows and it should not !
You can delete it (probably a bad idea if you are in production) using request like :
DELETE user
FROM `user`
LEFT JOIN country_type ON country_type.id = user.country
WHERE country_type.id is null;
and adding constraint should be done after !
add to your migrations
$this->addSql('SET foreign_key_checks = 0');
.....
$this->addSql('SET foreign_key_checks = 1');
Setup: Contact database using 4 tables
Contacts
Cities
States
Zips
Structure:
CREATE TABLE `contacts` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`last` varchar(100) CHARACTER SET latin1 COLLATE latin1_general_ci DEFAULT NULL,
`first` varchar(100) CHARACTER SET latin1 COLLATE latin1_general_ci DEFAULT NULL,
`prefix` varchar(50) CHARACTER SET latin1 COLLATE latin1_general_ci DEFAULT NULL,
`suffix` varchar(50) CHARACTER SET latin1 COLLATE latin1_general_ci DEFAULT NULL,
`address` varchar(100) CHARACTER SET latin1 COLLATE latin1_general_ci DEFAULT NULL,
`address_1` varchar(100) CHARACTER SET latin1 COLLATE latin1_general_ci DEFAULT NULL,
`city_id` int(100) DEFAULT NULL,
`state_id` int(20) DEFAULT NULL,
`alt_address_1` varchar(255) CHARACTER SET latin1 COLLATE latin1_general_ci DEFAULT NULL,
`alt_address_2` varchar(255) CHARACTER SET latin1 COLLATE latin1_general_ci DEFAULT NULL,
`alt_city` varchar(100) CHARACTER SET latin1 COLLATE latin1_general_ci DEFAULT NULL,
`alt_state` varchar(20) CHARACTER SET latin1 COLLATE latin1_general_ci DEFAULT NULL,
`alt_zip` varchar(15) CHARACTER SET latin1 COLLATE latin1_general_ci DEFAULT NULL,
`publish_name` varchar(255) CHARACTER SET latin1 COLLATE latin1_general_ci DEFAULT NULL,
`salutation` varchar(255) CHARACTER SET latin1 COLLATE latin1_general_ci DEFAULT NULL,
`mail_label` varchar(255) CHARACTER SET latin1 COLLATE latin1_general_ci DEFAULT NULL,
`solicitor` varchar(100) CHARACTER SET latin1 COLLATE latin1_general_ci DEFAULT NULL,
`is_volunteer` tinyint(1) DEFAULT NULL,
`is_sponsor` tinyint(1) DEFAULT '0',
`is_company` tinyint(1) DEFAULT '0',
`is_foundation` tinyint(1) DEFAULT '0',
`status` varchar(15) CHARACTER SET latin1 COLLATE latin1_general_ci DEFAULT NULL,
`created_on` datetime NOT NULL,
`created_by` varchar(30) CHARACTER SET latin1 COLLATE latin1_general_ci DEFAULT NULL,
`modified_on` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
`modified_by` varchar(100) CHARACTER SET latin1 COLLATE latin1_general_ci DEFAULT NULL,
`agency_id` int(25) DEFAULT NULL,
`primary_id` int(11) DEFAULT NULL,
PRIMARY KEY (`id`),
KEY `primary_id` (`primary_id`)
) ENGINE=InnoDB AUTO_INCREMENT=3008 DEFAULT CHARSET=utf8
CREATE TABLE `cities` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`city` varchar(50) CHARACTER SET latin1 COLLATE latin1_general_ci DEFAULT NULL,
`stateid` int(11) DEFAULT NULL,
PRIMARY KEY (`id`),
UNIQUE KEY `city` (`city`)
) ENGINE=InnoDB AUTO_INCREMENT=128 DEFAULT CHARSET=utf8
CREATE TABLE `states` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`abbreviation` varchar(2) CHARACTER SET latin1 COLLATE latin1_general_ci DEFAULT NULL,
`state` varchar(20) CHARACTER SET latin1 COLLATE latin1_general_ci DEFAULT NULL,
PRIMARY KEY (`id`),
UNIQUE KEY `state` (`state`),
UNIQUE KEY `abbreviation` (`abbreviation`),
KEY `id` (`id`)
) ENGINE=InnoDB AUTO_INCREMENT=52 DEFAULT CHARSET=utf8
CREATE TABLE `zips` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`zip` varchar(10) CHARACTER SET latin1 COLLATE latin1_general_ci DEFAULT NULL,
`cityid` int(100) DEFAULT NULL,
PRIMARY KEY (`id`),
UNIQUE KEY `zip` (`zip`)
) ENGINE=InnoDB AUTO_INCREMENT=128 DEFAULT CHARSET=utf8
I have filled the contacts with 111 contacts, the states are simply all of the states, and cities have corresponding id keys that relate to the state id, zip codes have a key to match to a city.
The query is to generate a list of people to match up to the proper fields. Here is the query.
SELECT concat(contacts.last,' ', contacts.first) as name
, cities.city
, zips.zip
FROM contacts
JOIN cities
ON cities.id = contacts.city_id
JOIN states ON states.id = contacts.state_id
JOIN zips ON zips.cityid = cities.id
This query, return 338 rows, of a possible 11 contacts. There obvious duplicates. This happens when I join zip codes, which because they belong to more than 1 city, it gets matched for each city (I think thats what is happening). Anyone have an answer on how to properly join these tables?
Thank you.
Rich
I believe that you should re-think the surrogate key usage on a lot of these tables and use natural keys where possible. Taking the state table for an example, in most cases it will be acceptable to simply use the state short (ie. TX vs. Texas) for data and display purposes. This means that if you removed the incrementing ID on the states table and used a natural key for each state, you would reduce the necessecity of a join in 90% of cases.
Then use state.abbriviation as the FK in tables that need to store state values. Extending this to zipcodes and cities, you can FK the state abbr to the city table, and make a compound FK form the cities table to the contacts table giving you a key for both the city and the state at the same time.
Example schema (excluded zipcodes table and shortened contacts table):
CREATE TABLE IF NOT EXISTS `states` (
`state_id` CHAR(2) NOT NULL ,
`name` VARCHAR(45) NULL ,
PRIMARY KEY (`state_id`) ,
UNIQUE INDEX `state_name` (`name` ASC)
)
ENGINE = InnoDB;
CREATE TABLE IF NOT EXISTS `cities` (
`state_id` CHAR(2) NOT NULL ,
`city_name` VARCHAR(255) NOT NULL ,
PRIMARY KEY (`state_id`, `city_name`) ,
INDEX `fk_city_state_id` (`state_id` ASC) ,
CONSTRAINT `fk_city_state_id`
FOREIGN KEY (`state_id` )
REFERENCES `states` (`state_id` )
ON DELETE NO ACTION
ON UPDATE NO ACTION
)
ENGINE = InnoDB;
CREATE TABLE IF NOT EXISTS `contacts` (
`contacts_id` INT NOT NULL AUTO_INCREMENT ,
`state` CHAR(2) NULL ,
`city` VARCHAR(255) NULL ,
PRIMARY KEY (`contacts_id`) ,
INDEX `fk_contact_city` (`state` ASC, `city` ASC) ,
INDEX `fk_contact_state` (`state` ASC) ,
CONSTRAINT `fk_contact_city`
FOREIGN KEY (`state` , `city` )
REFERENCES `cities` (`state_id` , `city_name` )
ON DELETE NO ACTION
ON UPDATE NO ACTION,
CONSTRAINT `fk_contact_state`
FOREIGN KEY (`state` )
REFERENCES `states` (`state_id` )
ON DELETE NO ACTION
ON UPDATE NO ACTION
)
ENGINE = InnoDB;
-- -----------------------------------------------------
-- Data for table `states`
-- -----------------------------------------------------
SET AUTOCOMMIT=0;
INSERT INTO `states` (`state_id`, `name`) VALUES ('TX', 'Texas');
INSERT INTO `states` (`state_id`, `name`) VALUES ('CA', 'California');
INSERT INTO `states` (`state_id`, `name`) VALUES ('OR', 'Oregon');
COMMIT;
-- -----------------------------------------------------
-- Data for table `cities`
-- -----------------------------------------------------
SET AUTOCOMMIT=0;
INSERT INTO `cities` (`state_id`, `city_name`) VALUES ('CA', 'modesto');
INSERT INTO `cities` (`state_id`, `city_name`) VALUES ('OR', 'protland');
INSERT INTO `cities` (`state_id`, `city_name`) VALUES ('TX', 'Dallas');
COMMIT;
Now your query is simplified except in the extreme case where you'd need a full state nomenclature:
SELECT
concat(contacts.last,' ', contacts.first) as name,
city,
state,
zip
FROM contacts
WHERE {INSERTWHERE}
your tables a properly joined. i think the problem you have here is that you have normalized your data improperly and gone too far. just store the address as provided. don't try and split it in to table with numeric ID's.
for example, what possible benefit do you get by storing the state as a number in the contact table instead of just the state code? and the same question applies to city.