MySQL performance with nested sub query - mysql

I have two tables messages and members. I tried joining tables without having a nested query but it does not reflect the join on members. So, I initially thought that I could do the following
SELECT M1.*, COUNT(M2.emid) AS replies FROM messages M1
LEFT JOIN messages M2
ON M2.thread = M1.emid
INNER JOIN members M
ON M.meid = M1.emitter
WHERE
M1.thread is NULL AND
M1.receiver = 2
GROUP BY
M1.emid
but it does not seem to join the corresponding member. Then I tried this and it gives me the result that I need but I would like to know if there is a way to accomplish the same result using joins without the nested query
SELECT * FROM (
SELECT M1.*, COUNT(M2.emid) AS replies FROM messages M1
LEFT JOIN messages M2
ON M2.thread = M1.emid
WHERE
M1.thread is NULL AND
M1.receiver = 2
GROUP BY
M1.emid
) O INNER JOIN members M ON O.receiver = M.meid
-- Table structure for table members
CREATE TABLE `members` (
`meid` bigint(64) NOT NULL,
`name` varchar(32) DEFAULT NULL,
`lastname` varchar(32) DEFAULT NULL,
`email` varchar(128) NOT NULL,
`mobile` char(10) DEFAULT NULL,
`college` bigint(64) NOT NULL,
`major` bigint(64) NOT NULL,
`password` varchar(256) NOT NULL,
`oauth` varchar(128) DEFAULT NULL,
`confirmed` tinyint(4) DEFAULT NULL,
`active` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
`joined` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP
) ENGINE=InnoDB DEFAULT CHARSET=latin1;
-- Table structure for table messages
CREATE TABLE `messages` (
`emid` bigint(20) NOT NULL,
`emitter` bigint(20) NOT NULL,
`receiver` bigint(20) NOT NULL,
`thread` bigint(20) DEFAULT NULL,
`opened` tinyint(4) DEFAULT '0',
`message` blob NOT NULL,
`timecard` datetime DEFAULT CURRENT_TIMESTAMP
) ENGINE=InnoDB DEFAULT CHARSET=latin1;

Related

MySql query is slow with join - how to speed it up

I have to export 554k records from our mysql db. At the current rate it will take 5 days to export the data and the slowness is mainly caused by the query below. The data structure consists of
Companies
--Contacts
----(Contact)Activities
For the contacts, we have an index on company_id. On the activities table, we have an index for contact_id and company_id which map back to the respective contacts and companies tables.
I need to grab each contact and the latest activity date that they have. This is the query that I'm running and it takes about .5 second to execute.
Select *
from contacts
left outer join (select occurred_at
,contact_id
from activities
where occurred_at is not null
group by contact_id
order by occurred_at desc) activities
on contacts.id = activities.contact_id
where company_id = 20
If I remove the join and just select * from contacts where company_id=20 the query executes in .016 sec.
If I use Explain for info on the join query I get this
Any ideas on how I can speed this up?
Edit:
Here are the table definitions.
CREATE TABLE `companies` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`name` varchar(255) COLLATE utf8_unicode_ci DEFAULT NULL,
`street_address` varchar(255) COLLATE utf8_unicode_ci DEFAULT NULL,
`city` varchar(255) COLLATE utf8_unicode_ci DEFAULT NULL,
`state` varchar(255) COLLATE utf8_unicode_ci DEFAULT NULL,
`county` varchar(255) COLLATE utf8_unicode_ci DEFAULT NULL,
`website` varchar(255) COLLATE utf8_unicode_ci DEFAULT NULL,
`external_id` int(11) DEFAULT NULL,
`created_at` datetime DEFAULT NULL,
`updated_at` datetime DEFAULT NULL,
`user_id` int(11) DEFAULT NULL,
`falloff_date` date DEFAULT NULL,
`zipcode` varchar(255) COLLATE utf8_unicode_ci DEFAULT NULL,
`phone` varchar(255) COLLATE utf8_unicode_ci DEFAULT NULL,
`company_id` int(11) DEFAULT NULL,
`order_count` int(11) NOT NULL DEFAULT '0',
`active_job_count` int(11) NOT NULL DEFAULT '0',
`duplicate_of` int(11) DEFAULT NULL,
`warm_date` datetime DEFAULT NULL,
`employee_size` int(11) DEFAULT NULL,
`dup_checked` tinyint(1) DEFAULT '0',
`rating` int(11) DEFAULT NULL,
`delinquent` tinyint(1) DEFAULT '0',
`cconly` tinyint(1) DEFAULT '0',
PRIMARY KEY (`id`),
KEY `index_companies_on_name` (`name`),
KEY `index_companies_on_user_id` (`user_id`),
KEY `index_companies_on_company_id` (`company_id`),
KEY `index_companies_on_external_id` (`external_id`),
KEY `index_companies_on_state_and_dup_checked` (`id`,`state`,`dup_checked`,`duplicate_of`),
KEY `index_companies_on_dup_checked` (`id`,`dup_checked`),
KEY `index_companies_on_dup_checked_name` (`dup_checked`,`name`),
KEY `index_companies_on_county` (`county`,`state`)
) ENGINE=InnoDB AUTO_INCREMENT=15190300 DEFAULT CHARSET=utf8 COLLATE=utf8_unicode_ci;
CREATE TABLE `contacts` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`first_name` varchar(255) COLLATE utf8_unicode_ci DEFAULT NULL,
`last_name` varchar(255) COLLATE utf8_unicode_ci DEFAULT NULL,
`title` varchar(255) COLLATE utf8_unicode_ci DEFAULT NULL,
`phone` varchar(255) COLLATE utf8_unicode_ci DEFAULT NULL,
`extension` varchar(255) COLLATE utf8_unicode_ci DEFAULT NULL,
`fax` varchar(255) COLLATE utf8_unicode_ci DEFAULT NULL,
`email` varchar(255) COLLATE utf8_unicode_ci DEFAULT NULL,
`active` tinyint(1) DEFAULT NULL,
`main` tinyint(1) DEFAULT NULL,
`company_id` int(11) DEFAULT NULL,
`created_at` datetime DEFAULT NULL,
`updated_at` datetime DEFAULT NULL,
`external_id` int(11) DEFAULT NULL,
`second_phone` varchar(255) COLLATE utf8_unicode_ci DEFAULT NULL,
PRIMARY KEY (`id`),
KEY `index_contacts_on_company_id` (`company_id`),
KEY `index_contacts_on_first_name` (`first_name`),
KEY `index_contacts_on_last_name` (`last_name`),
KEY `index_contacts_on_phone` (`phone`),
KEY `index_contacts_on_email` (`email`)
) ENGINE=InnoDB AUTO_INCREMENT=11241088 DEFAULT CHARSET=utf8 COLLATE=utf8_unicode_ci;
CREATE TABLE `activities` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`kind` int(11) DEFAULT NULL,
`contact_id` int(11) DEFAULT NULL,
`call_status` int(11) DEFAULT NULL,
`occurred_at` datetime DEFAULT NULL,
`notes` text COLLATE utf8_unicode_ci,
`user_id` int(11) DEFAULT NULL,
`scheduled_for` datetime DEFAULT NULL,
`priority` tinyint(1) DEFAULT NULL,
`company_id` int(11) DEFAULT NULL,
`created_at` datetime DEFAULT NULL,
`updated_at` datetime DEFAULT NULL,
`from_user_id` int(11) DEFAULT NULL,
`to_user_id` int(11) DEFAULT NULL,
PRIMARY KEY (`id`),
KEY `index_activities_on_contact_id` (`contact_id`),
KEY `index_activities_on_user_id` (`user_id`),
KEY `index_activities_on_company_id` (`company_id`)
) ENGINE=InnoDB AUTO_INCREMENT=515340 DEFAULT CHARSET=utf8 COLLATE=utf8_unicode_ci;
This is a greatest-n-per-group query, which comes up frequently on Stack Overflow.
Here's a solution that uses a MySQL 8.0 window function:
WITH latest_activities AS (
SELECT contact_id, occurred_at,
ROW_NUMBER() OVER (PARTITION BY contact_id ORDER BY occurred_at DESC) AS rn
FROM activities
)
SELECT *
FROM contacts AS c
LEFT OUTER JOIN latest_activities
ON c.id = latest_activities.contact_id AND latest_activities.rn = 1
WHERE c.company_id = 20
Here's a solution that should work on pre-8.0 versions:
SELECT c.*, a.*
FROM contacts AS c
LEFT OUTER JOIN activities AS a ON a.contact_id = c.id
LEFT OUTER JOIN activities AS a2 ON a2.contact_id = c.id
AND a2.occurred_at > a.occurred_at
WHERE c.company_id = 20
AND a2.contact_id IS NULL;
Another solution:
SELECT c.*, a.*
FROM contacts AS c
LEFT OUTER JOIN activities AS a ON a.contact_id = c.id
LEFT OUTER JOIN (
SELECT c2.contact_id, MAX(a2.occurred_at) AS occurred_at
FROM activities AS a2
INNER JOIN contacts AS c2 ON a2.contact_id = c2.id
WHERE c2.company_id = 20
GROUP BY c2.contact_id ORDER BY NULL
) AS latest_activities
ON latest_activities.contact_id = c.id
AND latest_activities.occurred_at = a.occurred_at
WHERE c.company_id = 20
It would be helpful to create a new index on activities (contact_id, occurred_at).
Don't use subqueries in the FROM clause if you can help it. They impede the MySQL optimizer. So, if you want one row:
Select c.*, a.occurred_at
from contacts c left outer join
from activities a
on c.id = a.contact_id and
a.occurred_at is not null
where c.company_id = 20
order by a.occurred_at desc
limit 1;
If you want one row per contact_id:
Select c.*, a.occurred_at
from contacts c left outer join
from activities a
on c.id = a.contact_id and
a.occurred_at is not null and
a.occurred_at = (select max(a2.occurred_at)
from activities a2
where a2.contact_id = a.contact_id
)
where c.company_id = 20
order by a.occurred_at desc
limit 1;
This can make use of an index on activities(contact_id, occured_at). and contact(company_id, contact_id).
Your query is doing one thing that is a clear no-no -- and no longer supported by the default settings in the most recent versions of MySQL. You have unaggregated columns in a select that are not in the group by. The contact_id should be generating an error.
I feel like I am overlooking something with how complicated the other answers are, but I would think this would be all you need.
SELECT c.*
, MAX(a.occurred_at) AS occurred_at
FROM contacts AS c
LEFT JOIN activities AS a
ON c.id = a.contact_id AND a.occurred_at IS NOT NULL
WHERE c.company_id = 20
GROUP BY c.id;
Notes: (1) this assumes you didn't actually want the duplicate contact_id from your original subquery to be in the final results. (2) This also assumes your server is not configured to require a full group by; if it is, you will need to manually expand c.* into the full column list, and copy that list to the GROUP BY clause as well.
Expanding on dnoeth's comments to your question; if you are not querying each company separately for a particular reason (chunking for load, code structure handling this also handles other stuff company by company, whatever), you could tweak the above query like so to get all your results in one query.
SELECT con.*
, MAX(a.occurred_at) AS occurred_at
FROM companies AS com
INNER JOIN contacts AS con ON com.id = con.company_id
LEFT JOIN activities AS a
ON con.id = a.contact_id AND a.occurred_at IS NOT NULL
WHERE [criteria for companies chosen to be queried]
GROUP BY con.id
ORDER BY con.company_id, con.id
;

Improve MySQL nested select performance with join

I have seen many samples to improve MySQL nested selects with joins, but I can't figure this out for this query:
SELECT * FROM messages WHERE answer = 'SuccessSubscribed' AND phone NOT IN
(SELECT phone FROM messages WHERE answer = 'SuccessUnSubscribed');
the query finds people who have subscribed but never unsubscribed.
Table structure:
CREATE TABLE `messages` (
`id` int(10) unsigned NOT NULL AUTO_INCREMENT,
`phone` varchar(12) COLLATE utf8_persian_ci NOT NULL,
`content` varchar(300) COLLATE utf8_persian_ci NOT NULL,
`flags` int(10) unsigned NOT NULL DEFAULT '0',
`answer` varchar(50) COLLATE utf8_persian_ci DEFAULT NULL,
....,
PRIMARY KEY (`id`),
....
) ENGINE=InnoDB CHARSET=utf8 COLLATE=utf8_persian_ci
Instead of the NOT IN, you can use LEFT JOIN with NULL check.
SELECT M1.*
FROM messages M1
LEFT JOIN messages M2 ON M2.phone = M1.phone AND M2.answer = 'SuccessUnSubscribed'
WHERE M1.answer = 'SuccessSubscribed' AND M2.phone IS NULL

inner joining on a update

I am trying to check if the user owns a course, I am trying to do this by getting the transaction_id (someone who's applied to a course) and then left join the courses and check the course_user against a session where the transaction_course is equal to course id
My SQL
UPDATE training_transactions
LEFT JOIN training
ON training_transactions.training_transaction_id =
training.course_id
SET training_transactions.training_transaction_status = 'declined'
WHERE training_transactions.training_transaction_id = ?
AND training.course_user = ?
training_transaction:
CREATE TABLE IF NOT EXISTS `training_transactions` (
`training_transaction_id` int(11) NOT NULL,
`training_transaction_user` int(11) NOT NULL,
`training_transaction_course` int(11) NOT NULL,
`training_transaction_status` varchar(50) NOT NULL,
`training_transaction_enabled` varchar(50) NOT NULL DEFAULT 'enabled',
`training_transaction_date` datetime NOT NULL DEFAULT CURRENT_TIMESTAMP
) ENGINE=InnoDB DEFAULT CHARSET=latin1;
training
CREATE TABLE IF NOT EXISTS `training` (
`course_id` int(11) NOT NULL,
`course_user` int(11) NOT NULL,
`course_type` varchar(255) NOT NULL,
`course_name` varchar(255) NOT NULL,
`course_location` varchar(255) NOT NULL,
`course_duration` varchar(255) NOT NULL,
`course_fitness_type` varchar(255) NOT NULL,
`course_instructor_name` varchar(255) NOT NULL,
`course_price` int(15) NOT NULL,
`course_start_date` date NOT NULL,
`course_max_attendees` int(8) NOT NULL,
`course_accommodation` varchar(255) NOT NULL,
`course_accommodation_price` varchar(255) NOT NULL,
`course_status` varchar(50) NOT NULL,
`course_enabled` varchar(10) NOT NULL DEFAULT 'enabled'
) ENGINE=InnoDB AUTO_INCREMENT=24 DEFAULT CHARSET=latin1;
So my question, how can I update if the transaction id matches the course_id and the user's session (below) matches the course_user?
Session::get('id') // user id
You should be testing training_transaction_course in the WHEN clause, not training_transaction_id. And you should use INNER JOIN, not LEFT JOIN; you only need LEFT JOIN if you need to get rows in the first table that have no matches in the second table.
UPDATE training_transactions AS tt
INNER JOIN training AS t ON tt.training_transaction_id = t.course_id
SET tt.training_transaction_status = 'declined'
WHERE tt.training_transaction_course = ?
AND t.course_user = ?

sql can't figure out the query

I have three tables:
CREATE TABLE IF NOT EXISTS `contacts` (
`id` int(10) unsigned NOT NULL AUTO_INCREMENT,
`providerId` int(10) unsigned NOT NULL DEFAULT '0',
`requestId` int(10) unsigned NOT NULL DEFAULT '0',
`status` binary(1) NOT NULL DEFAULT '0',
PRIMARY KEY (`id`)
)
CREATE TABLE IF NOT EXISTS `messages` (
`id` int(255) NOT NULL AUTO_INCREMENT,
`fromuid` int(255) NOT NULL,
`touid` int(255) NOT NULL,
`sentdt` datetime NOT NULL,
`read` tinyint(1) NOT NULL DEFAULT '0',
`readdt` datetime DEFAULT NULL,
`messagetext` longtext CHARACTER SET utf8 COLLATE utf8_bin NOT NULL,
PRIMARY KEY (`id`),
KEY `id` (`id`)
)
CREATE TABLE IF NOT EXISTS `users` (
`id` bigint(11) NOT NULL AUTO_INCREMENT,
`name` varchar(255) NOT NULL,
`email` varchar(255) DEFAULT NULL,
`mobile` varchar(15) NOT NULL,
`password` varchar(255) NOT NULL,
`city` varchar(255) NOT NULL,
`zip` varchar(15) DEFAULT NULL,
`device` varchar(50) DEFAULT NULL,
`version` varchar(10) DEFAULT NULL,
`photo` varchar(255) DEFAULT NULL,
`created` datetime NOT NULL,
`live` enum('0','1') NOT NULL DEFAULT '1',
`authenticationTime` datetime NOT NULL,
`userKey` varchar(255) DEFAULT NULL,
`IP` varchar(50) DEFAULT NULL,
`port` int(10) DEFAULT '0',
PRIMARY KEY (`id`),
KEY `firstname` (`mobile`,`city`,`zip`)
)
And this SQL query that finds out friends/contacts for specified user (user id 1 in this case):
SELECT u.id
,u.mobile
,u.name
,(NOW() - u.authenticationTime) AS authenticateTimeDifference
,u.IP
,f.providerid
,f.requestid
,f.status
,u.port
FROM contacts f
LEFT JOIN users u ON u.id =
IF (
f.providerid = 1
,f.requestid
,f.providerid
) WHERE (
f.providerid = 1
AND f.status = 1
)
OR f.requestid = 1
That works fine but I want to be able to also join messages table and show user's friends/contacts who have talked latest (meaning latest conversations first) with order by messages.sentdt desc option but I am unable to figure out how to do that, I tried all joins but none worked :(
Your help will be greatly appreciated. Thanks
Update
Here is sample data above query returns:
In that same resultset, I want to be able to sort based on order by messages.sentdt desc but I am not sure how to pull that in and sort resultset by latest message first
Try this:
select u.id
, u.mobile
, u.name
, (NOW() - u.authenticationTime) as authenticateTimeDifference
, u.IP
, f.providerid
, f.requestid
, f.status
, u.port
from contacts f
left join users u
on u.id = if (f.providerid = 1, f.requestid, f.providerid)
left join (select fromuid, max(sentdt) as sentdt from messages group by fromuid) m
on m.fromuid = if (f.providerid = 1, f.providerid, f.requestid)
where (f.providerid = 1 and f.status = 1)
or f.requestid = 1
order by m.sentdt

getting number of records from 2 tables - one to many relationship

I have problem with search query that i have to built on the fly to return records from the database.
I have 2 tables: adds andadds_filters`. For the sake of simplicity, i make the table adds shorter than it is, removing some of the (irrelevant) fields
My table structure:
CREATE TABLE IF NOT EXISTS `adds` (
`addid` int(11) NOT NULL AUTO_INCREMENT,
`memberid` int(11) NOT NULL,
`isnew` int(11) NOT NULL,
`catid` int(11) NOT NULL,
`manufacturerid` int(11) NOT NULL,
`modelid` varchar(255) DEFAULT NULL,
`colorid` int(11) DEFAULT NULL,
`geographicareaid` int(45) NOT NULL,
`addtypeid` varchar(45) NOT NULL,
`addcreatedon` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
`addvalidfrom` date NOT NULL,
`addvaliduntil` date NOT NULL,
`addcreatedfromip` varchar(255) NOT NULL,
`yearofmanufacturing` varchar(255) DEFAULT NULL,
`monthofmanufacturing` int(11) DEFAULT NULL,
`hoursused` int(11) DEFAULT NULL,
`cc2` int(11) DEFAULT NULL,
`horsepowers` int(11) DEFAULT NULL,
`metalic` tinyint(4) DEFAULT NULL,
`isdamaged` tinyint(4) DEFAULT NULL,
`price` float DEFAULT NULL,
`hasvat` tinyint(4) NOT NULL,
`canbenegotiated` tinyint(4) DEFAULT NULL,
`addtitle` varchar(255) DEFAULT NULL,
`addtext` text NOT NULL,
`youtubevideo` varchar(255) DEFAULT NULL,
`visible` tinyint(4) DEFAULT NULL,
`ff1` varchar(255) DEFAULT NULL,
`ff2` varchar(255) DEFAULT NULL,
`ff3` varchar(255) DEFAULT NULL,
`ff4` varchar(255) DEFAULT NULL,
PRIMARY KEY (`addid`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8 AUTO_INCREMENT=43 ;
CREATE TABLE IF NOT EXISTS `adds_filters` (
`addfilterid` int(11) NOT NULL AUTO_INCREMENT,
`addid` int(11) NOT NULL,
`filterid` int(11) NOT NULL,
PRIMARY KEY (`addfilterid`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8 AUTO_INCREMENT=45 ;
Here is the fiddle
Problem is that user can add more than one filter for each adds, that is if the
vehicle has AC,GPS, removable cabin and so on...This data is stored in the adds_filters. So, one add can have many filters.
How the query should look like when user choose that catid is 1, manufacturerid is 1 and then users check filters with ids
67 and 158 for example?
I would prefer view over the query, but i have no idea how can i build such view. Reason from preferring the view is that in such a case, i will be able to use select * instead of complicated queries...
Any help will be deeply appreciated.
With this table structure, you gonna need subqueries for each checked filterid, if you want filterids to be displayed.
Without displaying, you can just use DISTINCT and IN.
Here is the query without displaying filterids
SELECT DISTINCT a.*
FROM adds a
JOIN adds_filters f
ON a.`addid` = f.`addid`
WHERE a.`catid` = 1
AND a.`manufacturerid` = 1
AND f.`filterid` IN (67, 158);
Here is the query, that displays two different filterids:
SELECT t1.*, t2.filterid as filterid2
FROM
(
SELECT DISTINCT a.*,
f.`filterid`
FROM adds a
JOIN adds_filters f
ON a.`addid` = f.`addid`
WHERE a.`catid` = 1
AND a.`manufacturerid` = 1
AND f.`filterid` = 67
) t1
JOIN
(
SELECT DISTINCT a.`addid`,
f.`filterid`
FROM adds a
JOIN adds_filters f
ON a.`addid` = f.`addid`
WHERE a.`catid` = 1
AND a.`manufacturerid` = 1
AND f.`filterid` = 158
) t2
ON t1.addid = t2.addid;