I have two tables vtiger_crmentity and vtiger_crmentityrel (from open source project vtiger).
vtiger_crmentity
+--------------+--------------+------+-----+---------+-------+
| Field | Type | Null | Key | Default | Extra |
+--------------+--------------+------+-----+---------+-------+
| crmid | int(19) | NO | PRI | NULL | |
| smcreatorid | int(19) | NO | MUL | 0 | |
| smownerid | int(19) | NO | MUL | 0 | |
| modifiedby | int(19) | NO | MUL | 0 | |
| setype | varchar(30) | NO | | NULL | |
| description | text | YES | | NULL | |
| createdtime | datetime | NO | | NULL | |
| modifiedtime | datetime | NO | | NULL | |
| viewedtime | datetime | YES | | NULL | |
| status | varchar(50) | YES | | NULL | |
| version | int(19) | NO | | 0 | |
| presence | int(1) | YES | | 1 | |
| deleted | int(1) | NO | MUL | 0 | |
| label | varchar(255) | YES | MUL | NULL | |
+--------------+--------------+------+-----+---------+-------+
vtiger_crmentityrel
+-----------+--------------+------+-----+---------+-------+
| Field | Type | Null | Key | Default | Extra |
+-----------+--------------+------+-----+---------+-------+
| crmid | int(11) | NO | | NULL | |
| module | varchar(100) | NO | | NULL | |
| relcrmid | int(11) | NO | | NULL | |
| relmodule | varchar(100) | NO | | NULL | |
+-----------+--------------+------+-----+---------+-------+
I am trying to get a list of contacts which are not present in the crmentityrel table (in the relcrmid column to be specific). I can do this via a subquery but it is taking about 2 minutes to complete (for about 20k records in each table).
I tried to convert the query to a join but i am surely doing something wrong as i keep getting wrong values (compared to the subquery which i know is right).
Any help is greatly appreciated. Please tell me if you need any details from my side
Edit -
My working query (with subquery) is -
SELECT crmid, label from vtiger_crmentity
WHERE deleted = 0 and setype="Contacts"
and crmid not in (select relcrmid from vtiger_crmentityrel
where relmodule="Contacts")
To convert a not in to a join, the idea is to use left join and where:
SELECT c.crmid, c.label
FROM vtiger_crmentity c left join
vtiger_crmentityrel cr
on c.crmid = cr.relcrmid and relmodule = 'Contacts'
WHERE c.deleted = 0 and c.setype = 'Contacts' and cr.relcrmid is null;
I should point out that the above is not exactly equivalent. NOT IN returns no rows if the subquery returns even a single NULL value. The above behaves more intuitively.
Because of the behavior of NOT IN with NULL values, NOT EXISTS is a better choice. Plus, it often has better performance as well:
SELECT crmid, label
FROM vtiger_crmentity c
WHERE deleted = 0 and setype = 'Contacts' AND
NOT EXISTS (SELECT relcrmid
FROM vtiger_crmentityrel cr
WHERE cr.relmodule = 'Contacts' and cr.relcrmid = c.crmid
);
Related
I have two tables:
mysql> DESCRIBE swaps;
+-------------+----------+------+-----+---------+----------------+
| Field | Type | Null | Key | Default | Extra |
+-------------+----------+------+-----+---------+----------------+
| id | int(11) | NO | PRI | NULL | auto_increment |
| user1_id | int(11) | NO | | NULL | |
| user2_id | int(11) | NO | | NULL | |
| hasto | int(11) | NO | | NULL | |
| requested | datetime | NO | | NULL | |
| accepted | datetime | YES | | NULL | |
| swapped1 | datetime | YES | | NULL | |
| swapped2 | datetime | YES | | NULL | |
| rejected | datetime | YES | | NULL | |
| rejected_by | int(11) | YES | | NULL | |
+-------------+----------+------+-----+---------+----------------+
mysql> DESCRIBE messages;
+-----------+----------+------+-----+-------------------+----------------+
| Field | Type | Null | Key | Default | Extra |
+-----------+----------+------+-----+-------------------+----------------+
| msg_id | int(11) | NO | PRI | NULL | auto_increment |
| sender_id | int(11) | NO | | NULL | |
| msg | text | NO | | NULL | |
| msg_time | datetime | NO | | CURRENT_TIMESTAMP | |
| swap_id | int(11) | NO | | NULL | |
| seen | datetime | YES | | NULL | |
+-----------+----------+------+-----+-------------------+----------------+
and query that I adjusted from this question
SELECT s.*, m.*
FROM swaps as s
JOIN messages as m
ON (s.id= m.swap_id AND m.msg_time=
(SELECT MAX(msg_time) FROM messages WHERE messages.swap_id = s.id));
as a result I single row for every swap and information about last sent message within this swap. I want to add the count of messages that have not jet been seen (m.seen IS NULL).
I tried different approaches but always get error. What I want is to add count of messages in corresponding swap with seen IS NULL to my result set. Would appreciate any suggestions.
You can add the count as a subquery:
SELECT s.*, m.*,
(SELECT COUNT(*) FROM messages m2 WHERE m2.seen IS NULL) as seen_is_null
FROM swaps s JOIN
messages m
ON s.id= m.swap_id AND
m.msg_time =(SELECT MAX(m2.msg_time) FROM messages m2 WHERE m2.swap_id = s.id));
It seems curious to be counting over all messages, but that is what the question asks for. You can, of course, introduce a correlation clause to count for a given swap or something else.
I have two databases.
One is called INFO with three tables (Stories, Comments, Replies)
Stories has the following fields
+--------------+----------------+------+-----+---------------------+-----------------------------+
| Field | Type | Null | Key | Default | Extra |
+--------------+----------------+------+-----+---------------------+-----------------------------+
| storyID | int(11) | NO | PRI | NULL | |
| originalURL | varchar(500) | YES | | NULL | |
| originalDate | timestamp | NO | | CURRENT_TIMESTAMP | on update CURRENT_TIMESTAMP |
| numDiggs | int(11) | YES | | NULL | |
| numComments | int(11) | YES | | NULL | |
| diggURL | varchar(500) | YES | | NULL | |
| rating | varchar(50) | YES | | NULL | |
| title | varchar(200) | YES | | NULL | |
| summary | varchar(10000) | YES | | NULL | |
| uploaderID | varchar(50) | YES | | NULL | |
| imageURL | varchar(500) | YES | | NULL | |
| category1 | varchar(50) | YES | | NULL | |
| category2 | varchar(50) | YES | | NULL | |
| uploadDate | timestamp | NO | | 0000-00-00 00:00:00 | |
| num | int(11) | YES | | NULL | |
+--------------+----------------+------+-----+---------------------+-----------------------------+
Another database is called Data with one table (User). Fields shown below:
+-------------------+-------------+------+-----+---------+-------+
| Field | Type | Null | Key | Default | Extra |
+-------------------+-------------+------+-----+---------+-------+
| userID | varchar(50) | NO | PRI | NULL | |
| numStories | int(11) | YES | | NULL | |
| numComments | int(11) | YES | | NULL | |
| numReplies | int(11) | YES | | NULL | |
| numStoryDiggs | int(11) | YES | | NULL | |
| numCommentReplies | int(11) | YES | | NULL | |
| numReplyDiggs | int(11) | YES | | NULL | |
| numStoryComments | int(11) | YES | | NULL | |
| numStoryReplies | int(11) | YES | | NULL | |
+-------------------+-------------+------+-----+---------+-------+
User.userID is full of thousands of unique names. All other fields are currently NULL. The names in User.userID correspond to the names in Stories.uploaderID.
I need to, for each userID in User, count the number of stories uploaded from (i.e. num) Stories for the corresponding name and insert this value into User.numStories.
The query which I have come up with (which produces an error) is:
INSERT INTO DATA.User(numStories)
SELECT count(num)
FROM INFO.Stories
WHERE INFO.Stories.uploaderID=DATA.User.userID;
The error I get when running this query is
Unknown column 'DATA.User.userID' in 'where clause'
Sorry if this is badly explained. I will try and re-explain if need be.
You aren't creating new entries in the User table, you're updating existing ones. Hence, insert isn't the right syntax here, but rather update:
UPDATE DATA.User u
JOIN (SELECT uploaderID, SUM(num) AS sumNum
FROM INFO.Stories
GROUP BY uploadedID) i ON i.uploaderID = u.userID
SET numStories = sumNum
EDIT:
Some clarification, as requested in the comments.
The inner query sums the num in Stories per uploaderId. The updates statement updates the numStories in User the the calculated sum of the inner query of the matching id.
I would like to select random entries from a single table based on information coming from two other tables (one saved in a different database). The tables are as follows:
1- In databaseA called "islands" contains:
+-------------+------------+------+-----+---------+----------------+
| Field | Type | Null | Key | Default | Extra |
+-------------+------------+------+-----+---------+----------------+
| id | int(11) | NO | PRI | NULL | auto_increment |
| chrom | int(11) | NO | | NULL | |
| start | int(11) | NO | | NULL | |
| end | int(11) | NO | | NULL |
the indexes are:
+---------+------------+-----------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+---------------+
| Table | Non_unique | Key_name | Seq_in_index | Column_name | Collation | Cardinality | Sub_part | Packed | Null | Index_type | Comment | Index_comment |
+---------+------------+-----------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+---------------+
| islands | 0 | PRIMARY | 1 | id | A | 15991 | NULL | NULL | | BTREE | | |
| islands | 1 | locations | 1 | line_string | A | NULL | 32 | NULL | | SPATIAL | | |
To select from this database I normally use:
SELECT * FROM islands FORCE INDEX (locations)
WHERE MBRIntersects(GeomFromText('Linestring(1 120, 1 120)'), line_string)
2 - In databaseB call "Context" contains:
+---------+---------+------+-----+---------+----------------+
| Field | Type | Null | Key | Default | Extra |
+---------+---------+------+-----+---------+----------------+
| id | int(11) | NO | PRI | NULL | auto_increment |
| chrom | int(11) | NO | | NULL | |
| site | int(11) | NO | | NULL | |
| context | char(3) | NO | | NULL | |
There are only 4 possible contexts in this table (which are indexed)
3 - In databaseB called "Entries" Contains:
+-------------+---------+------+-----+---------+----------------+
| Field | Type | Null | Key | Default | Extra |
+-------------+---------+------+-----+---------+----------------+
| id | int(11) | NO | PRI | NULL | auto_increment |
| chrom | int(11) | NO | MUL | NULL | |
| site | int(11) | NO | MUL | NULL | |
| methylation | float | NO | | NULL | |
I would like to select a random entry from the entries table that is located in the context table with a context entry of "CpG", can be located in the "islands" table or not (I have two different searches).
Update:
Based on comments below I am using
SELECT * FROM
(SELECT t.chrom, t.site, t.methylation, c.context
FROM f1 as t INNER JOIN context as c on c.chrom = t.chrom AND c.site = t.site
WHERE c.context = 'CpG'
) AS s LIMIT 2;
To get:
+-------+-----------+-------------+---------+
| chrom | site | methylation | context |
+-------+-----------+-------------+---------+
| 1 | 10003735 | 69 | CpG |
| 1 | 100063074 | 98.79 | CpG |
+-------+-----------+-------------+---------+
I would like to now join these results to the islands table to get the sites from the first part that are found within island regions (or not). I am using:
SELECT * FROM
(SELECT t.chrom, t.site, t.methylation, c.context
FROM f1 as t INNER JOIN context as c on c.chrom = t.chrom AND c.site = t.site
WHERE c.context = 'CpG'
) AS s
CROSS JOIN islands as i
WHERE MBRINTERSECTS(GeomFromText('Linestring(s.chrom s.site, s.chrom s.site)'), i.Line_string)
LIMIT 2;
However, this gives me an empty set (it shouldn't).
I'm trying to select assets from RT's database with the the values for a set of custom fields as tables. Relevant tables are as follows:
mysql> describe AT_Assets;
+---------------+--------------+------+-----+---------+----------------+
| Field | Type | Null | Key | Default | Extra |
+---------------+--------------+------+-----+---------+----------------+
| id | int(11) | NO | PRI | NULL | auto_increment |
| Type | int(11) | NO | MUL | 0 | |
| Name | varchar(200) | NO | MUL | NULL | |
| Description | varchar(255) | YES | | NULL | |
| Status | varchar(20) | YES | | NULL | |
| URI | varchar(255) | YES | | NULL | |
| LastUpdatedBy | int(11) | NO | | 0 | |
| LastUpdated | datetime | YES | | NULL | |
| Creator | int(11) | NO | | 0 | |
| Created | datetime | YES | | NULL | |
+---------------+--------------+------+-----+---------+----------------+
10 rows in set (0.00 sec)
mysql> describe CustomFields;
+---------------+--------------+------+-----+---------+----------------+
| Field | Type | Null | Key | Default | Extra |
+---------------+--------------+------+-----+---------+----------------+
| id | int(11) | NO | PRI | NULL | auto_increment |
| Name | varchar(200) | YES | | NULL | |
| Type | varchar(200) | YES | | NULL | |
| RenderType | varchar(64) | YES | | NULL | |
| MaxValues | int(11) | YES | | NULL | |
| Pattern | text | YES | | NULL | |
| Repeated | smallint(6) | NO | | 0 | |
| BasedOn | int(11) | YES | | NULL | |
| ValuesClass | varchar(64) | YES | | NULL | |
| Description | varchar(255) | YES | | NULL | |
| SortOrder | int(11) | NO | | 0 | |
| LookupType | varchar(255) | NO | | NULL | |
| Creator | int(11) | NO | | 0 | |
| Created | datetime | YES | | NULL | |
| LastUpdatedBy | int(11) | NO | | 0 | |
| LastUpdated | datetime | YES | | NULL | |
| Disabled | smallint(6) | NO | | 0 | |
+---------------+-------------+------+-----+---------+----------------+
17 rows in set (0.00 sec)
mysql> describe ObjectCustomFieldValues;
+-----------------+--------------+------+-----+---------+----------------+
| Field | Type | Null | Key | Default | Extra |
+-----------------+--------------+------+-----+---------+----------------+
| id | int(11) | NO | PRI | NULL | auto_increment |
| CustomField | int(11) | NO | MUL | NULL | |
| ObjectType | varchar(255) | NO | | NULL | |
| ObjectId | int(11) | NO | | NULL | |
| SortOrder | int(11) | NO | | 0 | |
| Content | varchar(255) | YES | MUL | NULL | |
| LargeContent | longblob | YES | | NULL | |
| ContentType | varchar(80) | YES | | NULL | |
| ContentEncoding | varchar(80) | YES | | NULL | |
| Creator | int(11) | NO | | 0 | |
| Created | datetime | YES | | NULL | |
| LastUpdatedBy | int(11) | NO | | 0 | |
| LastUpdated | datetime | YES | | NULL | |
| Disabled | smallint(6) | NO | | 0 | |
+-----------------+--------------+------+-----+---------+----------------+
This query successfully gets me a listing of all assets, but repeats the asset in a separate column for each custom field value:
SELECT AT_Assets.Name, AT_Assets.description, CustomFields.Name, ObjectCustomFieldValues.content FROM ObjectCustomFieldValues inner join CustomFields on ObjectCustomFieldValues.CustomField = CustomFields.id inner join AT_Assets on AT_Assets.id = ObjectCustomFieldValues.ObjectID order by AT_Assets.description;
So I did some reading, and learned how to pivot. Now I have this:
SELECT at_assets.name,
at_assets.description AS "Asset Tag",
Max(CASE
WHEN customfields.name = "make" THEN
objectcustomfieldvalues.content
END) AS "Make",
Max(CASE
WHEN customfields.name = "model" THEN
objectcustomfieldvalues.content
END) AS "Model",
Max(CASE
WHEN customfields.name = "primary user" THEN
objectcustomfieldvalues.content
END) AS "Primary User",
Max(CASE
WHEN customfields.name = "hostname" THEN
objectcustomfieldvalues.content
END) AS "Hostname",
Max(CASE
WHEN customfields.name = "os" THEN objectcustomfieldvalues.content
END) AS "OS",
Max(CASE
WHEN customfields.name = "purchase date (if known)" THEN
objectcustomfieldvalues.content
END) AS "Purchase Date"
FROM objectcustomfieldvalues
INNER JOIN customfields
ON objectcustomfieldvalues.customfield = customfields.id
INNER JOIN at_assets
ON at_assets.id = objectcustomfieldvalues.objectid
WHERE at_assets.id = 5
ORDER BY at_assets.description;
Which works great, when I'm specifying a single asset. However, I would like this to run across every asset. Otherwise I need a Python script to run this hundreds of times manually incrementing the asset id, which is fairly inelegant. How do I go about getting a full list?
You learned everything about the pivot, except adding the group by.
Add the following line after the where:
group by at_assets.name, at_assets.description
To see more than one line, remove or adjust the where clause.
What is happening with your query is the MySQL recognizes it as an aggregation query, because it is using MAX(). There is no group by, so it produces one row -- an aggregation of all the rows.
What about the variables at_assets.name and at_assets.description in the select clause? you might ask. Well, most SQL engines would balk and produce an error. These variables are neither in the group by nor the argument to an aggregation function. MySQL has a (mis)feature called hidden columns that allows such references. However, the values come from arbitrary rows in the source data, so the value are not meaningful unless all values in the group are the same.
you just need group by here. and dont specify the asset name.
change this
WHERE at_assets.id = 5
ORDER BY at_assets.description;
to
group by at_assets.name
ORDER BY at_assets.description;
I have the following query:
SELECT DISTINCT `movies_manager_movie`.`id`,
`movies_manager_movie`.`title`,
`movies_manager_movie`.`original_title`,
`movies_manager_movie`.`synopsis`,
`movies_manager_movie`.`keywords`,
`movies_manager_movie`.`release_date`,
`movies_manager_movie`.`rating`,
`movies_manager_movie`.`poster_web_url`,
`movies_manager_movie`.`has_poster`,
`movies_manager_movie`.`number`,
`movies_manager_movie`.`has_sources`,
`movies_manager_movie`.`season_id`,
`movies_manager_movie`.`created`,
`movies_manager_movie`.`updated`,
`movies_manager_moviecache`.`activity_name`
FROM `movies_manager_movie`
LEFT OUTER JOIN `movies_manager_moviecache` ON (`movies_manager_movie`.`id` = `movies_manager_moviecache`.`movie_id`)
WHERE (`movies_manager_movie`.`has_sources` = 1
AND (`movies_manager_moviecache`.`team_member_id` IN (
SELECT U0.`id` FROM `movies_manager_movieteammember` U0
INNER JOIN `movies_manager_movieteammemberactivity` U1 ON (U0.`id` = U1.`team_member_id`)
WHERE U1.`movie_id` = 3588 )
AND `movies_manager_movie`.`number` IS NULL
)
AND NOT (`movies_manager_movie`.`id` = 3588 ))
ORDER BY `movies_manager_moviecache`.`activity_name` DESC LIMIT 3;
This query can take up to 3 seconds and I'm very surprise since I got indexes everywhere and no more than 35 rows in each of my MyIsam tables, using the latest MySQL version.
I cached everything I could but I have at least to run this one 20000 times every day, which is approximately 16 h of waiting for loading. And I'm pretty sure none of my user (nor Google Bot) appreciate a 4 secondes waiting time for each page loading.
What could I do to make it faster ?
I thought about duplicating field from movie to moviecache since the all purpose of movie cache is to denormalize to complex join already.
I tried inlining the subquery to a list of ID but it surprisingly doubled the time of the query.
Tables:
+----------------+--------------+------+-----+---------+----------------+
| Field | Type | Null | Key | Default | Extra |
+----------------+--------------+------+-----+---------+----------------+
| id | int(11) | NO | PRI | NULL | auto_increment |
| title | varchar(120) | NO | UNI | NULL | |
| original_title | varchar(120) | YES | | NULL | |
| synopsis | longtext | YES | | NULL | |
| keywords | varchar(120) | YES | | NULL | |
| release_date | date | YES | | NULL | |
| rating | int(11) | NO | | NULL | |
| poster_web_url | varchar(255) | YES | | NULL | |
| has_poster | tinyint(1) | NO | | NULL | |
| number | int(11) | YES | | NULL | |
| season_id | int(11) | YES | MUL | NULL | |
| created | datetime | NO | | NULL | |
| updated | datetime | NO | | NULL | |
| has_sources | tinyint(1) | NO | | NULL | |
+----------------+--------------+------+-----+---------+----------------+
+---------------------+--------------+------+-----+---------+----------------+
| Field | Type | Null | Key | Default | Extra |
+---------------------+--------------+------+-----+---------+----------------+
| id | int(11) | NO | PRI | NULL | auto_increment |
| name | varchar(120) | NO | UNI | NULL | |
| biography | longtext | YES | | NULL | |
| birth_date | date | YES | | NULL | |
| picture_web_url | varchar(255) | YES | | NULL | |
| allocine_link | varchar(255) | YES | | NULL | |
| created | datetime | NO | | NULL | |
| updated | datetime | NO | | NULL | |
| has_picture | tinyint(1) | NO | | NULL | |
| biography_linkyfied | longtext | YES | | NULL | |
+---------------------+--------------+------+-----+---------+----------------+
+----------------+--------------+------+-----+---------+----------------+
| Field | Type | Null | Key | Default | Extra |
+----------------+--------------+------+-----+---------+----------------+
| id | int(11) | NO | PRI | NULL | auto_increment |
| movie_id | int(11) | NO | MUL | NULL | |
| tag_slug | varchar(100) | YES | MUL | NULL | |
| team_member_id | int(11) | YES | MUL | NULL | |
| cast_rank | int(11) | YES | | NULL | |
| activity_name | varchar(30) | YES | MUL | NULL | |
+----------------+--------------+------+-----+---------+----------------+
Mysql tells me it's a slow query:
# Query_time: 3 Lock_time: 0 Rows_sent: 9 Rows_examined: 454128
Move movies_manager_movieteammemberactivity and movies_manager_movieteammember to your main join statement (so that you're doing a left outer between movies_manager_movie and the inner join product of the other 3 tables). This should speed up your query considerably.
Try this:
SELECT `movies_manager_movie`.`id`,
`movies_manager_movie`.`title`,
`movies_manager_movie`.`original_title`,
`movies_manager_movie`.`synopsis`,
`movies_manager_movie`.`keywords`,
`movies_manager_movie`.`release_date`,
`movies_manager_movie`.`rating`,
`movies_manager_movie`.`poster_web_url`,
`movies_manager_movie`.`has_poster`,
`movies_manager_movie`.`number`,
`movies_manager_movie`.`has_sources`,
`movies_manager_movie`.`season_id`,
`movies_manager_movie`.`created`,
`movies_manager_movie`.`updated`,
(
SELECT `movies_manager_moviecache`.`activity_name`
FROM `movies_manager_moviecache`
WHERE (`movies_manager_movie`.`id` = `movies_manager_moviecache`.`movie_id`
AND (`movies_manager_moviecache`.`team_member_id` IN (
SELECT U0.`id` FROM `movies_manager_movieteammember` U0
INNER JOIN `movies_manager_movieteammemberactivity` U1 ON (U0.`id` = U1.`team_member_id`)
WHERE U1.`movie_id` = 3588 )
AND `movies_manager_movie`.`number` IS NULL
) ) LIMIT 1) AS `activity_name`
FROM `movies_manager_movie`
WHERE (`movies_manager_movie`.`has_sources` = 1
AND NOT (`movies_manager_movie`.`id` = 3588 ))
ORDER BY `activity_name` DESC
LIMIT 3;
Let me know how that performs