How can I optimize the 'IN' query? - mysql

Question

**** EDIT ****
14ms may not seem a lot, however as you can see below in the "PostgresSQL Explain", PostgreSQL is doing a Seq Scan on 80,000 rows. There must be a way to avoid this Scan and do a couple of Index lookups instead.
**** EDIT END ****
I am playing around with the schemaless idea and I have the following three tables:
The tables are populated with 100,000 random entries.
entities(_primary_key SERIAL PRIMARY KEY, _id CHAR(32) UNIQUE,
data BYTEA)
index_username_profile_names(_id CHARE(32) PRIMARY KEY,
key VARCHAR UNIQUE)
index_username_email(_id CHAR(32) PRIMARY KEY, key VARCHAR)
with a non-unique index on index_username_email(key)
My SQL query is:
SELECT data FROM entities WHERE
_id IN (SELECT _id FROM index_users_email WHERE key = 'test')
OR
_id in (SELECT _id FROM index_users_profile_name WHERE key = 'test')
This takes a whooping 14ms although 'test' doesn't exits in either of the 'index' tables, no matter if I use PostgreSQL or MySQL, so it must be something that I am doing wrong.
Any idea how I can optimized it, or what I am doing wrong?
Thanks!
Postgres explain:
Seq Scan on entities (cost=16.88..4776.15 rows=80414 width=163) (actual time=15.169..15.169 rows=0 loops=1)
Filter: ((hashed SubPlan 1) OR (hashed SubPlan 2))
Rows Removed by Filter: 107218
SubPlan 1
-> Index Scan using index_users_email_key_idx1 on index_users_email (cost=0.42..8.44 rows=1 width=33) (actual time=0.039..0.039 rows=0 loops=1)
Index Cond: ((key)::text = 'test'::text)
SubPlan 2
-> Index Scan using index_users_profile_name_key_idx1 on index_users_profile_name (cost=0.42..8.44 rows=1 width=33) (actual time=0.071..0.071 rows=0 loops=1)
Index Cond: ((key)::text = 'test'::text)
Planning time: 0.202 ms
Execution time: 15.216 ms

Answer 1

ORed (join-) conditions are usually bad, try UNION instead:
SELECT data FROM entities
WHERE _id IN
( SELECT _id
FROM index_users_email
WHERE key = 'test'
)
UNION
SELECT data FROM entities
WHERE _id in
( SELECT _id
FROM index_users_profile_name
WHERE key = 'test'
)

Answer 2

14 milliseconds is quite fine. I have no idea why you think the query should run in less than a millisecond. There is "a lot" of work to set up a query, validate that the data is in memory, identify where the indexes are, and so on. I put that in quotes, because for most queries, this is trivial. But it can easily add up to milliseconds.
Second, if you are doing real timings, keep the following in mind:
Computers (as we use them) are not deterministic. You need to run the timings multiple times. For something that takes milliseconds, this would normally be thousands of times to get a stable reading.
Initialize the system to be in the same state for each timing. You need to decide if you want a cold cache or warm cache, but the timings should all be on the same system.
Isolate the system from any other work. Background tasks (even moving a mouse) can affect performance.
In terms of the query, the one thing I can think of is to use = and exists:
SELECT e.data
FROM entities e
WHERE _id = (SELECT _id FROM index_users_email WHERE key = 'test') OR
EXISTS (SELECT 1 FROM index_users_profile_name iupn WHERE iupn._id = e.id AND iupn.key = 'test');
At best, though, I'm guessing these would shave a millisecond or two off the query.

Answer 3

SELECT data
FROM entities e
LEFT OUTER JOIN index_users_email iue ON e._id=iue._id and iue.key = 'test'
LEFT OUTER JOIN index_users_profile_name iupn ON e._id=iupn._id and iupn .key = 'test'
WHERE
iue._id IS NOT NULL or iupn._id IS NOT NULL

Answer 4

Looks like one potential answer is union and join:
explain select data from entities as t join (select _id from index_users_email where key = 'test' union select _id from index_users_profile_name where key = 'test') u on t._id = u._id;
QUERY PLAN
-------------------------------------------------------------------------------------------------------------------------------------------
Nested Loop (cost=17.32..33.82 rows=2 width=163)
-> Unique (cost=16.90..16.91 rows=2 width=33)
-> Sort (cost=16.90..16.91 rows=2 width=33)
Sort Key: index_users_email._id
-> Append (cost=0.42..16.89 rows=2 width=33)
-> Index Scan using index_users_email_key_idx1 on index_users_email (cost=0.42..8.44 rows=1 width=33)
Index Cond: ((key)::text = 'test'::text)
-> Index Scan using index_users_profile_name_key_idx1 on index_users_profile_name (cost=0.42..8.44 rows=1 width=33)
Index Cond: ((key)::text = 'test'::text)
-> Index Scan using entities__id_key on entities t (cost=0.42..8.44 rows=1 width=196)
Index Cond: (_id = index_users_email._id)
(11 rows)
Time: 0.714 ms

Answer 5

This will have better performance as it takes advantage of the primary key in those tables:
select data
from entities
where
exists (
select _id
from index_users_email
where key = 'test' and _id = entities._id
) or exists (
select _id
from index_users_profile_name
where key = 'test' and _id = entities._id
)