I am in the middle of a uni project, when I discovered a huge problem with my database. Using wamp, and a massive (300Mb) database but with just a few tables my queries are very slow :( All tables are created with MyISAM engine. All settings are on default, I am not experienced in any optimisation. I will need to think of some better way to do it, but for now my question is what is the best substitute for the following query:
SELECT * FROM `payments` WHERE id IN (1, 2, 3, 4, 5, 6, 7, 8, 9, 10);
I can't use left join or any similar solution I have found here, because those IDs (1,2,3,4,5, ...) are not coming from the database. User select the payments he wants to delete, and on the next screen payment details are displayed.
FYI, payments table has more than a million records :)
For a continuos range:
SELECT * FROM payments WHERE id BETWEEN 1 AND 10
If the range is disjoint:
Create an indexed memory table with the values in it.
CREATE TABLE mem_table (
pk unsigned integer primary key
) ENGINE = MEMORY;
INSERT INTO mem_table (pk) VALUES (1),(2),...,(10);
SELECT p.* FROM payments p
INNER JOIN mem_table m ON (m.pk = p.id);
See: http://dev.mysql.com/doc/refman/5.0/en/memory-storage-engine.html
PS
Make sure you have an index on id (this should really be the primary key).
Related
There are a few similar questions on here. None provide a solution. I would like to INSERT a NEW record into table B, but only if a foreign key exists in table A. To be clear, I do not wish to insert the result of a select. I just need to know that the foreign key exists.
INSERT INTO tableB (tableA_ID,code,notes,created) VALUES ('24','1','test',NOW())
SELECT tableA_ID FROM tableA WHERE tableA_ID='24' AND owner_ID='9'
Clearly, the above does not work. But is this even possible? I want to insert the NEW data into tableB, only if the record for the row in tableA exists and belongs to owner_ID.
The queries I have seen so far relate to INSERTING the results from the SELECT query - I do not wish to do that.
Try this:
INSERT INTO tableB (tableA_ID,code,notes,created)
SELECT id, code, notes, created
FROM ( SELECT '24' as id, '1' as code, 'test' as notes, NOW() as created) t
WHERE EXISTS
(
SELECT tableA_ID
FROM tableA
WHERE tableA_ID='24' AND owner_ID='9'
)
I know it's a pretty much old answered question but it's highly ranked now in google search results and I think some addition may help someone in the future.
In some DB configuration, you may want to insert a row in a table that have two or more foreign keys. Let's say we have four tables in a chat application :
Users, Threads, Thread_Users and Messages
If we want a User to join a Thread we'll want to insert a row in Thread_Users in wich have two foreign keys : user_id, thread_id.
Then, we can use a query like this, to insert if both foreign keys exists, and silently fail otherwise :
INSERT INTO `thread_users` (thread_id,user_id,status,creation_date)
SELECT 2,3,'pending',1601465161690 FROM (SELECT 1 as nb_threads, 1 as nb_users) as tmp
WHERE tmp.nb_threads = (SELECT count(*) FROM `threads` WHERE threads.id = 2)
AND tmp.nb_users = (SELECT count(*) FROM `users` WHERE users.id = 3)
It's a little verbose but it does the job pretty well.
Application-side, we just have to raise an error if affectedRows = 0 and maybe trying to see which of the keys doesn'nt exists. IMHO, it's a better way to do the job than to execute two SELECT queries and THEN execute the INSERT especially when an inexistent foreign key probability is very low.
I'm building a db to hold friendships between users of my app.
The server I use to communicate with the MySQL instance is written using Node.js (Express).
My table 'friendships' consists mainly of two INTs which correspond (foreign keys) to user ids.
I want to avoid bidirectional duplicates ( 1,2 vs. 2,1) so I need to write a query which does the following:
INSERT INTO friendships f (id_1, id_2) VALUES (?, ?) IF (SELECT * FROM friendships s WHERE s.id_1=? AND s.id_2=?) IS NULL ;
Obviously this one doesn't really work. And of course I would have the last two question marks have opposite values compared to the first ones, and a UNIQUE key on the ids (id_1, id_2).
The usual answer for these kind of questions is "just order your ids by size to avoid duplicates" and it's a good answer. But in my case, I want to keep record of who sent the friend request (and who approved), without using any extra variables (and extra queries).
Also, I don't want to use code for this, in order to avoid "concurrent" problems.
Thanks!
In MySQL, you can do this with a trigger that does the check. Some other databases have functional indexes, indexes on computed columns, or check constraints that help implement this functionality.
If you want to do the check in the insert, you can do:
INSERT INTO friendships(id_1, id_2)
select new1, new2
from (select ? as new1, ? as new2) t
where not exists (select 1
from friendships f
where f.id_1 = new2 and f.id_2 = new1
);
You should also have a unique index on id_1 and id_2:
create unique index idx_friendsships_id1_id2 on (id_1, id_2);
EDIT:
The basic query is:
INSERT INTO friendships(id_1, id_2)
select ?, ?
from dual
where not exists (select 1
from friendships f
where f.id_1 = ? and f.id_2 = ?
);
But you have to get the arguments in the right order, so the earlier method is less prone to error.
I'm using MySQL and I have a database that I'm trying to use as the canonical version of data that I will be storing in different indexes. That means that I have to be able to retrieve a set of data quickly from it by primary key, but I also need to sort it on the way out. I can't figure out how to let MySQL efficiently do that.
My table looks like the following:
CREATE TABLE demo.widgets (
id INT AUTO_INCREMENT,
-- lots more information I need
awesomeness INT,
PRIMARY KEY (id),
INDEX IDX_AWESOMENESS (awesomeness),
INDEX IDX_ID_AWESOMENESS (id, awesomeness)
);
And I want to do something along the lines of:
SELECT *
FROM demo.widgets
WHERE id IN (
1,
2,
3,
5,
8,
13 -- you get the idea
)
ORDER BY awesomeness
LIMIT 50;
But unfortunately I can't seem to get good performance out of this. It always has to resort to a filesort. Is there a way to get better performance from this setup, or do I need to consider a different database?
This is explained in the documentation ORDER BY Optimization. In particular, if the index used to select rows in the WHERE clause is different from the one used in ORDER BY, it won't be able to use an index for ORDER BY.
In order to get an optimized query to fetch and retrieve like that, you need to have a key that orders by sort and then primary like so:
create table if not exists test.fetch_sort (
id int primary key,
val int,
key val_id (val, id)
);
insert into test.fetch_sort
values (1, 10), (2, 5), (3, 30);
explain
select *
from test.fetch_sort
where id in (1, 2, 3)
order by val;
This will give a query that only uses the index for searching/sorting.
I have a users table, and I want to define a "friends" relationship between two arbitrary users.
Up until now, I've used two different methods for this:
The friends table contains user1 and user2. Searching for users involves a query that looks like
... WHERE #userid IN (`user1`,`user2`), which is not terribly efficient
The friends table contains from and to fields. Initiating a friend request creates a row in that direction, and if it accepted then a second row is inserted with the opposite direction. There is additionally a status column that indicates that this has happened, making the search something like:
... WHERE `user1`=#userid AND `status`=1
I'm not particularly satisfied with either of these solutions. The first one feels messy with that IN usage, and the second seems bloated having two rows to define a single link.
So that's why I'm here. What would you suggest for such a link? Note that I don't need any more information saved with it, I just need two user IDs associated with each other, and preferably some kind of status like ENUM('pending','accepted','blocked'), but that's optional depending on what the best design for this is.
There are in general two approaches:
Store each friend pair once, storing the friend with the least id first.
CREATE TABLE
friend
(
l INT NOT NULL,
g INT NOT NULL,
PRIMARY KEY
(l, g),
KEY (g)
)
Store each friend pair twice, both ways:
CREATE TABLE
(
user INT NOT NULL,
friend INT NOT NULL,
PRIMARY KEY
(user, friend)
)
To store additional fields like friendship status, acceptance dates etc. you usually utilize a second table, for reasons I'll describe below.
To retrieve a list of friends for each user, you do:
SELECT CASE #myuserid WHEN l THEN g ELSE l END
FROM friend
WHERE l = #myuserid
OR
g = #myuserid
or
SELECT g
FROM friend
WHERE l = #myuserid
UNION
SELECT l
FROM friend
WHERE g = #myuserid
for the first solution; and
SELECT friend
FROM friend
WHERE user = #friend
To check if two users are friends, you issue this:
SELECT NULL
FROM friend
WHERE (l, g) =
(
CASE WHEN #user1 < #user2 THEN #user1 ELSE #user2 END,
CASE WHEN #user1 > #user2 THEN #user1 ELSE #user2 END
)
or
SELECT NULL
FROM friend
WHERE (user, friend) = (#user1, #user2)
Storage-wise, the two solutions are almost the same. The first (least/greatest) solution stores twice as few rows, however, for it to work fast you should have a secondary index on g, which, in fact, has to store g plus the part of the table's primary key which is not in the secondary index (that is, l). Thus, each record is effectively store twice: once in the table itself, once again in the index on g.
Performance-wise, the solutions are almost the same too. The first one, though, requires two index seeks followed by index scans (for "all friends"), the second one just one index seek, so for the L/G solution I/O amount might be slighly more. This might be mitigated a little by the fact that the one single index may become one level deeper than two independent ones, so the initial search may take one page read more. This may slow down "are they friends" query a little for the "both pairs" solution, compared to L/G.
As for the additional table for extra data, you most probably want it because it's usually much less used than the two query I described above (and usually only for history purposes).
Its layout also depends on the kind of queries you are using. Say, if you want "show my last ten friendships", then you may want to store the timestamp in "both pairs" so that you don't have to do filesorts, etc.
Consider the following schema:
CREATE TABLE `users` (
`uid` int(10) unsigned NOT NULL AUTO_INCREMENT,
`username` varchar(30) NOT NULL,
PRIMARY KEY (`uid`)
);
INSERT INTO `users` (`uid`, `username`) VALUES
(1, 'h2ooooooo'),
(2, 'water'),
(3, 'liquid'),
(4, 'wet');
CREATE TABLE `friends` (
`id` int(10) unsigned NOT NULL AUTO_INCREMENT,
`uid_from` int(10) unsigned NOT NULL,
`uid_to` int(10) unsigned NOT NULL,
`status` enum('pending','accepted','blocked') NOT NULL,
PRIMARY KEY (`id`),
KEY `uid_from` (`uid_from`),
KEY `uid_to` (`uid_to`)
);
INSERT INTO `friends` (`id`, `uid_from`, `uid_to`, `status`) VALUES
(1, 1, 3, 'accepted'), -- h2ooooooo sent a friend request to liquid - accepted
(2, 1, 2, 'pending'), -- h2ooooooo sent a friend request to water - pending
(3, 4, 1, 'pending'), -- wet sent a friend request to h2ooooooo - pending
(4, 4, 2, 'pending'), -- wet sent a friend request to water - pending
(5, 3, 4, 'accepted'); -- liquid sent a friend request to wet - accepted
I'd use something like the following:
SELECT
fu.username as `friend_username`,
fu.uid as `friend_uid`
FROM
`users` as `us`
LEFT JOIN
`friends` as `fr`
ON
(fr.uid_from = us.uid OR fr.uid_to = us.uid)
LEFT JOIN
`users` as `fu`
ON
(fu.uid = fr.uid_from OR fu.uid = fr.uid_to)
WHERE
fu.uid != us.uid
AND
fr.status = 'accepted'
AND
us.username = 'liquid'
Result:
friend_username | friend_uid
----------------|-----------
h2ooooooo | 1
wet | 4
Here us would be the user you want to query for friends, and fu would be the users friends. You could easily change the WHERE statement to select the user in whatever whay you want. The status could be changed to pending (and should only join on uid_to) if you want to find friends request that the users hasn't answered.
DEMO ON SQLFIDDLE
The EXPLAIN if we use us.uid to match the user (as it's indexed):
Performance considerations aside, another option might be a "friend" table in which one row represents a friend (does not matter which way around), together with a view which produces two result rows (one in each direction) for any friend row. In use, it would simplify queries because it could be used in the same way as the "two row" solution while only requiring one data row per "friendship".
The only drawback could be performance... depending on how the query optimizer works.
I tried to be creative, here are some results.
Easier drawn than said,
A simple trigger on table friends would do a nice service, ordering (user1,user2) without forgeting who requested friendship.
CREATE TRIGGER `friends_insert` BEFORE INSERT ON friends
FOR EACH ROW BEGIN
DECLARE X INT UNSIGNED;
IF NEW.user1 > NEW.user2 THEN
SET X = NEW.user1;
SET NEW.user1 = NEW.user2;
SET NEW.user2 = X;
SET NEW.invited_by = 1;
END IF;
END$$
Finally, let's say a user U has id = x. We can say U divides table users in two parts: users with id < x and ones with id > x. Before inserting a tuple into table friends, we order its ids, and so a certain information won't be explicitly written twice.
We obtain friends of our user U (id = x) through union of U's friends with id < x and ones with id > x:
SELECT user1 AS `friend_id` FROM friends
WHERE user1<#id AND user2=#id
UNION
SELECT user2 AS `friend_id` FROM friends
WHERE user2>#id AND user1=#id;
The main goal here is query performance. Dividing in these two cases would help MySQL to use the right index for each situation.
[ Time for questions & disagreement. Perhaps you want the complete SQL; it's shown here ]
You could try something like this SQLFiddle: http://sqlfiddle.com/#!2/219dae/3/0
Here is the code:
The SCHEMA:
-- This is the users table:
CREATE TABLE users
(
u_id int auto_increment,
username varchar(20),
PRIMARY KEY (u_id)
);
INSERT INTO users (username)
VALUES ('user1'),
('user2'),
('user3'),
('user4'),
('user5');
-- This is the friends table:
CREATE TABLE friends
(
f_id int auto_increment,
r_name varchar(20), -- the name of the user that requests for friendship
a_name varchar(20), -- the name of the user that answers the friendship request
status varchar(20), -- the status of the request
PRIMARY KEY (f_id)
);
-- below, user1 sends frind requests to user2, user3, user4 and user5; and receives one from user2:
INSERT INTO friends (r_name, a_name, status)
VALUES ('user1','user2', 'pending');
INSERT INTO friends (r_name, a_name, status)
VALUES ('user1','user3', 'pending');
INSERT INTO friends (r_name, a_name, status)
VALUES ('user1','user4', 'pending');
INSERT INTO friends (r_name, a_name, status)
VALUES ('user1','user5', 'pending');
INSERT INTO friends (r_name, a_name, status)
VALUES ('user2','user1', 'pending');
-- user1 accepts user2 request to be his friend:
UPDATE friends
SET status='accepted'
WHERE a_name='user1' AND r_name='user2';
-- user3 accepts user1 request to be his friend:
UPDATE friends
SET status='accepted'
WHERE a_name='user3' AND r_name='user1';
and the SELECT:
-- here we select all friend requests that the user1 received and all friend requests that he made
SELECT r_name, a_name, status FROM users
INNER JOIN friends ON users.username=friends.a_name
WHERE username='user1'
UNION
SELECT r_name, a_name, status FROM users
INNER JOIN friends ON users.username=friends.r_name
WHERE username='user1'
I've got two tables where I'm trying to insert data from one to another, I've been able to find a few examples of how this can be accomplished on the web, the problem is these examples mostly rely on identical table structure between the two ... you see I'm trying to insert some data from one table into another table with quite a different structure.
I'm trying to insert data from a table called 'catalog_product_entity_media_gallery' into a table called 'catalog_product_entity_varchar'. Below is a simple description of their structure
The 'catalog_product_entity_varchar' looks as follows:
value_id | entity_type_id | attribute_id | store_id | entity_id | value
PK INT INT INT INT VARCHAR
And the 'catalog_product_entity_media_gallery' table looks as follows:
value_id | attribute_id | entity_id | value
PK INT INT VARCHAR
I need to insert the entity, and value columns from catalog_product_entity_media_gallery into catalog_product_entity_varchar. However as you can see the structure is quite different.
The query I'm trying to use is as follows
USE magento_db;
INSERT INTO catalog_product_entity_varchar(entity_type_id, attribute_id, store_id, entity_id, value)
SELECT
4,
74,
0,
catalog_product_entity_media_gallery.entity_id,
catalog_product_entity_media_gallery.value
FROM catalog_product_entity_media_gallery;
I only need the entity_id and value from media_gallery and the other values are always the same, I have tried to do this using the above but this is just hanging in MySQL (no errors)
I think it's due to the fact that I'm trying to select 4, 74 and 0 from catalog_product_entity_media_gallery but I'm not 100% sure (apologies, I'm a bit of a novice with MySQL)
Can anybody point me in the right direction? Is there any way way I can insert some data from the media table whilst inserting static values for some columns? (I hope this all makes sense)
The query syntax is ok.
However, there may be issues with the unique and foreign keys in catalog_product_entity_varchar table, which doesn't allow you to insert data. Also the query may be waiting for some other query to complete (if your query is just a part of bigger scenario), so it is an issue with locking. Most probable is the first case.
Currently, the question lacks important details:
The MySQL client / programming code you use to perform query. So we
are not able to see the case in full and to reproduce it correctly
The scenario you perform. I.e. whether you do it inside the Magento application in some
module during a web-request. Or whether there are other queries in your script,
some opened transactions, other people accessing the DB server, etc.
Based on most probable assumption that you just don't see the actual error with unique/foreign keys, you may try the following queries.
1) Unique index failure.
Try this:
USE magento_db;
INSERT INTO catalog_product_entity_varchar(entity_type_id, attribute_id, store_id, entity_id, value)
SELECT
4 as etid,
74 as aid,
0 as sid,
catalog_product_entity_media_gallery.entity_id as eid,
catalog_product_entity_media_gallery.value as val
FROM
catalog_product_entity_media_gallery
GROUP BY
eid, aid, sid;
There is a huge possibility, that you insert non-unique entries, because catalog_product_entity_media_gallery can hold multiple entries for the same product, while catalog_product_entity_varchar can not. If the query above successfully completes, then the issue is really with unique key. In such a case you must re-verify what you want to achieve, because the initial aim (not the query itself) is wrong.
2) Wrong foreign key (non-existing attribute 74)
Try this (replacing ATTRIBUTE_CODE and ATTRIBUTE_ENTITY_TYPE_ID with the values you need, e.g. 'firstname' and 6):
USE magento_db;
INSERT INTO catalog_product_entity_varchar(entity_type_id, attribute_id, store_id, entity_id, value)
SELECT
4 as etid,
eav_attribute.attribute_id as aid,
0 as sid,
gallery.entity_id as eid,
gallery.value as val
FROM
catalog_product_entity_media_gallery AS gallery
INNER JOIN
eav_attribute
ON
eav_attribute.attribute_code = '<ATTRIBUTE_CODE>'
AND eav_attribute.entity_type_id = <ATTRIBUTE_ENTITY_TYPE_ID>
GROUP BY
eid, aid, sid;
If it executes successfully AND
Some rows are added to the catalog_product_entity_varchar - then it seems, that 74 was chosen as a wrong id of the attribute you needed, thus foreign key in catalog_product_entity_varchar didn't allow you to insert the records.
No rows are added to the catalog_product_entity_varchar - then it seems, that you mistake in attribute id, attribute code and entity type. Recheck, what you put as ATTRIBUTE_CODE and ATTRIBUTE_ENTITY_TYPE_ID.
If both queries still hang - then you have issues with your MySQL client or server or execution scenario.
Note: your initial query may make sense in your specific case, but some issues are signalling that something may be wrong with your approach, because:
You're using direct numbers for ids. But ids are different for different installations and Magento versions. It is expected to use more stable values, like attribute code in second query, by which you should extract the actual attribute id.
You copy data from the storage catalog_product_entity_media_gallery, which can store multiple entries for the same product, to the storage catalog_product_entity_varchar, which is able to store only one entry for the product. It means, that you cannot copy all the data in such a way. Probably, your query doesn't reflect the goal you want to achieve.
The entity type id, inserted to the catalog_product_entity_varchar is not related to attribute id. While in Magento these are deeply connected things. Putting the wrong entity type id in a table will either make Magento behave incorrectly, or it won't notice your changes at all.
try this
INSERT INTO catalog_product_entity_varchar( entity_id, value)
VALUES (
SELECT entity_id, value
FROM catalog_product_entity_media_gallery
WHERE value_id = here the row id of value_id which have those values 4,74,0 )
Assuming the valued_id in the catalog_product_entity_varchar table is an autoincrement, could you not do the following?
USE magento_db;
INSERT INTO catalog_product_entity_varchar(entity_type_id, store_id, entity_id, value)
SELECT
4,
74,
catalog_product_entity_media_gallery.entity_id,
catalog_product_entity_media_gallery.value
FROM catalog_product_entity_media_gallery;
Note that there is no attribute_id column in your catalog_product_entity_varchar table.