UPDATE `productData`.`productList`
SET `dateLastUpdated` = CURDATE(),
`totalReviews` = 100
WHERE `productList`.`name` = 'FirstchefproUltraSharpDualJuliennePeelerVegetablePeelerStainless';
This query takes 2.93 seconds and only affects a single row.
The table is ~75k rows long. I don't have much experience in SQL, is there a better way to do this? Updating a single row shouldn't take this long
I'm using the latest version of SQL on an Ubuntu server
If you need any more information just ask and I'll add it to the post.
Thanks
Edit:
I actually didn't know that SQL and mysql where different things.
The select statement
SELECT *
FROM `productList`
WHERE `productList`.`name` = 'FirstchefproUltraSharpDualJuliennePeelerVegetablePeelerStainless';
takes 2.88 seconds, so I guess its a problem with my index.
EDIT:
Thankyou for your help
When I try to create an index on name it says this:
ERROR 1170 (42000): BLOB/TEXT column 'name' used in key specification without a key length
EDIT:
switched name to a varchar(64)
its much faster now
Thanks everyone
For this query:
UPDATE `productData`.`productList`
SET `dateLastUpdated` = CURDATE(),
`totalReviews` = 100
WHERE `productList`.`name` = 'FirstchefproUltraSharpDualJuliennePeelerVegetablePeelerStainless';
You want an index on productList(name):
create index idx_productData_productList on productData(productList);
Either create an index on the name column (as written in some comments), or
use an indexed column in the WHERE statement, e.g. the product ID if your table contains something like that.
Create an index on
`productList`.`name`
Related
We have a table having around 10 million records and we are trying to update some columns using the id(primary key) in the where clause.
UPDATE table_name SET column1=1, column2=0,column3='2022-10-30' WHERE id IN(1,2,3,4,5,6,7,......etc);
Scenario 1: when there are 3000 or fewer ids in the IN clause and if I try for EXPLAIN, then the 'possible_keys' and 'key' show the PRIMARY, and the query gets executed very fast.
Scenario 2: when there are 3000 or more ids(up to 30K) in the IN clause and if I try for EXPLAIN, then the 'possible_keys' shows NULL and the 'key' shows the PRIMARY and the query runs forever. If I use FORCE INDEX(PRIMARY) then the 'possible_keys' and the 'key' shows the PRIMARY and the query gets executed very fast.
Scenario 3: when there are more than 30k ids in the IN clause and even if I use FORCE INDEX(PRIMARY), the 'possible_keys' shows NULL, and the 'key' shows the PRIMARY and the query runs forever.
I believe the optimizer is going for a full table scan instead of an index scan. Can we make any change such that the optimizer goes for an index scan instead of a table scan? Please suggest if there are any parameter changes required to overcome this issue.
The MySQL version is 5.7
As far as I know you need to just provide an ad-hoc table with all the ids and join table_name from it:
update (select 1 id union select 2 union select 3) ids
join table_name using (id) set column1=1, column2=0, column3='2022-10-30';
In mysql 8 you can use a values table constructor which is a little more terse (omit "row" for mariadb, e.g. values (1),(2),(3)):
update (select null id where 0 union all values row(1),row(2),row(3)) ids
join table_name using (id) set column1=1, column2=0, column3='2022-10-30';
fiddle
When UPDATEing a significant chunk of a table wit all the same update values, I see a red flag.
Do you always update the same set of rows? Could that info be in a smaller separate table that you JOIN to?
Or may some other structural schema change that focuses on helping the Updates be faster?
If you must have a long IN list, I suggest doing 100 at a time. And don't try to COMMIT all 3000+ in the same transaction. (Committing in chunks mak violate some business logic, so you may not want to do such.)
I have the following query that runs forever and I am looking to see if there is anyway that I can optimise it. This is running on a table that has in total 1,406,480 rows of data but apart from the Filename and Refcolumn, the ID and End_Date have both been indexed.
My Query:
INSERT INTO UniqueIDs
(
SELECT
T1.ID
FROM
master_table T1
LEFT JOIN
master_table T2
ON
(
T1.Ref_No = T2.Ref_No
AND
T1.End_Date = T2.End_Date
AND
T1.Filename = T2.Filename
AND
T1.ID > T2.ID
)
WHERE T2.ID IS NULL
AND
LENGTH(T1.Ref_No) BETWEEN 5 AND 10
)
;
Explain Results:
The reason for not indexing the Ref_No is that this is a text column and therefore I get a BLOB/TEXT error when I try and index this column.
Would really appreciate if somebody could advise on how I can quicken this query.
Thanks
Thanks to Bill in regards to multi column indexes I have managed to make some headway. I first ran this code:
CREATE INDEX I_DELETE_DUPS ON master_table(id, End_Date);
I then added a new column to show the length of the Ref_No but had to change it from the query Bill mentioned as my version of MySQL is 5.5. So I ran it in 3 steps:
ALTER TABLE master_table
ADD COLUMN Ref_No_length SMALLINT UNSIGNED;
UPDATE master_table SET Ref_No_length = LENGTH(Ref_No);
ALTER TABLE master_table ADD INDEX (Ref_No_length);
Last step was to change my insert query with the where clause for the length. This was changed to:
AND t1.Ref_No_length between 5 and 10;
I then ran this query and within 15 mins I had 280k worth of id's inserted into my UniqueIDs table. I did go change my insert script to see if I could add more values to the length by doing the following:
AND t1.Ref_No_length IN (5,6,7,8,9,10,13);
This was to bring in the values where length was also equal to 13. This query took a lot longer, 2hr 50 mins to be precise but the additional ask of looking for all rows that have length of 13 gave me an extra 700k unique ids.
I am looking at ways to optimise the query with the IN clause, but a big improvement where this query kept running for 24 hours. So thank you so much Bill.
For the JOIN, you should have a multi-column index on (Ref_No, End_Date, Filename).
You can create a prefix index on a TEXT column like this:
ALTER TABLE master_table ADD INDEX (Ref_No(10));
But that won't help you search based on the LENGTH(). Indexing only helps search by value indexed, not by functions on the column.
In MySQL 5.7 or later, you can create a virtual column like this, with an index on the values calculated for the virtual column:
ALTER TABLE master_table
ADD COLUMN Ref_No_length SMALLINT UNSIGNED AS (LENGTH(Ref_No)),
ADD INDEX (Ref_No_length);
Then MySQL will recognize that your condition in your query is the same as the expression for the virtual column, and it will automatically use the index (exception: in my experience, this doesn't work for expressions using JSON functions).
But this is no guarantee that the index will help. If most of the rows match the condition of the length being between 5 and 10, the optimizer will not bother with the index. It may be more work to use the index than to do a table-scan.
the ID and End_Date have both been indexed.
You have PRIMARY KEY(id) and redundantly INDEX(id)? A PK is a unique key.
"have both been indexed" -- INDEX(a), INDEX(b) is not the same as INDEX(a,b) -- they have different uses. Read about "composite" indexes.
That query smells a lot like "group-wise" max done in a very slow way. (Alas, that may have come from the online docs.)
I have compiled the fastest ways to do that task here: http://mysql.rjweb.org/doc.php/groupwise_max (There are multiple versions, based on MySQL version and what issues your code can/cannot tolerate.)
Please provide SHOW CREATE TABLE. One important question: Is id the PRIMARY KEY?
This composite index may be useful:
(Filename, End_Date, Ref_No, -- first, in any order
ID) -- last
This, as others have noted, is unlikely to be helped by any index, hence T1 will need a full-table-scan:
AND LENGTH(T1.Ref_No) BETWEEN 5 AND 10
If Ref_No cannot be bigger than 191 characters, change it to a VARCHAR so that it can be used in an index. Oh, did I ask for SHOW CREATE TABLE? If you can't make it VARCHAR, then my recommended composite index is
INDEX(Filename, End_Date, ID)
I'am having an optimalization problem. I am not really into MySQL Indexes yet. I read a few articles and questions but I haven't found what I am looking for.
I have an UPDATE statement with a WHERE condition on a table having nearly 1 million rows.
I want to speed up this UPDATE operation. I think most of the time is spent by looking for a the record specified in the WHERE.
My queries look like this, with different URLs:
UPDATE table SET updated = 1, updated_date = now() , [some more fields updated]
WHERE url = 'someurl.com?id=123532js23';
URLs are unique. Currently there is an AUTO INREMENT id field defined as a PRIMARY KEY. I dont need any <, > or BETWEEN operations - so maybe could I use some hashing?
What engine and indexing should I use for best performance?
One of my friends suggested to use InnoDB + UNIQUE on url field. Is there anything else I can do?
This UPDATE runs really many times - about 1 000 000 times each day and most of the executions results in updating - about 95%.
Thanks for any help!
One of my friends suggested to use InnoDB + UNIQUE on url field. Is there anything else I can do? This UPDATE runs really many times - about 1 000 000 times each day and most of the executions results in updating - about 95%.
You friend is right.
One thing is that URL may be long and the maximum possible length of an index key on InnoDB is 767 bytes.
Thus, you would better hash the urls (say, with MD5) and create a UNIQUE index on the field containing the url's hash (and of course use it in the WHERE condition):
INSERT
INTO mytable (url, hashed_url, ...)
VALUES ('someurl.com?id=123532js23', MD5('someurl.com?id=123532js23'))
UPDATE mytable
SET ...
WHERE hashed_url = MD5('someurl.com?id=123532js23')
I have a large MySQL photos table, and even with full-text indexing, normal indexing, and other performance tricks, I'm finding my query is getting a little slower (crappy MySQL full-text).
I have two fields, one called 'caption' and the other 'shortCaption'. The 'caption' field contains 500,000 rows of data between 150 and 2000 characters in length and the new 'shortCaption' field is empty.
The searchable information within my 'caption' field is mostly in the first 300 characters, so to speed up my full-text query I would like to have a 'shortCaption' field that contains the first 300 characters only, making less work for my DB/query.
Is there a way, in one query, to iterate through each photo record, grabbing the first 300 characters (length(field,300)) and updating the new field with the shorter version? I was just about to write a little server-side script that dealt with this, but curious to see if it can be done purely with SQL as it would be quicker.
Thanks in advance.
UPDATE tablename SET newfield = LEFT(oldfield, 300)
This query would update all the rows in tablename
Could try something like this:
UPDATE table
SET new_column = left(old_column, 300);
I ran a test against one of my test tables and it worked. Just know that the query will take some time to execute over 500K rows.
UPDATE table1
SET column1 = LEFT( column2, 300 )
I don't think this is possible as I couldn't find anything but I thought I would check on here in case I am not searching for the correct thing.
I have a settings table in my database which has two columns. The first column is the setting name and the second column is the value.
I need to update all of these at the same time. I wanted to see if there was a way to update these values at the same time one query like the following
UPDATE table SET col1='setting name' WHERE col2='1 value' AND SET col1='another name' WHERE col2='another value';
I know the above isn't a correct SQL format but this is the sort of thing that I would like to do so was wondering if there was another way that this can be done instead of having to perform separate SQL queries for each setting I want to update.
Thanks for your help.
You can use INSERT INTO .. ON DUPLICATE KEY UPDATE to update multiple rows with different values.
You do need a unique index (like a primary key) to make the "duplicate key"-part work
Example:
INSERT INTO table (a,b,c) VALUES (1,2,3),(4,5,6)
ON DUPLICATE KEY UPDATE b = VALUES(b), c = VALUES(c);
-- VALUES(x) points back to the value you gave for field x
-- so for b it is 2 and 5, for c it is 3 and 6 for rows 1 and 4 respectively (if you assume that a is your unique key field)
If you have a specific case I can give you the exact query.
UPDATE table
SET col2 =
CASE col1
WHEN 'setting1'
THEN 'value'
ELSE col2
END
, SET col1 = ...
...
I decided to use multiple queries all in one go. so the code would go like
UPDATE table SET col2='value1' WHERE col1='setting1';
UPDATE table SET col2='value2' WHERE col1='setting1';
etc
etc
I've just done a test where I insert 1500 records into the database. Do it without starting a DB transaction and it took 35 seconds, blanked the database and did it again but starting a transaction first, then once the 1500th record inserted finish the transaction and the time it took was 1 second, so definetely seems like doing it in a db transaction is the way to go.
You need to run separate SQL queries and make use of Transactions if you want to run as atomic.
UPDATE table SET col1=if(col2='1 value','setting name','another name') WHERE col2='1 value' OR col2='another value'
#Frits Van Campen,
The insert into .. on duplicate works for me.
I am doing this for years when I want to update more than thousand records from an excel import.
Only problem with this trick is, when there is no record to update, instead of ignoring, this method inserts a record and on some instances it is a problem. Then I need to insert another field, then after import I have to delete all the records that has been inserted instead of update.