I find this query and that takes 20 seconds for few rows...
SELECT DISTINCT(BINARY `meta_key`) as meta_key_binary, `meta_key`
FROM wp_postmeta
ORDER BY `meta_key` ASC
Is that deprecated in mysql 8? How to change it for mysql 8 than?
Or any idea why that is so slow?
I cant find something nearly same for mysql 8
Using the BINARY conversion operator is probably preventing use of an index for your query.
You could add a virtual column to the table with the binary value of meta_key and index that.
ALTER TABLE wp_postmeta
ADD COLUMN meta_key_binary VARCHAR(100) AS (BINARY meta_key),
ADD INDEX (meta_key_binary, meta_key);
Replace (100) with the correct length of meta_key from the original table schema.
----- 2021-10-19 8.0.27 General Availability -- -- Important Change -----
The BINARY operator is now deprecated, and subject to removal in a future release of MySQL. Use of BINARY now causes a warning. Use CAST(... AS BINARY) instead.
See Barmar's Comment on parentheses
For improved performance of WP, especially when using postmeta: WP Index Improvements
Related
I have the following query that runs forever and I am looking to see if there is anyway that I can optimise it. This is running on a table that has in total 1,406,480 rows of data but apart from the Filename and Refcolumn, the ID and End_Date have both been indexed.
My Query:
INSERT INTO UniqueIDs
(
SELECT
T1.ID
FROM
master_table T1
LEFT JOIN
master_table T2
ON
(
T1.Ref_No = T2.Ref_No
AND
T1.End_Date = T2.End_Date
AND
T1.Filename = T2.Filename
AND
T1.ID > T2.ID
)
WHERE T2.ID IS NULL
AND
LENGTH(T1.Ref_No) BETWEEN 5 AND 10
)
;
Explain Results:
The reason for not indexing the Ref_No is that this is a text column and therefore I get a BLOB/TEXT error when I try and index this column.
Would really appreciate if somebody could advise on how I can quicken this query.
Thanks
Thanks to Bill in regards to multi column indexes I have managed to make some headway. I first ran this code:
CREATE INDEX I_DELETE_DUPS ON master_table(id, End_Date);
I then added a new column to show the length of the Ref_No but had to change it from the query Bill mentioned as my version of MySQL is 5.5. So I ran it in 3 steps:
ALTER TABLE master_table
ADD COLUMN Ref_No_length SMALLINT UNSIGNED;
UPDATE master_table SET Ref_No_length = LENGTH(Ref_No);
ALTER TABLE master_table ADD INDEX (Ref_No_length);
Last step was to change my insert query with the where clause for the length. This was changed to:
AND t1.Ref_No_length between 5 and 10;
I then ran this query and within 15 mins I had 280k worth of id's inserted into my UniqueIDs table. I did go change my insert script to see if I could add more values to the length by doing the following:
AND t1.Ref_No_length IN (5,6,7,8,9,10,13);
This was to bring in the values where length was also equal to 13. This query took a lot longer, 2hr 50 mins to be precise but the additional ask of looking for all rows that have length of 13 gave me an extra 700k unique ids.
I am looking at ways to optimise the query with the IN clause, but a big improvement where this query kept running for 24 hours. So thank you so much Bill.
For the JOIN, you should have a multi-column index on (Ref_No, End_Date, Filename).
You can create a prefix index on a TEXT column like this:
ALTER TABLE master_table ADD INDEX (Ref_No(10));
But that won't help you search based on the LENGTH(). Indexing only helps search by value indexed, not by functions on the column.
In MySQL 5.7 or later, you can create a virtual column like this, with an index on the values calculated for the virtual column:
ALTER TABLE master_table
ADD COLUMN Ref_No_length SMALLINT UNSIGNED AS (LENGTH(Ref_No)),
ADD INDEX (Ref_No_length);
Then MySQL will recognize that your condition in your query is the same as the expression for the virtual column, and it will automatically use the index (exception: in my experience, this doesn't work for expressions using JSON functions).
But this is no guarantee that the index will help. If most of the rows match the condition of the length being between 5 and 10, the optimizer will not bother with the index. It may be more work to use the index than to do a table-scan.
the ID and End_Date have both been indexed.
You have PRIMARY KEY(id) and redundantly INDEX(id)? A PK is a unique key.
"have both been indexed" -- INDEX(a), INDEX(b) is not the same as INDEX(a,b) -- they have different uses. Read about "composite" indexes.
That query smells a lot like "group-wise" max done in a very slow way. (Alas, that may have come from the online docs.)
I have compiled the fastest ways to do that task here: http://mysql.rjweb.org/doc.php/groupwise_max (There are multiple versions, based on MySQL version and what issues your code can/cannot tolerate.)
Please provide SHOW CREATE TABLE. One important question: Is id the PRIMARY KEY?
This composite index may be useful:
(Filename, End_Date, Ref_No, -- first, in any order
ID) -- last
This, as others have noted, is unlikely to be helped by any index, hence T1 will need a full-table-scan:
AND LENGTH(T1.Ref_No) BETWEEN 5 AND 10
If Ref_No cannot be bigger than 191 characters, change it to a VARCHAR so that it can be used in an index. Oh, did I ask for SHOW CREATE TABLE? If you can't make it VARCHAR, then my recommended composite index is
INDEX(Filename, End_Date, ID)
I have a MySQL 5.7 table with a binary(16) id column used as PRIMARY key.
The engine is InnoDb.
Rows are created with Doctrine using https://github.com/ramsey/uuid-doctrine#innodb-optimised-binary-uuids as per https://www.percona.com/blog/2014/12/19/store-uuid-optimized-way/
For instance the uuid 55a54172-f5e4-11e8-aa0d-fe02fd3f406 is stored as HEX(id) = 11E8F5E455A54172AA0DFE02FD3F406 in the database
I'm trying to search for rows with the same string beginning.
Something like this with pseudo code
SELECT *
FROM log
WHERE id LIKE 11E8F5E4
I've tried to query
SELECT HEX(`id`) AS `id`
FROM `log`
WHERE (`id` & UNHEX('11E8F5E4')) = UNHEX('11E8F5E4')
LIMIT 10
which is fast but rows with a different prefix like 11E8E8AD47E47970B1C9525400B61105 are returned
How may I do it while using the Btree index (for perf reason)?
I tried the straightforward approach with LIKE, and it seems to be working fine.
You need to make a binary string which has a % at the end:
SELECT ...
FROM table
WHERE field LIKE CONCAT(UNHEX('11E8F5E4'),'%')
Is it possible to create an index in MySQL for the last digit of an int column?
Based on this answer i have created partitions based on last digit of an int column
CREATE TABLE partition_test(
textfiled INT,
cltext TEXT,
reindexedAt TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
indexedAt TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
status TINYINT(2),
postId INT)
PARTITION BY HASH(MOD(postId, 10))
PARTITIONS 10;
I'm trying to create an index for the last digit of postId for optimizing queries time. Is there any way to do this or a simple index on postId is enough?
Some failed tries:
CREATE INDEX postLastDigit USING HASH ON partition_test (MOD(postId, 10));
(1064, u"You have an error in your SQL syntax; check the manual that corresponds to your MariaDB server version for the right syntax to use near 'MOD(postId, 10))' at line 1")
and
CREATE INDEX postLastDigit ON partition_test (MOD(postId, 10));
(1064, u"You have an error in your SQL syntax; check the manual that corresponds to your MariaDB server version for the right syntax to use near 'MOD(postId, 10))' at line 1")
UPDATE:
The table has more than 100M row.
My goal is to optimize queries like:
1)
SELECT cltext FROM partition_tables
WHERE postId in (<INT>, <INT>)
AND status IS NOT NULL
2)
SELECT cltext FROM partition_tables
WHERE postId in (<INT>, <INT>)
AND status IS NOT NULL
AND reindexedAt BETWEEN (<DATE>, <DATE>)
MariaDB version: 10.1.23-MariaDB-9+deb9u1
What query are you trying to speed up? Without any indexes on the table, any query will have to scan the entire table! If you want speed, first look to indexing.
If your query is SELECT ... WHERE post_id = 123, your Partitioning might make it run about 10 times as fast. But INDEX(post_id), with or without partitioning, will make it run hundreds of times as fast.
Please provide the SELECTs so we can help you speed them up.
(OK, if you are just playing around with partitioning, the others have given you viable answers.)
"Partition Pruning" is rarely faster than a suitable index that starts with the pruning column.
After you solve your stated hashing problem, please report back whether the queries any faster than using an index. Even pitted against an index, I predict partitioning will not run faster, and may even run a little slower.
You have tagged your question with mariadb and mysql. If you are using a resonably recent version of MariaDB, you can use generated columns for indexing. If you are using MySQL, you can do the same if your MySQL version is at least 5.7.
If you are using a lower version of MySQL, you could create an additional column in your table where you store the last digit of postId for each row, and use that column for indexing / partitioning.
This would mean minimal changes to your application code: Before inserting or updating, get the last digit of postId first, and insert / update one more field. As an alternative, you eventually could use triggers to automatically fill that additional column.
Use virtual columns. In MariaDB 10.2, you can create index on virtual aka generated column, like this
CREATE TABLE t (
num int,
last_digit int(1) AS (num % 10) VIRTUAL,
KEY index_last_digit (last_digit)
)
Then you can use last_digit in your queries, i.e SELECT ... WHERE last_digit=1
In older versions of MariaDB, 5.2 to 10.1 , you'd need to specify PERSISTENT attribute rather than VIRTUAL, because non-persistent generated columns could not be indexed.
UPDATE `productData`.`productList`
SET `dateLastUpdated` = CURDATE(),
`totalReviews` = 100
WHERE `productList`.`name` = 'FirstchefproUltraSharpDualJuliennePeelerVegetablePeelerStainless';
This query takes 2.93 seconds and only affects a single row.
The table is ~75k rows long. I don't have much experience in SQL, is there a better way to do this? Updating a single row shouldn't take this long
I'm using the latest version of SQL on an Ubuntu server
If you need any more information just ask and I'll add it to the post.
Thanks
Edit:
I actually didn't know that SQL and mysql where different things.
The select statement
SELECT *
FROM `productList`
WHERE `productList`.`name` = 'FirstchefproUltraSharpDualJuliennePeelerVegetablePeelerStainless';
takes 2.88 seconds, so I guess its a problem with my index.
EDIT:
Thankyou for your help
When I try to create an index on name it says this:
ERROR 1170 (42000): BLOB/TEXT column 'name' used in key specification without a key length
EDIT:
switched name to a varchar(64)
its much faster now
Thanks everyone
For this query:
UPDATE `productData`.`productList`
SET `dateLastUpdated` = CURDATE(),
`totalReviews` = 100
WHERE `productList`.`name` = 'FirstchefproUltraSharpDualJuliennePeelerVegetablePeelerStainless';
You want an index on productList(name):
create index idx_productData_productList on productData(productList);
Either create an index on the name column (as written in some comments), or
use an indexed column in the WHERE statement, e.g. the product ID if your table contains something like that.
Create an index on
`productList`.`name`
Having this table:
CREATE TABLE `example` (
`id` int(11) unsigned NOT NULL auto_increment,
`keywords` varchar(200) NOT NULL,
PRIMARY KEY (`id`)
) ENGINE=InnoDB;
We would like to optimize the following query:
SELECT id FROM example WHERE keywords LIKE '%whatever%'
The table is InnoDB, (so no FULLTEXT for now) which would be the best index to use in order to optimize such query?
We've tried a simple :
ALTER TABLE `example` ADD INDEX `idxSearch` (`keywords`);
But an explain query shows that need to scan the whole table
if our queries where LIKE 'whatever%' instead, this index performs well, but otherwise has no value.
Is there anyway to optimize this for innoDB ?
Thanks!
Indexes are built from the start of the string towards the end. When you use LIKE 'whatever%' type clause, MySQL can use those start-based indexes to look for whatever very quickly.
But switching to LIKE '%whatever%' removes that anchor at the start of the string. Now the start-based indexes can't be used, because your search term is no longer anchored at the start of the string - it's "floating" somewhere in the middle and the entire field has to be search. Any LIKE '%... query can never use indexes.
That's why you use fulltext indexes if all you're doing are 'floating' searches, because they're designed for that type of usage.
Of major note: InnoDB now supports fulltext indexes as of version 5.6.4. So unless you can't upgrade to at least 5.6.4, there's nothing holding you back from using InnoDB *AND fulltext searches.
I would like to comment that surprisingly, creating an index also helped speed up queries for like '%abc%' queries in my case.
Running MySQL 5.5.50 on Ubuntu (leaving everything on default), I have created a table with a lot of columns and inserted 100,000 dummy entries. In one column, I inserted completely random strings with 32 characters (i.e. they are all unique).
I ran some queries and then added an index on this column.
A simple
select id, searchcolumn from table_x where searchcolumn like '%ABC%'
returns a result in ~2 seconds without the index and in 0.05 seconds with the index.
This does not fit the explanations above (and in many other posts). What could be the reason for that?
EDIT
I have checked the EXPLAIN output. The output says rows is 100,000, but Extra info is "Using where; Using index". So somehow, the DBMS has to search all rows, but still is able to utilise the index?