Aurora MySQL reader performance degrades over time - mysql

Question

I have an Aurora MySQL cluster and when running queries against the reader I see a degradation in performance over time. A reboot of the reader results in query performance that matches the writer. But after going a week without a reboot queries take 25x as long to run.
The replication lag for the reader instance is 20ms and none of the monitoring metrics are showing issues. The highest I have seen the CPU is 40%. I tried a suggestion to set block_nested_loop to off but that had no effect.
The reader does not get much activity so load should not be an issue. We do need to run a complex query against it that returns a lot of data which is used for analytics. I have found that queries that return a small number of records that are retrieved by an index do NOT have the performance problem. But a similar query that returns the same small number of records and requires a table scan does have the performance problem.
The rate of degradation seems consistent, so it seems like a resource issue related to replication, but I have not had any luck finding anything online documenting the issue.
Any help would be much appreciated.
Update: Additional details
Query execution plans
-- Fast query
explain select cpv.SHORT_TEXT_VALUE, c.UIDPK, c.GUID, c.SHARED_ID, cpv.*
from TCUSTOMERPROFILEVALUE cpv
inner join TCUSTOMER c on cpv.CUSTOMER_UID = c.UIDPK
where LOCALIZED_ATTRIBUTE_KEY = 'CP_EMAIL' and cpv.SHORT_TEXT_VALUE = 'some-email#gmail.com';
-- Slow query, using function to prevent use of index for email match
explain select cpv.SHORT_TEXT_VALUE, c.UIDPK, c.GUID, c.SHARED_ID, cpv.*
from TCUSTOMERPROFILEVALUE cpv
inner join TCUSTOMER c on cpv.CUSTOMER_UID = c.UIDPK
where LOCALIZED_ATTRIBUTE_KEY = 'CP_EMAIL' and LOWER(cpv.SHORT_TEXT_VALUE) = 'some-email#gmail.com';
Table definitions
CREATE TABLE `TCUSTOMERPROFILEVALUE` (
`UIDPK` bigint(20) NOT NULL,
`ATTRIBUTE_UID` bigint(20) NOT NULL,
`ATTRIBUTE_TYPE` int(11) NOT NULL,
`LOCALIZED_ATTRIBUTE_KEY` varchar(255) NOT NULL,
`SHORT_TEXT_VALUE` varchar(255) DEFAULT NULL,
`LONG_TEXT_VALUE` mediumtext,
`INTEGER_VALUE` int(11) DEFAULT NULL,
`DECIMAL_VALUE` decimal(19,2) DEFAULT NULL,
`BOOLEAN_VALUE` int(11) DEFAULT '0',
`DATE_VALUE` datetime DEFAULT NULL,
`CUSTOMER_UID` bigint(20) DEFAULT NULL,
`LAST_MODIFIED_DATE` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP,
`CREATION_DATE` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP,
PRIMARY KEY (`UIDPK`),
KEY `I_CPV_ATTR_UID` (`ATTRIBUTE_UID`),
KEY `I_CPV_CUID_ATTKEY` (`CUSTOMER_UID`,`LOCALIZED_ATTRIBUTE_KEY`),
KEY `I_CPV_STV_ATTVALUE` (`SHORT_TEXT_VALUE`),
KEY `I_CPV_ATTKEY_SHORTTEXT` (`LOCALIZED_ATTRIBUTE_KEY`,`SHORT_TEXT_VALUE`),
CONSTRAINT `FK_PROFILE_CUSTOMER` FOREIGN KEY (`CUSTOMER_UID`) REFERENCES `TCUSTOMER` (`UIDPK`) ON DELETE CASCADE,
CONSTRAINT `TCUSTOMERPROFILEVALUE_FK_1` FOREIGN KEY (`ATTRIBUTE_UID`) REFERENCES `TATTRIBUTE` (`UIDPK`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8 COMMENT='values associated with customer profiles.'
CREATE TABLE `TCUSTOMER` (
`UIDPK` bigint(20) NOT NULL,
`PREF_BILL_ADDRESS_UID` bigint(20) DEFAULT NULL,
`PREF_SHIP_ADDRESS_UID` bigint(20) DEFAULT NULL,
`CREATION_DATE` datetime NOT NULL,
`LAST_EDIT_DATE` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP,
`GUID` varchar(64) NOT NULL,
`STATUS` int(11) NOT NULL,
`AUTHENTICATION_UID` bigint(20) DEFAULT NULL,
`STORECODE` varchar(64) DEFAULT NULL,
`IS_FIRST_TIME_BUYER` tinyint(4) DEFAULT '1',
`CUSTOMER_TYPE` varchar(64) NOT NULL,
`SHARED_ID` varchar(255) NOT NULL,
`PARENT_CUSTOMER_GUID` varchar(64) DEFAULT NULL,
`DTYPE` varchar(40) DEFAULT 'ExtCustomerImpl',
`LAST_SESSION_DATE` timestamp NULL DEFAULT NULL,
PRIMARY KEY (`UIDPK`),
UNIQUE KEY `TCUSTOMER_UNIQUE` (`GUID`),
UNIQUE KEY `TCUSTOMER_SHARED_ID_TYPE_UNIQ` (`SHARED_ID`,`CUSTOMER_TYPE`),
UNIQUE KEY `I_CUST_AUTH_UID` (`AUTHENTICATION_UID`),
UNIQUE KEY `SHARED_ID` (`SHARED_ID`,`STORECODE`),
KEY `I_CUST_CR_DATE` (`CREATION_DATE`),
KEY `I_CUST_STORE_CODE` (`STORECODE`),
KEY `I_TYPE_LAST_EDIT` (`CUSTOMER_TYPE`,`LAST_EDIT_DATE`),
KEY `I_CUSTOMER_SHAREDID` (`SHARED_ID`),
KEY `I_CUSTOMER_PARENT` (`PARENT_CUSTOMER_GUID`),
CONSTRAINT `CUSTOMER_STORECODE_FK` FOREIGN KEY (`STORECODE`) REFERENCES `TSTORE` (`STORECODE`),
CONSTRAINT `TCUSTOMER_PARENT_GUID_FK` FOREIGN KEY (`PARENT_CUSTOMER_GUID`) REFERENCES `TCUSTOMER` (`GUID`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8 COMMENT='customer account information.'
Indexes

Answer 1

Well, I can't explain the shift in performance unless the Optimizer is randomly shifting between different query plans.
I do see that you are using the notorious Entity-Attribute-Value schema design. And doing it in a rather bulky and complex way -- with multiple columns for different datatypes.
I do see a few things that can probably help performance in general as the dataset grows. (I assume it will grow.)
The primary key, UIDPK of the attribute table TCUSTOMERPROFILEVALUE probably has no use. This will probably be better: PRIMARY KEY(CUSTOMER_UID, ATTRIBUTE_UID). Or maybe that should be LOCALIZED_ATTRIBUTE_KEY??? Why are there two columns for the attribute?
When changing the PK, this KEY I_CPV_ATTKEY_SHORTTEXT (LOCALIZED_ATTRIBUTE_KEY,SHORT_TEXT_VALUE) would implicitly have CUSTOMER_UID added on the end, thereby benefiting your JOIN.
BIGINT is usually overkill; consider using a smaller datatype.
Do you have another attribute table - TATTRIBUTE?
Having 5 UNIQUE keys for a table slows down inserts. Perhaps you can have fewer?
INDEX(SHARED_ID) is redundant since there are other keys starting with that column.

Answer 2

Have your tried removing the LOWER(xxxx) from the SLOW QUERY?
If this corrects the problem, and your results are the same, you were just wasting time with the LOWER(xxx) manipulation.