Getting error
Although my column CreatedAt is datetime still im getting this error on partitioning the table
**Error Code: 1486. Constant, random or timezone-dependent expressions in (sub)partitioning function are not allowed**
default data
1,"2022-07-21 11:30:31","2022-08-31 11:16:46"
2,"2022-07-21 11:35:50","2022-08-31 11:16:46"
Table structure
CREATE TABLE `click` (
`id` int(11) DEFAULT NULL,
`CreatedAt` datetime DEFAULT NULL,
`UpdatedAt` datetime
) ENGINE=InnoDB DEFAULT CHARSET=latin1
ALTER TABLE X.`click` PARTITION BY RANGE (UNIX_TIMESTAMP(CreatedAt))
PARTITIONS 4 (
PARTITION drop_old VALUES LESS THAN (UNIX_TIMESTAMP('2022-07-01')),
PARTITION p_20220801 VALUES LESS THAN (UNIX_TIMESTAMP('2022-08-01')),
PARTITION p_20220901 VALUES LESS THAN (UNIX_TIMESTAMP('2022-09-01')),
PARTITION future VALUES LESS THAN MAXVALUE);
he manual has an example of how to use a timestamp in partitioning https://dev.mysql.com/doc/refman/8.0/en/partitioning-range.html for your example to work you need to change createdat datatype to timestamp.
Related
I am working on a 10 year old web-app (!!!)
& currently running mysql locally, version 5.7.
This is the table I am currently working on:
CREATE TABLE `processes_history` (
`p_id` bigint(20) UNSIGNED NOT NULL DEFAULT '0',
`exec_id` bigint(20) UNSIGNED NOT NULL DEFAULT '0',
`feature` varchar(100) NOT NULL DEFAULT '',
`macro` tinyint(1) UNSIGNED NOT NULL DEFAULT '0',
`ts` date NOT NULL DEFAULT '0000-00-00',
`seen` int(10) UNSIGNED NOT NULL DEFAULT '1',
`seen_time` bigint(20) UNSIGNED NOT NULL DEFAULT '0',
`focus` int(10) UNSIGNED NOT NULL DEFAULT '0',
`focus_time` bigint(20) UNSIGNED NOT NULL DEFAULT '0',
`mouse` int(10) UNSIGNED NOT NULL DEFAULT '0',
`keyboard` int(10) UNSIGNED NOT NULL DEFAULT '0',
`interactive` int(10) UNSIGNED NOT NULL DEFAULT '0',
`interactive_time` bigint(20) UNSIGNED NOT NULL DEFAULT '0',
`last_seen` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP
) ENGINE=MyISAM DEFAULT CHARSET=utf8
PARTITION BY RANGE (TO_DAYS(`ts`))
(
PARTITION p0 VALUES LESS THAN (736695) ENGINE=MyISAM,
PARTITION p201701 VALUES LESS THAN (736726) ENGINE=MyISAM,
PARTITION p201702 VALUES LESS THAN (736754) ENGINE=MyISAM,
PARTITION p201703 VALUES LESS THAN (736785) ENGINE=MyISAM,
PARTITION p201704 VALUES LESS THAN (736815) ENGINE=MyISAM,
PARTITION p201705 VALUES LESS THAN (736846) ENGINE=MyISAM,
PARTITION p201706 VALUES LESS THAN (736876) ENGINE=MyISAM,
PARTITION p201707 VALUES LESS THAN (736907) ENGINE=MyISAM,
PARTITION p201708 VALUES LESS THAN (736938) ENGINE=MyISAM,
PARTITION p201709 VALUES LESS THAN (736968) ENGINE=MyISAM,
PARTITION p201710 VALUES LESS THAN (736999) ENGINE=MyISAM,
PARTITION p201711 VALUES LESS THAN (737029) ENGINE=MyISAM,
PARTITION p201712 VALUES LESS THAN (737060) ENGINE=MyISAM,
PARTITION p201801 VALUES LESS THAN (737091) ENGINE=MyISAM,
PARTITION pmax VALUES LESS THAN MAXVALUE ENGINE=MyISAM
);
--
-- Indexes for dumped tables
--
--
-- Indexes for table `processes_history`
--
ALTER TABLE `processes_history`
ADD PRIMARY KEY (`p_id`,`exec_id`,`feature`,`ts`),
ADD KEY `ts` (`ts`),
ADD KEY `exec_ts` (`exec_id`,`ts`),
ADD KEY `last_seen` (`last_seen`);
I keep getting an error when adding an index to p_id, exec_id, ts:
ALTER TABLE `dbname`.`processes_history` ADD INDEX `p_id,exec_id,ts` (`p_id`, `exec_id`, `ts`);
Error SQL query:
ALTER TABLE dbname.processes_history ADD INDEX p_id,exec_id,ts
(p_id, exec_id, ts) MySQL said: Documentation
1067 - Invalid default value for 'ts'
Following this post: https://dba.stackexchange.com/questions/192186/on-create-index-invalid-default-value
From what I understood, using 0000-00-00 as a default value breaks the 'date' type, and that's why it's not working.
But I just couldn't understand what is the solution for this situation. Using TIMESTAMP type instead?
Is there a way to solve this problem without breaking the structure (for now at least) unit I finish the whole web-app? Many things are dependent on that table and i reeeealy don't want to do something risky to index it the way I want.
Changing SQL_mode solved the problem:
I compared MySQL 5.7 and 5.6, and it seems like 5.7 has the default restrictions: ONLY_FULL_GROUP_BY, STRICT_TRANS_TABLES, NO_ZERO_IN_DATE, NO_ZERO_DATE, ERROR_FOR_DIVISION_BY_ZERO, NO_AUTO_CREATE_USER, and NO_ENGINE_SUBSTITUTION set (https://dev.mysql.com/doc/refman/5.7/en/sql-mode.html#sqlmode_no_zero_in_date).
And on 5.6: just NO_ENGINE_SUBSTITUTION
(mysql 5.6 manual https://dev.mysql.com/doc/refman/5.6/en/sql-mode.html#sql-mode-setting)
To get my local mysql (5.7) to work in sync with the production version (5.6) I just had to set mysql_mode to NO_ENGINE_SUBSTITUTION
SET GLOBAL sql_mode = 'NO_ENGINE_SUBSTITUTION';
SET SESSION sql_mode = 'NO_ENGINE_SUBSTITUTION';
You can review the post on db-stackexchange with more info.
You need to change to InnoDB -- The next release (8.0) has removed MyISAM.
Somewhere around 5.6, the handling of default for TIMESTAMP changed. Think about what values you have and what values you need.
PARTITIONing rarely improves performance; what query drove you to use it?
Temporarily making the column in question nullable also solves this issue.
I am trying to partition my table MySQL innoDB. Right now there are approximately 2 million rows in the location table (and growing always) rows of history data. I must perodicly delete the dataset old by year
I use MySQL 5.7.22 Community Server.
CREATE TABLE `geo_data` (
`ID` bigint(20) NOT NULL AUTO_INCREMENT,
`ID_DISP` bigint(20) DEFAULT NULL,
`SYS_TIMESTAMP` datetime DEFAULT NULL,
`DATA_TIMESTAMP` bigint(20) DEFAULT NULL,
`X` double DEFAULT NULL,
`Y` double DEFAULT NULL,
`SPEED` bigint(20) DEFAULT NULL,
`HEADING` bigint(20) DEFAULT NULL,
`ID_DATA_TYPE` bigint(20) DEFAULT NULL,
`PROCESSED` bigint(20) DEFAULT NULL,
`ALTITUDE` bigint(20) DEFAULT NULL,
`ID_UNIT` bigint(20) DEFAULT NULL,
`ID_DRIVER` bigint(20) DEFAULT NULL,
UNIQUE KEY `part_id` (`ID`,`DATA_TIMESTAMP`,`ID_DISP`),
KEY `Index_idDisp_dataTS_type` (`ID_DISP`,`DATA_TIMESTAMP`,`ID_DATA_TYPE`),
KEY `Index_idDisp_dataTS` (`ID_DISP`,`DATA_TIMESTAMP`),
KEY `Index_TS` (`DATA_TIMESTAMP`),
KEY `idx_sysTS_idDisp` (`ID_DISP`,`SYS_TIMESTAMP`),
KEY `idx_clab_geo_data_ID_UNIT_DATA_TIMESTAMP_ID_DATA_TYPE` (`ID_UNIT`,`DATA_TIMESTAMP`,`ID_DATA_TYPE`),
KEY `idx_idUnit_dataTS` (`ID_UNIT`,`DATA_TIMESTAMP`),
KEY `idx_clab_geo_data_ID_DRIVER_DATA_TIMESTAMP_ID_DATA_TYPE` (`ID_DRIVER`,`DATA_TIMESTAMP`,`ID_DATA_TYPE`)
) ENGINE=InnoDB AUTO_INCREMENT=584390 DEFAULT CHARSET=latin1;
I have to partition by DATA_TIMESTAMP (format timestamp date gps).
ALTER TABLE geo_data
PARTITION BY RANGE (year(from_unixtime(data_timestamp)))
(
PARTITION p2018 VALUES LESS THAN ('2018'),
PARTITION p2019 VALUES LESS THAN ('2019'),
PARTITION pmax VALUES LESS THAN MAXVALUE
);
Error Code: 1697. VALUES value for partition 'p2018' must have type INT
How can I do?
I would like to add later a subpartion range by ID_DISP. How can I do?
Thanks in advance!
Since data_timestamp was actually a BIGINT, you are not permitted to use date functions. It seemed there were two errors, and this probably fixes them:
ALTER TABLE geo_data
PARTITION BY RANGE (data_timestamp)
(
PARTITION p2018 VALUES LESS THAN (UNIX_TIMESTAMP('2018-01-01') * 1000),
PARTITION p2019 VALUES LESS THAN (UNIX_TIMESTAMP('2019-01-01') * 1000),
PARTITION pmax VALUES LESS THAN MAXVALUE
);
I am assuming your data_timestamp is really milliseconds, a la Java? If not, the decide what to do with the * 1000.
SUBPARTITIONs are useless; don't bother with them. If you really want to partition by Month or Quarter, then simply do it at the PARTITION level.
Recommendation: Don't have more than about 50 partitions.
How many "drivers" do you have? I suspect you do not have trillions. So, don't blindly use BIGINT for ids. Each takes 8 bytes. SMALLINT UNSIGNED, for example, would take only 2 bytes and allow 64K drivers (etc).
If X and Y are latitude and longitude, it might be clearer to name them such. Here are what datatype to use instead of the 8-byte DOUBLE, depending on the resolution you have (and need). 4-byte FLOATs are probably good enough for vehicles.
The table has several redundant indexes; toss them. Also, note that when you have INDEX(a,b,c), it is redundant to also have INDEX(a,b).
See also my discussion on Partitioning, especially related to time-series, such as yours.
Hmmm... I wonder if the 63 bits of precision for SPEED will let you record them when they go at the speed of light?
Another point: Don't create p2019 until just before the start of 2019. You have pmax in case you goof and fail to add that partition in time. And the REORGANIZE PARTITION mentioned in my discussion covers how to recover from such a goof.
Update:
Seems you cannot use from_unixtime in a PARTITION BY RANGE query because hash partitions must be based on an integer expression. More info see this answer
Its expecting an INT not a STRING (as per the error message), so try :
ALTER TABLE geo_data
PARTITION BY RANGE (year(from_unixtime(data_timestamp)))
(
PARTITION p2018 VALUES LESS THAN (2018),
PARTITION p2019 VALUES LESS THAN (2019),
PARTITION pmax VALUES LESS THAN MAXVALUE
);
Here I have specified the year in the partition values as an int ie 2018 / 2019, and not strings as in '2018' / '2019'
What's wrong with this MySQL query?
Result is [Err] 1654 - Partition column values of incorrect type
DROP TABLE IF EXISTS part;
CREATE TABLE `part` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`cnt` varchar(255) DEFAULT NULL,
`created` datetime NOT NULL,
PRIMARY KEY (`id`, `created`)
) ENGINE=InnoDB DEFAULT CHARSET=latin1
PARTITION BY RANGE COLUMNS (created)(
PARTITION p_2015_01 VALUES LESS THAN ('2015-01-30') ENGINE=InnoDB,
PARTITION p_2015_02 VALUES LESS THAN ('2015-02-30') ENGINE=InnoDB,
PARTITION p_2015_03 VALUES LESS THAN ('2015-03-30') ENGINE=InnoDB,
PARTITION p_catchall VALUES LESS THAN (MAXVALUE) ENGINE=InnoDB
);
If it matters, the version is 5.5
It took quite a long time to see the obvious:
The date '2015-02-30' does not exist.
Presumably it is converted to NULL or something, therefore the message about 'incorrect type'.
Hopefully it helps somebody someday.
If you're using the online-schema-change tool from the Percona toolkit to reorganize partitions, you can also get this error for using single quotes around an alter containing single quotes. Silly, but I just ran into it.
root#host:~# pt-online-schema-change --execute --no-drop-old-table --progress='time,1' --recursion-method=none --alter='REORGANIZE PARTITION pMAXVALUE INTO ( PARTITION p20161206 VALUES LESS THAN ('2016-12-06'),PARTITION pMAXVALUE VALUES LESS THAN (MAXVALUE))' D=somedb,t=sometable
vs
root#host:~# pt-online-schema-change --execute --no-drop-old-table --progress='time,1' --recursion-method=none --alter="REORGANIZE PARTITION pMAXVALUE INTO ( PARTITION p20161206 VALUES LESS THAN ('2016-12-06'),PARTITION pMAXVALUE VALUES LESS THAN (MAXVALUE))" D=somedb,t=sometable
An example of a variant that may lead to a similar error [#HY000 Partition column values of incorrect type]:
create table if not exists tbl (
col varchar(10))
partition by list columns (col)(
partition blablablabla values in ('blablablabla')
);
If length('blablablabla') > to the size specified in the column col (varchar(10)) we will get this error. In my case, increasing the max length of the characters of that specific column solve the issue.
I update MySQL versition from 5.0 to 5.5. and I am new for studying mysql partition. firstly, I type:
SHOW VARIABLES LIKE '%partition%'
Variable_name Value
have_partitioning YES
Make sure that the new version support partition. I tried to partition my table by every 10 minutes, then INSERT, UPDATE, QUERY huge data into this table for a test.
First, I need create a new table, I type my code:
CREATE TABLE test (
`id` int unsigned NOT NULL auto_increment,
`words` varchar(100) collate utf8_unicode_ci NOT NULL,
`date` varchar(10) collate utf8_unicode_ci NOT NULL,
PRIMARY KEY (`id`),
FULLTEXT KEY `index` (`words`)
)
ENGINE=MyISAM
DEFAULT CHARSET=utf8
COLLATE=utf8_unicode_ci
AUTO_INCREMENT=0
PARTITION BY RANGE (MINUTE(`date`))
(
PARTITION p0 VALUES LESS THAN (1322644000),
PARTITION p1 VALUES LESS THAN (1322644600) ,
PARTITION p2 VALUES LESS THAN (1322641200) ,
PARTITION p3 VALUES LESS THAN (1322641800) ,
PARTITION p4 VALUES LESS THAN MAXVALUE
);
It return alert: #1564 - This partition function is not allowed, so what is this problem? thanks.
UPDATE
Modify date into int NOT NULL, and PARTITION BY RANGE MINUTE(date) into PARTITION BY RANGE COLUMNS(date)
CREATE TABLE test (
`id` int unsigned NOT NULL auto_increment,
`words` varchar(100) collate utf8_unicode_ci NOT NULL,
`date` int NOT NULL,
PRIMARY KEY (`id`),
FULLTEXT KEY `index` (`words`)
)
ENGINE=MyISAM
DEFAULT CHARSET=utf8
COLLATE=utf8_unicode_ci
AUTO_INCREMENT=0
PARTITION BY RANGE COLUMNS(`date`)
(
PARTITION p0 VALUES LESS THAN (1322644000),
PARTITION p1 VALUES LESS THAN (1322644600) ,
PARTITION p2 VALUES LESS THAN (1322641200) ,
PARTITION p3 VALUES LESS THAN (1322641800) ,
PARTITION p4 VALUES LESS THAN MAXVALUE
);
Then caused new error: #1214 - The used table type doesn't support FULLTEXT indexes
I am so sorry, mysql not support fulltext and partition at the same time.
See partitioning limitations
FULLTEXT indexes. Partitioned tables do not support FULLTEXT indexes or searches. This includes partitioned tables employing the MyISAM storage engine.
One issue might be
select MINUTE('2008-10-10 56:56:98') returns null, the reason is Minute function returns minute from time or datetime value, where as in your case date is varchar
MINUTE function returns in either date/datetime expression. Again, A partitioning key must be either an integer column or an expression that resolves to an
integer but inyour case it's VARCHAR
I want to partition a mysql table by datetime column. One day a partition.The create table scripts is like this:
CREATE TABLE raw_log_2011_4 (
id bigint(20) NOT NULL AUTO_INCREMENT,
logid char(16) NOT NULL,
tid char(16) NOT NULL,
reporterip char(46) DEFAULT NULL,
ftime datetime DEFAULT NULL,
KEY id (id)
) ENGINE=InnoDB AUTO_INCREMENT=286802795 DEFAULT CHARSET=utf8
PARTITION BY hash (day(ftime)) partitions 31;
But when I select data of some day.It could not locate the partition.The select statement is like this:
explain partitions select * from raw_log_2011_4 where day(ftime) = 30;
when i use another statement,it could locate the partition,but I coluld not select data of some day.
explain partitions select * from raw_log_2011_4 where ftime = '2011-03-30';
Is there anyone tell me How I could select data of some day and make use of partition.Thanks!
Partitions by HASH is a very bad idea with datetime columns, because it cannot use partition pruning. From the MySQL docs:
Pruning can be used only on integer columns of tables partitioned by
HASH or KEY. For example, this query on table t4 cannot use pruning
because dob is a DATE column:
SELECT * FROM t4 WHERE dob >= '2001-04-14' AND dob <= '2005-10-15';
However, if the table stores year values in an INT column, then a
query having WHERE year_col >= 2001 AND year_col <= 2005 can be
pruned.
So you can store the value of TO_DAYS(DATE()) in an extra INTEGER column to use pruning.
Another option is to use RANGE partitioning:
CREATE TABLE raw_log_2011_4 (
id bigint(20) NOT NULL AUTO_INCREMENT,
logid char(16) NOT NULL,
tid char(16) NOT NULL,
reporterip char(46) DEFAULT NULL,
ftime datetime DEFAULT NULL,
KEY id (id)
) ENGINE=InnoDB AUTO_INCREMENT=286802795 DEFAULT CHARSET=utf8
PARTITION BY RANGE( TO_DAYS(ftime) ) (
PARTITION p20110401 VALUES LESS THAN (TO_DAYS('2011-04-02')),
PARTITION p20110402 VALUES LESS THAN (TO_DAYS('2011-04-03')),
PARTITION p20110403 VALUES LESS THAN (TO_DAYS('2011-04-04')),
PARTITION p20110404 VALUES LESS THAN (TO_DAYS('2011-04-05')),
...
PARTITION p20110426 VALUES LESS THAN (TO_DAYS('2011-04-27')),
PARTITION p20110427 VALUES LESS THAN (TO_DAYS('2011-04-28')),
PARTITION p20110428 VALUES LESS THAN (TO_DAYS('2011-04-29')),
PARTITION p20110429 VALUES LESS THAN (TO_DAYS('2011-04-30')),
PARTITION future VALUES LESS THAN MAXVALUE
);
Now the following query will only use partition p20110403:
SELECT * FROM raw_log_2011_4 WHERE ftime = '2011-04-03';
Hi You are doing the wrong partition in definition of the table the table definition would like this:
CREATE TABLE raw_log_2011_4 (
id bigint(20) NOT NULL AUTO_INCREMENT,
logid char(16) NOT NULL,
tid char(16) NOT NULL,
reporterip char(46) DEFAULT NULL,
ftime datetime DEFAULT NULL,
KEY id (id)
) ENGINE=InnoDB AUTO_INCREMENT=286802795 DEFAULT CHARSET=utf8
PARTITION BY hash (TO_DAYS(ftime)) partitions 31;
And your select command would be:
explain partitions
select * from raw_log_2011_4 where TO_DAYS(ftime) = '2011-03-30';
The above command would select all the date required, as if you use the TO_DAYS command as
mysql> SELECT TO_DAYS(950501);
-> 728779
mysql> SELECT TO_DAYS('2007-10-07');
-> 733321
Why to use the TO_DAYS AS The MySQL optimizer will recognize two date-based functions for partition pruning purposes:
1.TO_DAYS()
2.YEAR()
and this would solve your problem..
I just recently read a MySQL blog post relating to this, at http://dev.mysql.com/tech-resources/articles/mysql_55_partitioning.html.
Versions earlier than 5.1 required special gymnastics in order to do partitioning based on dates. The link above discusses it and shows examples.
Versions 5.5 and later allowed you to do direct partitioning using non-numeric values such as dates and strings.
Don't use CHAR, use VARCHAR. That will save a lot of space, hence decrease I/O, hence speed up queries. (Exception: If the column is really fixed length, then use CHAR. And it will probably be CHARACTER SET ascii.)
reporterip: (46) is unnecessarily big for an IP address, even IPv6. See My blog for further discussion, including how to shrink it to 16 bytes.
PARTITION BY RANGE(TO_DAYS(...)) as #Steyx suggested, but don't have more than about 50 partitions. The more partitions you have, the slower queries get, in spite of the "pruning". HASH partitioning is essentially useless.
More discussion of partitioning, especially the type you are looking at. That includes code for a sliding set of partitions over time.