Mysql - how to keep unique constraint while partitioning by RANGE (timestamp)? - mysql

I have one table, want to partition by RANGE (created_at timestamp), so can delete old data easily (by drop partition).
CREATE TABLE `orders` (
`order_id` NVARCHAR(64) NOT NULL,
`amount` INTEGER NOT NULL,
`created_at` TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
`modified_at` TIMESTAMP DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
KEY `order_id` (`order_id`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci;
ALTER TABLE dropship.orders
PARTITION BY RANGE (UNIX_TIMESTAMP(created_at)) (
PARTITION p0 VALUES LESS THAN ( UNIX_TIMESTAMP('2019-03-01 00:00:00') ),
PARTITION p1 VALUES LESS THAN ( UNIX_TIMESTAMP('2019-04-01 00:00:00') ),
PARTITION p2 VALUES LESS THAN ( UNIX_TIMESTAMP('2019-05-01 00:00:00') ),
PARTITION p3 VALUES LESS THAN ( UNIX_TIMESTAMP('2019-06-01 00:00:00') ),
PARTITION p4 VALUES LESS THAN ( UNIX_TIMESTAMP('2019-07-01 00:00:00') ),
PARTITION p5 VALUES LESS THAN (MAXVALUE)
);
This table only has two usages: get by order_id, or update by order_id.
select * from orders where order_id = '123';
update orders set amount = 10 where order_id = '123';
Due to the limitation of Mysql partitioning, I cannot add an unique key for order_id since will use created_at field for partitioning.
All columns used in the table's partitioning expression must be part of every unique key that the table may have, including any primary key.
Question:
Any way to make order_id unique in this table please?
I have thought about partitioning by order_id, but it's hard to delete old data in that way.
Any suggestion is welcome. (For example may be you have better design for this table).

BEGIN;
SELECT 1 FROM orders WHERE order_id = 234 FOR UPDATE;
if row exists, you have a dup error.
INSERT INTO orders ... order_id = 234;
COMMIT;
But, as Raymond points out, you may as well drop PARTITIONing and make the column the PRIMARY KEY. This would make all the stated operations slightly faster.

Related

Moving MANY partitions from one table to another -Mysql

I have a table with many partitions by date
I want to move some of the oldest partitions to another table.
I succeed moving the oldest partition by following the manual
but when I try to move more partitions I get: Error Code: 1737. Found a row that does not match the partition
So, I deleted the oldest partition and move the next partition -but then the rows from the first partitions returned to the original table (I did NOT see any documentation for record that go back...)
How can I move the three first partitions to another table?
THANKS
CREATE TABLE `TestPartA` (
`Name` VARCHAR(50) NOT NULL,
`Time` DATETIME NOT NULL DEFAULT CURRENT_TIMESTAMP,
`Slot` INT(11) NOT NULL DEFAULT '-1',
`Text` VARCHAR(50) NOT NULL,
PRIMARY KEY (`Name`, `Text`, `Time`),
INDEX `ClusterTimeIdx` (`Name`, `Time`, `Slot`),
INDEX `Time` (`Time`)
)
PARTITION BY RANGE (TO_DAYS(TIME))
(PARTITION p20190407 VALUES LESS THAN (TO_DAYS('2019-04-07')) ,
PARTITION p20190421 VALUES LESS THAN (TO_DAYS('2019-04-21')) ,
PARTITION p20190428 VALUES LESS THAN (TO_DAYS('2019-04-28')),
PARTITION p20190505 VALUES LESS THAN (TO_DAYS('2019-05-05'))) ;
CREATE TABLE TestPartB LIKE TestPartA;
ALTER TABLE TestPartB REMOVE PARTITIONING;
insert into TestPartA values ('A','2019-04-02',1,'W1');
insert into TestPartA values ('A','2019-04-04',1,'W1');
insert into TestPartA values ('A','2019-04-08',1,'W1');
insert into TestPartA values ('A','2019-04-20',1,'W1');
insert into TestPartA values ('A','2019-05-01',1,'W1');
SELECT PARTITION_NAME, TABLE_ROWS FROM INFORMATION_SCHEMA.PARTITIONS WHERE TABLE_NAME = 'TestPartA';
-- move the first partition
ALTER TABLE TestPartA EXCHANGE PARTITION p20190407 WITH TABLE TestPartB; -- Works GREAT
select * from TestPartA;
select * from TestPartB;
SELECT PARTITION_NAME, TABLE_ROWS FROM INFORMATION_SCHEMA.PARTITIONS WHERE TABLE_NAME = 'TestPartA'; -- this is not working any more - but according to documentation it happens sometimes
--move the second partition
ALTER TABLE TestPartA EXCHANGE PARTITION p20190421 WITH TABLE TestPartB; -- FAILED
ALTER TABLE TestPartA drop PARTITION p20190407;
ALTER TABLE TestPartA EXCHANGE PARTITION p20190421 WITH TABLE TestPartB; -- Succeed but the rows from the first partitions returned to table A
select * from TestPartA;
select * from TestPartB;

How to create partition mysql 8

I am trying to create a table with partition with following way
create table transaction_demo
(
trx_id bigint not null primary key,
created_datetime datetime
) partition by range (YEAR(created_datetime)) (
partition p1 values less than (2019),
partition p2 values less than (2020),
partition p3 values less than (2021)
);
But it is not working. I am getting following error
[2019-08-02 15:08:43] [HY000][1503] A PRIMARY KEY must include all columns in the table's partitioning function
[2019-08-02 15:08:43] [HY000][1503] A PRIMARY KEY must include all columns in the table's partitioning function
How to fix this?
Then define primary key for each column which will be part of the partition.
CREATE TABLE transaction_demo (
trx_id int NOT NULL,
created_datetime datetime NOT NULL,
other_column_1 int NULL,
other_column_2 varchar(222) NULL,
PRIMARY KEY(trx_id,created_datetime)
)
PARTITION BY RANGE( YEAR(created_datetime) )(
PARTITION p1 VALUES LESS THAN (2019),
PARTITION p2 VALUES LESS THAN (2020),
PARTITION p3 VALUES LESS THAN (2021)
);

Correctly optimising MySQL data for date range queries?

I have a table with lots of numeric data. I need to query this to get the closest row for a specific date_added and name.
My problem is that this data is not ordered by date, so when returning results I need to include ORDER BY date_added (or it doesn't return the correct row). Currently doing this takes a good 90 seconds to run because of this ORDER BY condition.
Are there any ways I can further optimise this? I've already indexed the date_added and name columns, so I'm not really sure what else can be done. I considered creating a new table with the data reordered in date_added order, but this isn't practical as new entries need to be added regularly.
I've stored the numeric data as decimal as it can potentially be very small, very large or both. Perhaps storing this data in a different way would be more efficient?
Add a compound index on name and date. The query above will run without using filesort.
An alternative way for the query:
SELECT date_added, data_1, data_2, data_3, data_4, data_5, data_6, data_7, data_8, data_9
FROM numeric_data
WHERE date_added =
(select min(date_added) from numeric_data where date_added >= '2018-05-03 11:00:00' and name = 'aaa')
and name = 'aaa'
limit 1;
Fiddle: http://sqlfiddle.com/#!9/4e8d89/1 .
You can use range partitioning:
https://dev.mysql.com/doc/refman/5.7/en/partitioning-range.html
You need to define your partitions depends on the date range you have.
CREATE TABLE `numeric_data` (
`id` int(255) NOT NULL AUTO_INCREMENT,
`date_added` datetime NOT NULL,
`name` varchar(8) COLLATE utf8mb4_unicode_ci NOT NULL,
`data_1` decimal(30,17) NOT NULL,
`data_2` decimal(30,17) NOT NULL,
`data_3` decimal(30,17) NOT NULL,
`data_4` decimal(30,17) NOT NULL,
`data_5` decimal(30,17) NOT NULL,
`data_6` decimal(30,17) NOT NULL,
`data_7` decimal(30,17) NOT NULL,
`data_8` decimal(30,17) NOT NULL,
`data_9` decimal(30,17) NOT NULL,
PRIMARY KEY (`id`),
KEY `date_added` (`date_added`),
KEY `name` (`name`)
) ENGINE=InnoDB AUTO_INCREMENT=60000000 DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci
PARTITION BY RANGE( TO_DAYS(date_added) ) (
PARTITION p1 VALUES LESS THAN (TO_DAYS('2018-01-01')),
PARTITION p2 VALUES LESS THAN (TO_DAYS('2018-02-01')),
PARTITION p3 VALUES LESS THAN (TO_DAYS('2018-03-01')),
PARTITION p4 VALUES LESS THAN (TO_DAYS('2018-04-01')),
PARTITION future VALUES LESS THAN MAXVALUE
);
For the below query will only use partition "future" :
SELECT date_added, data_1, data_2, data_3, data_4, data_5, data_6, data_7, data_8, data_9
FROM numeric_data
WHERE date_added >= '2018-05-03 11:00:00'
AND name = 'aaa'
ORDER BY date_added LIMIT 1

How do I create a partition for every day of the current year in mysql table

Actually I want to store data last three month from current date rest of record deleted. data having millions of record
you can use given query.
CREATE TABLE TABLE_NAME1 (
id BIGINT(20) NOT NULL AUTO_INCREMENT,
Ctime DATETIME DEFAULT NULL,
KEY id (id)
) ENGINE=INNODB AUTO_INCREMENT=286802795 DEFAULT CHARSET=utf8
PARTITION BY RANGE( TO_DAYS(Ctime) ) (
PARTITION p1 VALUES LESS THAN (TO_DAYS('2011-04-02')),
PARTITION p2 VALUES LESS THAN (TO_DAYS('2011-04-03')),
PARTITION p3 VALUES LESS THAN (TO_DAYS('2011-04-04')),
PARTITION p4 VALUES LESS THAN (TO_DAYS('2011-04-05')),
PARTITION p5 VALUES LESS THAN (TO_DAYS('2011-04-06')),
PARTITION p6 VALUES LESS THAN (TO_DAYS('2011-04-07'))
);

MySQL table partitioning on timestamp

I have partitioned a table (because of an out of memory error - table got too big). I have partitioned it on a timestamp column as shown below:
CREATE TABLE test (
fname VARCHAR(50) NOT NULL,
lname VARCHAR(50) NOT NULL,
dob timestamp NOT NULL
)
PARTITION BY RANGE( unix_timestamp(dob) ) (
PARTITION p2012 VALUES LESS THAN (unix_timestamp('2013-01-01 00:00:00')),
PARTITION p2013 VALUES LESS THAN (unix_timestamp('2014-01-01 00:00:00')),
PARTITION pNew VALUES LESS THAN MAXVALUE
);
I was hoping that the process of partitioning would also help in speeding up a couple of my queries whihc take a few hours to run; however, this type of partitioning doesn't seem to kick in and all partitions are still being used and scanned through for the queries. I have tried, and failed, with a couple more approaches:
1) Tried to use different range for the partitioning
CREATE TABLE t2 (
fname VARCHAR(50) NOT NULL,
lname VARCHAR(50) NOT NULL,
region_code TINYINT UNSIGNED NOT NULL,
dob timestamp NOT NULL
)
PARTITION BY RANGE( YEAR(dob) ) (
PARTITION p2012 VALUES LESS THAN (2013),
PARTITION p2013 VALUES LESS THAN (2014),
PARTITION pNew VALUES LESS THAN MAXVALUE
);
However, that results in an error: Error Code: 1486. Constant, random or timezone-dependent expressions in (sub)partitioning function are not allowed
2) Gave up on changing partitioning to be recognized by the query optimizer, and as suggested in MySQL's Doc - 18.5 Partition Selection tried specifying which partitions to use in the select statement instead:
select * from t2 partition (p2012)
But, that returns a syntax error Error Code: 1064. You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near '(p2012) LIMIT 0, 1000' at line 1
Does anybody have any suggestions what else I could try to utilize table partitioning to optimize the queries?
You can use UNIX_TIMESTAMP() function. Example from MySQL docs:
CREATE TABLE quarterly_report_status (
report_id INT NOT NULL,
report_status VARCHAR(20) NOT NULL,
report_updated TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP
)
PARTITION BY RANGE ( UNIX_TIMESTAMP(report_updated) ) (
PARTITION p0 VALUES LESS THAN ( UNIX_TIMESTAMP('2008-01-01 00:00:00') ),
PARTITION p1 VALUES LESS THAN ( UNIX_TIMESTAMP('2008-04-01 00:00:00') ),
PARTITION p2 VALUES LESS THAN ( UNIX_TIMESTAMP('2008-07-01 00:00:00') ),
PARTITION p3 VALUES LESS THAN ( UNIX_TIMESTAMP('2008-10-01 00:00:00') ),
PARTITION p4 VALUES LESS THAN ( UNIX_TIMESTAMP('2009-01-01 00:00:00') ),
PARTITION p5 VALUES LESS THAN ( UNIX_TIMESTAMP('2009-04-01 00:00:00') ),
PARTITION p6 VALUES LESS THAN ( UNIX_TIMESTAMP('2009-07-01 00:00:00') ),
PARTITION p7 VALUES LESS THAN ( UNIX_TIMESTAMP('2009-10-01 00:00:00') ),
PARTITION p8 VALUES LESS THAN ( UNIX_TIMESTAMP('2010-01-01 00:00:00') ),
PARTITION p9 VALUES LESS THAN (MAXVALUE)
);
You can find it in:
https://dev.mysql.com/doc/refman/5.7/en/partitioning-range.html.
You can do this if you use DATE or DATETIME instead of TIMESTAMP as the data type.
CREATE TABLE t2 (
fname VARCHAR(50) NOT NULL,
lname VARCHAR(50) NOT NULL,
region_code TINYINT UNSIGNED NOT NULL,
dob DATETIME NOT NULL
)
PARTITION BY RANGE( YEAR(dob) ) (
PARTITION p2012 VALUES LESS THAN (2013),
PARTITION p2013 VALUES LESS THAN (2014),
PARTITION pNew VALUES LESS THAN MAXVALUE
);
Using the partition-selection hint is only supported in MySQL 5.6 and later.
See http://dev.mysql.com/doc/refman/5.6/en/partitioning-selection.html
Note that the page of the manual is only for MySQL 5.6. If you try to click on MySQL 5.5 documentation link, it redirects you back to 5.6.
The wrong is that, This field dob is not unique key!
you can use this command:
CREATE TABLE t2 (
fname VARCHAR(50) NOT NULL,
lname VARCHAR(50) NOT NULL,
region_code TINYINT UNSIGNED NOT NULL,
dob timestamp NOT NULL,
unique 'dob' (dob)
)
or the table is exist:
alter table t2 add UNIQUE(dob)
You can try it!