I have a table with many partitions by date
I want to move some of the oldest partitions to another table.
I succeed moving the oldest partition by following the manual
but when I try to move more partitions I get: Error Code: 1737. Found a row that does not match the partition
So, I deleted the oldest partition and move the next partition -but then the rows from the first partitions returned to the original table (I did NOT see any documentation for record that go back...)
How can I move the three first partitions to another table?
THANKS
CREATE TABLE `TestPartA` (
`Name` VARCHAR(50) NOT NULL,
`Time` DATETIME NOT NULL DEFAULT CURRENT_TIMESTAMP,
`Slot` INT(11) NOT NULL DEFAULT '-1',
`Text` VARCHAR(50) NOT NULL,
PRIMARY KEY (`Name`, `Text`, `Time`),
INDEX `ClusterTimeIdx` (`Name`, `Time`, `Slot`),
INDEX `Time` (`Time`)
)
PARTITION BY RANGE (TO_DAYS(TIME))
(PARTITION p20190407 VALUES LESS THAN (TO_DAYS('2019-04-07')) ,
PARTITION p20190421 VALUES LESS THAN (TO_DAYS('2019-04-21')) ,
PARTITION p20190428 VALUES LESS THAN (TO_DAYS('2019-04-28')),
PARTITION p20190505 VALUES LESS THAN (TO_DAYS('2019-05-05'))) ;
CREATE TABLE TestPartB LIKE TestPartA;
ALTER TABLE TestPartB REMOVE PARTITIONING;
insert into TestPartA values ('A','2019-04-02',1,'W1');
insert into TestPartA values ('A','2019-04-04',1,'W1');
insert into TestPartA values ('A','2019-04-08',1,'W1');
insert into TestPartA values ('A','2019-04-20',1,'W1');
insert into TestPartA values ('A','2019-05-01',1,'W1');
SELECT PARTITION_NAME, TABLE_ROWS FROM INFORMATION_SCHEMA.PARTITIONS WHERE TABLE_NAME = 'TestPartA';
-- move the first partition
ALTER TABLE TestPartA EXCHANGE PARTITION p20190407 WITH TABLE TestPartB; -- Works GREAT
select * from TestPartA;
select * from TestPartB;
SELECT PARTITION_NAME, TABLE_ROWS FROM INFORMATION_SCHEMA.PARTITIONS WHERE TABLE_NAME = 'TestPartA'; -- this is not working any more - but according to documentation it happens sometimes
--move the second partition
ALTER TABLE TestPartA EXCHANGE PARTITION p20190421 WITH TABLE TestPartB; -- FAILED
ALTER TABLE TestPartA drop PARTITION p20190407;
ALTER TABLE TestPartA EXCHANGE PARTITION p20190421 WITH TABLE TestPartB; -- Succeed but the rows from the first partitions returned to table A
select * from TestPartA;
select * from TestPartB;
Related
I am tuning our database indices and discovered some strange behavior in Mysql 5.7.32. Here is a script to replicate the issue.
I have a table employees with three columns id, firstname and lastname. There are two indexes on the table for each of the varchar columns. For one of the SELECT statements below, the output is unexpectedly not using the key.
Why is one of those queries not using the index? Is it because Miller is the first value in the table? Or is this an inaccuracy of EXPLAIN?
DROP TABLE if EXISTS `employee`;
CREATE TABLE `employee` (
`id` INT(11) NOT NULL auto_increment,
`firstname` VARCHAR(50) NOT NULL,
`lastname` VARCHAR(50) NOT NULL,
PRIMARY KEY (`id`),
INDEX `index_firstname` (`firstname`),
INDEX `index_lastname` (`lastname`)
);
INSERT INTO `employee` (firstname,lastname) VALUES('alice','Miller');
INSERT INTO `employee` (firstname,lastname) VALUES('bob','Miller');
INSERT INTO `employee` (firstname,lastname) VALUES('charlie','Miller');
INSERT INTO `employee` (firstname,lastname) VALUES('doyle','Miller');
INSERT INTO `employee` (firstname,lastname) VALUES('evan','Smith');
INSERT INTO `employee` (firstname,lastname) VALUES('franz','Smith');
INSERT INTO `employee` (firstname,lastname) VALUES('gloria','Smith');
INSERT INTO `employee` (firstname,lastname) VALUES('helga','Unique');
EXPLAIN SELECT * FROM employee WHERE firstname='alice'; # uses the key 'index_firstname'
EXPLAIN SELECT * FROM employee WHERE lastname='Smith'; # uses the key 'index_lastname'
EXPLAIN SELECT * FROM employee WHERE lastname='Unique'; # uses the key 'index_lastname'
EXPLAIN SELECT * FROM employee WHERE lastname='Miller'; # does not use the key 'index_lastname'
Where a sampling of index values has over a ~25% ratio (not exact, see below) of sampling of the given value, the index isn't used.
There is a cost calculation that works out that scanning the full table is faster than using the secondary index (which needs to fetch from the primary table to retrieve *).
I am trying to do Multi-tenant based Mysql partitioning - need tenant_code as first level and second level created_at year and month combo partitioning
CREATE TABLE `_log` (
`id` BIGINT(20) NOT NULL AUTO_INCREMENT,
`message` LONGTEXT NULL,
`tenant_code` VARCHAR(20) NOT NULL,
`created_at` DATETIME NOT NULL,
PRIMARY KEY (`id`, `tenant_code`, `created_at`) USING BTREE
)
This is my table
ALTER TABLE weekly_audit_log
PARTITION BY LIST COLUMNS(company_code,YEAR(created_at),MONTH(created_at))(
PARTITION p_01_2019_1 VALUES IN('01',2019,1),
PARTITION p_01_2019_2 VALUES IN('01',2019,2),
PARTITION p_01_2019_3 VALUES IN('01',2019,3),
PARTITION p_01_2019_4 VALUES IN('01',2019,4),
PARTITION p_01_2019_5 VALUES IN('01',2019,5),
PARTITION p_01_2019_6 VALUES IN('01',2019,6),
PARTITION p_01_2019_7 VALUES IN('01',2019,7),
PARTITION p_01_2019_8 VALUES IN('01',2019,8),
PARTITION p_01_2019_9 VALUES IN('01',2019,9),
PARTITION p_01_2019_10 VALUES IN('01',2019,10),
PARTITION p_01_2019_11 VALUES IN('01',2019,11),
PARTITION p_01_2019_12 VALUES IN('01',2019,12),
PARTITION p_01_2020_1 VALUES IN('01',2020,1),
PARTITION p_01_2020_2 VALUES IN('01',2020,2),
PARTITION p_01_2020_3 VALUES IN('01',2020,3),
PARTITION p_01_2020_4 VALUES IN('01',2020,4),
PARTITION p_01_2020_5 VALUES IN('01',2020,5),
PARTITION p_01_2020_6 VALUES IN('01',2020,6),
PARTITION p_01_2020_7 VALUES IN('01',2020,7),
PARTITION p_01_2020_8 VALUES IN('01',2020,8),
PARTITION p_01_2020_9 VALUES IN('01',2020,9),
PARTITION p_01_2020_10 VALUES IN('01',2020,10),
PARTITION p_01_2020_11 VALUES IN('01',2020,11),
PARTITION p_01_2020_12 VALUES IN('01',2020,12));
I have a table:
+----+---------+----------+
| id | user_id | comment |
+----+---------+----------+
Where column type is:
id (bigint not null primary key autoincrement)
user_id (bigint not null)
comment (text)
How can I partition this table on user_id by range? I tried to partition this table by range in PHPMyAdmin but doesn't allow me because user_id isn't a primary key. If I have many 10 billion users and each has an infinite amount of comments this table will be very large. I want to partition it like:
partition 1 (user_id<500)
+----+---------+----------+
| id | user_id | comment |
+----+---------+----------+
partition 2 (user_id<1000)
+----+---------+----------+
| id | user_id | comment |
+----+---------+----------+
And so on.
Ensure you have satisfied the criteria of when to use partitioning. This is a rather rare case and needs to map closely to your queries. A 500 user range seems tiny. MySQL can handle large tables without partitioning so don't assume its necessary.
The form is:
CREATE TABLE tbl (
id bigint unsigned AUTO_INCREMENT NOT NULL,
user_id bigint unsigned NOT NULL,
COMMENT TEXT NOT NULL,
PRIMARY KEY (user_id, id),
key(id))
PARTITION BY RANGE (user_id) (
PARTITION p0 VALUES LESS THAN (500),
PARTITION p1 VALUES LESS THAN (1000),
PARTITION p2 VALUES LESS THAN (2000),
PARTITION p3 VALUES LESS THAN (3000)
);
ref: fiddle
Yes, since user_id is not part of the table primary key(s) or unique keys you can't create partitions solely for the user_id on your table as the DOCs states very clearly
every unique key on the table must use every column in the table's partitioning expression
So for your case what you can do is to add a unique key on your table for both id and user_id
alter table myTable add unique key uk_id_userid (id, user_id);
And then add the range partition for only user_id column as such:
alter table myTable partition by range (user_id) (
PARTITION p0 VALUES LESS THAN (10),
PARTITION p1 VALUES LESS THAN (20),
PARTITION p2 VALUES LESS THAN (30),
PARTITION p3 VALUES LESS THAN (40)
);
Note Since you already have a table with values in order to define your partition ranges you need to wrap around all existing values for your user_id column in your partitions. That is if you have a user_id of 1000 you can not define your last partition as PARTITION p3 VALUES LESS THAN (1000) that will fail. You will need one more partition i.e.: PARTITION p3 VALUES LESS THAN (2000) or PARTITION p3 VALUES LESS THAN MAXVALUE
See it working here: http://sqlfiddle.com/#!9/8ca7ed
Full working example:
create table myTable (
id bigint not null auto_increment,
user_id bigint not null,
comment text,
key (id)
) engine=InnoDb;
insert into myTable (user_id, comment) values
(1, 'bla'), (1, 'ble'), (1, 'bli'), (1, 'blo'),
(12, 'bla'), (12, 'ble'), (12, 'bli'), (12, 'blo'),
(23, 'bla'), (23, 'ble'), (23, 'bli'), (23, 'blo'),
(34, 'bla'), (34, 'ble'), (34, 'bli'), (34, 'blo');
alter table myTable add unique key uk_id_userid (id, user_id);
alter table myTable partition by range (user_id) (
PARTITION p0 VALUES LESS THAN (10),
PARTITION p1 VALUES LESS THAN (20),
PARTITION p2 VALUES LESS THAN (30),
PARTITION p3 VALUES LESS THAN (40)
);
I have one table, want to partition by RANGE (created_at timestamp), so can delete old data easily (by drop partition).
CREATE TABLE `orders` (
`order_id` NVARCHAR(64) NOT NULL,
`amount` INTEGER NOT NULL,
`created_at` TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
`modified_at` TIMESTAMP DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
KEY `order_id` (`order_id`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci;
ALTER TABLE dropship.orders
PARTITION BY RANGE (UNIX_TIMESTAMP(created_at)) (
PARTITION p0 VALUES LESS THAN ( UNIX_TIMESTAMP('2019-03-01 00:00:00') ),
PARTITION p1 VALUES LESS THAN ( UNIX_TIMESTAMP('2019-04-01 00:00:00') ),
PARTITION p2 VALUES LESS THAN ( UNIX_TIMESTAMP('2019-05-01 00:00:00') ),
PARTITION p3 VALUES LESS THAN ( UNIX_TIMESTAMP('2019-06-01 00:00:00') ),
PARTITION p4 VALUES LESS THAN ( UNIX_TIMESTAMP('2019-07-01 00:00:00') ),
PARTITION p5 VALUES LESS THAN (MAXVALUE)
);
This table only has two usages: get by order_id, or update by order_id.
select * from orders where order_id = '123';
update orders set amount = 10 where order_id = '123';
Due to the limitation of Mysql partitioning, I cannot add an unique key for order_id since will use created_at field for partitioning.
All columns used in the table's partitioning expression must be part of every unique key that the table may have, including any primary key.
Question:
Any way to make order_id unique in this table please?
I have thought about partitioning by order_id, but it's hard to delete old data in that way.
Any suggestion is welcome. (For example may be you have better design for this table).
BEGIN;
SELECT 1 FROM orders WHERE order_id = 234 FOR UPDATE;
if row exists, you have a dup error.
INSERT INTO orders ... order_id = 234;
COMMIT;
But, as Raymond points out, you may as well drop PARTITIONing and make the column the PRIMARY KEY. This would make all the stated operations slightly faster.
I have the following:
data points collected per day with associated date - unix timestamp.
I am trying to achieve the following:
Filter data points per day. i.e suppose I have 4 data points per day, I am trying to reduce it to n data points per day using GROUP BY.
I am using the following query:
SELECT *,FLOOR(UNIX_TIMESTAMP(date)/((1440/n)*60))
AS timekey
FROM `tbl`
where date < 'date'
GROUP BY timekey)m
UNION
(SELECT * ,0 As timekey FROM `tbl`
where date > 'date' );
This according to me should generate timekey for the data points for any row with date < given_date_in_query.
n determines the number of points to be grouped by.
Then the GROUP BY should ultimately reduce the number of data points with respect to n.
The above query works for a table with more than 20000 rows.
for n=2 the datapoints before the specified date are reduced to 2 points(to 2 rows from 4).
Here is the create script:
CREATE TABLE `tbl_a` (
`id` int(25) NOT NULL AUTO_INCREMENT,
`pid` bigint(11) NOT NULL,
---
`date` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP,
---
PRIMARY KEY (`id`),
UNIQUE KEY `phase_id_2` (`phase_id`,`date`),
KEY `id` (`id`,`pid`),
KEY `date` (`date`)
) ENGINE=InnoDB AUTO_INCREMENT=44 DEFAULT CHARSET=utf8 COLLATE=utf8_unicode_ci;
The query works perfectly for the above table.
However when I try the same query with another db with comparatively less rows it does not work as expected.
It reduces only the date specified in the query and the rest remain untouched.
.i.e only the row corresponding to the given_date in the query is
reduced by the query and the rest of the rows remain untouched.
Test table create script:
CREATE TABLE `test` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`date` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP,
PRIMARY KEY (`id`),
KEY `date` (`date`)
) ENGINE=InnoDB AUTO_INCREMENT=1 DEFAULT CHARSET=utf8 COLLATE=utf8_unicode_ci;
Could this be an index error or a database engine exception?
Steps to recreate the problem:
Create test tbl :
CREATE TABLE `test` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`date` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP,
PRIMARY KEY (`id`),
KEY `date` (`date`)
) ENGINE=InnoDB AUTO_INCREMENT=1 DEFAULT CHARSET=utf8 COLLATE=utf8_unicode_ci;
Insert data:
INSERT INTO test (date) VALUES ('2015-11-01 00:00:00');
INSERT INTO test (date) VALUES ('2015-11-01 06:00:00');
INSERT INTO test (date) VALUES ('2015-11-01 12:00:00');
INSERT INTO test (date) VALUES ('2015-11-01 18:00:00');
INSERT INTO test (date) VALUES ('2016-11-01 00:00:00');
INSERT INTO test (date) VALUES ('2016-11-01 06:00:00');
INSERT INTO test (date) VALUES ('2016-11-01 12:00:00');
INSERT INTO test (date) VALUES ('2016-11-01 18:00:00');
INSERT INTO test (date) VALUES ('2016-11-02 00:00:00');
INSERT INTO test (date) VALUES ('2016-11-02 06:00:00');
INSERT INTO test (date) VALUES ('2016-11-02 12:00:00');
INSERT INTO test (date) VALUES ('2016-11-02 18:00:00');
INSERT INTO test (date) VALUES ('2017-11-02 00:00:00');
INSERT INTO test (date) VALUES ('2017-11-02 06:00:00');
INSERT INTO test (date) VALUES ('2017-11-02 12:00:00');
INSERT INTO test (date) VALUES ('2017-11-02 18:00:00');
Run Query:
SELECT * FROM
(SELECT *,FLOOR(UNIX_TIMESTAMP(date)/((1440/2)*60))
AS timekey
FROM `test`
where date < '2017-11-02'
GROUP BY timekey) m
UNION
(SELECT * ,0 As timekey FROM `test`
where date > '2017-11-02');
The above should:
reduce all data points prior to 2017-11-02 which should give a result of 10.