I have an extremely large MySQL table that I would like to partition. A simplified create of this table is as given below -
CREATE TABLE `myTable` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`columnA` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP,
`columnB` varchar(50) NOT NULL ,
`columnC` int(11) DEFAULT NULL,
`columnD` varchar(255) DEFAULT NULL,
`columnE` int(11) DEFAULT NULL,
`columnF` varchar(255) DEFAULT NULL,
`columnG` varchar(255) DEFAULT NULL,
PRIMARY KEY (`id`),
UNIQUE KEY `UNIQ_B` (`columnB`),
UNIQUE KEY `UNIQ_B_C` (`columnB`,`columnC`),
UNIQUE KEY `UNIQ_C_D` (`columnC`,`columnD`),
UNIQUE KEY `UNIQ_E_F_G` (`columnE`,`columnF`,`columnG`)
)
I want to partition my table either by columnA or id, but the problem is that the MySQL Manual states -
In other words, every unique key on the table must use every column in the table's partitioning expression.
Which means that I cannot partition the table on either of those columns without changing my schema. For example, I have considered adding id to all my unique keys like so -
CREATE TABLE `myTable` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`columnA` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP,
`columnB` varchar(50) NOT NULL ,
`columnC` int(11) DEFAULT NULL,
`columnD` varchar(255) DEFAULT NULL,
`columnE` int(11) DEFAULT NULL,
`columnF` varchar(255) DEFAULT NULL,
`columnG` varchar(255) DEFAULT NULL,
PRIMARY KEY (`id`),
UNIQUE KEY `UNIQ_B` (`columnB`,`id`),
UNIQUE KEY `UNIQ_B_C` (`columnB`,`columnC`,`id`),
UNIQUE KEY `UNIQ_C_D` (`columnC`,`columnD`,`id`),
UNIQUE KEY `UNIQ_E_F_G` (`columnE`,`columnF`,`columnG`,`id`)
)
Which I do not mind doing except for the fact that it allows for the creation of rows that should not be created. For example, by my original schema, the following row insertion wouldn't have worked twice -
INSERT into myTable (columnC, columnD) VALUES (1.0,2.0)
But it works with the second schema as columnC and columnD by themselves no longer form a unique key. I have considered getting around this by using triggers to prevent the creation of such rows but then the trigger cost would reduce(or outweigh) the partitioning performance gain
Edited:
Some additional information about this table:
Table has more than 1.2Billion records.
Using Mysql 5.6.34 version with InnoDB Engine and running on AWS RDS.
Few other indexes are also there on this table.
Because of huge data and multiple indexes it is an expensive process to insert and retrieve the data.
There are no unique indexes on timestamp and float data types. It was just a sample table schema for illustration. Our actual table has similar schema as above table.
Other than Partitioning what options do we have to improve the
performance of the table without losing any data and maintaining the
integrity constraints.
How do I partition a MySQL table that contains several unique keys?
Sorry to say, you don't.
Also, you shouldn't. Remember that UPDATE and INSERT operations to a table with unique keys necessarily must query the table to ensure the keys stay unique. If it were possible to partition a table so unique keys weren't built in to the partititon expression, then every insert or update would require querying every partition. This would be likely to make the partitioning worse than useless.
Related
I'm working with SQL 5.7 and am trying to partition a table to improve performance over time. I'm new to partitioning and have come across this .
My table stores millions of rows and is used for reporting, so it quite query intensive, in particular, on my minute_rounded_timestamp column. In addition, almost every column has an index.
This is how I'm trying to do a partition:
CREATE TABLE `tlp_payout_partition_report_minute` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`minute_rounded_timestamp` int(11) NOT NULL,
`application_id` bigint(20) NOT NULL,
`commission` float NOT NULL DEFAULT '0',
`seller_code` int(11) DEFAULT NULL,
`tlp_aff_id` varchar(255) DEFAULT NULL,
`sub_aff_id` varchar(255) DEFAULT NULL,
`application_processing_duration_secs` int(11) DEFAULT NULL,
`bank_hash` varchar(20) DEFAULT NULL,
`application_result` varchar(20) DEFAULT NULL,
`user_id` int(11) NOT NULL,
`created` datetime DEFAULT NULL,
`modified` datetime DEFAULT NULL,
PRIMARY KEY (`id`) USING BTREE,
KEY `minute_rounded_timestamp` (`minute_rounded_timestamp`) USING BTREE,
KEY `application_id` (`application_id`) USING BTREE,
KEY `user_id` (`user_id`) USING BTREE,
KEY `seller_code` (`seller_code`) USING BTREE,
KEY `tlp_aff_id` (`tlp_aff_id`) USING BTREE,
KEY `sub_aff_id` (`sub_aff_id`) USING BTREE,
KEY `application_result` (`application_result`) USING BTREE,
KEY `created` (`created`),
KEY `modified` (`modified`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8
PARTITION BY RANGE( TO_DAYS(id, created) ) (
PARTITION p20110401 VALUES LESS THAN ( TO_DAYS('2011-04-02') ),
PARTITION p20110402 VALUES LESS THAN ( TO_DAYS('2011-04-03') ),
PARTITION p20110403 VALUES LESS THAN ( TO_DAYS('2011-04-04') ),
PARTITION p20110404 VALUES LESS THAN ( TO_DAYS('2011-04-05') ),
PARTITION future VALUES LESS THAN MAXVALUE
);
Which gives me the following error:
A PRIMARY KEY must include all columns in the table's partitioning function
So then, if I try to remove the PRIMARY KEY ( id ) USING BTREE row, I get:
Incorrect table definition; there can be only one auto column and it must be defined as a key
With (id, created) I get:
Incorrect parameter count in the call to native function 'TO_DAYS'
What am I missing?
TO_DAYS() takes only one argument, not two. So the following is an error:
PARTITION BY RANGE( TO_DAYS(id, created) )
Partitioning in MySQL is tricky, because of the rule described in this manual page: https://dev.mysql.com/doc/refman/5.7/en/partitioning-limitations-partitioning-keys-unique-keys.html
All columns used in the partitioning expression for a partitioned table must be part of every unique key that the table may have.
In other words, every unique key on the table must use every column in the table's partitioning expression. (This also includes the table's primary key, since it is by definition a unique key. This particular case is discussed later in this section.)
Read the page I linked to for more explanation and examples.
Honestly, this limitation prevents many sites from using partitioning on datetime columns. Either they need to make the datetime column their primary key, or else make a table with no primary key. Both of these choices are bad.
For what it's worth, partitioning isn't a magic "go fast" feature. It only helps improve performance for certain queries, if and only if the query has expressions that take advantage of partition pruning.
Using indexes is usually a much more flexible and effective way to optimize tables.
I have been facing to an issue with ON DELETE CASCADE in MySQL. It works perfectly when it is set on a primary key field, but not in other cases.
For example, I have a child table where I have a foreign key referring to a field in the parent table, but the child table has its own Auto-Incremental ID field which needs to be a primary key because grandchildren tables refer to it.
When I delete a row from the parent table, all records disappear as expected and no errors appear, however, the child's data depending on the deleted row of the parent table stays untouched.
I made researches without results. Although I assume it is something to do with the system identifying a row by its primary key, I could not find any relevant info about this.
The parent table:
CREATE TABLE IF NOT EXISTS table_parent (
ID TINYINT(3) UNSIGNED PRIMARY KEY AUTO_INCREMENT,
`level` TINYINT(1) NOT NULL,
updated DATETIME NOT NULL DEFAULT NOW()
);
The child table:
CREATE TABLE IF NOT EXISTS table_child (
ID TINYINT(3) UNSIGNED PRIMARY KEY AUTO_INCREMENT,
parentId TINYINT(3) UNSIGNED NOT NULL,
`name` VARCHAR(16) UNIQUE NOT NULL,
updated DATETIME NOT NULL DEFAULT NOW()
);
The relation:
ALTER TABLE table_child
ADD FOREIGN KEY (parentId) REFERENCES table_parent(ID) ON DELETE CASCADE
And in a nutshell, my goal would be to delete all records in table_child table where the parentId equals to the deleted row in table_parent.
Thank you for your help and have a nice day :)
It appears to me that what you are missing is that referential integrity constraints only work with InnoDB tables. Your DDL statements are missing the engine=InnoDB and most likely are defaulting to MyISAM.
While you will not receive an error on the declarations, MyISAM tables are by default what you get when you don't specify an engine, and MyISAM ignores constraint statements.
A corrected CREATE TABLE statement would be:
CREATE TABLE IF NOT EXISTS table_parent (
ID TINYINT(3) UNSIGNED PRIMARY KEY AUTO_INCREMENT,
`level` TINYINT(1) NOT NULL,
updated DATETIME NOT NULL DEFAULT NOW()
) ENGINE=InnoDB;
Here's a SQL Sandbox demonstrating that the constraint is correct and works as you expect it to.
This is not relevant to the question, but it seems odd to me that you declared all your keys to be TINYINTs. That would mean that you could have a maximum of 255 rows in your tables....
I have this table:
CREATE TABLE `sites_routing` (
`origin_site_id` int(11) NOT NULL,
`destination_site_id` int(11) NOT NULL,
`distance_meters` int(11) DEFAULT NULL,
`id` int(11) NOT NULL AUTO_INCREMENT,
`type` varchar(45) NOT NULL DEFAULT 'walking',
PRIMARY KEY (`id`)
)
I want to create two foreign keys for columns origin_site_id and destination_site_id and than another unique index on 3 columns: origin_site_id, destination_site_id and type. If I'm doing that mysql becomes really slow on reading and writing to the table and I can see in the error log stuff like:
sites_routing contains 4 indexes inside InnoDB, which is different from the number of ns 4 indexes inside InnoDB, which is different from the number of indexes 3 defined in the MySQL
Still, if I try to drop the fks and the indexs and recreate the fks when the unique key exists, mysql allows me to create only one fk using the unique key as index but for the other one it automatically creates another index.
What's the best way to go here?
Currently I have one table in a mySQL Database that I want to connect to another table I need to create. The second table is just going to have 2 columns, the member_id and product column. There is to be many entries in the product column for the same member_ID and the product column is to be a foreign key for another table.
Here is the SQL code I used to create the initial Login table:
'CREATE TABLE `members`(
`member_id` int(11) unsigned NOT NULL AUTO_INCREMENT,
`firstname` varchar(100) DEFAULT NULL,
`lastname` varchar(100) DEFAULT NULL,
`login` varchar(100) NOT NULL DEFAULT \'\',
`passwd` varchar(32) NOT NULL DEFAULT \'\',
PRIMARY KEY (`member_id`))
ENGINE=MyISAM AUTO_INCREMENT=8 DEFAULT CHARSET=utf8'
If someone could show me how to create the second table to connect to member_id that would be great.
The MyISAM storage engine does not support foreign keys. If You want to ensure data integrity by using foreign keys You need to use another storage engine eg. InnoDB
Using MyISAM You can't define a foreign key but You can still define the tables as following:
http://sqlfiddle.com/#!2/4cb9c/1/0
Please note that it is important to create the indexes to ensure decent performance for joins. If You want to use InnoDB and foreign keys then the explicit index creation is not needed as adding a foreign key creates an implicit index.
I am wondering if there is a better way to make some mysql tables than what I have been using in this project. I have a series of numbers which represent a specific time. Such as the number 101 would represent Jan 12, 2012 for example. It doesn't only represent time but that is the very basic of that information. So I created a lexicon table which has all the numbers we use and details such as time and meaning of that number. I have another table that is per customer which whenever they make a purchase I check off that the purchase is eligiable for a specific time. But the table where I check off each purchase and the lexicon table are not linked. I am wondering if there is a better way, maybe a way to have an sql statement take all the data from the Lexicon table and turn that into columns while the rows consist of customer ID and a true/false selector.
table structure
THIS IS THE CUSTOMER PURCHASED TABLE T/F
CREATE TABLE `group1` (
`100` TINYINT(4) NULL DEFAULT '0',
`101` TINYINT(4) NULL DEFAULT '0',
`102` TINYINT(4) NULL DEFAULT '0',
... this goes on for 35 times each table
PRIMARY KEY (`CustID`)
)
THIS IS THE LEXICON TABLE
CREATE TABLE `lexicon` (
`Number` INT(3) NOT NULL DEFAULT '0',
`Date` DATETIME NULL DEFAULT NULL,
`OtherPurtinantInfo` .... etc
)
So I guess instead of making groups of numbers every season for the customers I would prefer being able to use the updated lexicon table to automatically generate a table. My only concerns are that we have many many numbers so that would make a very large table all combined together but perhaps that could be limited into groups automatically as well so that it is not an overwhelming table.
I am not sure if I am being clear enough so feel free to comment on things that need to be clarified.
Here's a normalized ERD, based on what I understand your business requirements to be:
The classifieds run on certain dates, and a given advertisement can be run for more than one classifieds date.
The SQL statements to make the tables:
CREATE TABLE IF NOT EXISTS `classified_ads` (
`id` INT UNSIGNED NOT NULL AUTO_INCREMENT,
PRIMARY KEY (`id`)
);
CREATE TABLE IF NOT EXISTS `classified_dates` (
`id` INT UNSIGNED NOT NULL AUTO_INCREMENT,
`date` DATETIME NOT NULL,
`info` TEXT NULL,
PRIMARY KEY (`id`)
);
CREATE TABLE IF NOT EXISTS `classified_ad_dates` (
`classified_ad_id` INT UNSIGNED NOT NULL,
`classifiend_date_id` INT UNSIGNED NOT NULL,
PRIMARY KEY (`classified_ad_id`, `classifiend_date_id`),
INDEX `fk_classified_ad_dates_classified_ads1` (`classified_ad_id` ASC),
INDEX `fk_classified_ad_dates_classified_dates1` (`classifiend_date_id` ASC),
CONSTRAINT `fk_classified_ad_dates_classified_ads1`
FOREIGN KEY (`classified_ad_id`)
REFERENCES `classified_ads` (`id`)
ON DELETE CASCADE
ON UPDATE CASCADE,
CONSTRAINT `fk_classified_ad_dates_classified_dates1`
FOREIGN KEY (`classifiend_date_id`)
REFERENCES `classified_dates` (`id`)
ON DELETE CASCADE
ON UPDATE CASCADE
);