I wish to retrieve around 200 million profiles using PDO.
It is a partitioned table, so it does not have a primary key, it's sorted with INNODBs internal primary key.
I need to retrieve the data in internal sort order for performance, so using SORT BY some_id and to continue from there would not work for performance reasons.
Using LIMIT OFFSET would not work at all, OFFSET 100 million or more would take ages, the RAM is by far not enough to take all the terrabyte of table data into cache.
I would need something that fetches a chunk of rows, remembers the internal position and continues with next call.
Update
I was digging a bit and the problem seems familiar, it was solved in Oracle by having ROWID and ROWNUM which refer to the internal id.
It seems we have such an id in innodb as well but it's not queryable ?!
That sounds like a significant disadvantage of mysql for performant query.
Table def:
CREATE TABLE `members` (
`id` int(11) NOT NULL DEFAULT 0,
`intern_id` varchar(256) CHARACTER SET utf8 COLLATE utf8_bin DEFAULT NULL,
`first_name` varchar(48) CHARACTER SET utf8 COLLATE utf8_bin DEFAULT NULL,
`last_name` varchar(48) CHARACTER SET utf8 COLLATE utf8_bin DEFAULT NULL,
`middle_name` varchar(48) CHARACTER SET utf8 COLLATE utf8_bin DEFAULT NULL,
`l` varchar(196) CHARACTER SET utf8 COLLATE utf8_bin DEFAULT NULL,
`i` varchar(128) CHARACTER SET utf8 COLLATE utf8_bin DEFAULT NULL,
`ex` varchar(256) CHARACTER SET utf8 COLLATE utf8_bin DEFAULT NULL,
`country_code` varchar(4) CHARACTER SET utf8 COLLATE utf8_bin DEFAULT NULL,
`country_name` varchar(32) CHARACTER SET utf8 COLLATE utf8_bin DEFAULT NULL,
`state_name` varchar(32) CHARACTER SET utf8 COLLATE utf8_bin DEFAULT NULL,
`city_name` varchar(32) CHARACTER SET utf8 COLLATE utf8_bin DEFAULT NULL,
`edu` varchar(256) CHARACTER SET utf8 COLLATE utf8_bin DEFAULT NULL,
`nc` smallint(6) DEFAULT NULL,
`nj` smallint(6) DEFAULT NULL,
`jt` varchar(256) CHARACTER SET utf8 COLLATE utf8_bin DEFAULT NULL,
`cn` varchar(256) CHARACTER SET utf8 COLLATE utf8_bin DEFAULT NULL,
`email_address` varchar(128) CHARACTER SET utf8 COLLATE utf8_bin DEFAULT NULL,
`fpe` tinyint(4) DEFAULT NULL COMMENT '1/0',
`pbu` varchar(256) CHARACTER SET utf8 COLLATE utf8_bin DEFAULT NULL,
`cii` varchar(256) CHARACTER SET utf8 COLLATE utf8_bin DEFAULT NULL,
`unmatched_facts` varchar(2048) CHARACTER SET utf8 COLLATE utf8_bin DEFAULT NULL,
`dt_snapshot` datetime DEFAULT NULL,
`change_small` tinyint(4) DEFAULT NULL,
`change_significant` tinyint(4) DEFAULT NULL,
`jta` varchar(128) CHARACTER SET utf8 COLLATE utf8_bin DEFAULT NULL,
`cna` varchar(128) CHARACTER SET utf8 COLLATE utf8_bin DEFAULT NULL,
`cnia` varchar(256) CHARACTER SET utf8 COLLATE utf8_bin DEFAULT NULL,
`ut_created` int(11) DEFAULT NULL ,
`reserve_int_2` int(11) DEFAULT NULL,
`reserve_vc1` varchar(128) CHARACTER SET utf8 COLLATE utf8_bin DEFAULT NULL,
`reserve_vc2` varchar(128) CHARACTER SET utf8 COLLATE utf8_bin DEFAULT NULL,
`reserve_vc_3` varchar(256) CHARACTER SET utf8 COLLATE utf8_bin DEFAULT NULL,
KEY `id` (`id`),
KEY `intern_id` (`intern_id`),
KEY `state_name` (`state_name`),
KEY `city_name` (`city_name`),
KEY `i` (`i`),
KEY `country_name` (`country_name`)
) ENGINE=InnoDB DEFAULT CHARSET=latin1
PARTITION BY KEY (`country_name`)
PARTITIONS 30
Related
I am trying to insert some datetime values and I came up with a non sense data truncation problem, and I have no idea how to solve and what is causing it. Hope someone helps!
I have the following table:
CREATE TABLE tb_cliente (
id bigint unsigned NOT NULL AUTO_INCREMENT,
st_servidor tinyint(1) NOT NULL,
nm_nome varchar(255) COLLATE utf8mb4_unicode_ci NOT NULL,
nm_nome_social varchar(255) COLLATE utf8mb4_unicode_ci DEFAULT NULL,
cd_matricula varchar(255) COLLATE utf8mb4_unicode_ci NOT NULL,
cd_siape varchar(255) CHARACTER SET utf8mb4
COLLATE utf8mb4_unicode_ci DEFAULT NULL,
dt_nascimento timestamp NOT NULL,
sexo varchar(1) COLLATE utf8mb4_unicode_ci NOT NULL,
nacionalidade varchar(255) COLLATE utf8mb4_unicode_ci NOT NULL,
ano_chegada_no_pais varchar(255)
COLLATE utf8mb4_unicode_ci DEFAULT NULL,
naturalidade varchar(255) CHARACTER SET utf8mb4
COLLATE utf8mb4_unicode_ci DEFAULT NULL,
uf_nascimento varchar(255) COLLATE utf8mb4_unicode_ci NOT NULL,
estado_civil varchar(255) COLLATE utf8mb4_unicode_ci NOT NULL,
sn_conjuge_servidor tinyint(1) DEFAULT NULL,
nm_conjuge varchar(255) COLLATE utf8mb4_unicode_ci DEFAULT NULL,
nm_pai varchar(255) COLLATE utf8mb4_unicode_ci DEFAULT NULL,
nm_mae varchar(255) CHARACTER SET utf8mb4
COLLATE utf8mb4_unicode_ci DEFAULT NULL,
raca varchar(255) CHARACTER SET utf8mb4
COLLATE utf8mb4_unicode_ci DEFAULT NULL,
cor_dos_olhos varchar(255) CHARACTER SET utf8mb4
COLLATE utf8mb4_unicode_ci DEFAULT NULL,
cor_dos_cabelos varchar(255) CHARACTER SET utf8mb4
COLLATE utf8mb4_unicode_ci DEFAULT NULL,
sn_alergico tinyint(1) DEFAULT NULL,
grau_de_instrucao varchar(255) COLLATE utf8mb4_unicode_ci DEFAULT NULL,
dt_conclusao timestamp NULL DEFAULT NULL,
altura decimal(8,2) DEFAULT NULL,
tipo_sanguineo varchar(255) CHARACTER SET utf8mb4
COLLATE utf8mb4_unicode_ci DEFAULT NULL,
sn_doador_de_orgaos tinyint(1) DEFAULT NULL,
nome_anterior varchar(255) COLLATE utf8mb4_unicode_ci DEFAULT NULL,
endereco varchar(255) COLLATE utf8mb4_unicode_ci DEFAULT NULL,
nr_cep varchar(255) COLLATE utf8mb4_unicode_ci DEFAULT NULL,
nr_cpf varchar(255) COLLATE utf8mb4_unicode_ci NOT NULL,
telefone varchar(255) COLLATE utf8mb4_unicode_ci DEFAULT NULL,
telefone_celular varchar(255)
COLLATE utf8mb4_unicode_ci DEFAULT NULL,
email_institucional varchar(255)
COLLATE utf8mb4_unicode_ci DEFAULT NULL,
email_externo varchar(255)
COLLATE utf8mb4_unicode_ci DEFAULT NULL,
email_comunicados varchar(255)
COLLATE utf8mb4_unicode_ci DEFAULT NULL,
endereco_correspondencia varchar(255)
COLLATE utf8mb4_unicode_ci DEFAULT NULL,
nr_cep_correspondencia varchar(255) COLLATE utf8mb4_unicode_ci DEFAULT NULL,
created_at timestamp NULL DEFAULT NULL,
updated_at timestamp NULL DEFAULT NULL,
user_id bigint unsigned DEFAULT NULL,
PRIMARY KEY (id),
UNIQUE KEY tb_servidor_nr_cpf_unique (nr_cpf),
KEY tb_servidor_user_id_index (user_id),
CONSTRAINT tb_servidor_user_id_foreign FOREIGN KEY (user_id)
REFERENCES tb_users (id) ON DELETE CASCADE ON UPDATE CASCADE
) ENGINE=InnoDB AUTO_INCREMENT=7 DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci;
When I try to insert timestamp using year 1948, I get this error:
SQL Error [1292] [22001]: Data truncation: Incorrect datetime value: '1948-07-19 00:00:00' for column 'dt_nascimento' at row 1
But when I try to insert timestamp using year 1999, it inserts successfully using the following query:
insert into tb_cliente (
updated_at, created_at, st_servidor,
nm_nome, cd_matricula, dt_nascimento,
sexo, nacionalidade, uf_nascimento,
estado_civil, grau_de_instrucao, nr_cpf,
telefone, telefone_celular, email_institucional, email_externo)
values ('2022-10-25 00:00:00', '2022-10-25 00:00:00', '1',
'User name 1', '000000', '1999-07-19 00:00:00','M',
'1', 'AC', 'CASADO','ENSINO' , '000000', '000000', '000000',
'USER#UOL.COM.BR', 'USER#UOL.COM.BR' )
Mysql Server 8.0
https://dev.mysql.com/doc/refman/8.0/en/date-and-time-type-syntax.html says:
A timestamp. The range is '1970-01-01 00:00:01.000000' UTC to '2038-01-19 03:14:07.999999' UTC.
In other words, '1948-07-19 00:00:00' is more than 21 years earlier than the earliest date a TIMESTAMP can store.
You should use DATETIME if you want dates outside this range.
What is the Action keyword in MySQL? I'm converting a MySQL query into cockroachdb query and the MySQL query I'm seeing is:
CREATE TABLE tabDocType Action (
name varchar(140) COLLATE utf8mb4_unicode_ci NOT NULL,
creation TIMESTAMP(6) DEFAULT NULL,
modified TIMESTAMP(6) DEFAULT NULL,
modified_by varchar(140) COLLATE utf8mb4_unicode_ci DEFAULT NULL,
owner varchar(140) COLLATE utf8mb4_unicode_ci DEFAULT NULL,
docstatus INT NOT NULL DEFAULT 0,
parent varchar(140) COLLATE utf8mb4_unicode_ci DEFAULT NULL,
parentfield varchar(140) COLLATE utf8mb4_unicode_ci DEFAULT NULL,
parenttype varchar(140) COLLATE utf8mb4_unicode_ci DEFAULT NULL,
idx INT NOT NULL DEFAULT 0,
label varchar(140) COLLATE utf8mb4_unicode_ci DEFAULT NULL,
group varchar(140) COLLATE utf8mb4_unicode_ci DEFAULT NULL,
action_type varchar(140) COLLATE utf8mb4_unicode_ci DEFAULT NULL,
action text COLLATE utf8mb4_unicode_ci DEFAULT NULL,
PRIMARY KEY (name)
);
Also is Link also a keyword? It's not in the docs but I'm not quite sure what it's actually doing here:
CREATE TABLE `tabDocType Link` (
`name` varchar(140) COLLATE utf8mb4_unicode_ci NOT NULL,
`creation` TIMESTAMP(6) DEFAULT NULL,
`modified` TIMESTAMP(6) DEFAULT NULL,
`modified_by` varchar(140) COLLATE utf8mb4_unicode_ci DEFAULT NULL,
`owner` varchar(140) COLLATE utf8mb4_unicode_ci DEFAULT NULL,
`docstatus` INT NOT NULL DEFAULT 0,
`parent` varchar(140) COLLATE utf8mb4_unicode_ci DEFAULT NULL,
`parentfield` varchar(140) COLLATE utf8mb4_unicode_ci DEFAULT NULL,
`parenttype` varchar(140) COLLATE utf8mb4_unicode_ci DEFAULT NULL,
`idx` INT NOT NULL DEFAULT 0,
`group` varchar(140) COLLATE utf8mb4_unicode_ci DEFAULT NULL,
`link_doctype` varchar(140) COLLATE utf8mb4_unicode_ci DEFAULT NULL,
`link_fieldname` varchar(140) COLLATE utf8mb4_unicode_ci DEFAULT NULL,
PRIMARY KEY (`name`),
KEY `parent` (`parent`),
KEY `modified` (`modified`)
)
From the docs, https://dev.mysql.com/doc/refman/8.0/en/keywords.html, ACTION seems to be a keyword but it doesn't have details on it.
Perhaps they are not keywords at all but serves a different purpose? Thank you.
So looks like it's not a keyword, it's part of the table name. Spaces are allowed in a table name as long as the name is in the back tick.
https://www.tutorialspoint.com/can-we-create-a-table-with-a-space-in-name-in-mysql
It has nothing to do with reserved words, create table has no ACTION attribute
As you can read up on the home page about create table
Still group in your create table is a reserved word
CREATE TABLE tabDocType (
name varchar(140) COLLATE utf8mb4_unicode_ci NOT NULL,
creation TIMESTAMP(6) DEFAULT NULL,
modified TIMESTAMP(6) DEFAULT NULL,
modified_by varchar(140) COLLATE utf8mb4_unicode_ci DEFAULT NULL,
owner varchar(140) COLLATE utf8mb4_unicode_ci DEFAULT NULL,
docstatus INT NOT NULL DEFAULT 0,
parent varchar(140) COLLATE utf8mb4_unicode_ci DEFAULT NULL,
parentfield varchar(140) COLLATE utf8mb4_unicode_ci DEFAULT NULL,
parenttype varchar(140) COLLATE utf8mb4_unicode_ci DEFAULT NULL,
idx INT NOT NULL DEFAULT 0,
`label` varchar(140) COLLATE utf8mb4_unicode_ci DEFAULT NULL,
`group` varchar(140) COLLATE utf8mb4_unicode_ci DEFAULT NULL,
action_type varchar(140) COLLATE utf8mb4_unicode_ci DEFAULT NULL,
action text COLLATE utf8mb4_unicode_ci DEFAULT NULL,
PRIMARY KEY (name)
);
To answer your literal question, ACTION is a reserved word because one of the possible ON DELETE/ON UPDATE reference option values for a foreign key constraint is NO ACTION; see https://dev.mysql.com/doc/refman/8.0/en/create-table-foreign-keys.html
I am trying to import a csv file into a MySQL DB.
Said csv file is from a select * from the same table
LOAD DATA INFILE 'C:/ProgramData/MySQL/MySQL Server 5.7/Uploads/import.csv'
INTO TABLE fisc_hist_header
character set utf8
FIELDS TERMINATED BY ';'
LINES TERMINATED BY '\n'
(-- a whole lot of fields ...)
SET `CREATED_DATE`=STR_TO_DATE( #yourdatecolumn, '%d/%m/%Y %H:%i:%s' )
The file starts with
545752715002093599;3;1;503955117000124560;28/11/2019 14:38:51;0 -- all the other fields
What I get is
After searching here, all results are for
Incorrect integer value: ''
What is different and strange to me is that for me, it gets the value but does not inputs it as an integer ?
EDIT : As asked here is the create table statement:
CREATE TABLE `fisc_hist_header` (
`fiscal_idx` bigint(20) NOT NULL AUTO_INCREMENT,
`invc_sid` bigint(20) NOT NULL,
`sbs_no` int(11) NOT NULL,
`store_no` int(5) NOT NULL,
`workstation_id` bigint(20) NOT NULL,
`created_date` datetime NOT NULL,
`invc_type` int(11) NOT NULL,
`cashier_name` varchar(8) COLLATE utf8_bin DEFAULT NULL,
`associate_name` varchar(8) COLLATE utf8_bin DEFAULT NULL,
`store_code` varchar(5) COLLATE utf8_bin DEFAULT NULL,
`store_name` varchar(40) COLLATE utf8_bin DEFAULT NULL,
`store_address1` varchar(40) COLLATE utf8_bin DEFAULT NULL,
`store_address2` varchar(40) COLLATE utf8_bin DEFAULT NULL,
`store_address3` varchar(40) COLLATE utf8_bin DEFAULT NULL,
`store_address4` varchar(40) COLLATE utf8_bin DEFAULT NULL,
`store_address5` varchar(40) COLLATE utf8_bin DEFAULT NULL,
`store_address6` varchar(40) COLLATE utf8_bin DEFAULT NULL,
`store_zip` varchar(10) COLLATE utf8_bin DEFAULT NULL,
`customer_sid` bigint(20) DEFAULT NULL,
`customer_title` varchar(15) COLLATE utf8_bin DEFAULT NULL,
`customer_last_name` varchar(30) COLLATE utf8_bin DEFAULT NULL,
`customer_first_name` varchar(30) COLLATE utf8_bin DEFAULT NULL,
`customer_address1` varchar(40) COLLATE utf8_bin DEFAULT NULL,
`customer_address2` varchar(40) COLLATE utf8_bin DEFAULT NULL,
`customer_address3` varchar(40) COLLATE utf8_bin DEFAULT NULL,
`customer_address4` varchar(40) COLLATE utf8_bin DEFAULT NULL,
`customer_zip_code` varchar(10) COLLATE utf8_bin DEFAULT NULL,
`customer_country` varchar(35) COLLATE utf8_bin DEFAULT NULL,
`customer_phone` varchar(30) COLLATE utf8_bin DEFAULT NULL,
`customer_email` varchar(60) COLLATE utf8_bin DEFAULT NULL,
`item_count` int(5) NOT NULL,
`grand_total_receipt` decimal(10,0) NOT NULL,
`prism_version` varchar(19) COLLATE utf8_bin DEFAULT NULL,
`plugin_version` varchar(19) COLLATE utf8_bin DEFAULT NULL,
`flag_first_record` char(1) COLLATE utf8_bin DEFAULT NULL,
`signature_key` varchar(500) COLLATE utf8_bin DEFAULT NULL,
`signature` varchar(500) COLLATE utf8_bin DEFAULT NULL,
`signature_previous` varchar(500) COLLATE utf8_bin DEFAULT NULL,
`signature_short` varchar(10) COLLATE utf8_bin DEFAULT NULL,
PRIMARY KEY (`fiscal_idx`),
UNIQUE KEY `un_fhh` (`invc_sid`)
) ENGINE=InnoDB AUTO_INCREMENT=364 DEFAULT CHARSET=utf8 COLLATE=utf8_bin;
As per the documentation for integer values, the maximum value for an unsigned integer which can be held in an INT column is 4294967295. For a standard signed integer it's 2147483647.
Your number 545752715002093599 is larger than both of these. You'll need to declare the target column as BIGINT in your fisc_hist_header table - the max value for a signed integer in this column type is 9223372036854775807.
These restrictions exist because of the number of bytes used by MySQL to store each value in the database. If you know you won't need to store values above a certain limit, you can use a smaller integer type in order to save disk space and memory.
So, I have two tables filled with for the most part very similar data, for example, a row in each table may have the same first name, last name, and address, but have a different phone number or email address based on the most recently available data which was updated in a separate excel worksheet (out of my hands, my job is just to merge this data into our latest database which they plan to use from here on out, not the excel sheet). I just need a good way to merge these tables with the same column names without doing it manually (about 24,000+) records.
Here is the Create Table Syntax for both tables:
CREATE TABLE `UsersUpdated` (
`FULLNME` longtext,
`LSTNME` varchar(23) CHARACTER SET utf8 DEFAULT NULL,
`FSTNME` varchar(255) CHARACTER SET utf8 DEFAULT NULL,
`MID` varchar(255) CHARACTER SET utf8 DEFAULT NULL,
`SUFF` varchar(50) CHARACTER SET utf8 DEFAULT NULL,
`STAT` varchar(2) CHARACTER SET utf8 DEFAULT NULL,
`PTY` varchar(3) CHARACTER SET utf8 DEFAULT NULL,
`PH` bigint(20) DEFAULT NULL,
`ALTPH` bigint(20) DEFAULT NULL,
`DOB` datetime DEFAULT NULL,
`REGDTE` datetime DEFAULT NULL,
`ADDR` text,
`ST` int(11) DEFAULT NULL,
`STNME` varchar(19) CHARACTER SET utf8 DEFAULT NULL,
`APT` varchar(7) CHARACTER SET utf8 DEFAULT NULL,
`TWN` varchar(6) CHARACTER SET utf8 DEFAULT NULL,
`ZIP` int(11) DEFAULT NULL,
`W` int(11) DEFAULT NULL,
`d` int(11) DEFAULT NULL,
`G17` varchar(1) CHARACTER SET utf8 DEFAULT NULL,
`P17` varchar(1) CHARACTER SET utf8 DEFAULT NULL,
`G16` varchar(1) CHARACTER SET utf8 DEFAULT NULL,
`P16` varchar(1) CHARACTER SET utf8 DEFAULT NULL,
`G15` varchar(1) CHARACTER SET utf8 DEFAULT NULL,
`P15` varchar(1) CHARACTER SET utf8 DEFAULT NULL,
`G14` varchar(1) CHARACTER SET utf8 DEFAULT NULL,
`P14` varchar(1) CHARACTER SET utf8 DEFAULT NULL,
`SIGN` tinyint(1) NOT NULL DEFAULT '0',
`SUPP` tinyint(1) NOT NULL DEFAULT '0',
`NOTES` longtext,
`LTR` tinyint(1) DEFAULT NULL,
`REGISTERED` tinyint(1) DEFAULT NULL
) ENGINE=InnoDB DEFAULT CHARSET=latin1;
CREATE TABLE `Users` (
`FULLNME` longtext,
`LSTNME` varchar(23) CHARACTER SET utf8 DEFAULT NULL,
`FSTNME` varchar(255) CHARACTER SET utf8 DEFAULT NULL,
`MID` varchar(255) CHARACTER SET utf8 DEFAULT NULL,
`SUFF` varchar(50) CHARACTER SET utf8 DEFAULT NULL,
`STAT` varchar(2) CHARACTER SET utf8 DEFAULT NULL,
`PTY` varchar(3) CHARACTER SET utf8 DEFAULT NULL,
`PH` bigint(20) DEFAULT NULL,
`ALTPH` bigint(20) DEFAULT NULL,
`DOB` datetime DEFAULT NULL,
`REGDTE` datetime DEFAULT NULL,
`ADDR` text,
`ST` int(11) DEFAULT NULL,
`STNME` varchar(19) CHARACTER SET utf8 DEFAULT NULL,
`APT` varchar(7) CHARACTER SET utf8 DEFAULT NULL,
`TWN` varchar(6) CHARACTER SET utf8 DEFAULT NULL,
`ZIP` int(11) DEFAULT NULL,
`W` int(11) DEFAULT NULL,
`d` int(11) DEFAULT NULL,
`G17` varchar(1) CHARACTER SET utf8 DEFAULT NULL,
`P17` varchar(1) CHARACTER SET utf8 DEFAULT NULL,
`G16` varchar(1) CHARACTER SET utf8 DEFAULT NULL,
`P16` varchar(1) CHARACTER SET utf8 DEFAULT NULL,
`G15` varchar(1) CHARACTER SET utf8 DEFAULT NULL,
`P15` varchar(1) CHARACTER SET utf8 DEFAULT NULL,
`G14` varchar(1) CHARACTER SET utf8 DEFAULT NULL,
`P14` varchar(1) CHARACTER SET utf8 DEFAULT NULL,
`SIGN` tinyint(1) NOT NULL DEFAULT '0',
`SUPP` tinyint(1) NOT NULL DEFAULT '0',
`NOTES` longtext,
`LTR` tinyint(1) DEFAULT NULL,
`REGISTERED` tinyint(1) DEFAULT NULL
) ENGINE=InnoDB DEFAULT CHARSET=latin1;
As you can see, they are basically the same exact tables, I just need to merge them correctly.
Perhaps this is helpful.
update Users
set ADDR = (
select ADDR from UsersUpdated uu
where uu.FULLNME = Users.FULLNME and uu.DOB = Users.DOB
), STNME = (
select STNME from UsersUpdated uu
where uu.FULLNME = Users.FULLNME and uu.DOB = Users.DOB
) ...
;
You can add all the columns to a single update. Depending on the size of the database it might just be as easy to do them individually.
Many platforms allow for a from clause with update that permits a join and a shorter query but it can be problematic. This way you will get errors if any of the subqueries don't return just a single value.
I would modify the Users table to add a UNIQUE key on the fields that should be the same (presumably FSTNME, LSTNME and ADDR from your description but perhaps you might use some other columns e.g. DOB as suggested by #shawnt00), then INSERT the data from UsersUpdated into Users using an ON DUPLICATE KEY UPDATE clause to copy updated data into Users where the user already exists in that table. This query will also work when there are users in UsersUpdated who are not already in Users. So,
ALTER TABLE Users ADD UNIQUE KEY NameAddr (FSTNME, LSTNME, ADDR);
INSERT INTO Users
SELECT * FROM UsersUpdated
ON DUPLICATE KEY UPDATE
FULLNME=VALUES(FULLNME),
LSTNME=VALUES(LSTNME),
FSTNME=VALUKES(FSTNME),
...
LTR=VALUES(LTR),
REGISTERED=VALUES(REGISTERED);
I have two table the structure are given bellow, those table have lots of data but can't change the table stucture
Table "postsale"
CREATE TABLE IF NOT EXISTS `postsale` (
`id` bigint(20) NOT NULL AUTO_INCREMENT,
`group_id` varchar(255) CHARACTER SET latin1 COLLATE latin1_general_ci NOT NULL,
`season` varchar(25) CHARACTER SET latin1 COLLATE latin1_general_ci NOT NULL,
`sale_no` int(5) NOT NULL,
`auction_date` varchar(255) CHARACTER SET latin1 COLLATE latin1_general_ci NOT NULL,
`season_time` varchar(255) CHARACTER SET latin1 COLLATE latin1_general_ci NOT NULL,
`lot_no` varchar(255) CHARACTER SET latin1 COLLATE latin1_general_ci NOT NULL,
`invoice_no` varchar(255) CHARACTER SET latin1 COLLATE latin1_general_ci NOT NULL,
`origin` varchar(255) CHARACTER SET latin1 COLLATE latin1_general_ci NOT NULL,
`tea_type` varchar(255) CHARACTER SET latin1 COLLATE latin1_general_ci NOT NULL,
`sub_tea_type` varchar(255) CHARACTER SET latin1 COLLATE latin1_general_ci NOT NULL,
`category` varchar(255) CHARACTER SET latin1 COLLATE latin1_general_ci NOT NULL,
`mark` varchar(255) CHARACTER SET latin1 COLLATE latin1_general_ci NOT NULL,
`grade` varchar(255) CHARACTER SET latin1 COLLATE latin1_general_ci NOT NULL,
`no_of_packages` varchar(255) CHARACTER SET latin1 COLLATE latin1_general_ci NOT NULL,
`gross_wt` varchar(255) CHARACTER SET latin1 COLLATE latin1_general_ci NOT NULL,
`net_wt` varchar(255) CHARACTER SET latin1 COLLATE latin1_general_ci NOT NULL,
`auction_valuation` varchar(255) CHARACTER SET latin1 COLLATE latin1_general_ci NOT NULL,
`lsp_sp` varchar(255) CHARACTER SET latin1 COLLATE latin1_general_ci NOT NULL,
`package_type` varchar(255) CHARACTER SET latin1 COLLATE latin1_general_ci NOT NULL,
`package_no` varchar(255) CHARACTER SET latin1 COLLATE latin1_general_ci NOT NULL,
`quantity` varchar(255) CHARACTER SET latin1 COLLATE latin1_general_ci NOT NULL,
`auctioneer` varchar(255) CHARACTER SET latin1 COLLATE latin1_general_ci NOT NULL,
`auction_price` varchar(255) CHARACTER SET latin1 COLLATE latin1_general_ci NOT NULL,
`buyer` text CHARACTER SET latin1 COLLATE latin1_general_ci NOT NULL,
`area` varchar(255) CHARACTER SET latin1 COLLATE latin1_general_ci NOT NULL,
`broker_code` varchar(255) CHARACTER SET latin1 COLLATE latin1_general_ci NOT NULL,
`csv` varchar(255) CHARACTER SET latin1 COLLATE latin1_general_ci NOT NULL,
`session` int(11) NOT NULL,
PRIMARY KEY (`id`)
) ENGINE=InnoDB DEFAULT CHARSET=latin1 AUTO_INCREMENT=29623 ;
and Table finalesale
CREATE TABLE IF NOT EXISTS `finalsale` (
`id` bigint(20) NOT NULL AUTO_INCREMENT,
`group_id` varchar(255) CHARACTER SET latin1 COLLATE latin1_general_ci NOT NULL,
`season` varchar(25) CHARACTER SET latin1 COLLATE latin1_general_ci NOT NULL,
`sale_no` int(5) NOT NULL,
`auction_date` varchar(255) CHARACTER SET latin1 COLLATE latin1_general_ci NOT NULL,
`season_time` varchar(255) CHARACTER SET latin1 COLLATE latin1_general_ci NOT NULL,
`lot_no` varchar(255) CHARACTER SET latin1 COLLATE latin1_general_ci NOT NULL,
`invoice_no` varchar(255) CHARACTER SET latin1 COLLATE latin1_general_ci NOT NULL,
`origin` varchar(255) CHARACTER SET latin1 COLLATE latin1_general_ci NOT NULL,
`tea_type` varchar(255) CHARACTER SET latin1 COLLATE latin1_general_ci NOT NULL,
`sub_tea_type` varchar(255) CHARACTER SET latin1 COLLATE latin1_general_ci NOT NULL,
`category` varchar(255) CHARACTER SET latin1 COLLATE latin1_general_ci NOT NULL,
`mark` varchar(255) CHARACTER SET latin1 COLLATE latin1_general_ci NOT NULL,
`grade` varchar(255) CHARACTER SET latin1 COLLATE latin1_general_ci NOT NULL,
`no_of_packages` varchar(255) CHARACTER SET latin1 COLLATE latin1_general_ci NOT NULL,
`gross_wt` varchar(255) CHARACTER SET latin1 COLLATE latin1_general_ci NOT NULL,
`net_wt` varchar(255) CHARACTER SET latin1 COLLATE latin1_general_ci NOT NULL,
`auction_valuation` varchar(255) CHARACTER SET latin1 COLLATE latin1_general_ci NOT NULL,
`lsp_sp` varchar(255) CHARACTER SET latin1 COLLATE latin1_general_ci NOT NULL,
`package_type` varchar(255) CHARACTER SET latin1 COLLATE latin1_general_ci NOT NULL,
`package_no` varchar(255) CHARACTER SET latin1 COLLATE latin1_general_ci NOT NULL,
`quantity` varchar(255) CHARACTER SET latin1 COLLATE latin1_general_ci NOT NULL,
`auctioneer` varchar(255) CHARACTER SET latin1 COLLATE latin1_general_ci NOT NULL,
`auction_price` varchar(255) CHARACTER SET latin1 COLLATE latin1_general_ci NOT NULL,
`buyer` text CHARACTER SET latin1 COLLATE latin1_general_ci NOT NULL,
`area` varchar(255) CHARACTER SET latin1 COLLATE latin1_general_ci NOT NULL,
`broker_code` varchar(255) CHARACTER SET latin1 COLLATE latin1_general_ci NOT NULL,
`csv` varchar(255) CHARACTER SET latin1 COLLATE latin1_general_ci NOT NULL,
`session` int(11) NOT NULL,
PRIMARY KEY (`id`)
) ENGINE=InnoDB DEFAULT CHARSET=latin1 AUTO_INCREMENT=41365 ;
based on the above two table, the execution time of this query is huge so, I need to optimized the following query
UPDATE `finalsale`,`postsale`
SET
`finalsale`.`auction_price`=`postsale`.`auction_price`,
`finalsale`.`csv`=`postsale`.`csv`,
`finalsale`.`session`=`postsale`.`session`
WHERE `finalsale`.`lot_no`=`postsale`.`lot_no`
AND `finalsale`.`group_id`=`postsale`.`group_id`
AND `finalsale`.`group_id`='201217CLGuwahatiJT'
Please help
You will get a serious performance improvement from choosing the correct data types in your table definition.
There are several columns which appear to contain only numerical information, which is better stored in some int column.
Furthermore there are columns for some dates like auction_date that should be transformed to a date - datatype.
Also think about the length of your varchar columns. Most of the time you wont need the 255 characters there.
Have a close look at the Mysql Docu for datatypes.
If you have performance problems afterwards you may think about creating indexes on some columns. But I strongly recommend thinking about this just AFTER you reworked your table definitions!
Assuming it is not possible for you to improve on the schema (which as has been mentioned in the OP comments is rather... bloated), you probably would get a significant performance boost for this specific query by adding an index on lot_no and group_id on each of the tables.
Since you're using InnoDB, you might consider using a foreign key constraint (this also covers indexes), although it depends on the semantics & lifecycle of the tables, which I don't know about.
Add the following indexes:
Single column:
finalsale: (`group_id`)
Multi-column:
postsale: (`group_id, lot_no`)
These indexes allow MySQL to first limit the result set by applying the constant in the WHERE clause to the finalsale table, and then, with that minimal result set, perform a join to the postsale table, fully utilizing the index.