We have a MySQL table hosted on an Amazon AWS server, which is some times very very slow while querying. I'm considering restructuring the table with better indexing and columns data types.
Here is the create table structure :
SHOW CREATE TABLE export_users;
+------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Table | Create Table |
+------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| exportusers | CREATE TABLE `exportusers` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`UAccountID` varchar(50) CHARACTER SET utf8 NOT NULL,
`ApplyURL` text,
`CityName` varchar(64) CHARACTER SET utf8 DEFAULT NULL,
`ClassList` text,
`UCompanyID` varchar(20) DEFAULT NULL,
`UContactCompany` varchar(64) CHARACTER SET utf8 DEFAULT NULL,
`UContactEmail` varchar(512) CHARACTER SET utf8 DEFAULT NULL,
`UContactFax` varchar(32) CHARACTER SET utf8 DEFAULT NULL,
`UContactName` varchar(64) CHARACTER SET utf8 DEFAULT NULL,
`UContactPhone` varchar(100) CHARACTER SET utf8 DEFAULT NULL,
`CountryName` char(2) CHARACTER SET utf8 NOT NULL,
`DateCreated` datetime NOT NULL,
`DateModified` datetime NOT NULL,
`DateSysCreated` datetime NOT NULL,
`DateSysModified` datetime NOT NULL,
`DegreeCode` varchar(200) DEFAULT NULL,
`DegreeCodeDecoded` varchar(512) CHARACTER SET utf8 DEFAULT NULL,
`DisplayCity` varchar(64) CHARACTER SET utf8 DEFAULT NULL,
`DisplayuserID` varchar(32) CHARACTER SET utf8 DEFAULT NULL,
`ExperienceCode` varchar(200) DEFAULT NULL,
`ExperienceCodeDecoded` text,
`ExternalKey` varchar(32) CHARACTER SET utf8 DEFAULT NULL,
`GeoUSZip5` varchar(5) DEFAULT NULL,
`HostSite` char(2) CHARACTER SET utf8 NOT NULL,
`IndustryCode` varchar(128) CHARACTER SET utf8 DEFAULT NULL,
`IndustryCodeDecoded` text,
`IsBOFuser` tinyint(1) DEFAULT NULL,
`IsDiversityuser` tinyint(1) DEFAULT NULL,
`userID` varchar(20) NOT NULL,
`userDesc` text,
`userFunctionCode` text,
`userFunctionCodeDecoded` text,
`userReq` text,
`userSkinDID` varchar(20) DEFAULT NULL,
`userTitle` varchar(128) CHARACTER SET utf8 DEFAULT NULL,
`userType` varchar(512) CHARACTER SET utf8 DEFAULT NULL,
`userTypeDesc` text,
`userTypeCodeDecoded` text,
`Latitude` decimal(10,5) DEFAULT NULL,
`Longitude` decimal(10,5) DEFAULT NULL,
`Location` varchar(70) CHARACTER SET utf8 DEFAULT NULL,
`ManagementCode` varchar(20) CHARACTER SET utf8 DEFAULT NULL,
`MaximumExp` int(11) DEFAULT NULL,
`MinimunExp` int(11) DEFAULT NULL,
`Onet` varchar(10) DEFAULT NULL,
`OnetTitle` text,
`BeginDate` datetime NOT NULL,
`EndDate` datetime NOT NULL,
`PayBaseH` decimal(10,2) DEFAULT NULL,
`PayBaseL` decimal(10,2) DEFAULT NULL,
`PayBonus` decimal(10,2) DEFAULT NULL,
`PayComm` decimal(10,2) DEFAULT NULL,
`PayOther` varchar(64) CHARACTER SET utf8 DEFAULT NULL,
`PayPer` varchar(8) DEFAULT NULL,
`PayType` char(3) DEFAULT NULL,
`PostalCode` varchar(10) DEFAULT NULL,
`PostingPath` varchar(20) NOT NULL,
`Relocate` tinyint(1) DEFAULT NULL,
`RelocateOptions` varchar(5) DEFAULT NULL,
`ScreenerID` varchar(20) DEFAULT NULL,
`SiteID` varchar(1024) CHARACTER SET utf8 DEFAULT NULL,
`SliceList` text,
`StateName` char(30) CHARACTER SET utf8 DEFAULT NULL,
`Status` varchar(50) NOT NULL,
`TextPay` varchar(40) CHARACTER SET utf8 DEFAULT NULL,
`TotalPay` decimal(10,2) DEFAULT NULL,
`TownName` varchar(100) CHARACTER SET utf8 DEFAULT NULL,
`TravelCode` varchar(200) DEFAULT NULL,
`TravelCodeDecoded` text,
`UpgradeList` varchar(64) DEFAULT NULL,
PRIMARY KEY (`id`),
KEY `index_jobs_on_UAccountID` (`UAccountID`),
KEY `index_jobs_on_CountryName` (`CountryName`),
KEY `index_jobs_on_DateCreated` (`DateCreated`),
KEY `index_jobs_on_DateModified` (`DateModified`),
KEY `index_jobs_on_DateSysCreated` (`DateSysCreated`),
KEY `index_jobs_on_DateSysModified` (`DateSysModified`),
KEY `index_jobs_on_HostSite` (`HostSite`),
KEY `index_jobs_on_userID` (`userID`),
KEY `index_jobs_on_BeginDate` (`BeginDate`),
KEY `index_jobs_on_EndDate` (`EndDate`),
KEY `index_jobs_on_PostingPath` (`PostingPath`),
KEY `index_jobs_on_Status` (`Status`)
) ENGINE=InnoDB AUTO_INCREMENT=7907436 DEFAULT CHARSET=utf8mb4 |
+------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
1 row in set (0.27 sec)
Consider Normalization of your schema. Breaking up that one very large entity into much smaller related entities will have the greatest impact on your querying speed.
Normalization involves decomposing a table into less redundant (and smaller) tables but without losing information; defining foreign keys in the old table referencing the primary keys of the new ones. The objective is to isolate data so that additions, deletions, and modifications of an attribute can be made in just one table and then propagated through the rest of the database using the defined foreign keys.
Related
Today I've created a sql backup of a table using SQLyog Community Edition. Then I tried to run the created query against an up to date MySQL 8 managed database on Digital Ocean.
The query:
CREATE TABLE tblUsers (
id bigint unsigned NOT NULL AUTO_INCREMENT,
subscriberId bigint unsigned DEFAULT NULL,
user varchar(16) CHARACTER SET utf8mb4 COLLATE utf8mb4_0900_ai_ci DEFAULT NULL,
password varchar(64) DEFAULT NULL,
created timestamp NULL DEFAULT NULL,
modified timestamp NULL DEFAULT NULL,
status varchar(16) DEFAULT NULL,
privilege varchar(32) CHARACTER SET utf8mb4 COLLATE utf8mb4_0900_ai_ci DEFAULT NULL,
firstName varchar(64) DEFAULT NULL,
lastName varchar(64) DEFAULT NULL,
address1 varchar(256) CHARACTER SET utf8mb4 COLLATE utf8mb4_0900_ai_ci DEFAULT NULL,
city varchar(32) DEFAULT NULL,
state char(2) DEFAULT NULL,
postalCode char(6) DEFAULT NULL,
phone varchar(64) CHARACTER SET utf8mb4 COLLATE utf8mb4_0900_ai_ci DEFAULT NULL,
fax varchar(64) CHARACTER SET utf8mb4 COLLATE utf8mb4_0900_ai_ci DEFAULT NULL,
email varchar(256) CHARACTER SET utf8mb4 COLLATE utf8mb4_0900_ai_ci DEFAULT NULL,
notes varchar(1024) CHARACTER SET utf8mb4 COLLATE utf8mb4_0900_ai_ci DEFAULT NULL,
payRate varchar(16) DEFAULT NULL,
pay decimal(8,2) unsigned DEFAULT NULL,
PRIMARY KEY (id)
) ENGINE=InnoDB AUTO_INCREMENT=1 DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_0900_ai_ci;
The warning:
Warning Code : 1681
UNSIGNED for decimal and floating point data types is deprecated and support
for it will be removed in a future release.
The research:
https://dev.mysql.com/doc/refman/8.0/en/numeric-type-attributes.html
As of MySQL 8.0.17, AUTO_INCREMENT support is deprecated for FLOAT and
DOUBLE columns; you should expect it to be removed in a future version
of MySQL. Consider removing the AUTO_INCREMENT attribute from such
columns, or convert them to an integer type.
The question:
I'm not certain how to interpret what I'm reading. Is it really telling me to remove my primary key auto increments? Or is it telling me the pay field should no longer be define the way you see above? Or that AUTO_INCREMENT cannot exist together with unsigned?
This warning is not about your auto-increment column id. It's about the decimal column pay.
Change this:
pay decimal(8,2) unsigned DEFAULT NULL,
To this:
pay decimal(8,2) DEFAULT NULL,
The related excerpt from the manual page you linked to is the following. It tells you what to do if you want to restrict the column to nonnegative values:
As of MySQL 8.0.17, the UNSIGNED attribute is deprecated for columns of type FLOAT, DOUBLE, and DECIMAL (and any synonyms) and you should expect support for it to be removed in a future version of MySQL. Consider using a simple CHECK constraint instead for such columns.
In your case it would look like this:
pay decimal(8,2) DEFAULT NULL,
CHECK (pay >= 0),
Updated example:
id field remains unchanged
subscriberId field loses UNSIGNED and gains CHECK
pay field loses UNSIGNED and gains CHECK
CREATE TABLE tblUsers (
`id` BIGINT UNSIGNED NOT NULL AUTO_INCREMENT,
`subscriberId` BIGINT DEFAULT NULL CHECK (subscriberId >= 0),
`user` VARCHAR(16) CHARACTER SET utf8mb4 COLLATE utf8mb4_0900_ai_ci DEFAULT NULL,
`password` VARCHAR(64) DEFAULT NULL,
`created` TIMESTAMP NULL DEFAULT NULL,
`modified` TIMESTAMP NULL DEFAULT NULL,
`status` VARCHAR(16) DEFAULT NULL,
`privilege` VARCHAR(32) CHARACTER SET utf8mb4 COLLATE utf8mb4_0900_ai_ci DEFAULT NULL,
`firstName` VARCHAR(64) DEFAULT NULL,
`lastName` VARCHAR(64) DEFAULT NULL,
`address1` VARCHAR(256) CHARACTER SET utf8mb4 COLLATE utf8mb4_0900_ai_ci DEFAULT NULL,
`city` VARCHAR(32) DEFAULT NULL,
`state` CHAR(2) DEFAULT NULL,
`postalCode` CHAR(6) DEFAULT NULL,
`phone` VARCHAR(64) CHARACTER SET utf8mb4 COLLATE utf8mb4_0900_ai_ci DEFAULT NULL,
`fax` VARCHAR(64) CHARACTER SET utf8mb4 COLLATE utf8mb4_0900_ai_ci DEFAULT NULL,
`email` VARCHAR(256) CHARACTER SET utf8mb4 COLLATE utf8mb4_0900_ai_ci DEFAULT NULL,
`notes` VARCHAR(1024) CHARACTER SET utf8mb4 COLLATE utf8mb4_0900_ai_ci DEFAULT NULL,
`payRate` VARCHAR(16) DEFAULT NULL,
`pay` DECIMAL(8,2) DEFAULT NULL CHECK (pay >= 0),
PRIMARY KEY (`id`)
) ENGINE=INNODB AUTO_INCREMENT=465000 DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_0900_ai_ci;
I tried to query by join 2 tables giao_vien and lop_mon.
The following is my tables structure:
CREATE TABLE `lop_mon` (
`id` bigint(20) NOT NULL AUTO_INCREMENT,
`key_index` varchar(250) CHARACTER SET utf8 COLLATE utf8_general_ci NOT NULL,
`khoi_key_index` varchar(250) DEFAULT NULL,
`lop_key_index` varchar(250) DEFAULT NULL,
`gv_key_index` varchar(250) DEFAULT NULL,
`mon_key_index` varchar(250) DEFAULT NULL,
`ma_so` varchar(50) DEFAULT NULL,
`ma_phong` varchar(50) CHARACTER SET utf8 COLLATE utf8_general_ci DEFAULT NULL,
`nam_hoc` int(11) NOT NULL,
`ma_truong` varchar(50) CHARACTER SET utf8 COLLATE utf8_general_ci DEFAULT NULL,
`ma_khoi` varchar(50) CHARACTER SET utf8 COLLATE utf8_general_ci NOT NULL,
`ma_lop` varchar(50) NOT NULL,
`ma_giao_vien` varchar(50) NOT NULL,
`ma_mon_hoc` varchar(50) DEFAULT NULL,
`hoc_ky` tinyint(4) NOT NULL,
`guid` varchar(250) DEFAULT NULL,
`processid_monhoc` int(11) DEFAULT NULL,
`processid_giaovien` int(11) DEFAULT NULL,
`processid_canbo` int(11) DEFAULT NULL,
`source_data` varchar(50) DEFAULT NULL,
`ten_gv_chat_ph` varchar(255) CHARACTER SET utf8 COLLATE utf8_general_ci DEFAULT NULL,
`ma_giao_vien_bgd` varchar(50) CHARACTER SET utf8 COLLATE utf8_general_ci DEFAULT NULL,
`ma_lop_bgd` varchar(50) CHARACTER SET utf8 COLLATE utf8_general_ci DEFAULT NULL,
`ma_mon_hoc_bgd` varchar(50) CHARACTER SET utf8 COLLATE utf8_general_ci DEFAULT NULL,
`gv_guid` char(36) DEFAULT NULL,
`truong_key_index` varchar(255) DEFAULT NULL,
`create_date` datetime DEFAULT NULL,
`updatedAt` datetime DEFAULT NULL,
PRIMARY KEY (`id`) USING BTREE,
UNIQUE KEY `key_UNIQUE` (`key_index`),
UNIQUE KEY `unique_lop_mon_gv_keyindex_hoc_ky` (`lop_key_index`,`gv_key_index`,`mon_key_index`,`hoc_ky`) USING BTREE,
KEY `lopmon_keyindex` (`mon_key_index`),
KEY `lopmon_idx_mamonhoc` (`ma_mon_hoc`),
KEY `idx_gv_key_index` (`gv_key_index`),
KEY `lopmon_idx_lop_key_index` (`lop_key_index`),
KEY `lopmon_idx_key_index` (`khoi_key_index`),
KEY `lopmon_idx_ma_truong` (`ma_truong`),
KEY `lopmon_idx_nam_hoc` (`nam_hoc`),
KEY `lopmon_idx_ma_khoi` (`ma_khoi`),
KEY `lopmon_idx_hoc_ky` (`hoc_ky`),
KEY `lopmon_idx_key_lop_mon` (`lop_key_index`,`mon_key_index`),
KEY `lopmon_idx_key_lop_gv` (`lop_key_index`,`gv_key_index`),
KEY `lopmon_idx_key_mon_gv` (`mon_key_index`,`gv_key_index`),
KEY `lopmon_idx_key_lop_mon_gv` (`lop_key_index`,`mon_key_index`,`gv_key_index`),
KEY `lopmon_idx_key_truong_lop_mon_gv` (`ma_truong`,`lop_key_index`,`mon_key_index`,`gv_key_index`),
KEY `lop_mon__index_gv_guid` (`gv_guid`),
KEY `lopmon_idx_truong_key_index` (`truong_key_index`)
) ENGINE=MyISAM AUTO_INCREMENT=23142297 DEFAULT CHARSET=latin1
My Query:
select * from giao_vien inner join lop_mon on lop_mon.gv_key_index = giao_vien.key_index
where giao_vien.key_index = 'shcm_2019_01645344018'
When I explain it:
MySQL does not recognize the key:
KEY `idx_gv_key_index` (`gv_key_index`)
I check another query:
SELECT * FROM enetviet.lop_mon where gv_key_index = 'shcm_2019_01645344018';
The key idx_gv_key_index is still work. I don't know what is the problem.
I am trying to import a csv file into a MySQL DB.
Said csv file is from a select * from the same table
LOAD DATA INFILE 'C:/ProgramData/MySQL/MySQL Server 5.7/Uploads/import.csv'
INTO TABLE fisc_hist_header
character set utf8
FIELDS TERMINATED BY ';'
LINES TERMINATED BY '\n'
(-- a whole lot of fields ...)
SET `CREATED_DATE`=STR_TO_DATE( #yourdatecolumn, '%d/%m/%Y %H:%i:%s' )
The file starts with
545752715002093599;3;1;503955117000124560;28/11/2019 14:38:51;0 -- all the other fields
What I get is
After searching here, all results are for
Incorrect integer value: ''
What is different and strange to me is that for me, it gets the value but does not inputs it as an integer ?
EDIT : As asked here is the create table statement:
CREATE TABLE `fisc_hist_header` (
`fiscal_idx` bigint(20) NOT NULL AUTO_INCREMENT,
`invc_sid` bigint(20) NOT NULL,
`sbs_no` int(11) NOT NULL,
`store_no` int(5) NOT NULL,
`workstation_id` bigint(20) NOT NULL,
`created_date` datetime NOT NULL,
`invc_type` int(11) NOT NULL,
`cashier_name` varchar(8) COLLATE utf8_bin DEFAULT NULL,
`associate_name` varchar(8) COLLATE utf8_bin DEFAULT NULL,
`store_code` varchar(5) COLLATE utf8_bin DEFAULT NULL,
`store_name` varchar(40) COLLATE utf8_bin DEFAULT NULL,
`store_address1` varchar(40) COLLATE utf8_bin DEFAULT NULL,
`store_address2` varchar(40) COLLATE utf8_bin DEFAULT NULL,
`store_address3` varchar(40) COLLATE utf8_bin DEFAULT NULL,
`store_address4` varchar(40) COLLATE utf8_bin DEFAULT NULL,
`store_address5` varchar(40) COLLATE utf8_bin DEFAULT NULL,
`store_address6` varchar(40) COLLATE utf8_bin DEFAULT NULL,
`store_zip` varchar(10) COLLATE utf8_bin DEFAULT NULL,
`customer_sid` bigint(20) DEFAULT NULL,
`customer_title` varchar(15) COLLATE utf8_bin DEFAULT NULL,
`customer_last_name` varchar(30) COLLATE utf8_bin DEFAULT NULL,
`customer_first_name` varchar(30) COLLATE utf8_bin DEFAULT NULL,
`customer_address1` varchar(40) COLLATE utf8_bin DEFAULT NULL,
`customer_address2` varchar(40) COLLATE utf8_bin DEFAULT NULL,
`customer_address3` varchar(40) COLLATE utf8_bin DEFAULT NULL,
`customer_address4` varchar(40) COLLATE utf8_bin DEFAULT NULL,
`customer_zip_code` varchar(10) COLLATE utf8_bin DEFAULT NULL,
`customer_country` varchar(35) COLLATE utf8_bin DEFAULT NULL,
`customer_phone` varchar(30) COLLATE utf8_bin DEFAULT NULL,
`customer_email` varchar(60) COLLATE utf8_bin DEFAULT NULL,
`item_count` int(5) NOT NULL,
`grand_total_receipt` decimal(10,0) NOT NULL,
`prism_version` varchar(19) COLLATE utf8_bin DEFAULT NULL,
`plugin_version` varchar(19) COLLATE utf8_bin DEFAULT NULL,
`flag_first_record` char(1) COLLATE utf8_bin DEFAULT NULL,
`signature_key` varchar(500) COLLATE utf8_bin DEFAULT NULL,
`signature` varchar(500) COLLATE utf8_bin DEFAULT NULL,
`signature_previous` varchar(500) COLLATE utf8_bin DEFAULT NULL,
`signature_short` varchar(10) COLLATE utf8_bin DEFAULT NULL,
PRIMARY KEY (`fiscal_idx`),
UNIQUE KEY `un_fhh` (`invc_sid`)
) ENGINE=InnoDB AUTO_INCREMENT=364 DEFAULT CHARSET=utf8 COLLATE=utf8_bin;
As per the documentation for integer values, the maximum value for an unsigned integer which can be held in an INT column is 4294967295. For a standard signed integer it's 2147483647.
Your number 545752715002093599 is larger than both of these. You'll need to declare the target column as BIGINT in your fisc_hist_header table - the max value for a signed integer in this column type is 9223372036854775807.
These restrictions exist because of the number of bytes used by MySQL to store each value in the database. If you know you won't need to store values above a certain limit, you can use a smaller integer type in order to save disk space and memory.
So, I have two tables filled with for the most part very similar data, for example, a row in each table may have the same first name, last name, and address, but have a different phone number or email address based on the most recently available data which was updated in a separate excel worksheet (out of my hands, my job is just to merge this data into our latest database which they plan to use from here on out, not the excel sheet). I just need a good way to merge these tables with the same column names without doing it manually (about 24,000+) records.
Here is the Create Table Syntax for both tables:
CREATE TABLE `UsersUpdated` (
`FULLNME` longtext,
`LSTNME` varchar(23) CHARACTER SET utf8 DEFAULT NULL,
`FSTNME` varchar(255) CHARACTER SET utf8 DEFAULT NULL,
`MID` varchar(255) CHARACTER SET utf8 DEFAULT NULL,
`SUFF` varchar(50) CHARACTER SET utf8 DEFAULT NULL,
`STAT` varchar(2) CHARACTER SET utf8 DEFAULT NULL,
`PTY` varchar(3) CHARACTER SET utf8 DEFAULT NULL,
`PH` bigint(20) DEFAULT NULL,
`ALTPH` bigint(20) DEFAULT NULL,
`DOB` datetime DEFAULT NULL,
`REGDTE` datetime DEFAULT NULL,
`ADDR` text,
`ST` int(11) DEFAULT NULL,
`STNME` varchar(19) CHARACTER SET utf8 DEFAULT NULL,
`APT` varchar(7) CHARACTER SET utf8 DEFAULT NULL,
`TWN` varchar(6) CHARACTER SET utf8 DEFAULT NULL,
`ZIP` int(11) DEFAULT NULL,
`W` int(11) DEFAULT NULL,
`d` int(11) DEFAULT NULL,
`G17` varchar(1) CHARACTER SET utf8 DEFAULT NULL,
`P17` varchar(1) CHARACTER SET utf8 DEFAULT NULL,
`G16` varchar(1) CHARACTER SET utf8 DEFAULT NULL,
`P16` varchar(1) CHARACTER SET utf8 DEFAULT NULL,
`G15` varchar(1) CHARACTER SET utf8 DEFAULT NULL,
`P15` varchar(1) CHARACTER SET utf8 DEFAULT NULL,
`G14` varchar(1) CHARACTER SET utf8 DEFAULT NULL,
`P14` varchar(1) CHARACTER SET utf8 DEFAULT NULL,
`SIGN` tinyint(1) NOT NULL DEFAULT '0',
`SUPP` tinyint(1) NOT NULL DEFAULT '0',
`NOTES` longtext,
`LTR` tinyint(1) DEFAULT NULL,
`REGISTERED` tinyint(1) DEFAULT NULL
) ENGINE=InnoDB DEFAULT CHARSET=latin1;
CREATE TABLE `Users` (
`FULLNME` longtext,
`LSTNME` varchar(23) CHARACTER SET utf8 DEFAULT NULL,
`FSTNME` varchar(255) CHARACTER SET utf8 DEFAULT NULL,
`MID` varchar(255) CHARACTER SET utf8 DEFAULT NULL,
`SUFF` varchar(50) CHARACTER SET utf8 DEFAULT NULL,
`STAT` varchar(2) CHARACTER SET utf8 DEFAULT NULL,
`PTY` varchar(3) CHARACTER SET utf8 DEFAULT NULL,
`PH` bigint(20) DEFAULT NULL,
`ALTPH` bigint(20) DEFAULT NULL,
`DOB` datetime DEFAULT NULL,
`REGDTE` datetime DEFAULT NULL,
`ADDR` text,
`ST` int(11) DEFAULT NULL,
`STNME` varchar(19) CHARACTER SET utf8 DEFAULT NULL,
`APT` varchar(7) CHARACTER SET utf8 DEFAULT NULL,
`TWN` varchar(6) CHARACTER SET utf8 DEFAULT NULL,
`ZIP` int(11) DEFAULT NULL,
`W` int(11) DEFAULT NULL,
`d` int(11) DEFAULT NULL,
`G17` varchar(1) CHARACTER SET utf8 DEFAULT NULL,
`P17` varchar(1) CHARACTER SET utf8 DEFAULT NULL,
`G16` varchar(1) CHARACTER SET utf8 DEFAULT NULL,
`P16` varchar(1) CHARACTER SET utf8 DEFAULT NULL,
`G15` varchar(1) CHARACTER SET utf8 DEFAULT NULL,
`P15` varchar(1) CHARACTER SET utf8 DEFAULT NULL,
`G14` varchar(1) CHARACTER SET utf8 DEFAULT NULL,
`P14` varchar(1) CHARACTER SET utf8 DEFAULT NULL,
`SIGN` tinyint(1) NOT NULL DEFAULT '0',
`SUPP` tinyint(1) NOT NULL DEFAULT '0',
`NOTES` longtext,
`LTR` tinyint(1) DEFAULT NULL,
`REGISTERED` tinyint(1) DEFAULT NULL
) ENGINE=InnoDB DEFAULT CHARSET=latin1;
As you can see, they are basically the same exact tables, I just need to merge them correctly.
Perhaps this is helpful.
update Users
set ADDR = (
select ADDR from UsersUpdated uu
where uu.FULLNME = Users.FULLNME and uu.DOB = Users.DOB
), STNME = (
select STNME from UsersUpdated uu
where uu.FULLNME = Users.FULLNME and uu.DOB = Users.DOB
) ...
;
You can add all the columns to a single update. Depending on the size of the database it might just be as easy to do them individually.
Many platforms allow for a from clause with update that permits a join and a shorter query but it can be problematic. This way you will get errors if any of the subqueries don't return just a single value.
I would modify the Users table to add a UNIQUE key on the fields that should be the same (presumably FSTNME, LSTNME and ADDR from your description but perhaps you might use some other columns e.g. DOB as suggested by #shawnt00), then INSERT the data from UsersUpdated into Users using an ON DUPLICATE KEY UPDATE clause to copy updated data into Users where the user already exists in that table. This query will also work when there are users in UsersUpdated who are not already in Users. So,
ALTER TABLE Users ADD UNIQUE KEY NameAddr (FSTNME, LSTNME, ADDR);
INSERT INTO Users
SELECT * FROM UsersUpdated
ON DUPLICATE KEY UPDATE
FULLNME=VALUES(FULLNME),
LSTNME=VALUES(LSTNME),
FSTNME=VALUKES(FSTNME),
...
LTR=VALUES(LTR),
REGISTERED=VALUES(REGISTERED);
There are some columns in my mysql table which contain values such as "Sbeed Pharmaceuticals åÊ" . Two questions:
How do I prevent getting these characters from getting into the table in the first place ?
How do I find rows containing these characters ?
Here is the table:
CREATE TABLE `drugs` (
`drug-id` int(10) unsigned NOT NULL AUTO_INCREMENT,
`th-class` varchar(512) DEFAULT NULL,
`division` varchar(512) DEFAULT NULL,
`name` varchar(512) DEFAULT NULL,
`brand-name` varchar(64) DEFAULT NULL,
`pack` varchar(512) DEFAULT NULL,
`rx-type` varchar(16) DEFAULT NULL,
`pack-form` varchar(128) DEFAULT NULL,
`available-in` varchar(512) DEFAULT NULL,
`mrp` double DEFAULT '-1',
`mrp-per-unit` double DEFAULT NULL,
`manufacturer` varchar(512) DEFAULT NULL,
`schemes` text,
`drug-status` varchar(64) DEFAULT 'new',
`created-by` varchar(512) DEFAULT NULL,
`created-at` datetime DEFAULT CURRENT_TIMESTAMP,
`approved-at` datetime DEFAULT NULL,
`approved-by` varchar(512) DEFAULT NULL,
`dosage` varchar(64) DEFAULT NULL,
`manufacturer-id` int(10) DEFAULT NULL,
PRIMARY KEY (`drug-id`),
UNIQUE KEY `name` (`name`)
) ENGINE=InnoDB DEFAULT CHARSET=latin1;