I want to merge 5 identical-schema (okay, they are not exactly identical but I can edit the field names to make them identical) MySQL databases into one database. Is there any easy way?
CREATE TABLE `users` (
`name` VARCHAR (50) COLLATE utf8_turkish_ci DEFAULT NULL,
`surname` VARCHAR (50) COLLATE utf8_turkish_ci DEFAULT NULL,
`telephone` VARCHAR (50) COLLATE utf8_turkish_ci DEFAULT NULL,
PRIMARY KEY (`id`),
) ENGINE = MyISAM AUTO_INCREMENT = 1000 DEFAULT CHARSET = utf8 COLLATE = utf8_turkish_ci PACK_KEYS = 0 ROW_FORMAT = DYNAMIC
If the tables are exactly the same with column names/types and are named
user1
user2
user3
user4
user5
there are two approaches to handle this:
APPROACH #1 : Load the data into one table
CREATE TABLE user LIKE user1;
INSERT INTO user (name,surname,telephone,...)
SELECT name,surname,telephone,... FROM user1;
INSERT INTO user (name,surname,telephone,...)
SELECT name,surname,telephone,... FROM user2;
INSERT INTO user (name,surname,telephone,...)
SELECT name,surname,telephone,... FROM user3;
INSERT INTO user (name,surname,telephone,...)
SELECT name,surname,telephone,... FROM user4;
INSERT INTO user (name,surname,telephone,...)
SELECT name,surname,telephone,... FROM user5;
If the id is auto_increment all rows get new ids.
APPROACH #2 : Use the MERGE Storage Engine
CREATE TABLE user LIKE user1;
ALTER TABLE user
ENGINE=Mrg_MyISAM
UNION=(user1,user2,user3,user4,user5)
;
Give it a Try !!!
There's no easy way to do this.
You could attempt to alter your tables within the different databases to bring them to be in the most similar format.
Additionally, you could use statements such as
Create table as select
in order to further format the data.
Than you would have to do a MYSQL DUMP of all your databases.
Select only the create statements from the database schema you are interested in following, and add your insert statements (for the data) from all the different databases.
You may also have to perform text manipulation in Excel, or from with mysql in order to get the data in such a format that it is compatible and can be inserted in your final schema.
Any ETL tool, like Clover, would be well suited for your purpose. Just define your column mappings and you should be good to go. Leave a comment if you need further help.
Related
Mysql :5.7
General Log: 1
log_output : Table
I have written some code using java to store images in a blob column of a table (tbl_attachment_mst).
My General Log settings are turned on and is configured to write to 'table'.
Whenever i add an image to tbl_attachment_mst , mysql does log it in the mysql.general_log table with _binary('some unreadable characters not sure what this is').
I have accidently lost contents of the table tbl_attachment_mst . Is it possible to recover my data from the mysql.generaL_log table??
I think i am having some issue with the character set while trying to execute the query that is stored in mysql.general_log.
(From comment)
CREATE TABLE tbl_attachment_mst (
attachment_id int(11) NOT NULL AUTO_INCREMENT,
file_name varchar(200) DEFAULT NULL,
created_date datetime DEFAULT NULL,
activate_flag tinyint(4) DEFAULT NULL,
file_id int(11) DEFAULT NULL,
type varchar(300) DEFAULT NULL,
attachment_asblob longblob,
PRIMARY KEY (attachment_id)
) ENGINE=MyISAM DEFAULT CHARSET=latin1
insert into tbl_attachment_mst values
(default,'test.jpg',sysdate(),1,1,'Type',
_binary('some characters here'))
Grab it fast. I suspect the general_log, as a table, gets flushed in some fashion pretty fast.
It works just like a table, so SELECTs work. However, your SELECTs go into the general log unless you have turned it off.
Recommend doing SELECT ... HEX(col) ... to avoid the unprintable characters.
Please provide more details so I can try to simulate it and see what would work best.
SHOW CREATE TABLE
An approximation of the INSERT statement (or whatever was involved)
And if the grabbed hex is useful, you can use something like this to reverse the steps:
INSERT ... VALUES (... UNHEX(hex_string) ... )
using the below set of queries, i was able to get the exact data that was inserted into the db at that particular time.....
SELECT argument INTO #sql FROM mysql.general_log limit 1;
PREPARE sql_query FROM #sql;
EXECUTE sql_query;
These queries would insert the data back into the tbl_attachment_mst...
I need to get table full structure as one string.
I use SHOW CREATE TABLE my_table but lack column collation information
CREATE TABLE `my_table` (
`col_1` int(11) NOT NULL AUTO_INCREMENT,
`col_2` varchar(255) NOT NULL,
PRIMARY KEY (`col_1`),
KEY `col_2` (`col_2`)
) ENGINE=MyISAM AUTO_INCREMENT=15 DEFAULT CHARSET=utf8mb4
Espected:
Instead col_2 varchar(255) NOT NULL,
I want col_2 varchar(255) COLLATE utf8mb4_unicode_ci NOT NULL,
One strange thing:
I don't understand why certains tables show also the column colation with SHOW CREATE TABLE but others no. However if I look the structure table in phpmyadmin all tables show column collation (when the column is varchar or text type)
One last note:
If I use SHOW FULL COLUMNS ... all columns have also their collation info (even if are showing or not in SHOW CREATE TABLE)
So if I have not other option I try to recreate the creation table string manually from SHOW FULL COLUMNS result. But i think that this is a hard and risky work ...
All that I need to use in a personal tool to compare when have structure change (between two servers)
CAUSE:
I have a table and the columns are all suitably Collated as utf8mb4_unicode_ci,
CREATE TABLE IF NOT EXISTS `users` (
`user_id` int(8) NOT NULL AUTO_INCREMENT,
`username` varchar(100) NOT NULL,
`pass_word` varchar(512) NOT NULL ,
...etc etc...
PRIMARY KEY (`user_id`),
UNIQUE KEY `email_addr` (`email_addr`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 AUTO_INCREMENT=989 ;
...Including the column storing the password hash (generated from password_hash) such as $2y$14$tFpExwd2TXm43Bd20P4nkMbL1XKxwF.VCpL.FXeVRaUO3FFxGJ4Di.
BUT, I find that due to the case insensitivity of the column, that a hash of $2y$14$tFpExwd2tXm43Bd20P4NKmbL1XKxwF.VCpL.FxEVRaUO3FFxGJ4DI would still allow access.
This means that there are potentially hundreds of collisions possible by storing the data in a case insensitive manner. Not good.
ISSUE:
Now, Is there a way of forcing MySQL to treat pass_word column as a case sensitive column, when doing comparisons. I want to avoid having to edit every occurance of the PHP/SQL querying, and instead simply set the database table column to compare in a case sensitive manner by default.
The utf8mb4 character set does not give me any _cs options, and the only non-_ci option appears to be utf8mb4_bin.
So simple questions:
Does the UTF8mb4_bin character set & collation on MySQL treat standard comparisons case sensitively? [yes]
Dose the UTF8mb4_bin suit what I want to do. Should I use another set, and if so, why?
Are there any issues in storing password_hash outputs in a MySQL utf8mb4_bin column?
Does this approach conveniently sidestep the need to edit the query SQL of each login query? Can I change the column type and then move on?
EDIT
As detailed by nj_ , this is a silly issue that is not an issue at all because the value of pass_word is never directly edited when logging in.
... It's been a long day.
If you're really that worried about the potential 2^55 collisions in your 62^55 address space, you can simply change the column type to BLOB, which is always case-sensitive.
CREATE TABLE IF NOT EXISTS `users` (
`user_id` int(8) NOT NULL AUTO_INCREMENT,
`username` varchar(100) NOT NULL,
`pass_word` BLOB NOT NULL ,
...etc etc...
PRIMARY KEY (`user_id`),
UNIQUE KEY `email_addr` (`email_addr`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 AUTO_INCREMENT=989 ;
Example:
INSERT INTO `users` (..., `pass_word`) VALUES (..., 'AbC');
SELECT * FROM `users` WHERE `pass_word` = 'AbC' LIMIT 0,1000; -> 1 hit
SELECT * FROM `users` WHERE `pass_word` = 'abc' LIMIT 0,1000; -> 0 hits
Case sensitivity is no problem in this case, because you cannot verify the password directly with SQL anyway. A correctly salted password hash cannot be searched for in the database. Search by username only and extract the stored hash from the database:
$sql= 'SELECT * FROM users WHERE username = ?';
$db->prepare($sql);
$db->bind_param('s', $_POST['username']);
Afterwards you can extract the hash from the row and check the entered password against the found hash with the password_verify() function:
// Check if the hash of the entered login password, matches the stored hash.
// The salt and the cost factor will be extracted from $existingHashFromDb.
$isPasswordCorrect = password_verify($password, $existingHashFromDb);
Is there a proper place to store a high level description of a database? Something along the lines of "This database is used to store XYZ for use by ABC". It's not necessarily information one would need to query, but something that would useful for someone administering the system (i.e. me in a few months when I'm trying to remember what I was trying to accomplish a few months ago.).
This seems like something that someone would have asked before (or information that is readily findable), but none of my searching came up with anything relevant. Most of what I found was for displaying the structure of the database itself.
Comment metadata is not available for MySQL databases, but you can create a table to store some comments: (I have this table in a generic tools database)
-- Create a table to store db name and fields as need.
create table dbinfo(
id int not null primary key auto_increment,
db_name varchar(64) not null collate utf8_bin,
db_comment varchar(255),
unique (db_name)
) default charset utf8;
To fill/update dbinfo table with current databases:
insert into dbinfo (db_name)
select SCHEMA_NAME
from INFORMATION_SCHEMA.SCHEMATA
where SCHEMA_NAME not in (select db_name from dbinfo);
You will only need to maintain dbinfo table until MySQL enables database comments.
I have this table for users that stores their usernames and other data, thats done like this (stripped down):
CREATE TABLE `prod_users` (
`p_user_id` INT(11) NOT NULL AUTO_INCREMENT,
`p_user_name` VARCHAR(200) NOT NULL DEFAULT ''
`p_comp_name` VARCHAR(300) NOT NULL DEFAULT '',
PRIMARY KEY (`p_user_id`)
)
COLLATE='utf8_general_ci'
ENGINE=MyISAM
Each time a user signs up, he'll provide a company name as well.
There's another table called prod_profiles, which stores profile details like phone nos. fax nos. etc.
CREATE TABLE `prod_profiles` (
`pf_gen_id` INT(11) NOT NULL AUTO_INCREMENT,
`pf_user_id` INT(11) NOT NULL DEFAULT '0',
`pf_user_name` VARCHAR(200) NOT NULL DEFAULT ''
`pf_comp_name` VARCHAR(300) NOT NULL DEFAULT '',
PRIMARY KEY (`pf_gen_id`)
)
COLLATE='utf8_general_ci'
ENGINE=MyISAM
When a new user signs up and his details are added to prod_users, is it possible to automatically add his new user_id, user_name and comp_name details to prod_profile using MySql itself? Since each user will have a new p_user_id and we wont know it, it'll be difficult using php. Can this be achieved inside MySql itself without any problems?
It isn't difficult using PHP, since you have the LAST_INSERT_ID() available for use, be it via mysql_insert_id() or mysqli::$insert_id, PDO::lastInsertId() or whatever your API provides. As long as you call the two INSERT statements in immediate succession on the same script (it is connection dependent), MySQL will supply the correct p_user_id.
However, you can use an AFTER INSERT trigger to force MySQL to create the new row automatically:
CREATE TRIGGER build_profile AFTER INSERT ON prod_users
FOR EACH ROW
BEGIN
INSERT INTO prod_profiles
(pf_user_id, pf_user_name, pf_comp_name)
VALUES (NEW.p_user_id, NEW.p_user_name, NEW.p_comp_name)
END
Review the MySQL CREATE TRIGGER syntax reference for full details and options.
You can use the next mysql function: LAST_INSERT_ID(); which returns the last auto increased id.
Therefore , add a user and then add a prod_profile , while pf_user_id value will be the returned value of last_insert_id().
INSERT INTO `prod_users`(`p_user_name`,`p_comp_name`) VALUES('Dan' , 'Stackover')
INSERT INTO `prod_profiles`(`pf_user_id`,`pf_user_name`,`pf_comp_name`) VALUES(LAST_INSERT_ID(),'Dan','Stackover')
Please notice: I have to say , that storing the username and company_name twice for the same user in two different tables is a reall waste...
Consider re-thinking about your DB structre and logic.