Mysql :5.7
General Log: 1
log_output : Table
I have written some code using java to store images in a blob column of a table (tbl_attachment_mst).
My General Log settings are turned on and is configured to write to 'table'.
Whenever i add an image to tbl_attachment_mst , mysql does log it in the mysql.general_log table with _binary('some unreadable characters not sure what this is').
I have accidently lost contents of the table tbl_attachment_mst . Is it possible to recover my data from the mysql.generaL_log table??
I think i am having some issue with the character set while trying to execute the query that is stored in mysql.general_log.
(From comment)
CREATE TABLE tbl_attachment_mst (
attachment_id int(11) NOT NULL AUTO_INCREMENT,
file_name varchar(200) DEFAULT NULL,
created_date datetime DEFAULT NULL,
activate_flag tinyint(4) DEFAULT NULL,
file_id int(11) DEFAULT NULL,
type varchar(300) DEFAULT NULL,
attachment_asblob longblob,
PRIMARY KEY (attachment_id)
) ENGINE=MyISAM DEFAULT CHARSET=latin1
insert into tbl_attachment_mst values
(default,'test.jpg',sysdate(),1,1,'Type',
_binary('some characters here'))
Grab it fast. I suspect the general_log, as a table, gets flushed in some fashion pretty fast.
It works just like a table, so SELECTs work. However, your SELECTs go into the general log unless you have turned it off.
Recommend doing SELECT ... HEX(col) ... to avoid the unprintable characters.
Please provide more details so I can try to simulate it and see what would work best.
SHOW CREATE TABLE
An approximation of the INSERT statement (or whatever was involved)
And if the grabbed hex is useful, you can use something like this to reverse the steps:
INSERT ... VALUES (... UNHEX(hex_string) ... )
using the below set of queries, i was able to get the exact data that was inserted into the db at that particular time.....
SELECT argument INTO #sql FROM mysql.general_log limit 1;
PREPARE sql_query FROM #sql;
EXECUTE sql_query;
These queries would insert the data back into the tbl_attachment_mst...
Related
Im revisiting my database and noticed I had some primary keys that were of type INT.
This wasn't unique enough so I thought I would have a guid.
I come from a microsoft sql background and in the ssms you can
choose type to "uniqeidentifier" and auto increment it.
In mysql however Ive found that you have to make triggers that execute on insert for the tables you want
to generate a guide id for. Example:
Table:
CREATE TABLE `tbl_test` (
`GUID` char(40) NOT NULL,
`Name` varchar(50) NOT NULL,
PRIMARY KEY (`GUID`)
) ENGINE=MyISAM DEFAULT CHARSET=latin1;
Trigger:
CREATE TRIGGER `t_GUID` BEFORE INSERT ON `tbl_test`
FOR EACH ROW begin
SET new.GUID = uuid();
Alternatively you have to insert the guid yourself in the backend.
Im no DB expert but still remember that triggers cause performance problems.
The above is something I found here and is 9 years old so I was hoping something has changed?
As far as stated in the documentation, you can use uid() as a column default starting version 8.0.13, so something like this should work:
create table tbl_test (
guid binary(16) default (uuid_to_bin(uuid())) not null primary key,
name varchar(50) not null
);
This is pretty much copied from the documentation. I don't have a recent enough version of MySQL at hand to test this.
You can make a
INSERT INTO `tbl_test` VALUES (uuid(),'testname');
This would generate a new uuid, when you call it.
Or you can also use the modern uuid v4 by using one of these functions instead of the standard uuid(), which is more random than the uuid in mysql
How to generate a UUIDv4 in MySQL?
You can use since 8.0.13
CREATE TABLE t1 (
uuid_field VARCHAR(40) DEFAULT (uuid())
);
But you wanted more than unique, but here are only allowed internal functions and not user defined as for uuid v4, for that uyou need the trogger
As per the documentation, BINARY(x) adds some hidden padding bytes to the end of each entry, & VARCHAR(40) also wastes space by not being encoded directly in binary. Using VARBINARY(16) would be more efficient.
Also, more entropy (unguessability / security) per byte is available from RANDOM_BYTES(16) than standardized UUIDs, because they use some sections to encode constant metadata.
Perhaps the below will work for your needs.
-- example
CREATE TABLE `tbl_test` (
`GUID` VARBINARY(16) DEFAULT (RANDOM_BYTES(16)) NOT NULL PRIMARY KEY,
`Name` VARCHAR(50) NOT NULL
);
Im revisiting my database and noticed I had some primary keys that were of type INT.
This wasn't unique enough so I thought I would have a guid.
I come from a microsoft sql background and in the ssms you can
choose type to "uniqeidentifier" and auto increment it.
In mysql however Ive found that you have to make triggers that execute on insert for the tables you want
to generate a guide id for. Example:
Table:
CREATE TABLE `tbl_test` (
`GUID` char(40) NOT NULL,
`Name` varchar(50) NOT NULL,
PRIMARY KEY (`GUID`)
) ENGINE=MyISAM DEFAULT CHARSET=latin1;
Trigger:
CREATE TRIGGER `t_GUID` BEFORE INSERT ON `tbl_test`
FOR EACH ROW begin
SET new.GUID = uuid();
Alternatively you have to insert the guid yourself in the backend.
Im no DB expert but still remember that triggers cause performance problems.
The above is something I found here and is 9 years old so I was hoping something has changed?
As far as stated in the documentation, you can use uid() as a column default starting version 8.0.13, so something like this should work:
create table tbl_test (
guid binary(16) default (uuid_to_bin(uuid())) not null primary key,
name varchar(50) not null
);
This is pretty much copied from the documentation. I don't have a recent enough version of MySQL at hand to test this.
You can make a
INSERT INTO `tbl_test` VALUES (uuid(),'testname');
This would generate a new uuid, when you call it.
Or you can also use the modern uuid v4 by using one of these functions instead of the standard uuid(), which is more random than the uuid in mysql
How to generate a UUIDv4 in MySQL?
You can use since 8.0.13
CREATE TABLE t1 (
uuid_field VARCHAR(40) DEFAULT (uuid())
);
But you wanted more than unique, but here are only allowed internal functions and not user defined as for uuid v4, for that uyou need the trogger
As per the documentation, BINARY(x) adds some hidden padding bytes to the end of each entry, & VARCHAR(40) also wastes space by not being encoded directly in binary. Using VARBINARY(16) would be more efficient.
Also, more entropy (unguessability / security) per byte is available from RANDOM_BYTES(16) than standardized UUIDs, because they use some sections to encode constant metadata.
Perhaps the below will work for your needs.
-- example
CREATE TABLE `tbl_test` (
`GUID` VARBINARY(16) DEFAULT (RANDOM_BYTES(16)) NOT NULL PRIMARY KEY,
`Name` VARCHAR(50) NOT NULL
);
CAUSE:
I have a table and the columns are all suitably Collated as utf8mb4_unicode_ci,
CREATE TABLE IF NOT EXISTS `users` (
`user_id` int(8) NOT NULL AUTO_INCREMENT,
`username` varchar(100) NOT NULL,
`pass_word` varchar(512) NOT NULL ,
...etc etc...
PRIMARY KEY (`user_id`),
UNIQUE KEY `email_addr` (`email_addr`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 AUTO_INCREMENT=989 ;
...Including the column storing the password hash (generated from password_hash) such as $2y$14$tFpExwd2TXm43Bd20P4nkMbL1XKxwF.VCpL.FXeVRaUO3FFxGJ4Di.
BUT, I find that due to the case insensitivity of the column, that a hash of $2y$14$tFpExwd2tXm43Bd20P4NKmbL1XKxwF.VCpL.FxEVRaUO3FFxGJ4DI would still allow access.
This means that there are potentially hundreds of collisions possible by storing the data in a case insensitive manner. Not good.
ISSUE:
Now, Is there a way of forcing MySQL to treat pass_word column as a case sensitive column, when doing comparisons. I want to avoid having to edit every occurance of the PHP/SQL querying, and instead simply set the database table column to compare in a case sensitive manner by default.
The utf8mb4 character set does not give me any _cs options, and the only non-_ci option appears to be utf8mb4_bin.
So simple questions:
Does the UTF8mb4_bin character set & collation on MySQL treat standard comparisons case sensitively? [yes]
Dose the UTF8mb4_bin suit what I want to do. Should I use another set, and if so, why?
Are there any issues in storing password_hash outputs in a MySQL utf8mb4_bin column?
Does this approach conveniently sidestep the need to edit the query SQL of each login query? Can I change the column type and then move on?
EDIT
As detailed by nj_ , this is a silly issue that is not an issue at all because the value of pass_word is never directly edited when logging in.
... It's been a long day.
If you're really that worried about the potential 2^55 collisions in your 62^55 address space, you can simply change the column type to BLOB, which is always case-sensitive.
CREATE TABLE IF NOT EXISTS `users` (
`user_id` int(8) NOT NULL AUTO_INCREMENT,
`username` varchar(100) NOT NULL,
`pass_word` BLOB NOT NULL ,
...etc etc...
PRIMARY KEY (`user_id`),
UNIQUE KEY `email_addr` (`email_addr`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 AUTO_INCREMENT=989 ;
Example:
INSERT INTO `users` (..., `pass_word`) VALUES (..., 'AbC');
SELECT * FROM `users` WHERE `pass_word` = 'AbC' LIMIT 0,1000; -> 1 hit
SELECT * FROM `users` WHERE `pass_word` = 'abc' LIMIT 0,1000; -> 0 hits
Case sensitivity is no problem in this case, because you cannot verify the password directly with SQL anyway. A correctly salted password hash cannot be searched for in the database. Search by username only and extract the stored hash from the database:
$sql= 'SELECT * FROM users WHERE username = ?';
$db->prepare($sql);
$db->bind_param('s', $_POST['username']);
Afterwards you can extract the hash from the row and check the entered password against the found hash with the password_verify() function:
// Check if the hash of the entered login password, matches the stored hash.
// The salt and the cost factor will be extracted from $existingHashFromDb.
$isPasswordCorrect = password_verify($password, $existingHashFromDb);
I want to merge 5 identical-schema (okay, they are not exactly identical but I can edit the field names to make them identical) MySQL databases into one database. Is there any easy way?
CREATE TABLE `users` (
`name` VARCHAR (50) COLLATE utf8_turkish_ci DEFAULT NULL,
`surname` VARCHAR (50) COLLATE utf8_turkish_ci DEFAULT NULL,
`telephone` VARCHAR (50) COLLATE utf8_turkish_ci DEFAULT NULL,
PRIMARY KEY (`id`),
) ENGINE = MyISAM AUTO_INCREMENT = 1000 DEFAULT CHARSET = utf8 COLLATE = utf8_turkish_ci PACK_KEYS = 0 ROW_FORMAT = DYNAMIC
If the tables are exactly the same with column names/types and are named
user1
user2
user3
user4
user5
there are two approaches to handle this:
APPROACH #1 : Load the data into one table
CREATE TABLE user LIKE user1;
INSERT INTO user (name,surname,telephone,...)
SELECT name,surname,telephone,... FROM user1;
INSERT INTO user (name,surname,telephone,...)
SELECT name,surname,telephone,... FROM user2;
INSERT INTO user (name,surname,telephone,...)
SELECT name,surname,telephone,... FROM user3;
INSERT INTO user (name,surname,telephone,...)
SELECT name,surname,telephone,... FROM user4;
INSERT INTO user (name,surname,telephone,...)
SELECT name,surname,telephone,... FROM user5;
If the id is auto_increment all rows get new ids.
APPROACH #2 : Use the MERGE Storage Engine
CREATE TABLE user LIKE user1;
ALTER TABLE user
ENGINE=Mrg_MyISAM
UNION=(user1,user2,user3,user4,user5)
;
Give it a Try !!!
There's no easy way to do this.
You could attempt to alter your tables within the different databases to bring them to be in the most similar format.
Additionally, you could use statements such as
Create table as select
in order to further format the data.
Than you would have to do a MYSQL DUMP of all your databases.
Select only the create statements from the database schema you are interested in following, and add your insert statements (for the data) from all the different databases.
You may also have to perform text manipulation in Excel, or from with mysql in order to get the data in such a format that it is compatible and can be inserted in your final schema.
Any ETL tool, like Clover, would be well suited for your purpose. Just define your column mappings and you should be good to go. Leave a comment if you need further help.
I currently trying to use an Object Relational Mapper for CodeIgniter and I'm experiencing something I did not expect.
I have a table with a couple of fields, some of which are NOT NULL. An insert query is which is missing of the NOT NULL fields is generated -- a new row is added but with blanks for those fields.
I did not know MySQL would disregard the NOT NULL fields that aren't present in the query and insert the row anyways. Is there a way to restrict this?
-Edit-
Let me add a few more details and try to explain it a bit more
Here is a sample table:
CREATE TABLE `test` (
`id` int(11) unsigned NOT NULL AUTO_INCREMENT,
`color` varchar(40) COLLATE utf8_bin DEFAULT '',
`shape` varchar(40) COLLATE utf8_bin NOT NULL,
`size` varchar(40) COLLATE utf8_bin NOT NULL,
PRIMARY KEY (`id`)
) ENGINE=InnoDB AUTO_INCREMENT=2 DEFAULT CHARSET=utf8 COLLATE=utf8_bin
Here is a sample query:
INSERT INTO `test` (`shape`) VALUES ('foo')
I don't have size in my query yet it still adds the row - is this expected?
(The sample query was run in phpMyAdmin)
I believe the accepted answer is incorrect, given the question's test INSERT statement. It looks to me like MySQL's "strict mode" is turned off for this table or database. From the docs:
Strict mode controls how MySQL handles input values that are invalid or missing... A value is missing when a new row to be inserted does not contain a value for a non-NULL column that has no explicit DEFAULT clause in its definition...
If you are not using strict mode (that is, neither STRICT_TRANS_TABLES nor STRICT_ALL_TABLES is enabled), MySQL inserts adjusted values for invalid or missing values and produces warnings.
You can find out how your database is running with these queries:
SELECT ##global.sql_mode;
SELECT ##session.sql_mode;
Changing these values is discussed here: https://stackoverflow.com/a/5273824/27846
Empty string is not the same thing as NULL. Perhaps ORM inserts just '' for those fields.
Not a codeigniter dev, but I would hazard a guess that the issue is your ORM is passing blank values on to the database, I would check your logs to verify this and if its the case, check your ORM if it has some validation options.