I am using Python with MYSql string queries and trying to do something like
"SELECT id from Example
WHERE item_column = (%s)"
Where in my case the passed in value can be a number or NULL. I understand that in SQL NULL represents nothing so this equality will not work. Normally, I would have to do something like column IS NULL. However, how do I set up my statement to account for values that can be NULL or not NULL?
Thank you
You can use the NULL-safe equal operator <=>.
"SELECT id from Example WHERE item_column <=> (%s)"
If you want give a default value for your variable incase its null you can use coalese(say 'xx' in case of string and -99.99 in case of integer)
"SELECT id from Example
WHERE item_column = COALESCE((%s),-99.99)"
But if you want to modify your statement based on the whether variable is null or not then you have to do a if else in the code
if variable is None:
return "SELECT id from Example WHERE item_column is null"
else:
return "SELECT id from Example WHERE item_column = variable"
I'm using perl 5.20 and MySQL 5.7, but I think the question is about SQL in general:
perldoc DBI says:
Binding an undef (NULL) to the placeholder will not select rows which have a NULL age! At least for database engines that conform to the SQL standard. Refer to the SQL manual for your database engine or any SQL book for the reasons for this. To explicitly select NULLs you have to say "WHERE age IS NULL".
I don't even know what to google for... My question is: What are the reasons behind = ? not matching a binding to NULL/undef? (Beyond "that is how it is defined and documented".)
I've discovered that MySQL has an operator <=> that allows comparisons with NULL and so:
my $sth = $dbh->prepare('select count(*) from table where field <=> ?');
$sth->execute(345);
$sth->execute(undef);
both work as expected. Unfortunately, the doc says:
The <=> operator is equivalent to the standard SQL IS NOT DISTINCT FROM operator.
And MySQL doesn't support the IS NOT DISTINCT FROM operator :-(. So there seems to be no portable way to do this. Except for the very hackish:
my $sth = $dbh->prepare('
select count(*) from table
where field = ? OR ( ? IS NULL AND field IS NULL )
');
$sth->execute(345, 345);
$sth->execute(undef, undef);
or the even more hackish snippet from perldoc DBI
$sql_clause = defined $age? "age = ?" : "age IS NULL";
$sth = $dbh->prepare(qq{
SELECT fullname FROM people WHERE $sql_clause
});
$sth->execute(defined $age ? $age : ());
Is there a portable way to do WHERE FIELD = ? and have it do what I mean also with NULLs/undefs? What did I miss?
EDIT: I also came up with this workaround, which especially works great if field is a numeric type such as an INT, so we're sure the string "NULL" is not a possible non-NULL value.
my $sth = $dbh->prepare('
select count(*) from table
where COALESCE(field, "NULL") = COALESCE(?,"NULL")
');
$sth->execute(345);
$sth->execute(undef);
But performance goes out the window, as I don't think any indexes can be used....
I understand that this is not exactly what you asked for but if you use DBIx::Class, the ORM will do that lifting for you.
my $res = $schema->resultset('table')->search({ field => [345, undef] });
print $res->count;
It will be translated to this SELECT COUNT( * ) FROM table me WHERE ( ( field = ? OR field IS NULL ) ): '345'
I am trying to do a query that sees if fields are equivalent. However, whenever the field is NULL it returns a false result. I even tried doing the same thing with the column itself:
SELECT * FROM `mturk_completion` WHERE (`mturk_completion`.`imdb_url` =
`mturk_completion`.`imdb_url` AND `mturk_completion`.`worker_id` = 'A3NF84Q37D7F35' )
And it only returns results where the column is not NULL. Why is this so, and how do I get around it?
Your title is absolutely correct for any SQL implementation (not just MySQL). NULL is not equal to anything (including another NULL).
You need to use explicit IS NULL check or COALESCE() function (or its RDBMS-dependent alternatives) to set some default value in case of NULL.
Your comparison of mturk_completion.imdb_url to itself is redundant and should always return True, except when mturk_completion.imdb_urlis Null, in which case it will return Null.
That's because the operator = returns either True, False when comparisons can be made or Null, when either of the two operators is Null
Try this to illustrate the situation.
SELECT 1 = NULL; -- returns NULL
SELECT 1 != NULL; -- also return NULL
SELECT ISNULL(1 = NULL); -- returns 1
SELECT ISNULL(1 != NULL); -- returns 1
If you rewrite your query like below, your problems with ignoring NULLs will go away:
SELECT * FROM `mturk_completion` WHERE worker_id = 'A3NF84Q37D7F35'
I think you can use
(table.Field = table2.Field OR COALESCE(table.Field, table2.Field) IS NULL)
I was trying to en- and decypt data with the corresponding MySQL-functions.
This is what I did:
INSERT INTO dbsec.tbl_credent (U_Password) VALUES (AES_ENCRYPT('secretText', SHA2('pwd123',512)));
the Primary-Key (id) was 6. So I used
SELECT dbsec.tbl_credent.U_Password FROM dbsec.tbl_credent WHERE dbsec.tbl_credent.id = '6';
I got something like this:
Ž4•ý/2Ÿ½üyÙ¤Ý'
So encryption seems to work so far.
When I start the following query:
SELECT AES_DECRYPT(dbsec.tbl_credent.U_Password, 'pwd123') FROM dbsec.tbl_credent WHERE dbsec.tbl_credent.id = '6';
Result is NULL
I used hashing for the password so I tried
SELECT AES_DECRYPT(dbsec.tbl_credent.U_Password, SHA2('pwd123',512)) FROM dbsec.tbl_credent WHERE dbsec.tbl_credent.id = '6';
Result is 73656372657454657874
As all this didn't work I tested this directly:
SELECT AES_DECRYPT(AES_ENCRYPT('secretText', SHA2('pwd123',512)), 'pwd123');
Again the Result was NULL and
SELECT AES_DECRYPT(AES_ENCRYPT('secretText', 'pwd123'), 'pwd123');
returned 73656372657454657874 again.
What do I have to do to get back the 'secretText' I have encrypted?
The Type of U_Password is text (latin1_swedish_ci), btw.
Currently I'm manually creating a string where I concatenate all the values in each row in my table. I'm hashing this string for each row to get a hash value for the current values (/status) of the row, which I'm later is using to determine if the row has changed.
Instead of doing this manually, is there an build-in way i mySQL to get a unique hash value for each row?
you could do something like
SELECT MD5(concat(field1, field2, field3, ...)) AS rowhash
but you can't get away from listing which fields you want, as concat(*) is not an option (syntax error).
It's better to use concat_ws(). e.g. two adjacent column: 12,3 => 1,23 .
Sorry, this still has some problems. Think about the null value, empty string, string can contain ',', etc...
A program is required to generate the hash statement, which should replace null to specific value (for null-able columns), and also use the seldom used char/byte as separator.
There are problems with CONCAT, e.g. CONCAT('ab', 'c') vs CONCAT('a', 'bc'). Two different rows, but result is the same. You could use CONCAT_WS(';', 'ab', 'c') to get ab;c but in case of CONCAT_WS(';', ';', '') vs CONCAT_WS(';', '', ';') you still get the same result.
Also CONCAT(NULL, 'c') returns NULL.
I think the best way is to use QUOTE:
SELECT MD5(CONCAT(QUOTE(c1), QUOTE(c2), QUOTE(c3))) AS row_hash FROM t1;
Result of: select (concat(quote('a'), quote('bc'), quote('NULL'), quote(NULL), quote('\''), quote('')));
is: 'a''bc''NULL'NULL'\''''
Also, don't use GROUP_CONCAT() to get hash of table, it has limit: https://dev.mysql.com/doc/refman/8.0/en/server-system-variables.html#sysvar_group_concat_max_len
Instead, CHECKSUM TABLE might be better, but you can't skip columns with CHECKSUM TABLE https://dev.mysql.com/doc/refman/5.7/en/checksum-table.html
Well I made a little script that could do excactly what you want, and maybe what others want... so here it goes...for PHP that is...
first you have to make a list of columns of the table, then you make a "case when" statement for each column based on their type and put that in the concat_ws statement and then you hash it with sha1...i've used this method on very large tables (600000+ records) and the speed is quite good when selecting all records. also I think that it is faster to concat the required data in a concat_ws and explode it in php or whatever you are using, but that is just a hunch...
<?
$query= mysql_query("SHOW COLUMNS FROM $table", $linklive);
while ($col = mysql_fetch_assoc($query)) {
$columns[] = mysql_real_escape_string($col['Field']);
if ($col['Key'] == 'PRI') {
$key = mysql_real_escape_string($col['Field']);
}
$columnsinfo[$col['Field']] = $col;
}
$dates = array("date","datetime","time");
$int = array("int","decimal");
$implcols = array();
foreach($columns as $col){
if(in_array($columnsinfo[$col]['Type'], $dates)){
$implcols[] = "(CASE WHEN (UNIX_TIMESTAMP(`$col`)=0 || `$col` IS NULL) THEN '[$col EMPTY]' ELSE `$col` END)";
}else{
list($type, $rest) = explode("(",$columnsinfo[$col]['Type']);
if(in_array($columnsinfo[$col]['Type'], $dates)){
$implcols[] = "(CASE WHEN ( `$col`=0 || `$col` IS NULL ) THEN '[$col EMPTY]' ELSE `$col` END)";
}else{
$implcols[] = "(CASE WHEN ( `$col`='' || `$col` IS NULL ) THEN '[$col EMPTY]' ELSE `$col` END)";
}
}
}
$keyslive = array();
//echo "SELECT $key SHA1(CONCAT_WS('',".implode(",", $columns).")) as compare FROM $table"; exit;
$q = "SELECT $key as `key`, SHA1(CONCAT_WS('',".implode(", ",$implcols).")) as compare FROM $table";
?>