I am taking data from a csv file and throwing it all into a tempory table, so everything is in string format.
So even date fields are in string format, so I need to convert date from string to a date. All dates are in this format 28/02/2013
I used STR_TO_DATE for this, but I am having a problem.
Here is a snippet of my code.
INSERT INTO `invoice` (`DueDate`)
SELECT
STR_TO_DATE('','%d/%m/%Y')
FROM `upload_invoice`
There are of course more fields than this, but I am concentrating on the field that doesn't work.
Using this command if a date is invalid it should put in a null, but instead of a null being put in, it generates an error instead.
#1411 - Incorrect datetime value: '' for function str_to_date
I understand what the error means. It means it is getting an empty field instead of a properly formatted date, but after reading the documentation it should not be throwing an error, but it should inserting a null.
However if I use the SELECT statement without the INSERT it works.
I could do the following line which actually works to a point
IF(`DueDate`!='',STR_TO_DATE(`DueDate`,'%d/%m/%Y'),null) as `DueDate`
So STR_TO_DATE doesn't run if the field is empty. Now this works, but it can't check for a date which is not valid eg if a date was ASDFADFAS.
So then I tried
IF(TO_DAY(STR_TO_DATE(`DueDate`,'%d/%m/%Y') IS NOT NULL),STR_TO_DATE(`DueDate`,'%d/%m/%Y'),null) as `DueDate`
But this still gives the #1411 error on the if statement.
So my question is why isn't STR_TO_DATE not returning NULL on an incorrect date? I should not be getting the #1411 error.
This is not an exact duplicate of the other question. Also there was not a satisfactory answer. I solved this a while and I have added my solution, which is actually a better solution that was given in the other post, so I think because of my better answer this should stay.
An option you can try:
mysql> SELECT VERSION();
+-----------+
| VERSION() |
+-----------+
| 5.7.19 |
+-----------+
1 row in set (0.00 sec)
mysql> DROP TABLE IF EXISTS `upload_invoice`, `invoice`;
Query OK, 0 rows affected (0.00 sec)
mysql> CREATE TABLE IF NOT EXISTS `invoice` (
-> `id` BIGINT UNSIGNED NOT NULL AUTO_INCREMENT PRIMARY KEY,
-> `DueDate` DATE
-> );
Query OK, 0 rows affected (0.00 sec)
mysql> CREATE TABLE IF NOT EXISTS `upload_invoice` (
-> `DueDate` VARCHAR(10)
-> );
Query OK, 0 rows affected (0.00 sec)
mysql> INSERT INTO `upload_invoice`
-> (`DueDate`)
-> VALUES
-> ('ASDFADFAS'), (NULL), (''),
-> ('28/02/2001'), ('30/02/2001');
Query OK, 5 rows affected (0.01 sec)
Records: 5 Duplicates: 0 Warnings: 0
mysql> INSERT INTO `invoice`
-> SELECT
-> NULL,
-> IF(`DueDate` REGEXP '[[:digit:]]{2}/[[:digit:]]{2}/[[:digit:]]{4}' AND
-> UNIX_TIMESTAMP(
-> STR_TO_DATE(`DueDate`, '%d/%m/%Y')
-> ) > 0,
-> STR_TO_DATE(`DueDate`, '%d/%m/%Y'),
-> NULL)
-> FROM `upload_invoice`;
Query OK, 5 rows affected (0.00 sec)
Records: 5 Duplicates: 0 Warnings: 0
mysql> SELECT `id`, `DueDate`
-> FROM `invoice`;
+----+------------+
| id | DueDate |
+----+------------+
| 1 | NULL |
| 2 | NULL |
| 3 | NULL |
| 4 | 2001-02-28 |
| 5 | NULL |
+----+------------+
5 rows in set (0.00 sec)
See db-fiddle.
I forgot I posted this question, but I solved this problem a while ago like this
IF(`{$date}`!='',STR_TO_DATE(`{$date}`,'%d/%m/%Y'),null) as `{$date}`
So because the line is long and confusing I made a function like this
protected function strDate($date){
return "IF(`{$date}`!='',STR_TO_DATE(`{$date}`,'%d/%m/%Y'),null) as `{$date}`";
}
INSERT INTO `invoice` (`DueDate`)
SELECT
{$this->strDate('DueDate')}
FROM `upload_invoice`
I really forgot I posted this question. It seems like an eternity away, but this is how I solved the issue.
Related
I have a mysql table which has a data structure as follows,
create table data(
....
name char(40) NULL,
...
)
But I could insert names which has characters more than 40 in to name field. Can someone explain what is the actual meaning of char(40)?
You cannot insert a string of more than 40 characters in a column defined with the type CHAR(40).
If you run MySQL in strict mode, you will get an error if you try to insert a longer string.
mysql> create table mytable ( c char(40) );
Query OK, 0 rows affected (0.01 sec)
mysql> insert into mytable (c) values ('Now is the time for all good men to come to the aid of their country.');
ERROR 1406 (22001): Data too long for column 'c' at row 1
If you run MySQL in non-strict mode, the insert will succeed, but only the first 40 characters of your string is stored in the column. The characters beyond 40 are lost, and you get no error.
mysql> set sql_mode='';
Query OK, 0 rows affected (0.00 sec)
mysql> insert into mytable (c) values ('Now is the time for all good men to come to the aid of their country.');
Query OK, 1 row affected, 1 warning (0.01 sec)
mysql> show warnings;
+---------+------+----------------------------------------+
| Level | Code | Message |
+---------+------+----------------------------------------+
| Warning | 1265 | Data truncated for column 'c' at row 1 |
+---------+------+----------------------------------------+
1 row in set (0.00 sec)
mysql> select c from mytable;
+------------------------------------------+
| c |
+------------------------------------------+
| Now is the time for all good men to come |
+------------------------------------------+
1 row in set (0.00 sec)
I recommend operating MySQL in strict mode (strict mode is the default since MySQL 5.7). I would prefer to get an error instead of losing data.
I want to show rows that have updated_at more than 3 hours ago. MySQL seems to be completely ignoring the ORDER BY clause. Any idea why?
Edit: as pointed out by Sebastian, this only occurs in certain timezones, like GMT+5 or GMT+8.
mysql> SET time_zone='+08:00';
Query OK, 0 rows affected (0.00 sec)
mysql> CREATE DATABASE test1; USE test1;
Query OK, 1 row affected (0.01 sec)
Database changed
mysql> CREATE TABLE `boxes` (
-> `box_id` int unsigned NOT NULL AUTO_INCREMENT,
-> `updated_at` timestamp NULL DEFAULT NULL ON UPDATE CURRENT_TIMESTAMP,
-> PRIMARY KEY (`box_id`)
-> ) ENGINE=InnoDB;
Query OK, 0 rows affected (0.01 sec)
mysql> INSERT INTO `boxes` (`box_id`, `updated_at`) VALUES
-> (1, '2020-08-22 05:25:35'),
-> (2, '2020-08-26 18:49:05'),
-> (3, '2020-08-23 03:28:30'),
-> (4, '2020-08-23 03:32:55');
Query OK, 4 rows affected (0.00 sec)
Records: 4 Duplicates: 0 Warnings: 0
mysql> SELECT NOW();
+---------------------+
| NOW() |
+---------------------+
| 2020-08-26 20:49:59 |
+---------------------+
1 row in set (0.00 sec)
mysql> SELECT b.box_id, updated_at, (b.updated_at < NOW() - INTERVAL 3 HOUR) AS more_than_3hr
-> FROM boxes b
-> ORDER BY more_than_3hr DESC;
+--------+---------------------+---------------+
| box_id | updated_at | more_than_3hr |
+--------+---------------------+---------------+
| 1 | 2020-08-22 05:25:35 | 1 |
| 2 | 2020-08-26 18:49:05 | 0 | <--- WHY IS THIS HERE???
| 3 | 2020-08-23 03:28:30 | 1 |
| 4 | 2020-08-23 03:32:55 | 1 |
+--------+---------------------+---------------+
4 rows in set (0.00 sec)
Expectation: the rows with "1" should show up first.
Actual results: ORDER BY is ignored, and the resultset is sorted by primary key
I have a hunch it has something to do with MySQL storing timestamps as UTC and displaying them in the current timezone. My current timezone is GMT+8. However, it still doesn't make sense -- I am sorting the results based on the aliased expression, and the expression's value is clearly shown in the resultset.
MySQL version 8.0.21.
I also tried moving the expression to the ORDER BY clause, and the results are the same.
I don't know why but it compares wrong timezones in the background and thus values at the end are correct, but comparisons are invalid (for specific timezones).
When you query a TIMESTAMP value, MySQL converts the UTC value back to
your connection’s time zone. Note that this conversion does not take
place for other temporal data types such as DATETIME.
https://www.mysqltutorial.org/mysql-timestamp.aspx/
Changing type from TIMESTAMP to DATETIME fixes problem.
Other solution may be casting to the decimal number.
SELECT b.box_id, updated_at, FORMAT((b.updated_at < NOW() - INTERVAL 3 HOUR),0) AS more_than_3hr
FROM boxes b
ORDER BY more_than_3hr DESC;
From the documentation:
https://dev.mysql.com/doc/refman/8.0/en/user-variables.html
HAVING, GROUP BY, and ORDER BY, when referring to a variable that is assigned a value in the select expression list do not work as expected because the expression is evaluated on the client and thus can use stale column values from a previous row.
Basically, you can't use a variable name you created with "AS" in your sorting.
The solution is to use the verbose statement you used for the AS in sorting. Yeah, it's verbose. 🤷♂️ It is what it is.
Case
In our MySql database the data is stored in combined json-strings like this:
| ID | DATA |
| 100 | {var1str: "sometxt", var2double: 0,01, var3integer: 1, var4str: "another text"} |
| 101 | {var3integer: 5, var2double: 2,05, var1str: "txt", var4str: "more text"} |
Problem
Most of the DATA-fields hold over 2500 variables. The order of variables in the DATA-string is random (as shown in above example). Right now we only know how to extract data with the following querie:
select
ID,
json_extract(DATA,'var1str'),
json_extract(DATA,'var2double'),
FROM table
With this querie, only the values of var1str and var2double will be returned as result. Values of variable 3 and 4 are ignored. There is no overview of what possible variables are hiding in the data fields.
With almost 60.000 entries and over 3.000 possible unique variable names, I would like to create a query that loops through all of the 60.000 DATA-fields and extracts every unique variable name that is found in there.
Solution?
The querie I am looking for would give the following result:
var1str
var2double
var3integer
var4str
My knowledge of MySql is very limited. Any direction given to get to this solution is much appreciated.
What version of MySQL are you using?.
From MySQL 8.0.4 and later JSON_TABLE function is supported and can be useful in this case.
mysql> SELECT VERSION();
+-----------+
| VERSION() |
+-----------+
| 8.0.11 |
+-----------+
1 row in set (0.00 sec)
mysql> DROP TABLE IF EXISTS `table`;
Query OK, 0 rows affected (0.09 sec)
mysql> CREATE TABLE IF NOT EXISTS `table` (
-> `ID` BIGINT UNSIGNED AUTO_INCREMENT PRIMARY KEY,
-> `DATA` JSON NOT NULL
-> ) AUTO_INCREMENT=100;
Query OK, 0 rows affected (0.00 sec)
mysql> INSERT INTO `table`
-> (`DATA`)
-> VALUES
-> ('{"var1str": "sometxt", "var2double": 0.01, "var3integer": 1, "var4str": "another text"}'),
-> ('{"var3integer": 5, "var2double": 2.05, "var1str": "txt", "var4str": "more text"}');
Query OK, 2 rows affected (0.00 sec)
Records: 2 Duplicates: 0 Warnings: 0
mysql> SELECT
-> DISTINCT `der`.`key`
-> FROM
-> `table`,
-> JSON_TABLE(
-> JSON_KEYS(`DATA`), '$[*]'
-> COLUMNS(
-> `key` VARCHAR(64) PATH "$"
-> )
-> ) `der`;
+-------------+
| key |
+-------------+
| var1str |
| var4str |
| var2double |
| var3integer |
+-------------+
4 rows in set (0.01 sec)
Be aware of the Bug #90610 ERROR 1142 (42000) when using JSON_TABLE.
I have a database that has an array of data stored in a JSON column. I need to find all values that have a null value at a particular position in the JSON array. While pulling out the data with JSON_EXTRACT seemed trivial, none of my comparisons to null have worked, all of them claiming the value is null.
Here is the example code that should work as far as I can tell:
SELECT JSON_EXTRACT(`COLUMNS_HEADERS`, '$[1]') , (JSON_EXTRACT(`COLUMNS_HEADERS`, '$[1]') is null)
FROM ate.readings_columns_new;
The first few rows of my results table look like this:
null | 0
"INTERNALTEMPERATURE" | 0
"INPUT_VOLTAGE" | 0
null | 0
null | 0
"AH1" | 0
I have tried every comparison I can think of, and they all result in a 0:
(JSON_EXTRACT(`COLUMNS_HEADERS`, '$[1]') is null)
(JSON_EXTRACT(`COLUMNS_HEADERS`, '$[1]') <=> null)
ISNULL(JSON_EXTRACT(`COLUMNS_HEADERS`, '$[1]'))
(JSON_EXTRACT(`COLUMNS_HEADERS`, '$[1]') <=> 'null')
Is there some key to comparing null values pulled from a JSON_EXTRACT?
SELECT
JSON_EXTRACT(`COLUMNS_HEADERS`, '$[1]'),
(JSON_EXTRACT(`COLUMNS_HEADERS`, '$[1]') = CAST('null' AS JSON))
FROM ate.readings_columns_new;
or
SELECT
JSON_EXTRACT(`COLUMNS_HEADERS`, '$[1]'),
(JSON_TYPE(JSON_EXTRACT(`COLUMNS_HEADERS`, '$[1]')) = 'NULL')
FROM ate.readings_columns_new;
See the docs for JSON_TYPE.
A bit of a belated answer but I just hit this problem and couldn't find anything reasonably documented. The solution I ended ended up using was the json_type function as 'abl' pointed out above.
The trick was to compare with the string 'NULL' not null or NULL.
As a test throw the following into a mysql prompt and play around with the values
(if using phpMyAdmin don't forget to check 'show this query here again' and 'retain query box' - the universe is frustrating enough without losing edits..)
set #a='{"a":3,"b":null}';
select if(json_type(json_extract(#a,'$.b')) = 'NULL',1,0);
I ended up with the following.
mysql> set #a='{"a":3,"b":null}';
Query OK, 0 rows affected (0.00 sec)
mysql> select if(json_type(json_extract(#a,'$.b')) = 'NULL',1,0);
+----------------------------------------------------+
| if(json_type(json_extract(#a,'$.b')) = 'NULL',1,0) |
+----------------------------------------------------+
| 1 |
+----------------------------------------------------+
1 row in set (0.00 sec)
mysql> set #a='{"a":3,"b":1}';
Query OK, 0 rows affected (0.00 sec)
mysql> select if(json_type(json_extract(#a,'$.b')) = 'NULL',1,0);
+----------------------------------------------------+
| if(json_type(json_extract(#a,'$.b')) = 'NULL',1,0) |
+----------------------------------------------------+
| 0 |
+----------------------------------------------------+
1 row in set (0.00 sec)
As the bare bones of a stored procedure - which is what I needed it for - using the 'if' statements rather than the if() function.
drop procedure if exists test;
delimiter $$
create procedure test(in x json)
begin
if json_type(json_extract(x,'$.b')) = 'NULL' then
select 1;
else
select 0;
end if;
end$$
delimiter;
mysql> call test('{"a":3,"b":1}');
+---+
| 0 |
+---+
| 0 |
+---+
1 row in set (0.00 sec)
Query OK, 0 rows affected (0.00 sec)
mysql> call test('{"a":3,"b":null}');
+---+
| 1 |
+---+
| 1 |
+---+
1 row in set (0.00 sec)
Query OK, 0 rows affected (0.00 sec)
Well I had a suspicion but I found a workaround that confirms that a JSON null value is not the same as a MySQL null value.
I tried various methods to get a similar null value but the only one that works is to extract a null JSON value from an array like the value I'm attempting to check against:
SELECT JSON_EXTRACT(`COLUMNS_HEADERS`, '$[1]') , (JSON_EXTRACT(`COLUMNS_HEADERS`, '$[1]') = JSON_EXTRACT('[null]', '$[0]'))
FROM ate.readings_columns_new;
This seems like bad form, but was the only way I could get a value that evaluated as equal to the null values in my array.
Another trick is MySQL's NULLIF function
SELECT COLUMNS_HEADERS->>"$[1]", NULLIF(COLUMNS_HEADERS->>"$[1]",'null') IS NULL)
(I'm also using ->> which is an alias for JSON_UNQUOTE(JSON_EXTRACT())
That way querying a column containing {"id":1},{"id":2},{"id":null} & {"name":4} for the JSON path $.id will return 1,2,NULL,NULL instead of 1,2,null,NULL
I'm using the Play Framework 2.1.2, the JDBC MySQL Connector and Scala 2.10. The following query is my problem:
DB.withConnection { implicit connection =>
SQL("""SELECT SUM(r.dayFrequency)
FROM relationships AS r
WHERE r.id = {id}
AND
(r.date BETWEEN {from} AND {to})""").on(
'id -> id,
'from -> from,
'to -> to).as(scalar[Int](bigDecimalToInt).single)
}
It raises this exception:
Execution exception[[RuntimeException: UnexpectedNullableFound(ColumnName(.SUM(r.dayFrequency),Some(SUM(r.dayFrequency))))]]
The console logs the following query:
SELECT SUM(r.dayFrequency)
FROM relationships AS r
WHERE r.id = 26180
AND
(r.date BETWEEN 2014-08-04 12:00:00.0 AND 2014-08-04 12:00:00.0)
If I run this query on my MySQL Workbench it returns null, which confirms the exception. But with this change in the query it works:
(r.date BETWEEN '2014-08-04' AND '2014-08-04')
For the conversion of Joda DateTime, I use this piece of code: Joda DateTime Field on Play Framework 2.0's Anorm
and frequency and date field looks like the following:
date DATE NOT NULL,
dayFrequency INT
Can anyone help with this problem? Seems that something is wrong with the conversion.
EDIT after first POST below:
From the view I receive date strings like this 2014-08-04 and I convert them into Joda DateTime in my controller to compare them to other and use them in MySQL queries like this:
private def clientDateStringToTimestamp(date: String) = {
val Array(year, month, day) = date.split("-")
new DateTime(year.toInt, month.toInt, day.toInt, 12, 0, 0).getMillis()
}
new DateTime(clientDateStringToTimestamp("2014-08-04"))
For the MySQL queries I want to compare only the date part not the time part.
So i did a simple experiment in mysql:
mysql> create table t (v int);
Query OK, 0 rows affected (0.01 sec)
mysql> insert into t values (null);
Query OK, 1 row affected (0.00 sec)
mysql> select sum(v) from t;
+--------+
| sum(v) |
+--------+
| NULL |
+--------+
1 row in set (0.01 sec)
mysql> insert into t values (1);
Query OK, 1 row affected (0.00 sec)
mysql> select sum(v) from t;
+--------+
| sum(v) |
+--------+
| 1 |
+--------+
1 row in set (0.00 sec)
mysql> update t set v = NULL;
Query OK, 1 row affected (0.00 sec)
Rows matched: 2 Changed: 1 Warnings: 0
mysql> select sum(v) from t;
+--------+
| sum(v) |
+--------+
| NULL |
+--------+
1 row in set (0.00 sec)
So this tells us that a summing nulls to nulls gives us null but summing nulls to numbers gives us numbers.
I suspect that your first query (r.date BETWEEN 2014-08-04 12:00:00.0 AND 2014-08-04 12:00:00.0) returns just rows with null dayFrequency values, where the second query (r.date BETWEEN '2014-08-04' AND '2014-08-04'), which is offset 12 hours earlier returns at least one non-null frequency. So since null is possible, you will have to use scalar[Option[Int]] for the sum, then turn it to 0 with getOrElse. A better way, if you can is to make the dayFrequency column in the database NOT NULL DEFAULT 0. Then it will give you a 0, and you can sum away
Also related, direct support for Joda temporal types in Anorm: https://github.com/playframework/playframework/commit/bdbbbe90822a6fb150c7044e68b33e2e52a7323d