I'm importing a CSV file with dotted german dates into a MySQL database. I want the dates in the CSV to automatically be formatted correctly to the correct data type fields used by MySQL.
I'm using Sequel Pro for the import. I gather I'm supposed to use the STR_TO_DATE function, but I just can't wrap my head around how to use Add Value or Expression in the program.
German date
Here are the dates in the CSV file:
DD.MM.YYYY e.g.: 28.01.1978
MySQL date
Here is what I want to end up with in the database:
YYYY-MM-DD
e.g.: 1978-01-28
Here's what I've tried
I put in STR_TO_DATE('$5', '%d.%m.%Y'); into Add Value or Expression, but this only gives the following error message:
[ERROR in row 1] You have an error in your SQL syntax; check the manual that corresponds to your
MySQL server version for the right syntax to use near '06.04.1997'', '%d.%m.
%Y');,'2KMX','43354997')' at line 2
Any ideas?
You need import the date field in a varchar fields (temp_varchar_field) first, after that, you can use something like:
update table_name set final_date=STR_TO_DATE(temp_varchar_field,'%d.%m.%Y');
You should do something like:
Create a temporary field: alter table table_name add column temp_varchar_field varchar(10);
Import, using Sequel Pro, the CVS file but using the temp_varchar_field for the date.
update table_name set final_date=STR_TO_DATE(temp_varchar_field,'%d.%m.%Y');
Delete the temp field if everything was imported properly. Using: alter table_name drop column temp_varchar_field;
I just got it to work with this piece of SQL-code:
load data local infile 'myfile.csv' into table `mytable`
fields terminated by ','
enclosed by '"'
lines terminated by '\n'
(surname, name, #germandate, telephone, etc)
set birthyear = STR_TO_DATE(#germandate , "%d.%m.%Y")
;
The clue here being the #germandate variable which is turned into the default MySQL date by setting the respective column with STR_TO_DATE(). No hacks needed! :)
It is easier if your CSV import would contain a date as a MySQL string, but it can be done otherwise too:
Step 1:
Define a varchar(10) for your german dates and import the data.
Step 2:
Add another field to your table:
ALTER TABLE `yourtable`
ADD COLUMN `your_mysql_date` DATE NULL;
Step 3:
Move the data:
UPDATE yourtable
SET your_mysql_date = CONCAT(
RIGHT(your_german_date,4),
'-',
MID(your_german_date,4,2),
'-',
LEFT(your_german_date,2)
);
...done!
There might be an easier way to solve this, but this way you have alot of control over the data and the formatting.
Related
I code in the SQL, but I want to change the date format from default of My SQL to different format "DD/MM/YYYY" to use this format to code, but I recieved an error.
More about version of SQL:
SQL: My SQL Workbench 8.0.30 build 2054668
Window 11 Pro
Language: English
So, how can I change the date format or what is the true type of "dmy"? Please help me.
This is code that I used:
set dateformat dmy
And error is:
"Error Code: 1064. You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'dmy' at line 1"
At the moment, we don't really know the reason why you want to change the date format, but I have a couple of assumption:
You want to insert date data from front end into the table but couldn't do so due to the mismatched date format.
You want the end output to show the date format of DD/MM/YYYY instead.
Although, it may look like you need to change the table column date format, there's a way to avoid that operation entirely. However, since you've mentioned changing from YYYY/MM/DD in your title, I'm not sure if your date column is actually DATE datatype because the default MySQL should be YYYY-MM-DD. Nonetheless, I'll address the matter in this answer altogether.
Modify the date format in query and leave the table date column datatype as it is:
From DD\MM\YYYY to MySQL DATE datatype format YYYY-MM-DD
... STR_TO_DATE(data, '%d/%m/%Y')
.. from YYYY-MM-DD to DD/MM/YYYY
... DATE_FORMAT(data, '%d/%m/%Y')
You can use any of that anywhere in a query; whether in SELECT or WHERE.
If "in query" is not what you want and you still want to update the table:
Well, you have two options here:
Directly modify the column datatype then update the value:
ALTER TABLE mytable MODIFY COLUMN date_col VARCHAR(255);
UPDATE mytable SET date_col =DATE_FORMAT(date_col , '%d/%m/%Y');
Or you can add another column, populate the desired date format there and keep the original date column as it is:
ALTER TABLE mytable ADD COLUMN my_date VARCHAR(255);
UPDATE mytable SET my_date =DATE_FORMAT(date_col , '%d/%m/%Y');
this way you have the option to directly use MySQL date functions on the default MySQL date column without the hassle of converting your desired date format into the default before you can use date functions. What I'm saying is something like this:
DAY(mysql_default_dateformat)
is similar to
DAY(STR_TO_DATE(your_dateformat, '%d/%m/%Y'))
which means that you can use DAY() (date function) on the default date format directly without the need to convert what is not default first.
Here's demo fiddle examples
I have a dataset as follows,
Date,Time,ID,Name,Count
01-MAY-2009,00:00,4,Town Hall (West),209
01-MAY-2009,02:00,17,Collins Place (South),28
01-MAY-2009,23:00,18,Collins Place (North),36
For this I have created table with following schema,
CREATE TABLE table_name(date DATE, time TIME, id int, name VARCHAR(50), count int);
And for loading the table from ".csv" file as,
LOAD DATA INFILE '/home/cloudera/dataset.csv'
INTO TABLE table_name
FIELDS TERMINATED BY ','
LINES TERMINATED BY '\n'
(#var1, #var2, id, name, count)
SET date = STR_TO_DATE(#var1, '%d-%b-%Y')
SET time = TIME(#var2, "%H:%i");
But I get an error as,
ERROR 1064 (42000): You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'SET time = TIME(#var2, "%H:%i")' at line 1
I can't understand what the error is. I have tried going through MySQL website and documentation, but can't make out what the correct format is. Can anyone please me. Thanks in advance.
I don't think that you even need to be using the TIME function here. Your current hour:minute string time literals should be good enough, q.v. the documentation:
Be careful about assigning abbreviated values to a TIME column. MySQL interprets abbreviated TIME values with colons as time of the day. That is, '11:12' means '11:12:00', not '00:11:12'
As valid literals, your times would be interpreted as having a zero second component. So, if you just insert those time strings as is, I think it should work. Try the following code:
LOAD DATA INFILE '/home/cloudera/dataset.csv'
INTO TABLE table_name
FIELDS TERMINATED BY ','
LINES TERMINATED BY '\n'
(#var1, Time, id, name, count)
SET date = STR_TO_DATE(#var1, '%d-%b-%Y');
Tim is right in pointing out that you do not need to convert your time data in the load data infile statement.
Just to answer why you get a syntax error: load data infile can only have a single set clause, in which assignments to multiple columns are separated by comma. Your code has multiple set clauses, hence it fails.
Also, the time() function does not have a 2nd parameter for a pattern. The function you need to use is called str_to_date().
So, it should look like as follows:
...
SET date = STR_TO_DATE(#var1, '%d-%b-%Y'), time = TIME(str_to_date(#var2, "%H:%i"));
I have many files, for example: 20170319, 20170320 ...
For each file I have 2 columns, one for username and the other was data.
I've created a table
create table A(user varchar(35), date date, data varchar(35), primary key(user, date));
Then, I want to load those files into database, and use filename as specific date in date field.
Can I still use sth like:
Load data infile '20170320' into table A
The answer is that you cannot do this in MySQL alone, you need to use an external program or script that builds the load data infile statements with the appropriate SET clause derived from the name of the file:
The SET clause can be used to supply values not derived from the input
file. The following statement sets column3 to the current date and
time:
LOAD DATA INFILE 'file.txt'
INTO TABLE t1
(column1, column2)
SET column3 = CURRENT_TIMESTAMP;
The reason for this is that:
load data infile cannot use the file name as an input variable
neither MySQL prepared statements using the prepare statement, nor stored procedures are allowed to use the load data infile statement.
I am trying to load data from a CSV into a database in MySQL workbench. The table I am loading into has an auto increment ID column. I am trying to get the query to recognize that I want it to keep the first column as Null, and I put NULL as the value in the CSV, but I cannot get the SET ... NULL command to recognize the name of the ID column. Here is the SQL I am using:
load data infile 'filenam.csv'
INTO TABLE table_name
fields Terminated By ','
LINES TERMINATED BY ',,'
SET column_name = null
I suspect I am making a simple syntax error that is causing the problem. But I cannot figure out what it is.
If you put NULL as the value in the CSV file then you shouldn't need the "SET column_name = null" in the statement. AFAIK, the SET value should be used to supply values not derived from the input file or to perform calculations to the value before insertion. This statement should work fine since you said you specified NULL in the CSV. However, make sure you specified NULL "properly" according to the documentation. I always use \N in my import files.
LOAD DATA INFILE 'filename.csv'
INTO TABLE table_name
FIELDS TERMINATED BY ','
LINES TERMINATED BY ',,'
Secondly, you can discard the NULL specified in the CSV file by assigning it to a user variable and then specifying the column value with SET. However, you need to specify a column list
LOAD DATA INFILE 'filename.csv'
INTO TABLE table_name (#dummy, column_2, column_3)
FIELDS TERMINATED BY ','
LINES TERMINATED BY ',,'
SET column_name = NULL
I have one other thought based on the MySQL docs and dependent upon how your server is configured. Comment if this does not work and I will provide more options.
http://dev.mysql.com/doc/refman/5.1/en/load-data.html
I have a column in a MySQL table that consists of comma-delimited strings. I would like to convert this column into a set of distinct strings that occur in the column (for any row in the table -- the set includes strings that occur in any row of the table in this column). What is the easiest way to accomplish this?
The solution doesn't need to be pure MySQL. It could involve unix, perl, etc.
You could probably get a quick-and-dirty list of distinct strings from a comma-delimited column using SELECT INTO OUTFILE, sed, and LOAD DATA INFILE.
Basically you want to dump the text data to a flat file, using a comma as the line delimiter so each string will be treated as a separate row when you load it back into the database. Then load the extracted into a new table and select the distinct values.
Off the top of my head, the code would look something like this:
select str
into outfile
'/tmp/your_table_data.txt'
lines terminated by ','
from your_table;
sed -e 's/\\,/,/g' -e 's/,$//' /tmp/your_table_data.txt > /tmp/commadelimited.txt
create table your_table_parsed(str text);
load data infile '/tmp/commadelimited.txt'
ignore into table your_table_parsed
fields terminated by ','
lines terminated by ',';
select distinct str from your_table_parsed;
The way I chose was to run the select mysql command outside of the mysql shell and pipe the results into tr and sort --uniq
mysql my_db [-p -u -h] -se "select my_column from my_table;" | tr ',' '\n' | sort -u
This is pretty simple and seems to give the correct results as far as I can tell.