I need to import a large file of csv data into MySQL, and when I attempted to use MySQL's unix_timestamp function to import the dates, about half of the records didn't make it.
As far as I can tell, the datetime values are formatted with either a single first "month" digit or two of them, and the same goes with the day of the month (e.g. 6/6/2014 3:48PM vs. 12/16/2014 3:48PM) This throws off the import completely (well about half of the records won't import).
I'm trying to convert this into a unix_timestamp.
Now I know I could write a script with a regex to fix something like this, but I am wondering is there a simpler way to do a mass import like this? For the record, I am using my text editor to write the sql statements from the csv as "insert into" statements. This is where I tried to use date formatting but it seems to only accept one format.
Any way to do this with such a minor difference in input?
Actually, despite my comment, something like this might work:
COALESCE(STR_TO_DATE(val, "formatcandidate1")
, STR_TO_DATE(val, "formatcandidate2")
, STR_TO_DATE(val, "formatcandidate3")
, STR_TO_DATE(val, "formatcandidate4")
, [etc...]
) AS dateVal
There are online tools to do this kind of stuff
reports.zoho.com is one of them
In this tool you can import data applying a specific date format and skip the other rows.
and you can do the same for all the type of formats that are present in your file
and finally you can export the data with same date format for all the data
ask me any doubts if you have any regarding this :)
Related
I have a large dataset with employees' time entries. The current date format is MM/dd/yyyy. However, I need to convert all the dates into yyyy-MM-dd format.
I have tried the following:
Update human_resources.timekeeping
Set Actual_Date = str_to_date(Actual_Date,'%d-%m-%Y');
Got the errror messsage Error Code: 1411. Incorrect datetime value: '' for function str_to_date.
My SQL version is 5.7.18-log.
I tried to view SQL mode using SELECT ##sql_mode; and I got NO ENGINE SUBSTITUTION.
I have tried to retrieve the value like shown below and it was working fine.
Converting varchar mm/dd/yy to date format yyyy-mm-dd
However, updating the data would not work. I need to update the actual records, not insert new records.
Hope someone can help me regarding this. Thank you in advance!
EDIT: The data type for Actual_Date is VARCHAR.
Apologies if my explanation may be a bit confusing. But I am using this data set to display and filter time entries in a gridview. When I am filtering dates, say for example (01/15/2022-01/25/2022), data from 2021 is also being displayed. When I tried to manually change the format of some of my data in sql to yyyy-MM-dd, my code seemed to be working fine. The problem is there are a lot of data in this table, which is why manually updating the format is impossible. What is the first thing that I need to do? I'm sorry this is all still a bit confusing for me.
My apologies if you have already taken the following things into consideration but I thought them worth mentioning.
Given that you say this is a "large dataset" I assume this is a table that is currently in use. Does the existing application rely on the Actual_Date being in that string format? Does it rely on a fixed number of columns in the table? Some poorly written applications can be very brittle when it comes to changing underlying data structure.
You may want to consider creating a copy of the current table, modifying the structure of the copy, and replacing the original with a view with the same columns and formats as the original. This way you get improved data but reduce risk to existing application.
In the title and first line of your question you state that the current format is MM/dd/yyyy
Update human_resources.timekeeping Set Actual_Date = str_to_date(Actual_Date,'%m/%d/%Y');
Your separator is / not -
%d-%m-%Y >> %d/%m/%Y
I am working with Mysql workbench.
I have a huge database in a csv, that contains among other things, 3 columns with different formats of dates.
To be able of loading this csv file into my database, I have to set the 3 date columns as text, otherwise, it doesn't upload them properly.
Here an example of my data:
inDate, outDAte
19-01-10, 02-02-10
04-01-11 12:02, 2011-01-11 11:31
29-01-11 6:57, 29-03-2010
30-03-10, 01-04-2010
2012-12-03 05:39:27.040, 12-12-12 17:04
2012-12-04 13:47:01.040, 29-11-12
I want to homogenize them and to make 2 columns of each one of those one only with "date" and other only with "time".
I have tried working with "regular expressions" and with "case".
When I used "reg expressions" gave me nulls and with "case" gave me "truncated incorrect value".
I have tried to find something similar situations in the web. I have found that people got similar issues but with two date formats not with so many different formats as I do:
Convert varchar column to date in mysql at database level
Converting multiple datetime formats to single mysql datetime format
Format date in SELECT * query.
I am really new in this and I do not know how to write so many exceptions in mysql.
Load the CSV into a temporary table; massage the values in that table; finally copy to the 'real' table.
Have 2 columns in that table for each date; one for the raw value coming from the CSV; the other being a DATETIME(3) (or whatever the sanitized version will be).
Do one of these for each distinctly different format:
UPDATE tmp SET inDate = ...
WHERE raw_inDate REGEXP '...';
The WHERE may need things like AND LENGTH(inDate) = 8 and other ways to test other than REGEXP.
SUBSTRING_INDEX(inDate, '-', ...) may be a handy function for splitting up a date.
But, really, I would rather write the code in Perl or some other real programming language.
I am trying to import a CSV file with date information in the format MMM-YY (e.g., Jan-92). My ultimate goal is to have the information in a date format in SQL. The format doesn't matter that much to me, but I was thinking something like dd-mm-yy (e.g., 16-01-92). I've tried a lot and looked around the forum, but can't figure it out.
Right now, I am loading the data from the CSV into a column "period_month" as a VARCHAR field.
Then, my conversion code looks like
UPDATE PERIODS
SET period_month = DATE(str_to_date(period_month,'%M-%Y'));
I end up with a field of VARCHAR type in the format yyyy-mm-dd, with the dd field set to '00'. (e.g., 1992-01-00)
I really just want this to be in a date format that I can export to another program like R for analysis.
Any help appreciated.
try
UPDATE PERIODS
SET period_month =DATE(str_to_date('Jan-92','%M-%Y')+1)
or
UPDATE PERIODS
SET period_month =DATE(str_to_date('period_month','%M-%Y')+1)
use date_format function of mysql query
I use Workbench to query database at work. We have a field which indicates company size and has the following options :
1-9
10-49
50-99
100-499
500+
When I export the results containing this field in Excel(which I use for analysis), 1-9 becomes 9-Jan, when I change the format of the cell to text, it becomes 42013. Similarly, 10-49 becomes Oct-50 and in text - 18537. Is there a way to avoid this?
I know this may seem trivial but I take a download of the results every couple of hours or so, and currently, I use the Replace function in Excel to fix this which is a time cost. Also, adding manual intervention increases the probability of error which I want to minimize. I would ideally like the result to export as 1-9, as it exists in the database, based on which the analytical model is built to take input.
I would appreciate any help or pointers on how to fix this issue.
Thanks!
You are not saying how you are bringing the data into Excel. The simplest method is to bring the column in as "text". You can do this when you are importing the data into Excel, by setting the column type to "text".
Alternatively, when you create the output file, you can prepend the value with a single quote or some other character:
select concat('''', company_size)
select concat('_', company_size)
Appreciate the help! I used to export the results in CSV which caused the problem I think. I exported them in XML and that solved it, the fields appear as they exist in the database. Thanks a lot. The concatenation would work as well!
I have a csv file that has a date field in a format like (among other fields):
17DEC2009
When I do a mysqlimport, the other fields are imported properly, but this field remains 0000-00-00 00:00:00
How can I import this date properly? Do I have to run a sed/awk command on the file first to put it into a proper format? If so, what would that be like? Does the fact that the month is spelled out instead of a number matter?
STR_TO_DATE() enables you to convert a string to a proper DATE within the query. It expects the date string, and a format string.
Check the examples in the manual entry to figure out the correct format.
I think it should be along the lines of %d%b%Y (However the %b is supposed to produce Strings like Dec instead of DEC so you will have to try out whether it works).
I had this issue in the past. What I had to do was to utilize LOAD DATA and set the appropriate expression here -
[SET col_name = expr,...]
http://dev.mysql.com/doc/refman/5.1/en/load-data.html
Here is the approach I took to solve similar problem. My use case was bit complex with so many columns, but making here simple to present the solution.
I have Persons table with (Id int autogen, name varchar(100),DOB date), and few million of data(name,DOB) needs to be populated from CSV file.
Created additional column in persons table with name like (varchar_DOB varchar(25)).
Imported data using mysqlimport utility into columns(name,varchar_DOB).
Executed update query that updated DOB column using str_to_date(varchar_DOB,'format') function.
Now, I have expected data populated DOB column.
The same logic could be applied in doing even other kind of data formatting like double,time_stamp etc.