Each month I load new data to MySQL Database. Columns 'Event', Event Description' and 'Location' are standard columns. The Columns which hold the data for the months are dynamic. That is this month I am reading a file which has months 2019M01, 2019M02 and 2019M03. Next month I will read 2019M02, 2019M03, 2019M04.
Is there any way to easily prepare tables that can read these dynamic .csv files that can take into account the dynamic nature of these data columns ?
Currently for each month I create new tables and then as we move to the next month I drop the previous table and create a new one which has the correct column names for the new month.
Currently my code is as follows:
/*We are in Month January*/;
CREATE TABLE ex_1 (
`Event` TEXT,
`Event Description` TEXT,
`Location` TINYTEXT,
`2019M01` DECIMAL (15,10) NOT NULL, `2019M02` DECIMAL (15,10) NOT NULL, `2019M03` DECIMAL (15,10) NOT NULL
);
LOAD DATA LOCAL INFILE 'C:/Users/BPerei23/Desktop/WORK/Projects/MAPE/MI-STAT/MarketIntelligence_DS-Spine_20190124.csv' REPLACE INTO TABLE ex_1
CHARACTER SET Latin1 FIELDS TERMINATED BY ',' OPTIONALLY ENCLOSED BY '"' LINES TERMINATED BY '\r\n' IGNORE 1 LINES;
INSERT INTO FINAL_TABLE
SELECT * FROM ex_1;
/*We are now in Month February*/;
DROP TABLE ex_1;
CREATE TABLE ex_2 (
`Event` TEXT,
`Event Description` TEXT,
`Location` TINYTEXT,
`2019M02` DECIMAL (15,10) NOT NULL, `2019M03` DECIMAL (15,10) NOT NULL, `2019M04` DECIMAL (15,10) NOT NULL
);
LOAD DATA LOCAL INFILE 'C:/Users/BPerei23/Desktop/WORK/Projects/MAPE/MI-STAT/MarketIntelligence_DS-Spine_20190224.csv' REPLACE INTO TABLE ex_2
CHARACTER SET Latin1 FIELDS TERMINATED BY ',' OPTIONALLY ENCLOSED BY '"' LINES TERMINATED BY '\r\n' IGNORE 1 LINES;
INSERT INTO FINAL_TABLE
SELECT * FROM ex_2;
Related
I have this table:
CREATE TABLE `country` (
`name` VARCHAR(60) NOT NULL,
`code` VARCHAR(3) UNIQUE NOT NULL,
PRIMARY KEY (`code`)
);
As you can see the primary key of this table is the word code
When I try to select a specific code in this table, that is 2 characters long, it cannot find anything.
On the other hand, when I select a 3 characters long code like this:
select * from `country` where `code` = "TZA";
I get the result I want
I searched for my variable in the table (for example the code "AL") and it appears to be registered.
Why is this happening and how could I make it work?
Thank you in advance!
I am importing my data from a csv file that looks like this:
LOAD DATA LOCAL INFILE 'path_to_file\\countries.csv'
INTO TABLE `country`
FIELDS TERMINATED BY ','
LINES TERMINATED BY '\n'
IGNORE 1 ROWS
(`name`, `code`);
I have tried selecting with a space in the end of the code and on the front of it:
select * from `country` where `code` = 'AL ';
select * from `country` where `code` = ' AL';
But they output nothing
The real solution is:
When importing this CSV file you should use:
LOAD DATA LOCAL INFILE 'path_to_file\\countries.csv'
INTO TABLE `country`
FIELDS TERMINATED BY ','
LINES TERMINATED BY '\r\n'
IGNORE 1 ROWS
(`name`, `code`);
Because your lines seems to be terminated the way Windows terminates lines.
I have a CSV file with one column with the next data:
"2015-01-01",
...
...
"2015-03-27"
I created mysql table that way:
CREATE TABLE `my_tbl` ( `id` int(11) NOT NULL AUTO_INCREMENT, `date` DATE NOT NULL, PRIMARY KEY (`id`) );
I am trying to insert data using the next command:
LOAD DATA INFILE '/tmp/myFile.csv' INTO TABLE my_tbl FIELDS TERMINATED BY ',' LINES TERMINATED BY '\n' (#col1) set date=#col1;
The problem: When checking my_tbl I see that all records are '0000-00-00'
What am I doing wrong and how can I fix it?
Did I define the table as it should be (e.g. Maybe it was better to define timestamp etc.)
Spot the difference:
[..snip..] BY '\n' (#col1) set date=#co1;
^---------------^
It appears I should have added FIELDS ENCLOSED BY '\"'. So the LOAD query should be this way:
LOAD DATA INFILE '/tmp/myFile.csv' INTO TABLE my_tbl FIELDS ENCLOSED BY '\"' LINES TERMINATED BY '\n' (#col1) set date=#col1;
Tnx for user #Marc B for the hint
I am trying to import a csv file that is delimited by tabs.
Here is my query
LOAD DATA LOCAL INFILE 'c:/news.csv'
INTO TABLE news
FIELDS TERMINATED BY '\t'
LINES TERMINATED BY '\r'
(url, storyid, title, date, details, category, author);
What happens is only the first column is loaded, (url).
The rest shows NULL. I have tried lines terminated by \n as well. Same result.
Any advice?
Table structure for table `news`
--
CREATE TABLE IF NOT EXISTS `news` (
`url` varchar(62) DEFAULT NULL,
`storyid` int(15) DEFAULT NULL,
`title` varchar(255) DEFAULT NULL,
`date` date DEFAULT NULL,
`details` longtext,
`category` varchar(255) DEFAULT NULL,
`author` varchar(110) DEFAULT NULL
)
It depends on the exact format of your .csv file but for Windows .csv format I always use
LINES TERMINATED BY '\r\n'
also (again depending on the data) try
FIELDS ESCAPED BY '\\' TERMINATED BY '\t' OPTIONALLY ENCLOSED BY '\"'
If you're unsure of the exact nature of the data sometimes it is better to view it in hexadecimal to see how the lines are really terminated. I use Hexedit - http://www.hexedit.com/
Hope this helps.
Dermot
Like I said in the comments you can use '/r/n' for a new line.
However your csv file contains only 1 column, namely a full line of text.
That is probably also why only the first table column is filled and the rest is null.
LOAD DATA LOCAL INFILE 'c:/news.csv'
INTO TABLE news
COLUMNS TERMINATED BY ','
LINES TERMINATED BY '\r\n'
(url, storyid, title, date, details, category, author)
This worked.
Turned out that even though it looks tab separated, it is comma separated. Dermot was right that you need to view it in hexadecimal view to see how it is really deliminated.
I have ASCII files with a static number of characters for each line with no delimiters. I'd like to use LOAD DATA INFILE to import into my table.
Example of file:
USALALABAMA
USARARKANSAS
USFLFLORIDA
The structure for this table:
country Char(2)
state Char(2)
name Varchar(70)
CREATE TABLE `states` (
`country` char(2) COLLATE latin1_general_ci NOT NULL,
`state` char(2) COLLATE latin1_general_ci NOT NULL,
`name` varchar(70) COLLATE latin1_general_ci NOT NULL
) ENGINE=MyISAM DEFAULT CHARSET=latin1_general_ci COLLATE=latin1_general_ci;
Is it possible to specify a start and end position for each column?
According to the documentation, you can load a fixed format file without using a temporary table.
If the FIELDS TERMINATED BY and FIELDS ENCLOSED BY values are both empty (''), a fixed-row (nondelimited) format is used. With fixed-row format, no delimiters are used between fields (but you can still have a line terminator). Instead, column values are read and written using a field width wide enough to hold all values in the field. For TINYINT, SMALLINT, MEDIUMINT, INT, and BIGINT, the field widths are 4, 6, 8, 11, and 20, respectively, no matter what the declared display width is.
The positions are derived from the columns definitions, which in your case match the structure of the file. So you just need to do:
LOAD DATA INFILE 'your_file' INTO TABLE your_table
FIELDS TERMINATED BY ''
LINES TERMINATED BY '\r\n'
SET name = trim(name);
First create a temporary table which you will load all lines into it, then you can load the data from the temporary table into the main table and split to fields using substring
Something like this:
CREATE TEMPORARY TABLE tmp_lines
(countrystring TEXT);
LOAD DATA INFILE 'yourfilegoeshere' INTO TABLE tmp_lines
FIELDS TERMINATED BY ''
LINES TERMINATED BY '\r\n';
INSERT INTO main_table SELECT SUBSTRING(countrystring,1,2), SUBSTRING(countrystring,3, 2), SUBSTRING(countrystring,5) from tmp_lines;
Another way to do this is just assigning a variable and splitting it direct in your load.
LOAD DATA INFILE 'yourfilegoeshere' INTO TABLE main_table
LINES TERMINATED BY '\r\n' (#_var)
set
field1=TRIM(SUBSTR(#_var from 1 for 2)),
field2=TRIM(SUBSTR(#_var from 3 for 2)),
field3=TRIM(SUBSTR(#_var from 5 for 70));
Just be sure not to specify any field separator, otherwise you will have to use more variables, note that I'm using TRIM to clean data in the same statement.
There is a CSV file with the following data:
1;8-25-2010;0:05;210;4
2;8-25-2010;2:45;412;5
3;8-25-2010;3:40;300;3
4;8-25-2010;4:45;226;6
5;8-25-2010;5:20;206;4
6;8-25-2010;5:25;216;3
And there is MySQL Table:
CREATE TABLE IF NOT EXISTS `Schedule` (
`ID` SMALLINT NOT NULL AUTO_INCREMENT,
`Num` INT(10),
`PlannedDate` DATE,
`PlannedTime` TIME NOT NULL,
`resQty` INT(3) NOT NULL,
`stID` VARCHAR(10) NOT NULL,
PRIMARY KEY (`ID`),
FOREIGN KEY `stID` (`stID`) REFERENCES Stands (`stID`)
) ENGINE=MyISAM DEFAULT CHARSET=latin1;
Now I need to fill this table with the data from CSV file. For this I'm using the following code:
TRUNCATE TABLE testDB.Schedule;
LOAD DATA LOCAL INFILE 'C:\\temp\\Input.csv'
INTO TABLE testDB.Schedule FIELDS TERMINATED BY ';' ENCLOSED BY '"'
LINES TERMINATED BY '\r\n' (Num,PlannedDate,PlannedTime,stID,resQty);
But the error message says that "Data truncated for column PlannedDate at row1", ErrorNr. 1265. The same error message for all rows.
The date has the wrong format, it should be 2012-01-19
If you are stuck with that date format (which is not the MySQL default), you can load those dates into a temporary variable and then convert it to a date, like this:
TRUNCATE TABLE testDB.Schedule;
LOAD DATA LOCAL INFILE 'C:\\temp\\Input.csv'
INTO TABLE testDB.Schedule FIELDS TERMINATED BY ';' ENCLOSED BY '"'
LINES TERMINATED BY '\r\n'
(Num,#PlannedDate,PlannedTime,stID,resQty)
SET PlannedDate = STR_TO_DATE(#PlannedDate,'%m-%d-%Y');
in this line:
(Num,PlannedDate,PlannedTime,stID,resQty);
the two last parameters are reversed: stID,resQty
it SHOULD be:
resQty, stID
This is why you are getting a truncation error.