I am trying to load data from a csv file that is still in excel. So far this is the statement that I have as my sql query:
LOAD DATA LOCAL INFILE 'C:\\Documents and Settings\\J03299\\Desktop\\TMETER.csv'
INTO TABLE edata
COLUMNS TERMINATED BY ',' ENCLOSED BY "" LINES TERMINATED BY '\n'
(Year,Month,Day,MJD,xpiles,xstacks,Utilites);
The file is called TMETER and it has 7 columns. It has 366 rows. I am able to only read the first row and only the first four columns(till MJD) but everything else is null after that. Secondly in the second row it puts all the columns from my file (TMETER.csv) in row 124 into the first column of the second row in my edata table. I am confused as to
Why doesn't it read the data from column piles,stacks ,utilites? (mind you the column names in the csv file are weird and not the same as my edata table e.g in database table it is piles while in actual csv file it is x(piles), stacks in table but y(stacks) in csv file. Mysql doesn't not allow me to create etable names with this format so I had to improvise. Could this be why it is not reading and mapping from the csv file to the table in mysql?
Why is my statement putting the first row in my csv file in the first row in mysql table but then skipping all the down to row 124 and then inserting all columns from csv file into first column of mysql database table?
Sorry my English is not good.
Any assistance will be greatly appreciated.
When you run the command, it should give a message like Records: 1 Deleted: 0 Skipped: 0 Warnings: 0. If there are any warnings, type SHOW WARNINGS to see what they were.
This sounds like behavior I sometimes get when the line ending is wrong. Try opening your document in a code editor and checking to make sure your lines actually end with \n. If that's inconvenient, you could also just try dropping the table and reimporting with LINES TERMINATED BY '\r' or LINES TERMINATED BY '\r\n' and see if that fixes it.
Regarding field names: This command ignores field names in your text file. All it does is match the first column to the first field indicated in parentheses (in your case, Year) the second column to the second field in parentheses (Month), and so on. If you have field names at the top of your file, you should skip over them by adding IGNORE 1 LINES just before the parentheses with the list of fields.
Related
I am using the following command from the MySQL command line to try and import a csv file into one of my tables:
LOAD DATA INFILE 'file path' INTO TABLE table
FIELDS TERMINATED BY ','
LINES TERMINATED BY '\n'
IGNORE 1 LINES
(BASEID, BIGID, StartDate, EndDate)
Some of my rows have no value for EndDate which I want to be represented as NULL values in the database.
Unfortunately when I execute the above command I get the following error:
for column 'EndDate' at row 141lue:
If I remove the rows with blank cells the command works, so it is clearly the blank values for EndDate which are causing the problem.
I have also tried changing my csv file so that the blank cells say NULL or \N. I have also tried the following command instead but I still get the same error message:
LOAD DATA INFILE 'file path' INTO TABLE Table
FIELDS TERMINATED BY ','
LINES TERMINATED BY '\n'
IGNORE 1 LINES
(BASEID, BIGID, StartDate, #EndDate)
SET EndDate = nullif(#EndDate, ' ')
How can I load csv files which have some blank values? The suggest solutions I have seen on other posts don't seem to work as outlined above.
Is the issue that the value for the end date is missing, or that the column itself is missing? These are not the same thing. For the former case, I think LOAD DATA should be able to handle this, assuming that the target column for the end date can tolerate missing/null values.
What I suspect here is that some of your input lines look like this:
1,1,'2020-10-03'
That is, there is no fourth column present at all. If this be the case, then the most prudent thing to do here might be to run a simple regex over your input CSV flat file to fix these missing fourth column edge cases. You may try:
Find: ^([^,]+,[^,]+,'[^,]+')$
Replace: $1,
This would turn the sample line above into:
1,1,'2020-10-03',
Now, the date value is still missing, but at least LOAD DATA should detect that the line has four columns, instead of just three.
I'm new to Mysql and am using it to make use of several CSV files I have that are very large (some have over a million rows). I'm on Win7-64 Ultimate. I have installed MySql Workbench v. 6.3.6 build 511 64 bit. I read a similar question however I cannot comment since I am new. I am getting a different error anyway.
I have set up a database called crash0715, and created a table called driver_old with five columns. The first column is a report number (set up as INT(20)) that will be keyed to other files. It contains some duplicates depending upon the data in the other columns. The next four columns contain numeric data that is either 1 or 2 digits.
I set up the report_number column as INT(20), primary key, not null.
The other 4 were set up as INT or INT(2)
When I tried to import a little over 1 million rows in a 5-column CSV file (named do.csv in my c:\ root) via the GUI, the program hung. I had let it run over 12 hours and my task manager showed the program was using 25% cpu.
I next tried the command line. After switching to the database, I used
LOAD DATA LOCAL INFILE 'c:/do.csv' INTO TABLE driver_old FIELDS TERMINATED BY ',' ENCLOSED BY '"' LINES TERMINATED BY '\n';
I had removed the header row from the CSV before trying both imports.
I got the following message:
QUERY OK, 111 rows affected, 65535 warnings <3.97 sec> Records: 1070145 Deleted: 0 Skipped: 1070034 Warnings: 2273755
I read the first few lines of SHOW WARNINGS and they were as follows:
1264 Out of range value for column 'report_number' for row 1.
1261 Row 1 doesn't contain data for all columns
These two repeated for all of the other lines.
There was also a
1062 Duplicate entry '123456789' for key 'primary' (123456789 is a representative value)
It also reoccurred with the other two codes.
The CSV file has no blanks on the first column, however there are a few in the other ones.
Any idea what I'm doing wrong here?
i solved this by save and export sql insert statement
I would use bigint insted of int!
Inserting ignore or replace may help with duplicate primary key values!
LOAD DATA LOCAL INFILE 'c:/do.csv' ignore/replace INTO TABLE driver_old FIELDS TERMINATED BY ',' ENCLOSED BY '"' LINES TERMINATED BY '\n';
I cannot comment on this question,but it would be great if you could post an url to a picture showing few lines from csv file and code how you created table and inserted data ! That would be very helpful for answering the question!
I have now successfully imported the 1045767 records. As suggested by another member here, I imported a small 100 row file that gave the same errors. I then opened the csv in Libre Office and saved it. I was able to import it OK.
The problem was the spreadsheet program, GS-Calc. When saving csv files, it gives three options: UTF-8, UTF-16, and ANSI/OEM/ISO. I had initially saved it as UTF-8 and it returned the error.
I saved it as ANSI/OEM/ISO and it was able to be imported OK. I hope this helps others with large csv files in the future.
i change the separator default in mysql by comma
Im trying to import a text file containing:
http://pastebin.com/qhzrq3M7
Into my database using the command
Load data local infile 'C:/Users/Gary/Desktop/XML/jobs.txt'
INTO Table jobs
fields terminated by '\t';
But I keep getting the error Row 1-13 doesn't contain data for all columns
Make sure the last field of each row ends with \t. Alternatively, use LINES TERMINATED BY
LOAD DATA LOCAL INFILE 'C:/Users/Gary/Desktop/XML/jobs.txt' INTO TABLE jobs COLUMNS TERMINATED BY '\t' OPTIONALLY ENCLOSED BY '"' LINES TERMINATED BY '\r';
\r is a carriage return character, similar to the newline character (i.e. \n)
I faced same issue. How i fixed the issue:
Try to open the CSV file using Notepad++ (text editor)
I've seen a blank line at the end of my file, I've deleted it.
-- Hurrah, it resolved my issue.
Below URL also can help you out to resolve the issue.
http://www.thoughtspot.com/blog/5-magic-fixes-most-common-csv-file-problems
If you're on Windows, make sure to use the LINES TERMINATED BY \r\n as explained by the mariadb docs
sounds like load data local infile expects to see a value for each column.
You can edit the file by hand (to delete those rows -- could be blank lines), or you can create a temp table, insert the rows into a single column, and write a mysql command to split the rows on tab and insert the values into the target table
Make sure there are no "\"s at the end of any field. In the csv viewed as text this would look like "\," which is obviously a no-no, since that comma will be ignored so you won't have enough columns.
(This primarily applies when you don't have field encasings like quotes around each field.)
I'm doing a lad data infile in MySQL through MySQL Workbench. I'm pretty new to SQL in general, so this may be a simple fix, but I can't get it to work. It is throwing a a 1261 Error (doesn't contain data for all columns) on the last row, but the last row (like the rest of the CSV) doesn't have any blank or null values.
I've looked around for help and read the manual, but everything I've seen has been about dealing with null values.
I exported the CSV from Excel, to the extent that maters.
The code I'm using to import is (I've changed the field, file, and table names to be more generic):
load data infile '/temp/filename.csv'
into table table1
fields terminated by ","
lines terminated by '\r'
ignore 1 lines
(Col1,Col2,Col3,Col4,Col5,col6,col7,Col8,Col9);
The first two columns are varchar and char, respectively with the remaining columns all formatted as double.
Here's the last few lines of the csv file:
364,6001.009JR,43.96,0,0,0,0,0,0
364,6001.900FM,0,0,0,0,0,0,0
364,6001.900JR,0,0,0,0,0,0,0
The only thing I can think of is that I'm supposed to have some signal after the last line to indicate that the file is finished, but I haven't found anything to indicate what that would be.
Any help would be appreciated
When I've had similar errors, it's because there were unexpected newlines inside my data (a newline in one row would look like two too-short rows, upon import).
I am trying to do a load data infile to a database. The only issue I have is I am getting a Error 1261. I was getting an incorrect datetime value error earlier but i solved that with the code in the load data infile below (set date_time = ). My problem now is it says that I don't have enough data for all columns. I know that you are supposed to name the columns after the name of the table but I can't seem to get it to work.
There is one table and it has 15 columns. The first column is the primary key, the other fourteen are regular columns.
Here is the load file statement:
load data infile 'c:/proj/test.csv' into table base (#var1,event,failure,ue,mc,mn,cell,durat,cause,ne,ims,hier,hier3,hier32)
set date_time = STR_TO_DATE(#var1, '%Y%m%d %H%i%s')
;
Additional notes: pk column is called dataId and is an INT
It is auto increment.
Here is the data from the csv file:
2013-03-20 14:55:22,4098,1,21060800,344,930,4,1000,0,11B,344930000000011,4809532081614990000,8226896360947470000,1150444940909480000
Try this
load data infile 'c:/proj/test.csv' into table base (#var1,event,failure,ue,mc,mn,cell,durat,cause,ne,ims,hier,hier3,hier32)
set date_time = STR_TO_DATE(#var1, '%Y%m%d %H%i%s')
character set latin1
fields terminated by '\t' enclosed by '' escaped by '\\'
lines terminated by '\n' starting by ''
ignore 1 lines;
Take a look at here
I also had met the similar problems.
The error message is:
load data infile 'L:/new_ncbi' into table ncbi
fields terminated by '\t'
lines terminated by '\r\n' 1973 row(s) affected, 3 warning(s):
1261 Row 1629 doesn't contain data for all columns
1261 Row 1630 doesn't contain data for all columns
1261 Row 1630 doesn't contain data for all columns
Records: 1973 Deleted: 0 Skipped: 0 Warnings: 3 0.281 sec
so I come back to see the data what I load.
I find at line 1639-1630 in the file, I find this problem:
Sphingomonas phage PAU NC_019521 "Sphingomonas paucimobilis
" species
yes, as you see.
The two line would like to be one line, but it is not.
By the way, I state that my data is stored by a excel file.
While I need to handle my data, I transfer my data from excel file to a normal file.
One line data in excel file will be two just because this line my contain a spacial character like CRLF or others.
So I suggest you can copy your data from csv to a normal file and check whether having similar problems.
Maybe my English is bad, but I still hope be helpful.