I am trying to import a rather large (520k rows) .CSV file into a SQL Server 2012 table. The file uses a delimiter of ;.
Please do not edit my delimiter. It is ";" I know that may seem strange, but that is what they used. It is not just a semicolon.
I don't think the delimiter is the issue because I replaced it with a tab and it seemed to be okay. When I try importing the file, I get a text truncation error, but I set the column to 255 just to be sure it had plenty of room.
Even when I delete the row, the next row causes the error. I don't see any offending characters in the data, so I am at a loss as to what the issue is.
I ended up using the EOL Conversion and selected Windows format in Notepad++ and then created a script to import the data.
Related
I have a flat file with almost 300 columns that needs to be imported in SQL server. It seems that if I use SSIS it can't read the csv file if I mark it as unicode. It seems that it looses ability to recognize cr and lf.
The specified header or data row delimiter "{CR}{LF}" is not found
after scanning 524288 bytes of the file "...\contact.csv". Do you want
to continue scanning this file?
what am I doing wrong?
EDIT
Based on comments, it seems I need to clarify - yes I did check that {CR}{LF} is present at the end of each line, and that it's set as a row delimiter in the connector.
The problem is with the "unicode" checkbox. If I uncheck it, the file is read fine. If I check it, it doesn't find {CR}{LF} any more.
It also doesn't matter what encoding I set on the file as that only effects the default "code page" selection.
ok, after a while I found an answer.
The unicode checkbox is still not working, but if you can go to the advanced section of the flat file manager and set your string columns to unicode. It's kind of tedious, and I don't know what I would do if I had 200 columns, but for my small data set it worked.
I'm trying to import the following file into a MySQL Db:
https://drive.google.com/drive/folders/1WbRdNgqVre3wN4DpJZ-08jtGkJtCDJNQ?usp=sharing
Using the "data import wizard" on MySql Workbench, for some reason I'm getting "218\223 lines imported successfully", whereas the file contains close to 100K.
I tried looking for special chars around lines 210-230, also removing all of them, but still the same happens.
The file is a CSV of Microsoft Bing's geo locations, used in Microsoft Advertising campaigns, downloaded from Microsoft's website (using an ad account there).
I've been googling, reading, StackOverflowing, playing with the file and different import options...
I tried cutting the file into small bits, and the newly created file was completely corrupt somehow...
Encoding seems to be UTF-8, line breaks all "\n". I tried changing them all into "\r\n" using notepad++, but still the same happens.
File opens normally in Excel, looks normal, passes CSVlint.io...
The only weird thing is that the file contains quotes on some of the values but not on the rest (e.g. line 219. Yeah I know it sounds like this would be the problem, but I removed it, and all the rest of the lines with quotes, and it still happens... Also tried loading with ENCLOSED BY ", see below).
I also tried using SQL statements to import:
LOAD DATA LOCAL INFILE 'c:\\Users\\Gilad\\Downloads\\GeoLocations.csv'
INTO TABLE aw_geo_map_bmsl
FIELDS TERMINATED BY ','
(tried also with: ENCLOSED BY '"')
LINES TERMINATED BY '/n'
IGNORE 1 ROWS;
(had to add OPT_LOCAL_INFILE=1 to the connection on Advanced for MySQL Workbench to be allowed access to local files on my computer)
This gives 0 rows affected.
Help?
Epilogue: In the end I just gave up on all these import wizards and did it the old "make your SQL statements from Excel" way.
I imported the CSV data into Excel. Watch out: in this case I found I needed to use a data import wizard from Excel (but that one worked perfectly) to be able to change the encoding to UTF, which Excel 2010 chose as "windows" which was wrong.
After processing the data a bit to my liking, I used the following Excel code:
=CONCATENATE("INSERT INTO aw_geo_map_bmsl (`Location Id`,Name,`Canonical Name`,`Location Type`,Status,`Adwords Location Id`)
VALUES (",
A2,
",""",B2,"""",
",""",C2,"""",
",""",D2,"""",
",""",E2,"""",
",",F2,");")
to generate INSERT statements for every line, then copy-pasted and pasted only values, then pasted into an editor, removed additional quotes that Excel adds, and ran it in MySQL Workbench, which runs it line by line (takes some time), and you can see the progress.
Saved me hours of unsuccessfully playing around with "automatic tools" which fail for unknown reasons and don't give proper logs ootb.
Warning: do NOT do this for unsanitized code as it's vulnerable to SQL injection. In this case it was data from Microsoft so I know it's fine.
I have problem with mysql database. I can't import a database from my friend.
I need some help.
SET SQL_MODE="NO_AUTO_VALUE_ON_ZERO";
SET time_zone = "+00:00";
ERROR:
Unexpected beginning of statement. (near "phpMyAdmin" at position 0)
Unrecognized statement type. (near "SQL" at position 11)
#1064 - Something is wrong in your syntax obok 'phpMyAdmin SQL Dump
SET SQL_MODE="NO_AUTO_VALUE_ON_ZERO"' w linii 1
There´s nothing wrong with you syntax, but probably with your file:
most likely the file was edited and the text-editor (of course Windows notepad.exe) was too clever and added a BOM on saving.
Remove the first 3 bytes (HEX: EF BB BF), save the file without it (either use a hex editor or use PSPad and switch format to UNIX), and the importer should have no problem anymore.
The BOM fools the importer, the first - gets eaten and the importer no longer recognizes the first comment as such.
Wikipedia about BOM:
File comparison (w/o BOM)
I encountered exactly the same problem. Apparently, you use a version of phpMyAdmin which has bugs in the import module (in my case it was phpMyAdmin 4.5.5.1 packaged in Wamp 3.0.4). More precisely, it interprets comments (valid syntax with space after --) as SQL code. This is the case at the beginning of a dump created by phpMyAdmin: it typically starts with
-- phpMyAdmin SQL Dump
which explains your error message.
The import module of phpMyAdmin 4.5.5.1 was not able to parse escaped single quotes either (see https://github.com/phpmyadmin/phpmyadmin/issues/11721).
There are many possible workarounds to this problem:
Update phpMyAdmin
Use another tool to import your DB dump, for example MySQL Command Line or MySQL workbench
Less advisable: execute the contents of the .sql file as a query in your current version of phpMyAdmin (it has fewer bugs)
Less advisable: strip all comments from your .sql file
windows notepad and other editor, change encoding of file.
for change it to utf-8 open your file with "notepad++" and use Encoding menu then select UTF-8
now save your file
AM using the copy method for cpying the .csv file into the cassandra tables..
But am getting records mismatch error..
Record 41(Line 41) has mismatched number of records (85 instead of 82)
This is happening for all the .csv files & all the .csv files are system generated..
Any work around for this error..?
Based on your error message, it sounds like the copy command is working for you, until record 41. What are you using as a delimiter? The default delimiter for the COPY command is a comma, and I'll bet that your data has some additional commas in it on line 41.
A few options:
Edit your data and remove the extra commas.
Alter your .csv file to encapsulate the values of all of your fields in double-quotes, as COPY's default QUOTE value is ". This will allow you to leave the in-text commas.
Alter your .csv file to delimit with pipes | instead of a comma, and set the COPY command's DELIMITER option to |.
Try using either the Cassandra bulk loader or json2sstable utility to import your data. I've never used them, but I would bet you'll have similar problems if you have commas in your data set.
(SQL Server 2008)
So here's my task ..
I need to export query results to file, and then import that file using SSIS to another DB.
Specific to the task, the data contains every awkward unicode character you can think of, so delimiting with commas, pipes etc is out of the question.
Here are the options SSMS gives me for export format:
Column Aligned
Comma/Tab/Space delimited
Custom delimiter
And here are the options SSIS gives me for a flat file data source:
Delimited (custom)
Fixed Width
Ragged Right
So given that a delimiter character is out of the question ... I cannot see another method that both SSMS & SSIS agree on.
Such as fixed width ?
Seems strange that the 2 closely related MS products have such different options.
Or have I missed something here ?
Any advice appreciated !!
It seems you need to try out different combination of options while creating delimited flat file(for your exported query result).
Try setting Code page to UTF-8 with and without Unicode. Also use Text qualifier as " or any of your choice which you thought might work. Also try using different option for column delimiter.
Once you are able to create delimited file then you have to apply same setting on file while importing to another DB.