I have a file of 700 000 rows and 90 columns and I had converted it to csv to upload it (using qualifier: "). The import worked but when I look at the table in SQL I see few text has moved to the next column.
In the actual file there is column called Comments and it has all type of characters(like *,|,| etc). Can this be the reason?
I tried doing the same import from .xlsb but it throws an error:
Failed to start
project(Microsoft.DataTransformationServices.VsIntegration)
If your fields have the same characters s the separator then of course the whole thing will break.
If you create the CSV with , as the separator and quotes (") around the fields, you should be able to handle it in SSIS using these instructions by adding " as the text qualifier.
I haven't tested this myself, but I would assume it works just fine.
Related
I'm trying to import the following file into a MySQL Db:
https://drive.google.com/drive/folders/1WbRdNgqVre3wN4DpJZ-08jtGkJtCDJNQ?usp=sharing
Using the "data import wizard" on MySql Workbench, for some reason I'm getting "218\223 lines imported successfully", whereas the file contains close to 100K.
I tried looking for special chars around lines 210-230, also removing all of them, but still the same happens.
The file is a CSV of Microsoft Bing's geo locations, used in Microsoft Advertising campaigns, downloaded from Microsoft's website (using an ad account there).
I've been googling, reading, StackOverflowing, playing with the file and different import options...
I tried cutting the file into small bits, and the newly created file was completely corrupt somehow...
Encoding seems to be UTF-8, line breaks all "\n". I tried changing them all into "\r\n" using notepad++, but still the same happens.
File opens normally in Excel, looks normal, passes CSVlint.io...
The only weird thing is that the file contains quotes on some of the values but not on the rest (e.g. line 219. Yeah I know it sounds like this would be the problem, but I removed it, and all the rest of the lines with quotes, and it still happens... Also tried loading with ENCLOSED BY ", see below).
I also tried using SQL statements to import:
LOAD DATA LOCAL INFILE 'c:\\Users\\Gilad\\Downloads\\GeoLocations.csv'
INTO TABLE aw_geo_map_bmsl
FIELDS TERMINATED BY ','
(tried also with: ENCLOSED BY '"')
LINES TERMINATED BY '/n'
IGNORE 1 ROWS;
(had to add OPT_LOCAL_INFILE=1 to the connection on Advanced for MySQL Workbench to be allowed access to local files on my computer)
This gives 0 rows affected.
Help?
Epilogue: In the end I just gave up on all these import wizards and did it the old "make your SQL statements from Excel" way.
I imported the CSV data into Excel. Watch out: in this case I found I needed to use a data import wizard from Excel (but that one worked perfectly) to be able to change the encoding to UTF, which Excel 2010 chose as "windows" which was wrong.
After processing the data a bit to my liking, I used the following Excel code:
=CONCATENATE("INSERT INTO aw_geo_map_bmsl (`Location Id`,Name,`Canonical Name`,`Location Type`,Status,`Adwords Location Id`)
VALUES (",
A2,
",""",B2,"""",
",""",C2,"""",
",""",D2,"""",
",""",E2,"""",
",",F2,");")
to generate INSERT statements for every line, then copy-pasted and pasted only values, then pasted into an editor, removed additional quotes that Excel adds, and ran it in MySQL Workbench, which runs it line by line (takes some time), and you can see the progress.
Saved me hours of unsuccessfully playing around with "automatic tools" which fail for unknown reasons and don't give proper logs ootb.
Warning: do NOT do this for unsanitized code as it's vulnerable to SQL injection. In this case it was data from Microsoft so I know it's fine.
When I export the data from excel as a CSV format it encapsulates some data in double quotes.
E.g.
8" becomes "8""". And I believe this operation is trying to get the database to understand the inside quote later on.
but for single quote 8', it keeps the same and this causes problem(see the pic below) while I am importing the csv.
Why not quoting the 8' into "8'" too?
8' becomes ' while importing, while "8'" will result in 8' And not quoting single quote leads some data loss.
question related:
what does quotechar mean in mysql while importing data?
Excel adds extra quotes on CSV export
After doing so many experiments, I finally found a pretty close answer.
Conclusion first:
It is Mysql Workbench's problem. Its import wizard works badly. I test under Navicat for every test data, and Navicat get all things right.
Single quote can caused unexpected behavior.
Test:
By default, Mysql workbench import wizard takes the first row value as column name, while in Navicat, I can configure that.
(All tests files are excel-exported csv utf8 encoding.)
test1:
e.g.: 8'（only 1 record）, without column name.
Mysql: Pop out some unknown error and whatever configuration you change, can't get the original data.
Navicat:works fine.
test2:
e.g.: 8' , only 1 record with column name or have extra records without column name
Mysql: Can handle single quote properly.
Navicat: No problem.
test3:
If single quote exists, for most situation import wizard can't handle double quotes well.
e.g.: Single quote data comes before double quotes data.
Mysql: Fails totally.
Navicat: No problem.
I have a large CSV file (nearly 10,000 rows) and I am trying to upload it on the BigQuery but it gives me this error:
ile-00000000: CSV table references column position 8, but line starting at position:622 contains only 8 columns. (error code: invalid)
Can anyone please tell me a possible to reason to it? I have double checked my Schema and it looks alright.
Thanks
I had this same issue when trying to import a large data set in a csv to a BigQuery table.
The issue turned out to be some ascii control characters (\b, \t, \r, \n) in the data that was written in the csv. When the csv was being sent to BigQuery these characters caused the BiqQuery csv parser to misinterpret the line and break because the data didn't match with the number of columns in the header.
Replacing these characters with a space (to preserve formatting as best as possible) allowed me to import the data without further issues.
The error message suggests that the load job failed because at least one row has fewer columns than the automatically detected schema dictates.
Add
allow_jagged_rows=true
in the options.
I am trying to import a csv file into my mysql database using phpmyadmin but keep getting errors.
Here is how the csv looks:
Then I import like this:
And get the error: "Invalid parameter for CSV import: Fields enclosed by". I have tried to put the columns in quotes " or put a semicolon after each column, but keep getting errors.
Yeah, you have an extra field in there. For instance, with your example line of:
itemId,date,description,amount
,1,2/13/2013,Fabrics,44
the date maps to "description" because of the leading comma, which basically gives an empty (or null, depending on how the import is handled) value to itemId, which doesn't seem to be what you want. Where'd that extra comma come from -- was this an export from some program?
Also, in this case you don't have anything enclosing the fields so you should just be able to leave that value empty, which seems to have worked for you once you got the column count corrected.
I had to remove the first line of the csv (containing the column names) and that solved the issue. Everything got imported properly.
Note, the date field needed reformatting to match SQL's date format yyyy-mm-dd.
How can I load 10,000 rows of test.xls file into mysql db table?
When I use below query it shows this error.
LOAD DATA INFILE 'd:/test.xls' INTO TABLE karmaasolutions.tbl_candidatedetail (candidate_firstname,candidate_lastname);
My primary key is candidateid and has below properties.
The test.xls contains data like below.
I have added rows starting from candidateid 61 because upto 60 there are already candidates in table.
please suggest the solutions.
Export your Excel spreadsheet to CSV format.
Import the CSV file into mysql using a similar command to the one you are currently trying:
LOAD DATA INFILE 'd:/test.csv'
INTO TABLE karmaasolutions.tbl_candidatedetail
(candidate_firstname,candidate_lastname);
To import data from Excel (or any other program that can produce a text file) is very simple using the LOAD DATA command from the MySQL Command prompt.
Save your Excel data as a csv file (In Excel 2007 using Save As) Check
the saved file using a text editor such as Notepad to see what it
actually looks like, i.e. what delimiter was used etc. Start the MySQL
Command Prompt (I’m lazy so I usually do this from the MySQL Query
Browser – Tools – MySQL Command Line Client to avoid having to enter
username and password etc.) Enter this command: LOAD DATA LOCAL INFILE
‘C:\temp\yourfile.csv’ INTO TABLE database.table FIELDS TERMINATED
BY ‘;’ ENCLOSED BY ‘”‘ LINES TERMINATED BY ‘\r\n’ (field1, field2);
[Edit: Make sure to check your single quotes (') and double quotes (")
if you copy and paste this code - it seems WordPress is changing them
into some similar but different characters] Done! Very quick and
simple once you know it :)
Some notes from my own import – may not apply to you if you run a different language version, MySQL version, Excel version etc…
TERMINATED BY – this is why I included step 2. I thought a csv would default to comma separated but at least in my case semicolon was the deafult
ENCLOSED BY – my data was not enclosed by anything so I left this as empty string ”
LINES TERMINATED BY – at first I tried with only ‘\n’ but had to add the ‘\r’ to get rid of a carriage return character being imported into the database
Also make sure that if you do not import into the primary key field/column that it has auto increment on, otherwhise only the first row will be imported
Original Author reference