I am exporting data from sql table to CSV file. Few of my columns in the table has "Comma(,)" separated data, while loading the same into CSV file, data has been splitted into two columns.
Example
data in Sql table
ename desig Industry
Roy PM Business,Analyst
Rem PL Marketting and Production
King PM Marketting, Analyst
while exporting the same data to CSv File it is coming in this way
ename desig Industry
Roy PM Business Analyst
Rem PL Marketting and Production
King PM Marketting Analyst
Since this is CSV format, it is delimiting after comma and taking Analyst as another column instead of same with Industry column.
My required output in CSV File
ename desig Industry
Roy PM Business,Analyst
Rem PL Marketting and Production
King PM Marketting, Analyst
my FaltFileConnectionManager Settings are below
in General tab
Header Row Delimiter {CR}-{LF}
Columns Tab
Row Delimiter {CR}-{LF}
Column Delimiter Comma{,}
I changed these setting , but still facing the same issue.
EDIT
Since it appears from new information that you are not using a CSV (comma separated values) file, but are instead using a pipe delimited file, the issue appears to be that whatever is processing beyond this file is using both | and , as delimiters and therefore delimiting in the middle of the last field where there is an internal comma. Since you don't want to use the industry standard where there are embedded comma's in a field and use a text qualifier on that field, then I am not sure what I can suggest without more definition about what you are using to test the file, what is processing the file after you are done etc.
Update your question with more information and I will refine my answer. First guess without new information is that since you are using the .CSV name on your file that the comma handling is automatic by whatever is processing downstream.
/EDIT
Change the Text Qualifier to " (the double quotation mark), this should qualify text fields with "" to follow the standard practice for CSV files of quoting strings so that imbedded commas don't cause a field break.
Related
I am dumping some csv data into mysql with this query
LOAD DATA LOCAL INFILE 'path/LYRIC.csv' INTO TABLE LYRIC CHARACTER SET euckr FIELDS TERMINATED BY '|';
When I did this, I can see follow logs from console.
...
[2017-09-13 11:24:10] ' for column 'SONG_ID' at row 3
[2017-09-13 11:24:10] [01000][1261] Row 3 doesn't contain data for all columns
[2017-09-13 11:24:10] [01000][1261] Row 3 doesn't contain data for all columns
...
I think csv got some line feed as a column data so it breaks all parsing process.
A single record in csv looks like ...
000001|2014-11-17 18:10:00|2014-11-17 18:10:00|If I were your September
I''d whisper a sunset to fly through
And if I were your September
|0|dba|asass|2014-11-17 18:10:00||||2014-11-17 18:10:00
So LOAD DATA pushes line 1 as a record and then try line 2 and so on, even if this is a single data.
How can I fix it? Should I request different type of the file to the client?
P.S. I am so new with this csv work.
Multiline fieds in csv should be surrounded with double quotes, like this:
000001|2014-11-17 18:10:00|2014-11-17 18:10:00|"If I were your September
I''d whisper a sunset to fly through
And if I were your September
"|0|dba|asass|2014-11-17 18:10:00||||2014-11-17 18:10:00
And any double quote inside that field should be escaped with another double quote.
Of course, the parser has to support (and maybe be instructed to use) multiline fields.
We have a table dumped from a database as a plain text file. There is a field called Address in that table. Usually when we see the information of that field inside database it is like:
12A StreetName CityName
or
13G StreetName, CityName
Now that comma in the second situation is causing the problem. Since data is exported as a text file which is also comma delimited file, we know the comma will be treated as delimiter.
So when I open that text file in Notepad, it looks like this:
previous_field_info, 13G StreetName\, CityName, next_field_info
Then later if I want to open this text file in excel it splits the address field into two columns (for those have commas in the address) and that makes the information in the rest fields shifted. What I see in excel now becomes like this:
previous_field_info | 12A StreetName CityName | next_field_info
previous_field_info | 13G StreetName\ | CityName | next_field_info
So my question is how can I make excel think \, is a whole symbol not just a slash and a comma? The current dummy solution is I do a find and replace to remove \, but what I want is open that up directly in excel using Text Import Wizard
The reason I do things like this (seems like a big detour) is because I don't have access to that database and the only way to get information is asking the company who maintains the database to send recurring files to me (ideally in csv but now they are sending txt file)
Hope my question is clear and thanks for the help in advance.
I’m importing a SQL view to SSIS using the Flat File Connection Manager. One of my columns in SQL has comma(s) in it. (123 Main St, Boston, MA) . When I import the data to SSIS, the commas within the column are being treated as delimiters, and my column is being broken into several columns. I have done a lot of research online, and have followed some workarounds which aren't working for me.
In SQL Server, I added double quotes around the values that have comma(s) in it.
' "'+CAST(a.Address as varchar(100))+'" '
So, 123 Main St, Boston, MA now reads “123 Main St, Boston, MA”
Then in my SSIS Flat File Connection Manager,
In the General tab:
Text Qualifier is set to “
Header Row Delimiter is set to {CR}-{LF}
In the columns tab:
Row delimiter is set to {LF}
Column delimiter is set to Comma {,}
And in the advanced Tab, all of my columns have the Text Qualified set to True.
After all of this, my column with commas in it, is still being separated into multiple columns. Am I missing a step? How can I get the SSIS package to treat my address column as one column and not break it out to several columns?
EDIT: Just to add more specifics. I am pulling from a SQL view that has double quotes around any field that has commas in it. I am then emailing that file and opening it in MS Excel. When I open it the file it read as follows:
123 Main St Boston MA" " (In three cells)
And I need it to read as
123 Main St, Boston, MA (in one cell)
Have a look of this - Commas within CSV Data
If there is a comma in a column then that column should be surrounded
by a single quote or double quote. Then if inside that column there is
a single or double quote it should have an escape charter before it,
usually a \
Example format of CSV
ID - address - name
1, "Some Address, Some Street, 10452", 'David O\'Brian'
Change every comma values with another unique delimiter which values haven't any of the characters inside,like : vertical bar ( | )
Change column delimiter to this new delimiter , and set text qualifier with double quote ( " )
You can automate the replace process using a Script Task before Dataflow Task for replacing delimiters. You can use replace script form here.
Also have a look of these resources.
Fixing comma problem in CSV file in SSIS
How to handle extra comma inside double quotes while processing a CSV file in SSIS
I ended up recreating the package, using the same parameters that are listed in my question. I also replaced this
' "'+CAST(a.Address as varchar(100))+'" '
with this in my SQL view
a.Address
And it now runs as desired. Not sure what was going on there. Thanks to everyone for their comments and suggestions.
I need to export a result set from a SQL Server stored procedure to a csv file. One of the fields being exported is a notes field which can contain quotes and carriage return/line feeds.
I'm using the SSIS data flow task to get the result set from the sproc and then to a flat file destination.
The problem I'm having is how to deal with the carriage return/line feeds. With the row delimiter being {CR/LF} it starts a new row when it encounters this in the notes field. I'm viewing the output with the preview when creating the flat file destination.
The database notes fields is datatype NVARCHAR(MAX).
I'm also having the same problem when exporting record details to an SSRS report. The notes fields are not persisting the carriage return/line feeds resulting in garbled bunch of text.
Any help would be much appreciated. Been at this for hours.
Thanks
Change field datatype to text or ntext.
You can also do double substitution:
Replace CR and LF with 2 unique character combinations in SP
Replace these char sets with CR and LF in SSIS/SSRS.
I have a database built from CSV files loaded from an external source. For some reason, an ID number in many of the tables is loaded into the CSV / database encased in single quotes - here's a sample line:
"'010010'","MARSHALL MEDICAL CENTER NORTH","8000 ALABAMA HIGHWAY 69","","","GUNTERSVILLE","AL","35976","MARSHALL","2565718000","Acute Care Hospitals","Government - Hospital District or Authority","Yes"
Is there any SQL I can run on the already-established database to strip these single quotes, or do I have to parse every CSV file and re-import?
I believe the following would do it (test it first):
UPDATE U
SET YourID = REPLACE(YourID, '''', '')
FROM MyTable AS U
WHERE YourID LIKE '''%'''
If it works right, do a full backup before running it in production.