Column headers are importing, but I don't know where they are coming from as they are not included in the csv file as headers. I have tried unchecking them in the initial "columns to import" box, unsuccessfully. My imports keep failing.
[columns being added at import level for csvfailure of csv import](https://i.stack.imgur.com/Y3Mfu.png)
columns being autogenerated and not part of the data set
Related
I am trying to load a big file which basically is a json format flat file from my local drive to SQL Server by using SSIS. It's a one line file and I don't need to specify columns and rows as I am going to parse it as soon as it's in SQL Server by OPENJSON.
but when I tried to create Flat File Source in Visual Studio SSIS, I was not able to do that as even I used 'fixed width' format according to the solution here: import large flat file with very long string as SSIS package, as the max width seems to be 32000, while the json file could be much bigger.
here are my settings:
There are other options of loading the data by t-sql like OPENROWSET but we have SQL Server instance installed on another server rather than the same one we are doing our dev work. So there are some security limits between them.
So just wondering if this is the limitation of Flat File Source in SSIS or I didn't do it right?
You're likely looking for the Import Column transformation. https://learn.microsoft.com/en-us/sql/integration-services/data-flow/transformations/import-column-transformation?view=sql-server-ver15
Define a Data Flow as OLE Source -> Import Column -> OLE Destination.
OLE Source
Really, any source but this is the easiest to reproduce
SELECT 'C:\curl\output\source_data.txt' AS SourceFilePath;
That will add a column named SourceFilePath with a single row.
Import Column
Reference the article on Import Column Transformation but the summary is
Check the column that will provide the path
Add a column to the Import Column Collection to hold the file content. Change the data type to DT_TEXT/DT_NTEXT depending on your unicode-ness and note the LineageID value
Click back to Import Column Input and find the column name. Scroll down to the Custom Properties and use the LineageID above for FileDataColumnID where it says 0. Otherwise, you have an error of
The "Import Column.Outputs[Import Column Output].Columns[FileContent]" is not referenced by any input column. Each output column must be referenced by exactly one input column.
OLE DB Destination
Any data sink will do but the important thing will be to map our column from the previous step to a n/varchar(max) in the database.
I am completely new to SQL. I asked that question 2 weeks ago but got no reply, so I am afraid I have to ask again.
I am trying to import a file into mySQl using Workbench. When I import the full file into a new table, this works perfectly. But what I am trying to do is to import the file into an existing table, specifying column matching. I use Table Import Wizard and get the message that the file was imported in 0.25s but that 0 rows were imported. I thought it my be because of different data type between the file and the existing table, so I tried to simplify my import to just one column, even so, it doesn't work. No error, but nothing imported. However, when configurating the import settings, I see the data in the file perfectly.
What can it be ?
I have added some screen shots:
Situation 1, screenshot 1-2-3: i import the file as it is, without touching the field names, imports all data perfectly
Situation 2, screenshot 4-5: i try to import some fields only, and force them into an existing table, therefore I have to indicate to the wizard where it fields has to go. 0 rows imported
I tried removing the header line, doesn't make any difference, still 0 imported
(if you do not see the links, I have pasted them again in comment below)
I have been trying to import CSV into a process node which doesnt care about rest of the fields (dynamic in count of fields as well) but for 2 or 3. But in those other fields I have date fields that is being imported in a wrong way. The field gets automatically assigned as Date20. while it is actually datetime. Also another field that is supposed to be a 16 digit character is being imported as number and is getting truncated (shows in the form 9.401153E15). After processing, this node exports the data into CSV and I see all these errors there.
I checked few links like http://www2.sas.com/proceedings/sugi30/038-30.pdf which is relevent to the topic but irrelevent in the context. How can I solve this?
PROC IMPORT for CSV simply generates datastep code, so I would recommend simply copying the datastep code into your program (it should be visible in the log) and editing it to reflect your needs.
I'm importing data from a CSV file into an Access table. The number is something like
-21000000 (-2.1E7). I'm using TransferText to do the import.
DoCmd.TransferText acImportDelim, , "matching report temp", Source_folder & "\" & Source
In the "matching report temp" table, the field is set up as Double. The import generates a type conversion failure. However, when I open the CSV file in Excel and copy the offending row, I can successfully use Paste Append to add it to the table manually - so the number doesn't exceed the capacity of the field.
As far as I can tell, this only happens with large negative numbers. Smaller numbers, and positive numbers of the same magnitude seem to import fine. I know I can specify an Import Specification in TransferText, but I don't see a way to set the field type to Double.
How can I get around this problem? I don't want to have to manually track down the import errors and append them by hand.
Don't let TransferText create the Access table for you. Create the Access destination table first and assign the field types you need.
When you run TransferText, the CSV data will be appended to that existing table. It should work without error as long as you choose compatible data types for the Access fields.
I examined your CSV file. The header row is troublesome because some field names are missing:
Reporting Unit,,$ Dollars,Offset Unit,,$ Dollars,Variance
That seemed to be a complication for DoCmd.TransferText. So I imported manually from the Access UI. And that gave me an import error on row 49 with the 3rd and 6th columns. In Access, the destination fields were both created as Long Integer. However the values for those fields in the CSV row are 2262169190 and -2262169190 ... both beyond the capacity of Access' Long Integer, -2,147,483,648 to 2,147,483,647.
As a cheap workaround, I selected Text for the data type of those 2 columns when I imported. That allowed the import to work without error. Once you get the data into Access successfully, you could cast those (string) values to a numeric type when you need to use them.
It also worked when I chose Double as the type for those 2 columns. It sounds like that's what you actually want.
If you want to get the import working with DoCmd.TransferText, I think you'll need to create an import specification. If possible, I would also first modify the header line of the CSV file so that all the fields have names.
I actually tested that approach without altering the CSV file. After creating an import specification in the Access UI (see screen capture below), this TransferText operation worked in Access 2007 without import errors.
DoCmd.TransferText acImportDelim, _
"IC_Y1301_Specification", _
"IC_Y1301_LD10279_F25210001", _
"C:\Users\hans\Downloads\IC_Y1301_LD10279_F25210001.CSV", _
True
I'm trying to manipulate a .csv file to remove text at the beginning of the file before the data starts. The file contains a fixed text string followed by a date field, which will change from file to file and then another fixed text string.
eg.
"Text1"
"------"
"date"
"Text2"
"data column1","data column2" etc
How can I remove this text so i can then use SSIS to import the data to the SQL database?
If I understand the question correctly you want to skip the first line of the file. When you set up the Flat file connection there is an option in the format section of the properties Header rows to skip:. You can set this to the number of rows you need to skip and the file should import. If you have a an actual header row you will need to skip that as well and then map the columns manually.
Within the SSIS import configuration, is there not an option to tell SSIS that the "first row has headers," or something roughly similar? That's what I've used when importing through SSIS, at least.