i have a scenario to create SSIS package Load different excel files in a folder to different SQL table with file name as table name, can any one help me
Thanks
If All Sheets have same columns ,
a)Get List of Execelfile paths and destination server and table name into object variable
b) Get object record through foreach loop
c) Give Excel filepath variable in input expression and server ,table info at destination expression
each file will be loaded to corresponding table
Related
I have a SSIS package - which within a FOR LOOP CONTAINER I look in a particular location, for a particular file format and import it into a database.
This is working fine - when I have two files the contents of both files are being imported.
So I have a Variable Mapping under my ForLoop which records the fully qualified name. What I want to do is when I import the file is I am also recording the file path of where it has come from.
I'm unsure in my dataflow task where I would put that ? Under the data flow I have my source file and a destination.
I tried to have a sql task after the data flow that updated the field in the database with the variable (via Parameter Mapping), but that set the field to the same value for everything (the last file path found) which is not what I'm after.
Any advice would be welcome
In your dataflow task, in between your source and destination add a Derived Column transformation. This will add columns to your dataset with a name and value that you specify. If you reference variables in which you are storing the file name for your loop container, the name of the file being accessed will be appended to an additional column in your dataset. Obviously you need to make sure that this column is present in your destination table.
I'm trying to get all the file names from a folder directory along with their row counts. (Also file size in bytes if possible) I am using Microsoft Visual Studio 2010 Shell. Here's what I've done so far:
I have created a Foreach Loop Container, set the Enumerator to Foreach File Enumerator and Expressions to a variable to the folder I want to loop over. I left the Files section with *.* and asked to retrieve Name Only. I have changed the Variable Mappings to a New Variable called FullFilePath, Container is Package, Value type is String and Value: is blank.
I then added a Data Flow to the Loop. Added a flat file source, row count, and OLE DB Destination. I changed the Flat file Source properties expression to the same Folder Variable in the Foreach Loop Container Expression. I added the Variable RecordCount to the Row Count function (Int32, value 0). The OLE DB Destination creates a new table with the name OLE DB Destination.
The next step is a Execute SQL Task that does and Insert Into DBO.FileData (FileName,RowCount) Values (?,?). I set 2 parameter mappings - 1) Variable Name from the Foreach Loop Container, FullFilePath and Data Type VarChar, 2) Variable from Row Count, RecordCount and Data Type Long.
I then have another Execute SQL Task that drops the table created by the data flow task. The problem is that with all the these step the Package still does not complete. It actually gets hung up and fails on the pre-execute. It says:
Warning: Access is denied. Error: Cannot open the datafile 'FullFilePath' Error: Flat File Source failed the pre-execute phase and returned error code 0xC020200E.
Anything you see I could be doing wrong? Let me know if pictures would help.
So I figured this out finally. In order to loop over all of the files with varying headers and column counts I decided to change the option in the Flat File Source to unselect "File contains headers." Doing this allowed the all the files to have the same #1 Column, which by default is Column 0(the first column in all of my files is some sort of a numeric field or ID). I was able to map this through row count and insert into a SQL table. Then I was able to finish the Foreach Loop and scribe the file name and row count into another SQL table to record the counts. It is however taking a really really really long time, i.e. it has been running for over 14 hours and it has only counted through 13 files. Granted some files are 250K+ rows but I wouldn't think it would take this long.
I have an SSIS package that creates and populates an excel sheet, it creates the sheet from a template file.
I want to check the Excel file by "startdate",if a new product to be added, must this new product be deleted. and then send a warning e-mail
thanks
Is "Start Date" held on a database somewhere? If so you could read this value using an "Execute SQL task" or "Lookup" and store the value as a variable.
A "Conditional split" could then compare the date with that coming from the flat file. If the Start Date in the file is later than your stored value then output this data to a flat file/alternate database.
In a separate data flow task check for records in the flat file/alternate database.
If there are records then send an email with the flat file data as an attachment.
I have a source folder which contains 4 csv files with different no of columns in each of the file. I need to fetch only 3 columns(metadata same this 3 columns in all the 4 files) from each csv and load the columns inside Raw Destination from all the files avaiable in source folder. And Raw destination Output file name has to be like wht the inputfilename we are fetching + time stamp.
And at next level, i need to fetch this output raw as raw source and insert this records into oledb destination . and the destination table also has to be in dynamic.
for example i have 4 csv files called, test1.csv(10 columns). test2.csv(8), test3.csv(6), test4.csv(10) along with time stamps.
all this 4 files has columns position_id, asofdate, sumassured in common, now i want to load only these 3 columns to raw destination. If i load test1.csv then my raw destination outputfile name has to be RW_test1_20120119_222222.RW. similalrly if i load second file its filename as raw destination output..
Thanks
Satish
As always, decompose your problems until you've got it into a something you can manage.
Processing CSVs via queries
Following the two questions and answers below will result in a package with an OLEDB Connection Manager configured to operate on CSVs in the folder #[User::InputFolder]. 3 variables CurrentFileName, InputFolder and Query have been defined with an expression set on Query.
The expression for your #[User::Query] would look like "SELECT position_id, asofdate, sumassured FROM " + #[User::CurrentFileName]
Reference answers
SSIS FlatFile Acces via Jet
SSIS Task for inconsistent column count import?
At this point, your package should resemble the center piece below. Verify you can correctly enumerate all of the CSVs in the folder and the OLEDB query piece works.
RAW files
I'm not an expert on RAW file usage so there may be better ways of interacting with them. This will use the fourth variable, RawFileName. Set an expression on it like #[User::InputFolder] + "RawFile.raw" which would result in the file being written to C:\ssisdata\so\satishkumar\RawFile.raw
My general approach is to have a dataflow with a script task that sends no rows into a RAW File Destination.
Configure your destination as
Access mode: File name from variable
Variable name: User::RawFileName
Write option: Create Always
Process CSVs
The concept here is to append all the data into the RAW file that was created in the initial step.
Your source should already be configured as
OLE DB connection manager: FlatFile
Data access mode: SQL command from variable
Variable name: User::Query
Configure your destination as
Access mode: File name from variable
Variable name: User::RawFileName
Write option: Append
Extract from RAW
At this point, the foreach enumerator has completed and all the data has been loaded into the staging file. Now it is time to consume that and send data on to the destination.
Drag a Raw File Source Transformation onto your data flow. Unsurprisingly, you will configure as
Access mode: File name from variable
Variable name: User::RawFileName
Instead of Simulate destination, wire it up to the correct data destination.
Caveat
Be careful when using an expression with GETDATE/GETUTCDATE to define filenames as they are constantly evaluated. In 2005, we had used FileName_HHMMSS and had issues because processing didn't complete in the same second between the creation of a file and the next task that consumed the file. Instead, I have had better success using a dynamic but fixed starting point and generally, that is the system variable, StartTime #[System::StartTime]
You can use ForEach Loop Container on the Control Flow Diagram to iterate txt and csv files.
I need to add an footer in flat file destinatiion as an row count. am importing data from oledb to flat file, i need the count of the records placed as a footer in flat file.
I would solve this similar to this:
ssis package format in excel
But in this case you can:
Create variable of type int (package scope).
Add Row Count Component in your data flow and in Variable Name put you var.
Create script task that will connect to your txt file and add additonal row with rowcount.
Hope it will help.