I have created SSIS packages which imports Excel files from a folder. The name of the Excel file changes based on months.
e.g.: I:\Test\User_09-05-2016.xlsx or
I:\Test\User_09-06-2016.xlsx etc.....
I want to create SQL jobs to run the packages because I will get a new file every month.
I want to archive the excel file after successful package execution(to keep only one excel file in folder at a time).
How can I automate the process?
Just use a ForEach container within SSIS.
Import each spreadsheet within that folder.
Delete (or move) each spreadsheet after processing.
Alternative approach to FE loop - define path and filename of Excel file on the package run.
If you know rules for naming Excel files, create a string variable Excel_FilePath containing complete path with Expression like [User::Folder Path]+"\\"+[User::Filename]+".xlsx". Then take your Excel file Connection Manager and add the following expression to the ConnectionString property "Provider=Microsoft.Jet.OLEDB.4.0;Data Source=" + #[User::Excel_FilePath] + ";Extended Properties=\"Excel 8.0;HDR=YES\";"
In Package properties - set DelayValidation=true.
More details on this approach with screenshots and exactly your question reviewed.
Related
Used foreachloopcontainer and file system task to create sub folders in main folders
used data flow task and creating dynamically csv file
i need to loop through each sub folder and create csv file
could you please help
Save the filename and Folder path in a variable. Create a flat file connection. In the properties tab, dynamically define the ConnectionString using Expressions.
how to load multiple Excel files using SSIS. I have had Packages in the past where I was looping through multiple Text files in a folder and loading them into SQL server tables.
Create a variable named FileName, set the scope to ImportMultipleExcelFiles, Data type is String.
Add a Foreach Loop Container in the Control Flow Task.
Edit the Foreach Loop Container, in the Collection section change the Enumerator value to Foreach File Enumerator
You need to change the Enumerator configuration as shown below :
Folder: Provide a complete folder path location where all our Excel
source files are stored.
Files: you need to read the Excel files from our source folder, so
enter *.xls in the Files section, this will make sure our SSIS
package will read all available .xls files from the source folder.
Here * indicates that the Excel file name can be anything, but file
extension will be .xls. If we need to read data from a specific Excel
file name then we have to configure it accordingly.
Retrieve File Name: select the Fully Qualified radio button.
Then, create variable mappings for the Foreach Loop container, select the "User::FileName" variable and set the Index value to 0 in the Variable Mappings section.
Add a Data Flow Task inside the Foreach Loop Container.
Right click on the recently added Data Flow task and click on Properties and mark the DelayValidation property to True.
Add an Excel Source in the Data Flow Task and create a new connection to any of the Excel source files.
You have to make the Excel connection dynamic so that it can connect to each Excel file in the source folder. To make the Excel source connection dynamic, right click on Excel Source Connection and then click on Properties.
Expand the Expression Properties, then select the Connection String property and then click on the expression icon, in the expression window :
"Provider=Microsoft.Jet.OLEDB.4.0;Data Source="
+#[User::FileName]+";Extended Properties=\"Excel 8.0;HDR=YES\";"
Finally, execute the SSIS package and see the result.
I want to create a package in SSIS to copy all the file names from 1 folder to excel file?
Create a dataflow task
Open the dataflow task and add a script component as a source
Open the script transformation editor
Add an output column, ex DT_WSTR with size of 255
Loop through your folder and fetch the filenames
Add the filename to the output column you just created
Close the script transformation editor
Add an excel destination
Connect the script compnent to the excel destination
Add a connection to the excel file you want to store the filenames in
Set the mapping accordingly to store your filename values to the file
column
Something like that...
You can test it in BIDS GUI, run it by using dtexec or add it as a job to the sql agent
I have created a package to fetch data from two SQL Server tables, and using merge join combined this data, then stored the result into an Excel destination.
The first time it works fine. The second time it stores repeated data in the Excel file.
How do I overwrite the Excel file rows?
Yes, Possible!
Here is the solution:
First go to your Excel Destination Click to New Button next to Name of Excel Sheet, copy the DML query inside.
Then put an Execute SQL Task into your Control Flow and connect it to your data flow that contains Excel destination. Set the Connection Type To Excel, Set the Connection to your Excel Destination's Excel Connection Manager, go to SQL Statement and type :
Drop TABLE `put the name of the sheet in the excel query you just copied`
Go
finally paste the query after it.
It is all you need to do to solve the problem.
You can refer to this link for a complete info:
http://dwhanalytics.wordpress.com/2011/04/07/ssis-dynamically-generate-excel-tablesheet/
Yes, Possible!
Using SSIS we can solve this problem:
first of all, crate an Excel format file (Structure Format using Excel Connection Manager) at one location as a template file. Then create a copy of that excel file using FILE SYSTEM TASK in another location and make sure that SET OverwriteDestination=True. Finally, using a data flow task, insert data into the new copied file. whenever we want insert data, it will create a copy of the template excel file and then load the data.
Unfortunately the Excel connection manager does not have a setting that allows overwriting the data. You'll need to set up some file manipulation using the File System Task in the Control Flow.
There are several possibilities, here's one of them. You can create a template file (which just contains the sheet with the header) and prior to the Data Flow Transformation a File System Task copies it over the previously exported file.
The File System Task (MSDN)
For Excel it will append data. There is no such option available for overwriting data.
You have to delete and recreate the file through the File System task.
Using a CSV file with flat-file connection manager would serve your purpose of overwriting.
The best solution for me was using File System Tasks to delete and recreate the Excel files from a template.
What I was trying to do was to send every employee a report with Excel attachment in the same format but different data. In a foreach container for each employee, I get the required data, create an Excel file and send a mail with the Excel file attached.
I first:
Create an Excel template (manually)
Create an original Excel file to be used (manually)
Then in the foreach container:
Delete the original file (SSIS File System Task )
Copy the template as the original file (SSIS File System Task)
Get the data from SQL Server and write them to the original file (SSIS Data Flow Task)
Send the mail (SSIS -> SQL Stored Procedure)
I have a script that downloads data from a database into a series of CSV files. After they're downloaded, they must be loaded into an Access database for reporting (I use DoCmd.TransferText, and have a saved text import specification). Every time I run the job that generates the data and downloads into CSV, I usually need to load into a fresh copy of the unpopulated version of the Access database. Is there a way to automate this in a batch script?
In short, I need to be able to:
copy the unpopulated Access file to a new file with the timestamp in the name
load certain CSV files that match a pattern (such as "data_for_reporting_2_20111024_135142.csv") in the directory into the Access file.
I think you can use VBScript to do what you need.
copy the unpopulated Access file to a new file with the timestamp in
the name
FileSystemObject.CopyFile "c:\somefolder\template.mdb", "c:\dest\new.mdb"
See CopyFile Method.
load certain CSV files that match a pattern (such as
"data_for_reporting_2_20111024_135142.csv") in the directory into the
Access file.
You can examine the Files Collection of your CSV folder, determine which of those file names match your target pattern, then run DoCmd.TransferText with each matching file name.
You would run DoCmd.TransferText from an Access application instance:
Option Explicit
Dim appAccess
Dim strMdb
Const cstrFolder = "c:\dest\"
strMdb = "new.mdb"
Set appAccess = CreateObject("Access.Application")
appAccess.OpenCurrentDatabase cstrFolder & strMdb, False
So, do the Transfertext from that instance variable:
appAccess.DoCmd.TransferText [your options]
Edit: This would be faster for me to create and test in VBA. So I think I would use that instead of VBScript.
Create a function, SnarfCSV, in a standard module in your template MDB. Then create a macro, mcrSnarfCSV, with the SnarfCSV function as its runcode action. Then after you copy the template MDB to the new MDB, open the new one with the /x command line switch to run the macro.
"Path to MSACCESS.EXE" "Path to your db file" /x mcrSnarfCSV