I have a process that I want to create in SSIS. What I have so far is below.
So the scenario is that a number of different file types are downloaded to a specific location - which has been created as a variable "RootFolder"
Within that RootFolder, is a folder called "Archive"
The first part of the process is to create a folder within the variable of ArchiveFolder, which is basically todays date.
Then in a For Loop Container, it copies all the files in the RootFolder to the Archive Folder. So we have a history of the files received.
The next step that I want to carry out in SSIS is to add a header line at the top of each of those text files. So all the files that still exist in the Root Folder get a header added to the top of the file. Is there something direct in SSIS to do this, or would I need a .bat file to do it and have an Execute Process Task, to call that bat file. Either way I have no idea how to achieve this bit. Your help will be appreciated.
Related
Well, I am facing one weird issue with Copy Files task i SSIS.
I have folder A in which I have sub folders and I am using Copy files task to copy all the Excel files in sub-folders to the folder A. I noticed that with some of the excel files where I have custom date column which is restricted by list to select dates from as I don't want user to input wrong date/format.
When I run the package the excel file is copied but that date column is not getting populated.
I am not getting the reason why its happening.
Can some one help? What approach I should follow to resolve it?
So, what I did is in place of Copy file from the drop down of the component I selected move file and ran it once. And then again I ran with Copy command and it some how worked. I am not happy this though. I have put in a solution at the end of SSIS package where after the files are loaded to SQL table I am deleting them from the source folder using looping component.
For now it seem to be working. but exact reason for not copying the particular custom date field is still a question.
my question is how we will create SSIS package to get the particular file from multiple files and copy to different folder
Need Help
You can use File System Task in your package control flow.
Configure the properties as below:
DestinationConnection: Enter the full file path of the folder you want to move your file to e.g c:\Users\ToTest
Operation: Change to Move file.
SourceConnection: Enter the full file path of the folder where your file is, including the file name and extension e.g c:\Users\Test\testfile.csv
You'll need at least two components.
Use a Foreach Loop container with the default Foreach File Enumerator on the Collection tab. Set Folder: to \YourFolder and build an expression under Files: to identify the particular file you're interested in.
Inside the Foreach Loop container, add a File System Task to Copy the file from the Source folder to the Destination folder.
I'm trying to create a package that imports excel files into a database using SSIS.
As the operation has to perform this regularly and the file names follow a convention but are not the same, and equally the sheet/tab names are not always the same, the SSIS package is set up as follows:
Main Container
->
First For Each container (call it FE1)
Obtains filenames (assigns to a variable)
->
Second For Each Container (call it FE2)
Obtains worksheet name and starts the process to import.
What I have done is create a "failure" precedence constraint from FE2 to a file system task process in FE1.
The idea is that the file move is done if the import is unsuccessful for whatever reason.
(once it works I'd like to create a "success" process that moves the file to the archive folder)
The file task process works when there is only one "for each container" (i.e. not nested the way it is now) but it fails when all the processes are in the nested container citing "file in use". I'm assuming this is because the first for each container is locking the file, hence why I moved the file task process to the first for each container and used a precedent control.
Any help and advice much appreciated.
For the benefit of anyone else who may have the same problem:
for love nor money could I get the excel connector to release the files, even when the move file task was outside of the loop.
In the end I recorded the files that were moved into a table in the DB and then executed a second package that contained the move file task and would iterate through the table rows with the list of successfully imported (and failed import) files and moved them to their destination based on fail/success flag.
It was the only way I got to successfully make this happen.
When having to iterate through the worksheet of each excel file (i.e. two excel connections effectively) SSIS would not release the files so I forever received the file in use error and failure.
I have a folder which contains files like:
A_ddmmyyyy, b_ddmmyyyy and c_ddmmyyyy.
I need to read all these files for a date and again all these files for the next date present in the same folder. Also the number of files present in the folder varies (may contain data for three days or of five dates), but the date on the folder remains the same.
Is it possible?
You can create a Foreach Loop Container, inside the container create a data flow task that processes all the files in a specific folder. You create a flat file connection with the needed delimiter, and a variable that has the folder path for your files. And then as a last step to your tasks, create a File System Task to move the file to a processed or completed folder so that the your main folder empties out once the files are processed .
File name doesn't matter, you have to be certain how many kinds of schemas (number of columns, column names, types) of files in the folder. Say you have 3 schemas, then you will need to define 3 type of flat file connections. There are many ways to do the job, the easiest of i can think of is to use powershell to separate files of different schema into different folders, you have to know what file names are mapped to what schema, there may be a pattern or business rule. You then put your powershell in a execute process task to run the script. Then the following is simple, for each folder you create a package, inside is a for each container where you loop through the folder to load each file. Or you can have one package with three for each loop container to do the job.
I need to visit a folder and all of its children with SSIS (SQL Server Integration Services). At the moment by setting the folder path into a variable after reading it, I able to loop through all the .txt files of the current folder and fill a pre-generated (with head info) xml file.
What I would need now is to be able to create one per each accessed folder, a new xml file (the beginning content will be always the same). Once I would be able to create it, as first action once a new folder is accessed, I can then simply apply the logic I developed so far.
However I am blocked at the moment, since within the loop where i read the files (with their full path) I cannot find a way to express "create the xml file if the accessed folder is new".
Assuming I understand the problem, you need to walk the entirety of a directory structure and for each folder you find, you need to create a base XML file. Then for each text file you find in that folder, you will perform some operation on the XML file. The trick being how do you only create the XML file once.
I would envision a process like this.
A script task that makes use of the System.IO.GetDirectories to populate a variable (directoryXML> that contains the folder structure, something like
<Dir>
<D>C:\ssisdata</D>
<D>C:\ssisdata\a</D>
<D>C:\ssidata\a\b</D>
</Dir>
Use a Foreach Nodelist Enumerator to shred that XML out into a variable (currentDirecotry).
You'd perform your one-time task of creating the XML file in currentDirectory.
Further using the currentDirectory variable as an expression on the Foreach File Enumerator (assign to Directory with a FileSpec of *.txt) you can then perform your task on all the files meeting that specification. Do not check the traverse subfolder option as that will not give the desired results.
This is a fairly high level approach to the problem as I'm assuming you have some familiarity with SSIS but the approach should be sound. Let me know if you have any particular sticking points.