I have a working SSIS task that executes once in a month and runs trough a given path, iterating though various XML files and inserting their contents on an SQL Server database.
On the last meeting with the area who determines the rules for that SSIS task, it was stated that the task can only run after all the files expected are present on the path. The files in question are in a number of 35. Their name is TUPSTATUS_SXX.xml, where XX estates for a number from 01 to 35 (35 telecommunication sectors here in Brazil). Those files are generated by the telecom companies that operate each one of those sectors.
So, my question is: how to run an SSIS task only after all the 35 files are present on our directory?
Instead of doing a busy wait on the folders with your SSIS process running continuously, why not set up a FileSystemWatcher which will trigger on file system conditions. Use it to invoke your SSIS package. you can implement this watcher in either a service(if required for a longer time), or a simple application .
Another way is to have a simple application/script to check for the count, which will be invoked from task scheduler.
You could use a Script Task that will count the files in the folder, and only execute further down the pipeline when 35 files exist. Although if the files are large and being transferred by FTP, the final file may exist, but not have fully transferred at that point.
Related
I currently have an ETL package that downloads files and then loads them into a database.
When I have downloaded the files and processed them, I place the files into an archive folder. However, as I truncate the staging table each time I run the process if any new files are downloaded, I need to reload the archive files in a separate process.
I have 2 foreach containers, 1 loads the new files whilst the new container loads the archived files.
What I want to do is create an expression on the archive foreach loop that will only run if the SQL task that truncates the table is completed.
Do I create a variable of type boolean that changes the value to 1 if the task is run and use that in my expression? If so how could I achieve this?
I have a SSIS 2019 Package which imports an Excel as soon as the file is placed to a folder and then moves the Excel file to another folder (agent will run it every 30 minutes). At the beginning I have a Script Task which tests if the file is in the expected folder, if true then the control flow goes to the Data Flow task, if not then it goes nowhere (process ends there). In testing the package it works fine so long as the file is in the expected folder when I open or do anything with the package. In testing when I run the package a second time, after the file has been moved, then the Control Flow doesn't go past the Script Task but I get an error from the Data Flow because in the folder a new empty file with the expected filename but incorrect sheet has been automatically created. I have no idea what task caused this or if the excel file connection manager did it at the moment the package was executed. It will even do this as soon as I open the package if the file is not in the folder. How do I stop this? I don't want it to create an empty excel file at all. I want it to just do nothing if that file is not in the designated folder.
Here's a complete sequence of events:
Script Task checks if the file is in the folder – No: does nothing
Yes – Executes SQL Task deletes records in Target Table One
Then goes into Data Flow
a.Excel Data Source goes into Data Conversion Task
b.Then data goes into OLE DB source (Target Table One)
Exists Data Flow – On Error sends email
Then Executes SQL Task which – On Error sends email
a. Deletes records in Target Table Two
b. Inserts records into Target Table Two which do not have a null part number
c.Inserts a record into a Log History Table
Then Executes a File System Task which moves the Excel file from IMBOX folder to OUTBOX folder
– On Error sends email
– On Success sends email
The solution that worked was to move the Script Task (if file in folder query) to a seperate solution and add to it an Execute Package Task, setup as an external reference from file system, the original package .dtsx file. It runs perfectly from within Visual Studio as well as from the Agent in management studio. The Agent will be scheduled to run ever 5 minutes and most of the time there will be no Excel file to import, and no error will be generated. Then the user places the file into the folder then within 5 minutes it will be processed into the ETL database, Excel file will be moved to an outbox folder and user will receive an email indicating the process completed or failed.
I am sure there has to be another solution for preventing the Data Flow Task from creating an Excel data source when it isn't there when the package is opened. But the above solution definitely works.
Using Microsoft Visual Studio Community 2015.
Goal of project
-create "*\temp\email" directory
-start program to extract all emails that include xls attachments to the previously created folder
-use for each loop to cycle through each file in the folder, process, and shift to sql table.
The problem I am running into is caused by either a blank excel document (which is occasionally sent from a remote location) or some of the original xls reports only contain 5 columns instead of 6 that I have mapped now. Is there any way to separate files that include the correct columns from those that do not match?
** as Long as these two problems do not exist I can run the ssis package and everything runs without issue.
Control flow;
File System Task (creates directory --->Execute Process Task (xls extraction)-->ForEach Loop(Data flow Task "email2Sql")
Data Flow;
Excel Source (uses expression ExcelFilePath,#user:filepath) delay validation ==true
(columns are initially set to f1-f6 and are mapped to for ex. a,b,c,d,e,f. The Older files that get mixed in only include a,b,c,d,e.) This is where I want to be able to separate the xls files
Conditional Transformation split (column names are not in row 1, this helps remove "null" values)
Ole Db destination (sql table)
Sorry for the amount of reading, but for the first post I tried to include anything that I thought may be relevant.
There are some tools out there which would allow you to open the excel doc and read it. However, I think the simplest thing to do would be to use SSIS out of the box:
1 - add a file system task after the data flow which reads the file.
2 - Make the precedence constraint from the data flow to the file system task "failure." This will cause that to only fire when the data flow task fails.
3 - set the file task to move the "bad" files to another folder
This will allow you to loop through all the files and move the failed ones. Ultimately, the package will end in failure. If you don't want that behavior you can change the ForceExecutionResult property to be success. However, it might be good to know that there were problems with some files so that they can be addressed.
m
I'm trying to create a package that imports excel files into a database using SSIS.
As the operation has to perform this regularly and the file names follow a convention but are not the same, and equally the sheet/tab names are not always the same, the SSIS package is set up as follows:
Main Container
->
First For Each container (call it FE1)
Obtains filenames (assigns to a variable)
->
Second For Each Container (call it FE2)
Obtains worksheet name and starts the process to import.
What I have done is create a "failure" precedence constraint from FE2 to a file system task process in FE1.
The idea is that the file move is done if the import is unsuccessful for whatever reason.
(once it works I'd like to create a "success" process that moves the file to the archive folder)
The file task process works when there is only one "for each container" (i.e. not nested the way it is now) but it fails when all the processes are in the nested container citing "file in use". I'm assuming this is because the first for each container is locking the file, hence why I moved the file task process to the first for each container and used a precedent control.
Any help and advice much appreciated.
For the benefit of anyone else who may have the same problem:
for love nor money could I get the excel connector to release the files, even when the move file task was outside of the loop.
In the end I recorded the files that were moved into a table in the DB and then executed a second package that contained the move file task and would iterate through the table rows with the list of successfully imported (and failed import) files and moved them to their destination based on fail/success flag.
It was the only way I got to successfully make this happen.
When having to iterate through the worksheet of each excel file (i.e. two excel connections effectively) SSIS would not release the files so I forever received the file in use error and failure.
I am new to SSIS. I am assigned a POC development. Please help. Requirement is there are 'n' number of countries and each country will load 27 files in to 'n' number of folders. So I need to create a SSIS package to fetch the .csv files from the location and load them in staging and target tables. All countries will load same 27 files. How do I set the file connection manager dynamically. and how do i set package configuration to run it dynamically. each run should ensure the 27 files of one country gets processed. only then it has to execute the next country files. Everything has to be automated. That is at run time the files to be fetched has to be configured for a single country. can this be done? Somebody please help. I am from webmethods back ground and this is totally new.
I had faced this kind of problem in the past and did solve this issue with Creating two Foreach Loop Container 1) Loop the folder and 2) loop the file dynamically.You can modify this logic accordingly to your requirements. I have listed 2 references which will give you step by step process.
How to loop through Excel files and load them into a database using SSIS package?
http://www.codeproject.com/Tips/378129/Dynamically-Configure-Excel-in-Foreach-Loop-Contai