I want to load different config files for different conditions. for example, if I have "vendor" in my variable value, then load a vendor's config file, similarly for ext_warehouse.
not possible, SSIS will load all the file you specify on the package configuration organizer on the order they are listed and it will use the values on the configuration files to load your variables.
What you could do, lets say you have a "vendors.dtsConfig" and a"customer.dtsConfig", is to create a parent package that read a config file called "parent.dtsConfig" that will only have your variable containing "vendor" or "customer" and based on that value you copy the correct config file to another folder and call the actual package that reads the recently copied (and renamed) file
Related
my question is how we will create SSIS package to get the particular file from multiple files and copy to different folder
Need Help
You can use File System Task in your package control flow.
Configure the properties as below:
DestinationConnection: Enter the full file path of the folder you want to move your file to e.g c:\Users\ToTest
Operation: Change to Move file.
SourceConnection: Enter the full file path of the folder where your file is, including the file name and extension e.g c:\Users\Test\testfile.csv
You'll need at least two components.
Use a Foreach Loop container with the default Foreach File Enumerator on the Collection tab. Set Folder: to \YourFolder and build an expression under Files: to identify the particular file you're interested in.
Inside the Foreach Loop container, add a File System Task to Copy the file from the Source folder to the Destination folder.
One folder is having thousand files and each files will be loaded to different sql server tables. How to design a SSIS package to do the task ?
For ex:
File name: Location_12345.xlsx will be loaded into Location table
Employee_1233.txt will be loaded into Employee table
Department_123456.csv will be loaded into Department table
The answer is yes you can. Loop through the files in the folder and get whatever you need. A simple google search would give you everything you need. For example here
You will need a package for each flow. You defined a minimum of 3 in your question.
For example, if all Locations follow the same flow then:
Add a foreach and choose files to loop through.
Define the folder
Define the criteria (Location*.xlsx)
Set the full file path to a variable
Add an excel connection
Make an expression for that variable
Design your data flow.
Delay validation on Excel Source before running.
This is a sample. You will have to do this for each file type.
I have a folder which contains files like:
A_ddmmyyyy, b_ddmmyyyy and c_ddmmyyyy.
I need to read all these files for a date and again all these files for the next date present in the same folder. Also the number of files present in the folder varies (may contain data for three days or of five dates), but the date on the folder remains the same.
Is it possible?
You can create a Foreach Loop Container, inside the container create a data flow task that processes all the files in a specific folder. You create a flat file connection with the needed delimiter, and a variable that has the folder path for your files. And then as a last step to your tasks, create a File System Task to move the file to a processed or completed folder so that the your main folder empties out once the files are processed .
File name doesn't matter, you have to be certain how many kinds of schemas (number of columns, column names, types) of files in the folder. Say you have 3 schemas, then you will need to define 3 type of flat file connections. There are many ways to do the job, the easiest of i can think of is to use powershell to separate files of different schema into different folders, you have to know what file names are mapped to what schema, there may be a pattern or business rule. You then put your powershell in a execute process task to run the script. Then the following is simple, for each folder you create a package, inside is a for each container where you loop through the folder to load each file. Or you can have one package with three for each loop container to do the job.
I am working in BIDS. I have an SSIS package that loops over all files in a given directory and imports all of them to a table. So, my control flow has a Foreach Loop Container with a Data Flow Task inside of it. The Data Flow Task goes straight from a Flat File Source to an OLE DB Destination. Because the directory can change, I have made the Foreach Directory dynamic (i.e., I gave it a package variable, which I can change when the directory changes). The Files field has something like stuff*.pip, such that all files in the directory that match this pattern get imported. The Variable Mappings tab has a file name variable, which is the same variable used in the Expressions of the flat file connection manager.
When I originally set up the flat file connection manager, I had to point it to an already existing file to pick up the file's metadata. I then changed the file connection manager to dynamic (i.e., I added a file name variable in the Expressions property) such that the Foreach Loop Container will pick up all files (that match the pattern).
Since then, the metadata on the file has changed. So, when I open the solution file, it throws a warning on the Flat File Source in the Data Flow Task. To resolve this, I have to temporarily change the flat file connection back to a hard-coded path such that I can change the metadata on the new file. I then make it dynamic again (i.e., use the file name variable in the Expressions, which overrides the hard-coded file path). (Finally, I double-click on the Flat File Source in the Data Flow Task to automatically update the metadata in the Flat File Source.)
Is this the correct way to update the metadata on a flat file? It seems a bit clunky to me.
You can prevent this error from occurring by editing the properties on the Flat File Manager and setting ValidateExternalMetaData=False.
You may also need to set DelayValidation=True on the Data Flow Task that contains the Flat File Manager.
This will prevent the warning - but if the file's metadata changes for some reason (columns change, data types change) and you do not remember to update the metadata then you will get a runtime error.
I have an SSIS package that needs to lookup two different types of excel files, type A and type B and load the data within to two different staging tables, tableA and tableB. The formats of these excel sheets are different and they match their respective tables.
I have thought of putting typeA.xls and typeB.xls in two different folders for simplicity(folder paths to be configureable). The required excel files will then be put here through some other application or manually.
What I want is to be able to have my dtsx package to scan the folder and pick the latest unprocessed file and load it ignoring others and then postfix the file name with '-loaded' (typeAxxxxxx-loaded.xls). The "-loaded" in the filename is how I plan to differentiate between the already loaded files and the ones yet to be loaded.
I need advice on:
a) How to check that configured folder for the latest file ie. without the '-loaded' in the filename and load it? ..and then after loading it, rename the same file in that configured folder with the '-loaded' postfixed.
b) Is this the best approach to doing this or is there a better way?
Thanks.
You can do it this way, but it might require several complex string expressions.
E.g. create a ForEach loop over .xls files, inside the loop add an empty script task, then a data flow to load this file. Connect them with a precedence constraint and make it conditional: precedence constraint expression will the check if file name does not end with -loaded.xls. You may either do it in script task or purely using SSIS expression on precedence constraint. Finally, add File System Task to rename the file. You may need to build new file name with another expression.
It might be easier to create two folders: Incoming for new unprocessed files, and Loaded for the files you've processed, and just move the .xls to this folder after processing without renaming. This will avoid the first conditional expression (and dummy script task), and simplify the configuration of File System task.
You can get the SQL File watcher Task and add it to your SSIS. I think this is a cleaner way to do what you want.
SQL File Watcher