Well, I am facing one weird issue with Copy Files task i SSIS.
I have folder A in which I have sub folders and I am using Copy files task to copy all the Excel files in sub-folders to the folder A. I noticed that with some of the excel files where I have custom date column which is restricted by list to select dates from as I don't want user to input wrong date/format.
When I run the package the excel file is copied but that date column is not getting populated.
I am not getting the reason why its happening.
Can some one help? What approach I should follow to resolve it?
So, what I did is in place of Copy file from the drop down of the component I selected move file and ran it once. And then again I ran with Copy command and it some how worked. I am not happy this though. I have put in a solution at the end of SSIS package where after the files are loaded to SQL table I am deleting them from the source folder using looping component.
For now it seem to be working. but exact reason for not copying the particular custom date field is still a question.
Related
Using Microsoft Visual Studio Community 2015.
Goal of project
-create "*\temp\email" directory
-start program to extract all emails that include xls attachments to the previously created folder
-use for each loop to cycle through each file in the folder, process, and shift to sql table.
The problem I am running into is caused by either a blank excel document (which is occasionally sent from a remote location) or some of the original xls reports only contain 5 columns instead of 6 that I have mapped now. Is there any way to separate files that include the correct columns from those that do not match?
** as Long as these two problems do not exist I can run the ssis package and everything runs without issue.
Control flow;
File System Task (creates directory --->Execute Process Task (xls extraction)-->ForEach Loop(Data flow Task "email2Sql")
Data Flow;
Excel Source (uses expression ExcelFilePath,#user:filepath) delay validation ==true
(columns are initially set to f1-f6 and are mapped to for ex. a,b,c,d,e,f. The Older files that get mixed in only include a,b,c,d,e.) This is where I want to be able to separate the xls files
Conditional Transformation split (column names are not in row 1, this helps remove "null" values)
Ole Db destination (sql table)
Sorry for the amount of reading, but for the first post I tried to include anything that I thought may be relevant.
There are some tools out there which would allow you to open the excel doc and read it. However, I think the simplest thing to do would be to use SSIS out of the box:
1 - add a file system task after the data flow which reads the file.
2 - Make the precedence constraint from the data flow to the file system task "failure." This will cause that to only fire when the data flow task fails.
3 - set the file task to move the "bad" files to another folder
This will allow you to loop through all the files and move the failed ones. Ultimately, the package will end in failure. If you don't want that behavior you can change the ForceExecutionResult property to be success. However, it might be good to know that there were problems with some files so that they can be addressed.
m
Daily SQL Job will start at 12.00. It will run a package that fetch a CSV file from a folder(using for each loop container in ssis).
Suppose if there no files in that specific folder. You should not run the package until the csv files load into that folder? How we can do this using SSIS .
Please help me on this.
Have the job run on a schedule. If there are no files in the folder, it won't do anything. The next time it runs, if the files are there, it will process them.
Using a script task, you can check if the file exists in that location with that file extension and then build an expression in the precedence constraint editor. Set the evaluation operation to expression and constraint value to success. Something like the one shown in the screenshot below.
I have a process that I want to create in SSIS. What I have so far is below.
So the scenario is that a number of different file types are downloaded to a specific location - which has been created as a variable "RootFolder"
Within that RootFolder, is a folder called "Archive"
The first part of the process is to create a folder within the variable of ArchiveFolder, which is basically todays date.
Then in a For Loop Container, it copies all the files in the RootFolder to the Archive Folder. So we have a history of the files received.
The next step that I want to carry out in SSIS is to add a header line at the top of each of those text files. So all the files that still exist in the Root Folder get a header added to the top of the file. Is there something direct in SSIS to do this, or would I need a .bat file to do it and have an Execute Process Task, to call that bat file. Either way I have no idea how to achieve this bit. Your help will be appreciated.
I am facing a problem while deleting a file on a network location using SSIS, since its a zip file, contains monthly SQL Database backup file, so I need to delete the last month file before copying current month file.
May be there is some app which were using this file, I am not sure, but I wanna get rid of this file, so that I can copy new file.
Thanks
Use a file task and you should be able to delete pretty much anything on any location as long as you have the rights to do so.
I have an SSIS package that needs to lookup two different types of excel files, type A and type B and load the data within to two different staging tables, tableA and tableB. The formats of these excel sheets are different and they match their respective tables.
I have thought of putting typeA.xls and typeB.xls in two different folders for simplicity(folder paths to be configureable). The required excel files will then be put here through some other application or manually.
What I want is to be able to have my dtsx package to scan the folder and pick the latest unprocessed file and load it ignoring others and then postfix the file name with '-loaded' (typeAxxxxxx-loaded.xls). The "-loaded" in the filename is how I plan to differentiate between the already loaded files and the ones yet to be loaded.
I need advice on:
a) How to check that configured folder for the latest file ie. without the '-loaded' in the filename and load it? ..and then after loading it, rename the same file in that configured folder with the '-loaded' postfixed.
b) Is this the best approach to doing this or is there a better way?
Thanks.
You can do it this way, but it might require several complex string expressions.
E.g. create a ForEach loop over .xls files, inside the loop add an empty script task, then a data flow to load this file. Connect them with a precedence constraint and make it conditional: precedence constraint expression will the check if file name does not end with -loaded.xls. You may either do it in script task or purely using SSIS expression on precedence constraint. Finally, add File System Task to rename the file. You may need to build new file name with another expression.
It might be easier to create two folders: Incoming for new unprocessed files, and Loaded for the files you've processed, and just move the .xls to this folder after processing without renaming. This will avoid the first conditional expression (and dummy script task), and simplify the configuration of File System task.
You can get the SQL File watcher Task and add it to your SSIS. I think this is a cleaner way to do what you want.
SQL File Watcher