I have a scenario where there are two source file ABC.txt and DEF.txt. i want to make sure that if any files are present in that folder with naming convention as ABC.txt, it has to be moved to the different folder prior to loading to the SQL server, and the other (incorrect file) should be moved to a error log folder (separate folder). Can someone advice what task would i use to do that. i tried it using File system task but it just allow you to move or copy files from one location to other prior to loading the file.can someone advice how to i copy that incorrect file to the different folder ( error log folder)?
try to use a bat file.
move ABC.txt /directory
and after
xcopy "source" "destination_err\" /exclude:%temp%\ABC.txt /y
or you can use robocopy command in bat file, this is the command with parameters http://itsvista.com/2007/03/robocopy/
Related
I am extremely confused.
I have a destination directory: \\Client\D$\Data Feed\Archive. I set this as my Destination Connection in File System Task Editor.
I have a source directory: \\Client\D$\Data Feed\Plan 24-01-2020. I set this as my Source Connection in File System Task Editor. It also contains one CSV file.
For the Operation in File System Task Editor I choose Move Directory as the Operation.
All setup just click run. When I execute I get the following error message: "Cannot create a file when that file already exists."
Curiously enough the CSV file inside the source folder is copied to the Archive folder.
I was expecting only that the folder Plan 24-01-2020 would be moved to the folder Archive.
What I'm I doing wrong?
Because according to this tutorial the folder should be moved without any issues: https://www.tutorialgateway.org/move-directory-using-file-system-task-in-ssis/
I think there are two things you'll want to do here.
First, create an expression in the File System Task that sets OverwriteDestinationFile to TRUE.
Next, you'll want to slightly modify your Destination path. Instead of just \\Client\D$\Data Feed\Archive\, you'll probably want to specify your destination as \\Client\D$\Data Feed\Archive\Plan 24-01-2020\. Otherwise, it will just copy the contents of \\Client\D$\Data Feed\Plan 24-01-2020\ into \\Client\D$\Data Feed\Archive\ without creating the Plan 24-01-2020 sub-folder.
I have a folder 'DATA' at SFTP location from where I need to download the set of files to some common location and then copy the respective files to different folder location.
File Names are:
Test1.csv
Test2.csv
Test3.csv
Test4.csv
Test5.csv
I want that files first gets downloaded to below location:
G:\USER_DATA\USER_USER_SYNC\Download
Since these files are related to different schema and have to processed separately by each different ssis packages for further transformations and loading.
For some reasons we have to first keep it at some common location and then move or copy afterwards.
Here's my command line argument.
/log=G:\USER_DATA\USER_USER_SYNC\SFTP_LOG\user_sync_winscp.log /command "open sftp://username:password#stransfer.host.com/" -hostkey=""ssh-rsa 2048 9b:63:5e:c4:26:bb:35:0d:49:e6:74:5e:5a:48:c0:8a""" "get /DATA/Test1.csv G:\USER_DATA\USER_USER_SYNC\Download\" "exit"
Using above, I am able to download a given file one file at a time.
Since, I need to have first it at some common folder location. Hence I am planning to add another Execute process task to copy the files.
/C copy /b G:\USER_DATA\USER_USER_SYNC\Download\Test1.csv G:\USER_DATA\USER_USER_SYNC\Testing1
/C copy /b G:\USER_DATA\USER_USER_SYNC\Download\Test1.csv G:\USER_DATA\USER_USER_SYNC\Testing2
and so on...
I am looking for some way, using which we can download all the available files to some common folder location and then move or copy to different folder locations.
I have changed the design and followed a new approach. Thanks to Martin for fixing the sftp related issues and continuous support.
New SSIS package has below tasks:
Step1. It will look for latest updated files on sftp server and download the given files Test1.csv and Test2.csv to location G:\USER_DATA\USER_USER_SYNC\Download\
Here's my command line arguments:
/log=G:\USER_DATA\USER_USER_SYNC\SFTP_LOG\user_sync_winscp.log /command "open sftp://bisftp:*UFVy2u6jnJ]#hU0Zer5AjvDU4#K3m#stransfer.host.com/ -hostkey=""ssh-rsa 2048 9b:63:5e:c4:26:bb:35:0d:49:e6:74:5e:5a:48:c0:8a""" "cd /DATA" "get -filemask=">=today" Test1.csv Test2.csv G:\USER_DATA\USER_USER_SYNC\Download\" "exit"
Step-2. Since my requirement was to further copy each file to different folder location, so that respective process can pick corresponding file and start transformation and loading it into sql server.
This step will execute the Window cmd process and copy Test1.csv to new location as
G:\USER_DATA\USER_USER_SYNC\Testing1
command line arguments as:
/C copy /b G:\USER_DATA\USER_USER_SYNC\Download\Test1.csv G:\USER_DATA\USER_USER_SYNC\Testing1
Like wise I have another Execute process task to copy Test2.csv to new location as
G:\USER_DATA\USER_USER_SYNC\Testing2
command line arguments as:
/C copy /b G:\USER_DATA\USER_USER_SYNC\Download\Test2.csv G:\USER_DATA\USER_USER_SYNC\Testing2
The given solution is working fine, However there are couple of things which still needs to be handle.
Since I am downloading the latest file only using -filemask=">=today". Everything runs fine if execute process task is able to find the latest files on sftp server. If it's not there, than the next subsequent execute process task is failing with below error message.
The returning The process exit code was "1" while the expected was "0"
Here what I understand is that it's failing as it has nothing to copy or move.
Is there any way by which we can capture the exit code returned from first execute process task and store it into some variable, so that we can use expression to decide that whether to start next task or not.
Second, as you can see that I am using two execute process task to copy files from one location to another. Can we do anything to combine both these two commands into one execute process task?
Any suggestion most welcome and also i think that this issue needs to be addressed as a separate question.
I am having couple of files present at WinSCP folder location USERDATA.
Files present are staring with following names: ABC_XXXX.txt and XYZ_XXXX.txt.
I am interested in downloading all the files which are starting with ABC_ prefix only.
Below is my Execute Process Task configuration:
Executable- C:\Program Files (x86)\WinSCP\WinSCP.exe
/log=c:\path\to\log\winscp.log /command "open sftp://username:password#example.com/" "get /USERDATA/User_file.txt C:\User\Local\" "exit"
It's running fine when I am downloading a single file. How can we parametrize it to fetch the specific files which starts with prefix ABC_ only. Is there any way we can set the expression for this?
Use file mask ABC_*:
... "get /USERDATA/ABC_* C:\User\Local\" ...
How do I implement an execute process task to extract all files in a directory using 7z. exe in SSIS ?
I am trying to unzip all files in a certain directory using 7z executable but it gives a process error code 2.
Make sure you are specifying a WorkingDirectory rather than specifying the full path for the file in the arguments.
I have the need to copy the entire contents of a directory on a FTP location onto a shared networked location. FTP Task has you specify the exact file name (not a directory) and File System Task does not allow accessing a FTP location.
EDIT: I ended up writing a script task.
Nothing like reviving a really old thread... but there is a solution to this.
To copy the all files from a directory then specify your remote path to be /[directory name]/*
Or for just files and not directories /[directory name]/.
Or specific file types; /[directory name]/*.csv
I've had some similar issues with the FTP task before. In my case, the file names changed based on the date and some other criteria. I ended up using a Script Task to perform the FTP operation.
It looks like this is what you ended up doing as well. I'd be curious if anyone else can come up with a better way to use the FTP task. It's nice to have...but VERY limited.
When I need to do this sort of thing I use a batch file to call FTP on the command line and use the mget command. Then I call the batch from the DTS/DTSX package.