Kindly see this screen cast to get better idea about our requirement:
https://www.screenr.com/QmDN
We want to automate the Text Datasource Generation and connection to MS Excel in order to make it easier to the end-user to connect to the Text Datasource (CSV) to MS Excel so that they can generate their own reports.
The steps I have in mind:
Use WinSCP FTP Client with Scripting
Write script to get the most recent updated file from FTP Folder
Or instead of step 2, download all generated files from FTP to a Shared Folder on the Network.
Get the most recent version of the Generated CSV File
Rename the file to the Standard Naming Convention. This must be the name used in MS Excel as the CSV Text Datasource.
Delete all other files
I developed sample script that can be used by WinSCP to download the files from FTP folder:
# Automatically abort script on errors
option batch abort
# Disable overwrite confirmations that conflict with the previous
option confirm off
# Connect
open CSOD
# Change remote directory
cd /Reports/CAD
# Force binary mode transfer
option transfer binary
# Download file to the local directory d:\
#get "Training Attendance Data - Tarek_22_10_21_2014_05_05.CSV" "D:\MyData\Business\Talent Management System\Reports\WinCSP\"
get "*.CSV" "D:\MyData\Business\Talent Management System\Reports\WinCSP\Files\"
# Disconnect
close
exit
Then, I can schedule the above code to run periodically using this command:
winscp.com /script=example.txt
The above sample is working fine, but the main problem is how to identify the most recent file, so that I can rename it, and delete all the other files.
Appreciate your help.
Tarek
Just add the -latest switch to the get command:
get -latest "*.CSV" "D:\MyData\Business\Talent Management System\Reports\WinCSP\Files\"
For more details, see WinSCP article Downloading the most recent file.
You don't specify the language you use, here a Ruby script that downloads the most recent file of an FTP path. Just to demonstrate how easy and terse this can be done with a scripting language like Ruby.
require 'net/ftp'
Net::FTP.open('url of ftpsite') do |ftp|
ftp.login("username", "password")
path = "/private/transfer/*.*"
# file[55..-1] gives the filename part of the returned string
most_recent_file = ftp.list(path)[2..-1].sort_by {|file|ftp.mtime(file[55..-1])}.reverse.first[55..-1]
puts "downloading #{most_recent_file}"
ftp.getbinaryfile(most_recent_file, File.basename(most_recent_file))
puts "done"
end
Related
I have a folder 'DATA' at SFTP location from where I need to download the set of files to some common location and then copy the respective files to different folder location.
File Names are:
Test1.csv
Test2.csv
Test3.csv
Test4.csv
Test5.csv
I want that files first gets downloaded to below location:
G:\USER_DATA\USER_USER_SYNC\Download
Since these files are related to different schema and have to processed separately by each different ssis packages for further transformations and loading.
For some reasons we have to first keep it at some common location and then move or copy afterwards.
Here's my command line argument.
/log=G:\USER_DATA\USER_USER_SYNC\SFTP_LOG\user_sync_winscp.log /command "open sftp://username:password#stransfer.host.com/" -hostkey=""ssh-rsa 2048 9b:63:5e:c4:26:bb:35:0d:49:e6:74:5e:5a:48:c0:8a""" "get /DATA/Test1.csv G:\USER_DATA\USER_USER_SYNC\Download\" "exit"
Using above, I am able to download a given file one file at a time.
Since, I need to have first it at some common folder location. Hence I am planning to add another Execute process task to copy the files.
/C copy /b G:\USER_DATA\USER_USER_SYNC\Download\Test1.csv G:\USER_DATA\USER_USER_SYNC\Testing1
/C copy /b G:\USER_DATA\USER_USER_SYNC\Download\Test1.csv G:\USER_DATA\USER_USER_SYNC\Testing2
and so on...
I am looking for some way, using which we can download all the available files to some common folder location and then move or copy to different folder locations.
I have changed the design and followed a new approach. Thanks to Martin for fixing the sftp related issues and continuous support.
New SSIS package has below tasks:
Step1. It will look for latest updated files on sftp server and download the given files Test1.csv and Test2.csv to location G:\USER_DATA\USER_USER_SYNC\Download\
Here's my command line arguments:
/log=G:\USER_DATA\USER_USER_SYNC\SFTP_LOG\user_sync_winscp.log /command "open sftp://bisftp:*UFVy2u6jnJ]#hU0Zer5AjvDU4#K3m#stransfer.host.com/ -hostkey=""ssh-rsa 2048 9b:63:5e:c4:26:bb:35:0d:49:e6:74:5e:5a:48:c0:8a""" "cd /DATA" "get -filemask=">=today" Test1.csv Test2.csv G:\USER_DATA\USER_USER_SYNC\Download\" "exit"
Step-2. Since my requirement was to further copy each file to different folder location, so that respective process can pick corresponding file and start transformation and loading it into sql server.
This step will execute the Window cmd process and copy Test1.csv to new location as
G:\USER_DATA\USER_USER_SYNC\Testing1
command line arguments as:
/C copy /b G:\USER_DATA\USER_USER_SYNC\Download\Test1.csv G:\USER_DATA\USER_USER_SYNC\Testing1
Like wise I have another Execute process task to copy Test2.csv to new location as
G:\USER_DATA\USER_USER_SYNC\Testing2
command line arguments as:
/C copy /b G:\USER_DATA\USER_USER_SYNC\Download\Test2.csv G:\USER_DATA\USER_USER_SYNC\Testing2
The given solution is working fine, However there are couple of things which still needs to be handle.
Since I am downloading the latest file only using -filemask=">=today". Everything runs fine if execute process task is able to find the latest files on sftp server. If it's not there, than the next subsequent execute process task is failing with below error message.
The returning The process exit code was "1" while the expected was "0"
Here what I understand is that it's failing as it has nothing to copy or move.
Is there any way by which we can capture the exit code returned from first execute process task and store it into some variable, so that we can use expression to decide that whether to start next task or not.
Second, as you can see that I am using two execute process task to copy files from one location to another. Can we do anything to combine both these two commands into one execute process task?
Any suggestion most welcome and also i think that this issue needs to be addressed as a separate question.
I am having couple of files present at WinSCP folder location USERDATA.
Files present are staring with following names: ABC_XXXX.txt and XYZ_XXXX.txt.
I am interested in downloading all the files which are starting with ABC_ prefix only.
Below is my Execute Process Task configuration:
Executable- C:\Program Files (x86)\WinSCP\WinSCP.exe
/log=c:\path\to\log\winscp.log /command "open sftp://username:password#example.com/" "get /USERDATA/User_file.txt C:\User\Local\" "exit"
It's running fine when I am downloading a single file. How can we parametrize it to fetch the specific files which starts with prefix ABC_ only. Is there any way we can set the expression for this?
Use file mask ABC_*:
... "get /USERDATA/ABC_* C:\User\Local\" ...
I've created an Access database to be shared through the entire department, which I've split into a front end and a back end. Unfortunately, there's no easy way I can figure out to ensure all users are consistently using the newest version of the front end on their local machine as I add requested updates.
To overcome this, I created an install batch script that creates a shortcut on their desktop. as well as nesting the front end and an "update" batch script in a custom folder on their PC. The shortcut actually links to the "update" batch script, which then downloads the newest version of the front end (overwriting the existing one), then loads it.
Ideally, this would not download it every time and instead only downloads it if the version of the front end on the network is greater than that on the local machine. Unfortunately, I can't seem to figure out how to do this with an accdb file (though I've seen information for executable files). We are using Access 2010 and an Access 2007 filetype. I still have not figured out how to append a version number to the front end, but I'm open to including a text file as well simply to store that version number. Any suggestions?
Below is the script I currently have for the update file.
#ECHO OFF
CLS
XCOPY "\\NetworkPath\Install\*.accdb" c:\Reserved\Database /y /q
XCOPY "\\NetworkPath\Install\Update.bat" c:\Reserved\Database /y /q
CLS
ECHO Starting database...
START "" "C:\Reserved\Database\FrontEnd.accdb"
I've done the exact same thing, and solved the problem of only re-downloading the frontend when it has changed by using the xcopy command with the /d switch:
xcopy /yqd \\network\frontend.accdb frontend.accdb
Xcopy reference: http://www.microsoft.com/resources/documentation/windows/xp/all/proddocs/en-us/xcopy.mspx?mfr=true
That works, but leaves a small gap in the logic: when someone is using their local copy of the frontend, and you push a new version to the network, and then they exit the frontend and run the script again: it won't download the new version because the user's local copy will have a later modification time.
To overcome this, I actually make a copy of the local frontend and start that from the script, instead of starting the downloaded copy. That way the downloaded copy retains its original modification time and xcopy's time check works correctly. You do have to train your users though to ignore the local copies of the accdb file and only use the script.
I am creating a script on the fly to ftp some files from a remote computer. I create a file which is then called from the command line with
ftp -s:filename proxy
where filename is the file I just created. The file has code similar to the following:
anonymous#ip address
username
prompt off
binary
cd c:\destination directory
mget c:\source directory\*.*
quit
That doesn't work. Neither does the following:
anonymous#ip address
username
prompt off
binary
cd c:\source directory
mput c:\destination directory
quit
Obviously, I'm not so good at ftp. How, in what order, where in my file do I specify the place where I want the files to be put (destination directory, and also from where the ftp process is running), and where I want the files to come from (ip address computer which has files I want). Do I need to set the directory before starting the ftp process?
I'm running this in an SSIS package, and I'm not using the SSIS ftp task, because I don't want a failure if no files are found. If there's nothing there, that's cool. If there is something there, I want a copy.
(It was working in my development area, and now, when I'm trying to get files from a server that I truely have no access to except ftp, I'm not getting anything. See How to avoid SSIS FTP task from failing when there are no files to download? for an earlier, related question.)
Update:
Both of the answers below, listing lcd and cd, are correct. However, my example still failed, until I replaced the backslashes with forward slashes. In other words, my final, working result is as follows:
anonymous#ip address
username
prompt off
binary
lcd /destination directory
cd /source directory
mget *.*
quit
Are you looking for LCD and CD where LCD changes directory on the local machine? EG:
LCD c:\destination directory
mget c:\source directory\*.*
In most ftp clients you can set the working directory on the server with the command cd, and you set the working directory on the client with the command lcd.
But it is not clear to me what you are trying to do.
Are you trying to move or copy files that are on the ftp server to another location on the ftp server? As far as I know you cannot do that with ftp. If you wished to copy files from one folder on the ftp server to another, then I believe you would get a copy to the local system, and then reupload them to a new folder. If you wish to move files you can use the rename command.
I have the need to copy the entire contents of a directory on a FTP location onto a shared networked location. FTP Task has you specify the exact file name (not a directory) and File System Task does not allow accessing a FTP location.
EDIT: I ended up writing a script task.
Nothing like reviving a really old thread... but there is a solution to this.
To copy the all files from a directory then specify your remote path to be /[directory name]/*
Or for just files and not directories /[directory name]/.
Or specific file types; /[directory name]/*.csv
I've had some similar issues with the FTP task before. In my case, the file names changed based on the date and some other criteria. I ended up using a Script Task to perform the FTP operation.
It looks like this is what you ended up doing as well. I'd be curious if anyone else can come up with a better way to use the FTP task. It's nice to have...but VERY limited.
When I need to do this sort of thing I use a batch file to call FTP on the command line and use the mget command. Then I call the batch from the DTS/DTSX package.