SSIS copy all files into a variable destination via execute process task - ssis

I have read several posts that have given examples of similar processes.
I have put a lot of time trying to make this work so i would really like to get this to work so I can put all this time into the "I learned something Category"
But if its just not possible... I guess I will learn that at least.
I am tring to copy all files from a folder to another folder using ssis 2008r2.
-using execute process task
-using C:\Windows\System32\cmd.exe in the executable field
-the destination folder is a variable I want to create using the starttime variable - which is working
-then i want to copy all the working files into that folder.
-this dest folder uses the start time down to the second so i can capture each attempted run with minimal errors or overwrites
With so many attempts and failures i have dumbed it down to using straight string variables to try to just move all the files in the simplest way that still uses a variable.
the variables I am currently using in my mutiple failed attempts are...
[user::testtofolder] = p:\Utilities\SWUMOps\Archive\20160720-0921-03-SAWSImports
[user::testtofolder1] = p:\\Utilities\\SWUMOps\\Archive\\20160720-0921-03-SAWSImports
[user::testtofolder2] = p:\\Utilities\\SWUMOps\\Archive\\20160720-0921-03-SAWSImports
The below are some of the attempts I have tried in the arguments field
/c copy /y p:\utilities\swumops\backend\saws p:\utilities\swumops\archive\20160720-0921-03-sawsimports -- works (no variables)
/c copy /y p:\utilities\swumops\backend\saws + #[user::testtofolder] -- fails
"/c copy /y \"p:\utilities\swumops\backend\saws\" + #[user::testtofolder] -- fails
"/c copy /y \"p:\utilities\swumops\backend\saws\"" + #[user::testtofolder] -- fails
"/c copy /y \"p:\utilities\swumops\backend\saws \"" + #[user::testtofolder] -- fails
"/c copy /y \"p:\\utilities\\swumops\\backend\\saws \"" + #[user::testtofolder] --fails
/c copy /y p:\\utilities\\swumops\\backend\\saws + #[user::testtofolder] --fails
"/c copy /y \"p:\\utilities\\swumops\\backend\\saws \" + #[user::testtofolder] --fails feel this is the closest
"/c copy /y \"p:\\utilities\\swumops\\backend\\saws \" + #[user::testtofolder1] -- fails
"/c copy /y \"p:\\utilities\\swumops\\backend\\saws \" + #[user::testtofolder2] -- fails
the frustration thing about this is I can't(don't know how to) see what is truly passed to cmd because the window flashes to fast and the errors only read the line that is in the argument field not the past string. the errors look like this...
Error: 0xC0029151 at Copy files to archive, Execute Process Task: In Executing "C:\Windows\System32\cmd.exe" ""/c copy /y \"p:\utilities\swumops\backend\saws \" + #[user::testtofolder]" at "", The process exit code was "1" while the expected was "0".
Hope I put enough details. Let me know if you need more.
Thank you

In SSIS File System task should be a much easier way to do your task. I am not sure how come you need to use command line to move/copy the file.
For your question, I believe you need to use xcopy.exe instead of cmd.exe. The executable should be "C:\Windows\System32\xcopy.exe", and arguments should be looked like
" /y p:\utilities\swumops\backend\saws " + #[user::testtofolder]
Hope it helps.

Thanks for the attention, It helped me look elsewhere for my problem. I have not used the expressions area of a task before. iNoob. Everywhere I researched said place the formatted line IN the arguments. So that's where I have been placing it, directly in the arguments line... Not in the expressions window by adding a line for arguments. The first simple test worked. So I think I will be ok now.
I appreciate y'alls time it got me to where I needed to be!

Related

SSIS Execute process call to dtexec.exe failing

I am trying to set up an SSIS package that calls other packages via a execute process task. All was working fine until I decided to try to execute the packages with a configuration file. The original expression was:
"/REP E /f " + #[User::screenName] + ".dtsx "
I changed it to
"/REP E /conf .\\<myconfigfile>.dtsConfig /f " + #[User::screenName] + ".dtsx"
which got me The process exit code was "4" while the expected was "0".
and
"/REP E /conf d:\\<dir1>\\<dir2>\\packages\\<myconfigfile>.dtsConfig /f " + #[User::screenName] + ".dtsx"
which got me The process exit code was "6" while the expected was "0".
If I take the command and run it from a command line, like so:
dtexec.exe /REP E /conf .\<myconfigfile>.dtsConfig /f <myssispackage>.dtsx
or
dtexec.exe /REP E /conf "w:\<dir1>\<dir2>\<myconfigfile>.dtsConfig" /f <mySSISpackage>.dtsx
it works fine.
Is there any way at all I can get a better error message? I think there's something about the expression that's causing problems.
Make sure to target the correct dtexec executable. Maybe your PATH is different when executed from the command line to the one used in Execute Process task.
A good suggestion from http://www.sqlservercentral.com/articles/Integration+Services+(SSIS)/126293/ is to search the hard drive for all appearances of it. There are very detailed instructions in that article for more than that.
Edit: Ouch, maybe exit code has something to tell us?
https://learn.microsoft.com/en-us/sql/integration-services/packages/dtexec-utility#exit
4: The utility was unable to locate the requested package. The package could not be found.
6: The utility encountered an internal error of syntactic or semantic errors in the command line.
Errors of type 6 could be related to the escape characters.
What happens if you try to add some extra-escaped double quotes?
"/REP E /conf \"d:\\<dir1>\\<dir2>\\packages\\<myconfigfile>.dtsConfig\" /f " + #[User::screenName] + ".dtsx"

Editing Command Outputs

Basically, I am working on a large project, with some new concepts to challenge myself. I was planning on mapping out my computers directories and folders from what folder my document was in through HTML. Well, as it turns out (not surprisingly), I need some other files to do the mapping so that the HTML file can visualize it. I am stuck. Basically, I decided I would create a batch script that looks like this:
echo off
cd %cd%
for /d %%a in (*) do dir /ad /on /s /b "%%a" >> get_dirs.txt
This is a simple but cool script that I discovered at https://grumpybear.wordpress.com/2011/05/22/batch-files-to-list-all-files-and-directories-in-a-folder/ Which describes the components of the script very nicely. The output text file looks something like this:
C:\Users\Jason\Desktop\G3\DCIM\CardboardCamera
C:\Users\Jason\Desktop\G3\DCIM\Facebook
C:\Users\Jason\Desktop\G3\DCIM\Flickr
C:\Users\Jason\Desktop\G3\DCIM\.thumbnails\otg
C:\Users\Jason\Desktop\G3\Download\Adobe Acrobat
C:\Users\Jason\Desktop\G3\Download\Adobe Reader
C:\Users\Jason\Desktop\G3\Download\TrimbleMapBundles
C:\Users\Jason\Desktop\G3\Pictures\Facebook
C:\Users\Jason\Desktop\G3\Pictures\Instagram
C:\Users\Jason\Desktop\G3\Pictures\Messenger
C:\Users\Jason\Desktop\G3\Pictures\Old Phone
This is when the script is placed on the desktop. This is pretty much as far as I got when I came upon a serious problem. I needed to get these files to be displayed in an html document as
C:/Users/etc.
So I was going to edit the echo output, silly me didn't realize that it wasn't using echo for output but instead the dir command. So I tried to edit the output with echo by sending the output as input to a variable. Which I'm sure there is solutions to on Batch - Command output to variable and Save command output to variable, but the answers on those pages are a little vague and somewhat confusing to someone with a little less experience. Plus, I'm not sure it matters, because the directories all come out as one full output, and not line by line (editable line by line). I'm not sure if it is possible to make one line after another in the output file into separate variables, which would be great. It would be helpful if I could make one line of the text document a variable, then the next, and so on. which I could the format my html links using the variables in separate links, by making a different output with something caused by:
echo %dir1%
echo %dir2%
echo %dir3%
echo %dir4%
echo %dir5%
The script could calculate the amount of variables needed by reading how many lines are in the .txt file as seen here:How to count no of lines in text file and store the value into a variable using batch script?.
If anyone can share any insight (I know this is a lot to ask for and a lot of information) I would greatly appreciate it. Mainly just getting the variables line by line is the main part. I think I might be able to get some of the other stuff. Thank You.
--Edit: So, I was kind of thinking, is there a way for the batch script to read the .txt file line by line? It seems there is but no one has really broken down the method. It all seems a little vague to me.
--Edit: I did some more thinking and looking around and came up with this code:
#echo off
set /a file=file
cd %cd%
for /d %%b in (*) do dir /ad /on /s /b "%%b" >> get_dirs.txt
for /F "tokens=*" %%A in (get_dirs.txt) do set /a text=%%A
echo %text%
pause
This script, has something wrong with it, or as batch script output says, "Missing Operator," a bunch of times, then closes the window. I am not sure what is wrong with it but I do think I am getting somewhere. I think it may have to do with the angle brackets when echoing my html code, thinking it's trying to output or redirect the echo. Anyone know how I can solve this problem (assuming that it is the actual problem)?
<and > are commands to redirect in- or output. To disable this behaviour you have to escape ("disarm") those characters with a caret ^:
echo ^<a href="%file%:///%text%"^>%text%^</a^>
Also your second for reassigns each line to the variable (one after the other), resulting in your varaible containing the last line only. Extend the for to contain your echo line:
for /F "tokens=*" %%A in (get_dirs.txt) do (
echo ^<a href="%file%:///%%A"^>%%A^</a^>
)
PS: are you sure set /a file=file does what you want? According to your desired output, you should just use set file=file

IF and ! = ns2 error

I have a problem with path in a tcl file. I tried to use
source " /tmp/mob.tcl "
and this path in bash file :
/opt/ns-allinone-2.35/ns-2.35/indep-utils/cmu-scen-gen/setdest/setdest -v 1 -n $n -p 10 -M 64 -t 100 -x 250 -y 250 >> /tmp/mob.tcl
The terminal give me this output:
..."
(procedure "source" line 8)
invoked from within
"source "/tmp/mob.tcl" "
(file "mobilita_source.tcl" line 125)
How I can do this?
Firstly, this:
source " /tmp/mob.tcl "
is very unlikely to be correct. The spaces around the filename inside the quotes will confuse the source command. (It could be correct, but only if you have a directory in your current directory whose name is a single space. That's really unlikely, unless you're a great deal more evil than I am.)
It really helps a lot if you stop making this error.
Secondly, the error message is both
Incomplete, with just an ellipsis instead of a full error on the first line
Really worrying, with source claimed to be a procedure (second line of that short trace).
It's legal to make a procedure called source, and sometimes the right thing to do, but if you're doing it then you have to be ever so careful to duplicate the semantics of the standard Tcl command or odd things will happen.
Thirdly, you've got a file of what is apparently generated code, and you're hitting a problem in it, and you're not telling us what is on/around line 125 of the file (the error trace is pretty clear on that front) or in the contents of the source procedure (which is non-standard; the standard source is implemented in C) and you're expecting us to guess what's going wrong for you??? Seriously?
Tcl error traces are usually quite clear enough for you to figure out what went wrong and where. If there's an unclear error, and it didn't come from user code (by calling error or return -code error) then let us know; we'll help (or possibly even change Tcl to make things clearer in the future). But right now, there's a complete shortage of information.
Here's an example of what a normal source error looks like:
% source /tmp/foo/bar/boo
couldn't read file "/tmp/foo/bar/boo": no such file or directory
% puts $errorInfo
couldn't read file "/tmp/foo/bar/boo": no such file or directory
while executing
"source /tmp/foo/bar/boo"
If a script generates an error directly, it's encouraged to be as clear as that, but we cannot enforce it. Sometimes you have to be a bit of a detective yourself…

How do I create a simple Octave distributable without installing Octave

The Octave documentation on this subject is both intimidating and sparse.
I did not know where else to document the solution I found, so I am posting here. I apologize if that's inappropriate, but I want to help the next guy.
The following solution is for a simple windows distributable.
Use Case:
A solution is developed in Octave 3.2.4, and needs to be distributed to end-users with few computer skills. Installing and explaining Octave is impossible, the solution must be "one-click" or "brain-dead-simple."
Known Issues:
imread fails in 3.2.4 because file_in_path.m is wrong. You will need to update the file file_in_path.m to the following (just replace it):
function name=file_in_path(p,file)
idx=[1 findstr(p,pathsep) length(p)+1];
for i=1:length(idx)-1
if idx(i+1)-idx(i)<=1
dir=strcat(pwd,"/");
else
dir=p(idx(i)+1:idx(i+1)-1);
end
name = fullfile(dir, file);
fid = fopen(name,"r");
if fid >= 0
fclose(fid);
return
end
end
fid = fopen(file,"r");
if fid >= 0,
fclose(fid);
name=file;
return
end
name=[];
Solution: Create a distributable exe using mkoctfile, and package this exe with the core Octave files, and other .oct and .m files as necessary.
Step 1: Create a stand-alone executable.
You can see code that works here:
http://www.gnu.org/software/octave/doc/interpreter/Standalone-Programs.html
Particularly the file "embedded.cc".
I have simplified that file as follows:
#include <iostream>
#include <octave/oct.h>
#include <octave/octave.h>
#include <octave/parse.h>
int
main (int argc, char *argvc[])
{
string_vector argv (2);
argv(0) = "embedded";
argv(1) = "-q";
octave_main (2, argv.c_str_vec(), 1);
octave_value_list in = octave_value (argvc[1]);
octave_value_list out = feval ("your_custom_m_file", in);
if (!error_state && out.length () > 0)
{
}
else
{
std::cout << "invalid\n";
}
return 0;
}
Build this file with the command
mkoctfile --link-stand-alone embedded.cc -o embedded
It may throw warnings, but as long as it throws no errors, you should be fine. The file embedded.exe will be built, and can be run. The only issue is that it will lack all the goodies that make octave awesome. You will have to provide those.
Step 2: Create a distribution folder
You will need to create a copy of many of the Octave files. I suggest a directory specifically for this. At a minimum, you will need a copy of all or most of the DLLs in \bin. Additionally, place your distributable executable in this directory.
Step 3: Other files whack-a-mole
You will now need to find out what other files will be necessary to run your .m script. You can simplify this step by copying \oct\i686-pc-mingw32*.oct and \share\octave\3.2.4\m\*\*.m to the distribution directory, although this will be overkill, and will not actually prevent the whack-a-mole step.
Now, you must play whack-a-mole or the time-honored tradition of "where my includes be at, yo?"
Open a cmd prompt and navigate to your distribution folder.
Get rid of any useful PATH strings. Your customers won't have them.
Attempt to run the program embedded.exe. You will get an error such as the following:
embedded.exe
error: `max' undefined near line 83 column 22
error: evaluating argument list element number 1
error: evaluating argument list element number 1
error: called from:
error: T:\sms\Development\research\c2\disttest\strcat.m at line 83, column 3
error: T:\sms\Development\research\c2\disttest\file_in_path.m at line 5, column 10
error: T:\sms\Development\research\c2\disttest\imread.m at line 50, column 6
A Search in your Octave installation for "max". It will either be a .oct or a .m file. In this case, it is a .oct file, max.oct. Copy it to your distribution directory.
B You search for something obvious like "min", and get no results. This is because the Loadable Function "min" is in the .oct file "max.oct". Make a copy of max.oct, and rename it to min.oct. It will work now. How do you know where the functions are? I'm not sure. Most of them are in obvious places like "max.oct" for min, and "fft2.oct" for "ifft2.oct". Good luck with all that.
Repeat until your executable runs.
Just to add that if you want to run a script instead of an m function, then the line of the embedded.cc:
octave_value_list out = feval ("your_custom_m_file", in);
should be:
octave_value_list out = feval ("your_custom_m_script");
Also use 'which' to find where the missing functions are packed. For example for the min function:
octave:22> which min
min is a function from the file C:\Octave\Octave3.6.2_gcc4.6.2\lib\octave\3.6.2\oct\i686-pc-mingw32\max.oct
Something I found when linking my custom m file into an Octave standalone:
Needed #include <octave/toplev.h>
Replace return 0; (as above) with clean_up_and_exit(0);
Without these steps my program repeatedly crashed on exit.
Run mkoctfile --link-stand-alone embedded.cc -o embedded
from the octave solution and not from a batch file.
Just saved you half day (-;
In the above solution in bullet 4 B:
B You search for something obvious like "min", and get no results.
This is because the Loadable Function "min" is in the .oct file
"max.oct". Make a copy of max.oct, and rename it to min.oct. It will
work now.
This might not work if some function is being called from #folder function.m and also to avoid unnecessary duplicated files, just add the following code somewhere in your m file outside #folder
autoload ("min", "max.oct");
Likewise, it can be removed via
autoload ("min", "max.oct", "remove");
Ensure that the path to max.oct is provided here.
The above understanding is based on a file PKG_ADD and PKG_DEL in the communications package located at \Octave-4.0.1\lib\octave\packages\communications-1.2.1\i686-w64-mingw32-api-v50+\
Check out Stratego Octave Compiler.
(I've not tested it yet, but plan to do so in the next few days.)
I had that very same requirement (one-click, brain-dead-simple), so I made a setup that contained only curl.exe, the batch file below, an exe which was a .bat in disguise (simply calling the batch file below) and the .vbs script below (not writen by me). And of course my m-file.
This will download Octave 4.2.1 as a portable program (32 bit, otherwise we'dd have to download again if the system turns out to be 32 bit), unpack using the vbs script, move the contents to the same folder as the batch file and run it in GUI mode. Every next time the same script is called, it will only check if octave.bat is still there.
Of course this results in a huge waste of disk space, downloading the 280MB zip, which unpacks to over 1GB (which I make even worse by not deleting the zip afterwards), and you're stuck with a cmd window that is not easy to hide.
But it does offer the simplest solution I could find. It is also less likely to break in the future (either with an update of your own, or an update from Octave). Some glorious day, mkoktfile will actually be easy to use and will solve dependencies on its own, but until that day this remains the least headache-inducing solution I could find. And aspirins are more expensive than someone else's disk space.
::this file will test if the octave portable is downloaded and unpacked
#ECHO OFF
SET my_m_file=your_mfile.m
SET name_of_this_script=run_me.bat
::if the file exists, skip to the actual running.
IF EXIST "octave.bat" goto OctaveIsExtracted
IF EXIST "octave-4.2.1-w32.zip" goto OctaveIsDownloaded
ECHO The runtime (Octave portable 4.2.1) will now be downloaded.
ECHO This may take a long time, as it is about 280MB.
ECHO .
ECHO If this download restarts multiple times, you can manually download the octave-4.2.1-w32.zip from the GNU website. Make sure to unpack the contents.
::if this errors, you can uncomment the line with archive.org (which doesn't report total size during download)
curl http://ftp.gnu.org/gnu/octave/windows/octave-4.2.1-w32.zip > octave-4.2.1-w32.zip
::curl http://web.archive.org/web/20170827205614/https://ftp.gnu.org/gnu/octave/windows/octave-4.2.1-w32.zip > octave-4.2.1-w32.zip
:OctaveIsDownloaded
::check to see if the file size is the correct size to assume a successful download
::if the file size is incorrect, delete the file, restart this script to attempt a new download
::file size should be 293570269 bytes
call :filesize octave-4.2.1-w32.zip
IF /I "%size%" GEQ "293560000" goto OctaveIsDownloadedSuccessfully
del octave-4.2.1-w32.zip
::start new instance and exit and release this one
start %name_of_this_script%
exit
:OctaveIsDownloadedSuccessfully
IF EXIST "octave.bat" goto OctaveIsExtracted
::unzip and move those contents to the current folder
ECHO Unzipping octave portable, this may take a moment.
cscript //B j_unzip.vbs octave-4.2.1-w32.zip
SET src_folder=octave-4.2.1
SET tar_folder=%cd%
for /f %%a IN ('dir "%src_folder%" /b') do move %src_folder%\%%a %tar_folder%
pause
:OctaveIsExtracted
octave.bat %my_m_file%
goto :eof
:filesize
set size=%~z1
exit /b 0
And j_unzip.vbs
' j_unzip.vbs
'
' UnZip a file script
'
' By Justin Godden 2010
'
' It's a mess, I know!!!
'
' Dim ArgObj, var1, var2
Set ArgObj = WScript.Arguments
If (Wscript.Arguments.Count > 0) Then
var1 = ArgObj(0)
Else
var1 = ""
End if
If var1 = "" then
strFileZIP = "example.zip"
Else
strFileZIP = var1
End if
'The location of the zip file.
REM Set WshShell = CreateObject("Wscript.Shell")
REM CurDir = WshShell.ExpandEnvironmentStrings("%%cd%%")
Dim sCurPath
sCurPath = CreateObject("Scripting.FileSystemObject").GetAbsolutePathName(".")
strZipFile = sCurPath & "\" & strFileZIP
'The folder the contents should be extracted to.
outFolder = sCurPath
'original line: outFolder = sCurPath & "\"
WScript.Echo ( "Extracting file " & strFileZIP)
Set objShell = CreateObject( "Shell.Application" )
Set objSource = objShell.NameSpace(strZipFile).Items()
Set objTarget = objShell.NameSpace(outFolder)
intOptions = 256
objTarget.CopyHere objSource, intOptions
WScript.Echo ( "Extracted." )

how can i write a script to find the latest updated file and copy to certain directory

i have a process that downloads a file from a webbrower. it has the same name always (can't change that) so each file gets downloaded as file([latestnumber])
so in this directory i have:
joe.pdf
joe(1).pdf
joe(2).pdf
etc . . .
I now would like a script to take the "latest file" (joe(2).pdf in this case) and copy it to another directory.
something like GetLatestFile("joe") and copy to "X:\mydirectory"
can anyone think of an easy way to do this.
Do you have a preference as to what language you write your script in?
I wouldn't go by the name of the file, I'd choose whatever scripting language you are going to use, loop through the directory and look at the file attributes for each file to pick out the latest one, then move it to your target directory. This would be fairly trivial in a .NET console application with the classes available in the System.IO namespace. (namely the DirectoryInfo, FileInfo and File classes)
Try this: XCOPY C:\BATCH\*.* C:\UPLOAD /M
Put the code in a text file and rename it as whateveryouwant.bat and execute.
Be sure to edit the source and destination folder to your liking.
Is this what you're looking for ?
So, as it is enough to get the latest filename sorted by date, I suggest something like:
#echo off & setLocal enabledelayedexpansion
for /f "tokens=* delims= " %%a in ('dir /b/a-d/o-d') do (
set N=%%~Fa
goto :done
)
:done
echo !N!
Replace the last echo command for the "copy ..." or whatever you want to do with the newest file.
HTH!
Edit> If the files are not in the current directory, change the "dir" command accordingly
this uses sed, and regular expressions
http://gnuwin32.sourceforge.net/packages/sed.htm
it generates a bat file that does the job.
i've put the bat file in c:\crp so it doesn't become a latest file.
as a demonstration, i've created a latest file latestfile.txt
you can see the line that generates copyit.bat and you can amend it so the files goes exactly where you want.
C:>md c:\crp
C:>copy /y con latestfile.txt
fgfdgd^Z
1 file(s) copied.
C:>dir /o-d/a-d/b | find /N /V "QWERTY" | find "[1]" | sed -e s/[1](.*)/cop
y\d32\1\d32c:\newdir/>c:\crp\copyit.bat
C:>type c:\crp\copyit.bat
copy latestfile.txt c:\newdir