Copying all files in SSIS without Foreach - ssis

Is it possible to copy all the files from one folder to another without using foreach?
I have source as c:\test1*.txt
and destination as c:\test2
When I execute this using File System Task, I get the following error
An error occurred with the following error message: "Illegal characters in path.".

Yes, it's possible to copy all files from one folder to another. Below, my source is C:\test1 and my destination is C:\test2. The task below will copy all files from C:\test1 to C:\test2.
The error you are getting is due to the asterisk in your source. Are you trying to use a wildcard? The File System Task doesn't allow wildcards. Check out the documentation on the File System Task, below is an excerpt:
The File System task operates on a single file or directory.
Therefore, this task does not support the use of wildcard characters
to perform the same operation on multiple files. To have the File
System task repeat an operation on multiple files or directories, put
the File System task in a Foreach Loop container, as described in the
following steps:
Configure the Foreach Loop container On the Collection page of the
Foreach Loop Editor, set the enumerator to Foreach File Enumerator and
enter the wildcard expression as the enumerator configuration for
Files. On the Variable Mappings page of the Foreach Loop Editor, map a
variable that you want to use to pass the file names one at a time to
the File System task.
Add and configure a File System task Add a
File System task to the Foreach Loop container. On the General page of
the File System Task Editor, set the SourceVariable or
DestinationVariable property to the variable that you defined in the
Foreach Loop container.
The other option is to write a copy routine in a Script Task:
string fileName = string.Empty;
string destFile = string.Empty;
string sourcePath = #"C:\test1";
string targetPath = #"C:\test2";
// Create a new target folder, if necessary.
if (!System.IO.Directory.Exists(targetPath))
{
System.IO.Directory.CreateDirectory(targetPath);
}
if (System.IO.Directory.Exists(sourcePath))
{
string wildcard = "*.txt";
string[] files = System.IO.Directory.GetFiles(sourcePath, wildcard);
// Copy the files and overwrite destination files if they already exist.
foreach (string s in files)
{
fileName = System.IO.Path.GetFileName(s);
destFile = System.IO.Path.Combine(targetPath, fileName);
System.IO.File.Copy(s, destFile, true);
}
}
else
{
throw new Exception("Source path does not exist!");
}

Or an execute process task with this in it:
COPY c:\test1*.txt c:\test2
This code is 25x more efficient at reducing keyboard wear than a script task. :p

Related

How to export .csv files using a script in Dymola?

I am running a big set of simulations in Dymola using a script, so far, it works well.
However, it remains incomplete because all the results are still in .mat and I have not find a way to automatically save them as .csv.
I found the DataFiles.convertMATtoCSV() function, but it requires me to specify a list of variables to export. I would like it to export all the variables without writing them one by one, is it possible?
In the Dymola Manual, there is a section "Saving all values into a CSV file".
It contains the following example code:
// Define name of trajectory file (fileName) and CVS file
// (CSVfile)
fileName="PID_Controller.mat";
CSVfile="AllVariables.csv";
// Read the size of the trajectories in the result file and
// store in 'n'
n=readTrajectorySize(fileName);
// Read the names of the trajectories
names = readTrajectoryNames(fileName);
// Read the trajectories 'names' (and store in 'traj')
traj=readTrajectory(fileName,names,n);
// transpose traj
traj_transposed=transpose(traj);
// write the .csv file using the package 'DataFiles'
DataFiles.writeCSVmatrix(CSVfile, names, traj_transposed);
This should do what you want. Also it gives room for customization if necessary later...

Opensmile: unreadable csv file while extracting prosody features from wav file

I am extracting prosody features from an audio file while using Opensmile using Windows version of Opensmile. It runs successful and an output csv is generated. But when I open csv, it shows some rows that are not readable. I used this command to extract prosody feature:
SMILEXtract -C \opensmile-3.0-win-x64\config\prosody\prosodyShs.conf -I audio_sample_01.wav -O prosody_sample1.csv
And the output of csv looks like this:
[
Even I tried to use the sample wave file given in Example audio folder given in opensmile directory and the output is same (not readable). Can someone help me in identifying where the problem is actually? and how can I fix it?
You need to enable the csvSink component in the configuration file to make it work. The file config\prosody\prosodyShs.conf that you are using does not have this component defined and always writes binary output.
You can verify that it is the standart binary output in this way: omit the -O parameter from your command so it becomesSMILEXtract -C \opensmile-3.0-win-x64\config\prosody\prosodyShs.conf -I audio_sample_01.wav and execute it. You will get a output.htk file which is exactly the same as the prosody_sample1.csv.
How output csv? You can take a look at the example configuration in opensmile-3.0-win-x64\config\demo\demo1_energy.conf where a csvSink component is defined.
You can find more information in the official documentation:
Get started page of the openSMILE documentation
The section on configuration files
Documentation for cCsvSink
This is how I solved the issue. First I added the csvSink component to the list of the component instances. instance[csvSink].type = cCsvSink
Next I added the configuration parameters for this instance.
[csvSink:cCsvSink]
reader.dmLevel = energy
filename = \cm[outputfile(O){output.csv}:file name of the output CSV
file]
delimChar = ;
append = 0
timestamp = 1
number = 1
printHeader = 1
\{../shared/standard_data_output_lldonly.conf.inc}`
Now if you run this file it will throw you errors because reader.dmLevel = energy is dependent on waveframes. So the final changes would be:
[energy:cEnergy]
reader.dmLevel = waveframes
writer.dmLevel = energy
[int:cIntensity]
reader.dmLevel = waveframes
[framer:cFramer]
reader.dmLevel=wave
writer.dmLevel=waveframes
Further reference on how to configure opensmile configuration files can be found here

Knime loop to write csv file in different folder?

I am running a knime chunk loop to write always the same procedure in different csv files:
The Part with the python script until the csv write is working, when I do it without loop, but somehow he is not writing in the customized folder path, if I have the loop inside.
The target is to write a new csv-file for every loop (the output is a list).
The nodes are:
Chunk Loop: Rows per chunk: 51
Create file name:
Options Selected directioy: C:/....
Flow Variables: FileName: currentIteration
CSV Writer: Flow Variables: filename: CurrentIteration
How can I change the folder path of the file? He is always saving it in the default folder
Here is an example workflow (apologies for the weird blurry Windows 10 screenshots):
Create File Name config:
CSV Writer config:
You may need to run each node individually in order to create the flow variable before you can select it in the following node.

How to get SSIS to wait for a file to exist and/or become available

Scenario:
Package#1 creates a flat file that contains multiple messages (one per line) that is dropped in an external system's "INPUT" folder. The file is picked up by the external system and processed and the responses are written in the same format to a file in the "OUTPUT" folder. The file starts to be written while the external system is still processing messages, so it is written as foo.rsppro. When processing is complete and all response messages are written it is renamed foo.rsp.
I need to pick up that file once it is complete (i.e. after the rename) and process using Package#2, which will start immediatly following Package#1. When Package#2 starts, the external system could be in three states:
Processing the first message not yet complete and no response file written yet, in which case I need to wait for foo.rsppro to be written, then renamed to foo.rsp
Processing in progress and foo.rsppro written, in which case I need
to wait for foo.rsppro to be renamed to foo.rsp
Processing completed, foo.rsppro has been written and been renamed to foo.rsp, in which case I just need to process foo.rsp.
I have tried:
using a file in use task but that errors if the expected file isn't present when the task begins (i.e. errors for scenario 1 and 2)
using a file watcher task but that appears to ignore file renames by design, so will never handle scenario 1 or 2
Aside from building a script task, is there a custom task that will handle all three scenarios?
Edit: SSIS 2008 R2
only a script task can help in your case.
consider using FileSystemWatcher within the script if possible or have an application/windows service which can monitor file system using FileSystemWatcher and invoke your packages when the event is triggered.
humm, it seems that you can solve it by using a for each loop container on the output folder and set it to read only .rsp files. That would deal with your .rsp files.
how can scenario 1 and 2 happens if package 2 will only run after package1 is finish? As I understand, package1 renames the file so it will only end when all the files are processes and renamed
EDIT:
ok, no worry, there is a solution for everything.
How about, you create a variable on package1 called #TotalNumberOfFiles with the total number of files to be processed, then you use package one to call pacakge2 (not sure if you are doing this already, but if not is very simple, just use a execute pacakge task) and on package2 you create a "parent package variable" (this is very simple too in case you have never done it) and package 2 just start processing when there are #TotalNumberOfFiles files on the output folder with the .rsp extension?
EDIT2:
I dont know jf there is a command to get that, maybe google it, but if you dont find out you can add a foreachloop container pointing to the output directory and do something like this on a script task:
Public Sub Main()
Dts.Variables("User::filesCount").Value = Dts.Variables("User::FilesCount").Value + 1
Dts.TaskResult = ScriptResults.Success
End Sub
after it finishes counting, just compare with TotalNumberOfFiles. If equal, move to the next task, else sleep for a while and count again
Final code used as follows. Basically loops through until either the file is found or the max specified number of attempts is hit.
Imports System.Threading is required for Thread.sleep. It may not be the most processor efficient method but this is 100% dedicated hardware and the packages is are running in serial.
'loop until number of required attempts is hit or file is found
Do Until iCounter = iAttempts Or bFileFound = True
'Check if the file exists
If File.Exists(sFilename) Then
'Switch bFileFound to true
bFileFound = True
'Report that file has been found to VERIFY_Input_File_Exists_INT variable
Dts.Variables("VERIFY_Input_File_Exists_INT").Value = True
Dts.Events.FireInformation(1, "DEBUG:", sFilename & " found successfully.", "", 0, False)
Else
'sleep for specified time
Thread.Sleep(iInterval * 1000)
Dts.Events.FireInformation(1, "DEBUG:", sFilename & " not found successfully. Sleeping for " & iInterval & "* 1000", "", 0, False)
End If
'increment counter
iCounter = iCounter + 1
Loop

How do I create a simple Octave distributable without installing Octave

The Octave documentation on this subject is both intimidating and sparse.
I did not know where else to document the solution I found, so I am posting here. I apologize if that's inappropriate, but I want to help the next guy.
The following solution is for a simple windows distributable.
Use Case:
A solution is developed in Octave 3.2.4, and needs to be distributed to end-users with few computer skills. Installing and explaining Octave is impossible, the solution must be "one-click" or "brain-dead-simple."
Known Issues:
imread fails in 3.2.4 because file_in_path.m is wrong. You will need to update the file file_in_path.m to the following (just replace it):
function name=file_in_path(p,file)
idx=[1 findstr(p,pathsep) length(p)+1];
for i=1:length(idx)-1
if idx(i+1)-idx(i)<=1
dir=strcat(pwd,"/");
else
dir=p(idx(i)+1:idx(i+1)-1);
end
name = fullfile(dir, file);
fid = fopen(name,"r");
if fid >= 0
fclose(fid);
return
end
end
fid = fopen(file,"r");
if fid >= 0,
fclose(fid);
name=file;
return
end
name=[];
Solution: Create a distributable exe using mkoctfile, and package this exe with the core Octave files, and other .oct and .m files as necessary.
Step 1: Create a stand-alone executable.
You can see code that works here:
http://www.gnu.org/software/octave/doc/interpreter/Standalone-Programs.html
Particularly the file "embedded.cc".
I have simplified that file as follows:
#include <iostream>
#include <octave/oct.h>
#include <octave/octave.h>
#include <octave/parse.h>
int
main (int argc, char *argvc[])
{
string_vector argv (2);
argv(0) = "embedded";
argv(1) = "-q";
octave_main (2, argv.c_str_vec(), 1);
octave_value_list in = octave_value (argvc[1]);
octave_value_list out = feval ("your_custom_m_file", in);
if (!error_state && out.length () > 0)
{
}
else
{
std::cout << "invalid\n";
}
return 0;
}
Build this file with the command
mkoctfile --link-stand-alone embedded.cc -o embedded
It may throw warnings, but as long as it throws no errors, you should be fine. The file embedded.exe will be built, and can be run. The only issue is that it will lack all the goodies that make octave awesome. You will have to provide those.
Step 2: Create a distribution folder
You will need to create a copy of many of the Octave files. I suggest a directory specifically for this. At a minimum, you will need a copy of all or most of the DLLs in \bin. Additionally, place your distributable executable in this directory.
Step 3: Other files whack-a-mole
You will now need to find out what other files will be necessary to run your .m script. You can simplify this step by copying \oct\i686-pc-mingw32*.oct and \share\octave\3.2.4\m\*\*.m to the distribution directory, although this will be overkill, and will not actually prevent the whack-a-mole step.
Now, you must play whack-a-mole or the time-honored tradition of "where my includes be at, yo?"
Open a cmd prompt and navigate to your distribution folder.
Get rid of any useful PATH strings. Your customers won't have them.
Attempt to run the program embedded.exe. You will get an error such as the following:
embedded.exe
error: `max' undefined near line 83 column 22
error: evaluating argument list element number 1
error: evaluating argument list element number 1
error: called from:
error: T:\sms\Development\research\c2\disttest\strcat.m at line 83, column 3
error: T:\sms\Development\research\c2\disttest\file_in_path.m at line 5, column 10
error: T:\sms\Development\research\c2\disttest\imread.m at line 50, column 6
A Search in your Octave installation for "max". It will either be a .oct or a .m file. In this case, it is a .oct file, max.oct. Copy it to your distribution directory.
B You search for something obvious like "min", and get no results. This is because the Loadable Function "min" is in the .oct file "max.oct". Make a copy of max.oct, and rename it to min.oct. It will work now. How do you know where the functions are? I'm not sure. Most of them are in obvious places like "max.oct" for min, and "fft2.oct" for "ifft2.oct". Good luck with all that.
Repeat until your executable runs.
Just to add that if you want to run a script instead of an m function, then the line of the embedded.cc:
octave_value_list out = feval ("your_custom_m_file", in);
should be:
octave_value_list out = feval ("your_custom_m_script");
Also use 'which' to find where the missing functions are packed. For example for the min function:
octave:22> which min
min is a function from the file C:\Octave\Octave3.6.2_gcc4.6.2\lib\octave\3.6.2\oct\i686-pc-mingw32\max.oct
Something I found when linking my custom m file into an Octave standalone:
Needed #include <octave/toplev.h>
Replace return 0; (as above) with clean_up_and_exit(0);
Without these steps my program repeatedly crashed on exit.
Run mkoctfile --link-stand-alone embedded.cc -o embedded
from the octave solution and not from a batch file.
Just saved you half day (-;
In the above solution in bullet 4 B:
B You search for something obvious like "min", and get no results.
This is because the Loadable Function "min" is in the .oct file
"max.oct". Make a copy of max.oct, and rename it to min.oct. It will
work now.
This might not work if some function is being called from #folder function.m and also to avoid unnecessary duplicated files, just add the following code somewhere in your m file outside #folder
autoload ("min", "max.oct");
Likewise, it can be removed via
autoload ("min", "max.oct", "remove");
Ensure that the path to max.oct is provided here.
The above understanding is based on a file PKG_ADD and PKG_DEL in the communications package located at \Octave-4.0.1\lib\octave\packages\communications-1.2.1\i686-w64-mingw32-api-v50+\
Check out Stratego Octave Compiler.
(I've not tested it yet, but plan to do so in the next few days.)
I had that very same requirement (one-click, brain-dead-simple), so I made a setup that contained only curl.exe, the batch file below, an exe which was a .bat in disguise (simply calling the batch file below) and the .vbs script below (not writen by me). And of course my m-file.
This will download Octave 4.2.1 as a portable program (32 bit, otherwise we'dd have to download again if the system turns out to be 32 bit), unpack using the vbs script, move the contents to the same folder as the batch file and run it in GUI mode. Every next time the same script is called, it will only check if octave.bat is still there.
Of course this results in a huge waste of disk space, downloading the 280MB zip, which unpacks to over 1GB (which I make even worse by not deleting the zip afterwards), and you're stuck with a cmd window that is not easy to hide.
But it does offer the simplest solution I could find. It is also less likely to break in the future (either with an update of your own, or an update from Octave). Some glorious day, mkoktfile will actually be easy to use and will solve dependencies on its own, but until that day this remains the least headache-inducing solution I could find. And aspirins are more expensive than someone else's disk space.
::this file will test if the octave portable is downloaded and unpacked
#ECHO OFF
SET my_m_file=your_mfile.m
SET name_of_this_script=run_me.bat
::if the file exists, skip to the actual running.
IF EXIST "octave.bat" goto OctaveIsExtracted
IF EXIST "octave-4.2.1-w32.zip" goto OctaveIsDownloaded
ECHO The runtime (Octave portable 4.2.1) will now be downloaded.
ECHO This may take a long time, as it is about 280MB.
ECHO .
ECHO If this download restarts multiple times, you can manually download the octave-4.2.1-w32.zip from the GNU website. Make sure to unpack the contents.
::if this errors, you can uncomment the line with archive.org (which doesn't report total size during download)
curl http://ftp.gnu.org/gnu/octave/windows/octave-4.2.1-w32.zip > octave-4.2.1-w32.zip
::curl http://web.archive.org/web/20170827205614/https://ftp.gnu.org/gnu/octave/windows/octave-4.2.1-w32.zip > octave-4.2.1-w32.zip
:OctaveIsDownloaded
::check to see if the file size is the correct size to assume a successful download
::if the file size is incorrect, delete the file, restart this script to attempt a new download
::file size should be 293570269 bytes
call :filesize octave-4.2.1-w32.zip
IF /I "%size%" GEQ "293560000" goto OctaveIsDownloadedSuccessfully
del octave-4.2.1-w32.zip
::start new instance and exit and release this one
start %name_of_this_script%
exit
:OctaveIsDownloadedSuccessfully
IF EXIST "octave.bat" goto OctaveIsExtracted
::unzip and move those contents to the current folder
ECHO Unzipping octave portable, this may take a moment.
cscript //B j_unzip.vbs octave-4.2.1-w32.zip
SET src_folder=octave-4.2.1
SET tar_folder=%cd%
for /f %%a IN ('dir "%src_folder%" /b') do move %src_folder%\%%a %tar_folder%
pause
:OctaveIsExtracted
octave.bat %my_m_file%
goto :eof
:filesize
set size=%~z1
exit /b 0
And j_unzip.vbs
' j_unzip.vbs
'
' UnZip a file script
'
' By Justin Godden 2010
'
' It's a mess, I know!!!
'
' Dim ArgObj, var1, var2
Set ArgObj = WScript.Arguments
If (Wscript.Arguments.Count > 0) Then
var1 = ArgObj(0)
Else
var1 = ""
End if
If var1 = "" then
strFileZIP = "example.zip"
Else
strFileZIP = var1
End if
'The location of the zip file.
REM Set WshShell = CreateObject("Wscript.Shell")
REM CurDir = WshShell.ExpandEnvironmentStrings("%%cd%%")
Dim sCurPath
sCurPath = CreateObject("Scripting.FileSystemObject").GetAbsolutePathName(".")
strZipFile = sCurPath & "\" & strFileZIP
'The folder the contents should be extracted to.
outFolder = sCurPath
'original line: outFolder = sCurPath & "\"
WScript.Echo ( "Extracting file " & strFileZIP)
Set objShell = CreateObject( "Shell.Application" )
Set objSource = objShell.NameSpace(strZipFile).Items()
Set objTarget = objShell.NameSpace(outFolder)
intOptions = 256
objTarget.CopyHere objSource, intOptions
WScript.Echo ( "Extracted." )