I have searched for this question everywhere and can't seem to find it so here we go.
I set up a Foreachloop container which is using the "Foreach File Enumerator" and in the Files section where you're naming the format of the file as well as a wildcard if you want to return only certain format files, I have in as F_*.csv which works fine, however, I can't seem to find a way to also return files who's name begin with D_. I'm aware this can get done by having 2 separate Foreachloop containers but is there any way it can be done in the same one so that it checks for both those files?
The reason I need this is because there are other csv files in that folder which don't begin with a D_ nor an F_ so I'm trying to exclude those.
Thanks in advance !
I do not believe you can specify a regex in the "Files" criteria. I would try to read in all files matching the broadest criteria ("*.csv") and then in the foreach loop, evaluate the filename into a variable and test the variable in the control flow. Add a sequence container to perform the desired ETL. On the connector to the sequence container, add a constraint for the filename test, if the test fails, do nothing, if it passes, move to the sequence container and perform actions.
Related
I want to be able to do Vimdiffs and Vimfolds on Bookmarks files that have been converted to CVS files ie with one description and one uri per line. However, because the Bookmarks file has multiple levels for the folders, the CSV file will also need fields for the different levels of folder names on each line.
I am new to jq but it seems like it should be able to do this sort of conversion?
Thanks,
Phil.
Have you tried to use any free tools like: https://json-csv.com/
or json2csv: https://www.npmjs.com/package/json2csv
If neither of those works, perhaps this approach.
When I need to reconstruct data I write a set of loops that identify each property I want for each line in my CSV. Let's say my JSON has Name, Email, Phone but for some reason all are at different object levels in my JSON.
First right a loop that resolves Name, then a loop for Email, and one for Phone. At the end of the first loop call the second, and from the second call the third.
Then you can use jq -n which allows to create JSON with no input.
So your CSV output would be like jq -n '{NewName: .["'$Name'"]}'
once you have a clean JSON with all data points at the same level CSV conversion is smooth.
Hope this helps
I am working on TCL script that will open another tcl file and I want to get value of variable from second opened file and use it in the first file.
Two file abc.tcl and xyz.tcl
abc.tcl opens file xyz.tcl and reads value of variable and use it in abc.tcl.
If xyz.tcl sets a global variable, abc.tcl will be able to see it if it used source to load in xyz.tcl.
Here's a simple example. This is xyz.tcl:
set SomeVariable 12345
This is abc.tcl:
source xyz.tcl
puts "The password on my luggage is $SomeVariable"
The source command is really very simple internally. It just reads in the contents of the file (into a string), and then internally evals that string. Yes, this means that you probably shouldn't put source inside a procedure, at least not unless you're sure what the consequences of this are.
I am trying to load data from a flat file. The file is around 2.5 GB in size and row count is close to billion. I am using a flat file source inside DFT. Few rows inside the file does not follow the column pattern, for example there is a extra delimiter or say text qualifier as value of one column. I want to skip those rows and load rest of the rows which has correct format. I am using SSIS 2014. Flat file source inside DFT is failing. I have set alwaysCheckforrowdelimiter property to false but still does not work. Since the file is too huge manually opening and changing is not possible. Kindly help.
I have the same idea as Nick.McDermaid but I can maybe help you a bit more.
You can clean your file with a regular expression. (In a script)
You just need to define a regex to match lines with the number of delimiter you want. Other lines should be deleted.
Here is a visual example executed in Notepad++
Notepad++ Example screenshot
Here is the pattern used for my example:
^[A-Z]*;[A-Z]*;[A-Z]*;[A-Z]*$
And the data sample:
AA;BB;CC;DD
AA;BB;CC;DD
AA;BB;CC;DD;EE
AA;BB;CC;DD
AA;BB;CC
AA;BB;CC;DD
AA;BB;CC;DD
You can try it on line: https://regex101.com/r/PIYIcY/1
Regards,
Arnaud
I'd like to loop two csv files in Jmeter. I found this which is close, but I'd like the outer file to give me the CSV filename for the inner CSV.
So the outer file might have
filename
A
B
C
And this would lead to the inner loop looping
A.csv
B.csv
C.csv
When I try the technique referenced above, I get an error that the filename does not exist and I can see in the error that the problem is that jmeter is not substituting the variable in the filename for CSV data set under the Loop Controller. I suspect jmeter evaluates all the variables at a time when the variable introduced by the outer CSV file are not yet defined.
JMeter is not substituting a variable, but it will substitute a property.
Convert your variable into property and the approach will start working.
See Knit One Pearl Two: How to Use Variables in Different Thread Groups. guide to learn how you can do it and this SO answer for working JMeter script realizing alike scenario.
I have files being generated by another program/user that have names such as "jh-1.txt, jh-2.txt, ..., jh-100.txt, ..., jh-1024.txt". I'm extracting a column from these files, manipulating the data, and outputting to a new matrix. The only problem is that Octave is using ASCII ordering and not natural ordering when reading in the files. Thus, the output matrix is not ordered in a natural way. My question is, can Octave sort file names in a natural order? I'm getting file names in the standard method:
fileDirectory = '/path/to/directory';
filePattern = fullfile(fileDirectory, '*.txt'); % Selects only the txt files.
dataFiles = dir(filePattern); % Gets the info from the txt files in the directory.
baseFileName = {dataFiles.name}'; % Gets all the txt file names.
I can't rename the files because this is a script for another user. They are on a Windows machine and already have Octave installed with Cygwin and I don't want to make them use the command line more than they have to because they are unfamiliar with it. Alternatively, it would be nice to have the output with the file names in a column but, I haven't figured that one out either (bit of a noob with Octave myself). That way the user could use Excel (which they are familiar with) to sort the columns.
I don't think there's a built in natural sort in Octave. However, there is a natural sort submission on Mathwork's File Exchange. I've not used it, but the comments imply it works in Octave too.