I'm using 'CSV Data Set Config' with "While Controller". The CSV file has multiple values in one line (as against simple example of one variable per line - which is all over the place in net).
Example-
Id,BobId,TarFulDate,SSRId,EDPNumber,SiteCode,CrBy,CrDate,ModBy,ModDate,Status,Version,ToolVer,ShipDate,TMDate,MaintComments,ParentId,TOName
990:548254,18ATR0002,2018-04-02T10:00:00+05:30,548254,MEATLM-18ATR0002-001,NEATOM,LVerlli,2018-03-01T16:12:37.7230000+05:30,PFibacher,2018-05-15T12:19:33+05:30,Submitted,12,0,1,2018-04-02T10:00:00+05:30,,547011,18ATR0002-0600-0-2
I'm using a "While Controller". The "While Condition" is not working. It keeps on running in infinite loop. However, the values it picked from CSV file are correct. It reads all the values from the file and keeps on repeating.
I tried the following option, but none of them worked -
${__javaScript("${Id}" != "<EOF>",)}
${__javaScript(${Id} != null,)}
${__groovy(!vars.get('Id').equals('<EOF>'),)}
${__jexl3("${Id}" != "<EOF>")}
${__jexl3(${Id} != null)}
${__jexl3(${Id} != "<EOF>")}
${__jexl3('${Id}' != '<EOF>')}
I expect the while controller to exit and Application to stop when all the rows from csv file is processed.
You should set CSV Data Set Config parameters Recycle on EOF? as false, and Stop thread on EOF? as false, to get EOF value
When the end of file (EOF) is reached, and the recycle option is true, reading starts again with the first line of the file.
If the recycle option is false, and stopThread is false, then all the variables are set to when the end of file is reached.
I succeeded to make it works following this:
1- in the while condition put this condition:
${__groovy(vars.get('myVar') == null || (vars.get('myVar') as String) !="<EOF>",)}
2- add a if controller inside the while and put the sampler inside it, then in the if condition put this:
${__javaScript(vars.get("myVar")!="<EOF>")}
Related
I have been using the following Lua code for a while to do simply csv to array conversions, but everything previously had a value in every column, but this time on a csv formatted bank statement there are empty values, which this does not handle.
Here’s an example csv, with debit and credits.
Transaction Date,Transaction Type,Sort Code,Account Number,Transaction Description,Debit Amount,Credit Amount,Balance
05/04/2022,DD,'11-70-79,6033606,Refund,,10.00,159.57
05/04/2022,DEB,'11-70-79,6033606,Henry Ltd,30.00,,149.57
05/04/2022,SO,'11-70-79,6033606,NEIL PARKS,20.00,,179.57
01/04/2022,FPO,'11-70-79,6033606,MORTON GREEN,336.00,,199.57
01/04/2022,DD,'11-70-79,6033606,WORK SALARY,,100.00,435.57
01/04/2022,DD,'11-70-79,6033606,MERE BC,183.63,,535.57
01/04/2022,DD,'11-70-79,6033606,ABC LIFE,54.39,,719.20
I’ve tried different patterns (https://www.lua.org/pil/20.2.html), but none seem to work, I’m beginning to think I can’t fix this via the pattern as it’ll break how it works for the rest? I appreciate it if anyone can share how they would approach this…
local csvfilename = "/mnt/nas/Fireflyiii.csv"
local MATCH_PATTERN = "[^,]+"
local function create_array_from_file(csvfilename)
local file = assert(io.open(csvfilename, "r"))
local arr = {}
for line in file:lines() do
local row = {}
for match in string.gmatch(line, MATCH_PATTERN) do
table.insert(row, match)
end
table.insert(arr, row)
end
return arr
end
I've got a simple SSIS package that I want to include a section that determines if the file existed in the folder or not and email out either a fail or pass email. I've tried to use a script task after the main for loop section,and use precedence constraints to email either the file was there or not depending on success or failure. Every time I run it the script it picks up a success flag (File is present) from somewhere, when the file isn't there.
I put a message box output in the script code to show if the file exists variable was set to False or True, and it came up False, as well as on the mail send it shows false for variable value, but is sending a success email.
Iv'e got the following variables in my script task, the FileExist is a boolean variable set at package level to default false. In my script code I am checking if the xlFileName is existing and then setting a flag to print out Y or N
The code in the script is as follows:
Public Sub Main()
Dim fileExists As String
fileExists = CType(File.Exists(Dts.Variables("User::xlFileName").Value.ToString()), String)
If fileExists = "False" Then
MessageBox.Show("N")
Else
MessageBox.Show("Y")
End If
End Sub
Do I need to set the ScriptResult in the code to success/Failure, depending on outcome?
Dts.TaskResult = ScriptResults.Success
I tried this earlier and the script task just failed.
In the precedence constraints (Set to Evaluate Operation: Expression) I also had the following but removed them as they were not making any difference, but tested ok.
#[User::FileExists]="False" - For No File
#[User::FileExists]="True" - For File Present
I'm not a regular user of ssis, nor have I used script tasks as far back as I can remember, so be kind!
Thanks
Andrew
Edit: Precedent Constraint settings images below. I've now got these set to check the variable for true/false depending on if file is present or not:
Changed the variable to populate with True as a Boolean rather than a string value, and set taskresult to Success/Failure.
Public Sub Main()
Dim fileExists As Boolean
fileExists = CType(File.Exists(Dts.Variables("User::xlFileName").Value.ToString()), Boolean)
If fileExists = False Then
Dts.TaskResult = ScriptResults.Failure
Dts.Variables("User::FileExists").Value = False
Else
Dts.TaskResult = ScriptResults.Success
Dts.Variables("User::FileExists").Value = True
End If
End Sub
When I run the script and the file is not available the script task fails but sends the mail as if success:
I'm trying to load csv containing knn data (3 columns no names)
e.g
4 3 a
1 3 a
3 3 a
4 5 b
I have been able to load the file into a string.
When I try to move that into a table I get no errors, however when I print the table to screen I get values of nil.
I tried changing contents of file which gives the same result and if changed to (knn_data) I get the path address of the csv in all keys.
I'm trying to get the csv data to appear within the indexed table and in its 3 columns.
Here is the code:
--load kNN file.
local knn_data = system.pathForFile("knn.csv", system.ResourceDirectory)
local file, errorString = io.open(knn_data, "r")
if not file then
print("File Error: File Unavailable")
else
local contents = file:read("*a")
print(contents)
io.close(file)
end
file = nil
-- load data into table
dataset = {}
for line in io.lines(knn_data) do
dataset[#dataset+1] = (contents)
Previously attached screenshot of code
...
else
local contents= file:read("*a")
print(contents)
--io.close(file)
end
contents is a local variable in your else statement.
Outside of it, contents is nil.
dataset = {}
for line in io.lines(iknn_data) do
dataset[#dataset+1] = (contents)
So dataset[#dataset+1]= (contents) is equivalent to dataset[#dataset+1]= nil
Within that generic for loop, line contains the line read from the file. So actually you should work with that.
I received a flat file that cannot be generated in other way. The delimited is a comma and the text qualifier is a double quote. The problem is that sometimes a have a double quote in the value. In example:
"0","12345", "Centre d"edu et de recherche", "B8E7"
Because of the double quote in the value, I received this error:
[Flat File Source [58]] Error: The column delimiter for column "XYZ" was not found.
[Flat File Source [58]] Error: An error occurred while processing file "C:\somefile.csv" on data row 296.
What can I do to process this file?
I use SSIS 2016 with Visual Studio 2015
You can use the Flat File Source error output to redirect bad rows to another flat file and correct values manually while all valid rows will be processed.
There are many links online to learn more about Flat File Source Error Output:
Flat File source Error Output connection in SSIS
How to Avoid Package Design Flaws When Sourcing Data From Flat Files
Flat File Source Editor (Error Output Page)
Update 1 - Workaround using Script Component and conditional split
Since Flat File error output is not working you can use a script component with a conditional split to filter bad rows, the following update is a step by step guide to implement that:
Add a Flat File connection manager, Go To advanced Tab, Delete all columns except one column and change it length to 4000
Add a script component, Go to Input and Output Column Tab, add desired output columns (in this example 4 columns) and add a Flag Column of type DT_BOOL
Inside the Script Component write the following script to check if the number of columns is 4 then Flag = True which means this is a valid row else set Flag as False which mean that this is a bad row:
[Microsoft.SqlServer.Dts.Pipeline.SSISScriptComponentEntryPointAttribute]
public class ScriptMain : UserComponent
{
public override void Input0_ProcessInputRow(Input0Buffer Row)
{
if (!Row.Column0_IsNull && !String.IsNullOrWhiteSpace(Row.Column0))
{
string[] cells = Row.Column0.Split(new string[] { "\",\"" }, StringSplitOptions.None);
if (cells.Length == 4)
{
Row.Col1 = cells[0].TrimStart('\"');
Row.Col2 = cells[1];
Row.Col3 = cells[2];
Row.Col4 = cells[3].TrimEnd('\"');
Row.Flag = true;
}
else
{
bool cancel;
Row.Flag = false;
}
}
else
{
Row.Col1_IsNull = true;
Row.Col2_IsNull = true;
Row.Col3_IsNull = true;
Row.Col4_IsNull = true;
Row.Flag = true;
}
}
}
Add a conditional split to split rows based on Flag column
Map the Valid Rows output to the OLEDB Destination, and the Bad Rows output to another flat file where you only map Column0
For this diagram:
The "Get Score Files" Script obtains a list of files and puts them into a User Variable filelist (datatype object). That list is THrown into the "Find Score Files" Loop, and will process each item on the list.
I need it to run ONLY if their's files to be had. If the "Get Score Files" Script returns NO objects, I want the Package to End Successfuly. How do I tell it to do that?
Thanks
In "get score file" try this code
if (files.Count == 0)
{
Dts.Variables["files_present"].Value = false;
}
else
{
Dts.Variables["file_list"].Value =files;
Dts.Variables["files_present"].Value = true;
}`
In SSIS u should create one more variable(files_present) with bool type
Now in the precedence constraints expression before for each loop use files_present variable to check any file present or not`(if true file present else no files)