For this diagram:
The "Get Score Files" Script obtains a list of files and puts them into a User Variable filelist (datatype object). That list is THrown into the "Find Score Files" Loop, and will process each item on the list.
I need it to run ONLY if their's files to be had. If the "Get Score Files" Script returns NO objects, I want the Package to End Successfuly. How do I tell it to do that?
Thanks
In "get score file" try this code
if (files.Count == 0)
{
Dts.Variables["files_present"].Value = false;
}
else
{
Dts.Variables["file_list"].Value =files;
Dts.Variables["files_present"].Value = true;
}`
In SSIS u should create one more variable(files_present) with bool type
Now in the precedence constraints expression before for each loop use files_present variable to check any file present or not`(if true file present else no files)
Related
I'm using 'CSV Data Set Config' with "While Controller". The CSV file has multiple values in one line (as against simple example of one variable per line - which is all over the place in net).
Example-
Id,BobId,TarFulDate,SSRId,EDPNumber,SiteCode,CrBy,CrDate,ModBy,ModDate,Status,Version,ToolVer,ShipDate,TMDate,MaintComments,ParentId,TOName
990:548254,18ATR0002,2018-04-02T10:00:00+05:30,548254,MEATLM-18ATR0002-001,NEATOM,LVerlli,2018-03-01T16:12:37.7230000+05:30,PFibacher,2018-05-15T12:19:33+05:30,Submitted,12,0,1,2018-04-02T10:00:00+05:30,,547011,18ATR0002-0600-0-2
I'm using a "While Controller". The "While Condition" is not working. It keeps on running in infinite loop. However, the values it picked from CSV file are correct. It reads all the values from the file and keeps on repeating.
I tried the following option, but none of them worked -
${__javaScript("${Id}" != "<EOF>",)}
${__javaScript(${Id} != null,)}
${__groovy(!vars.get('Id').equals('<EOF>'),)}
${__jexl3("${Id}" != "<EOF>")}
${__jexl3(${Id} != null)}
${__jexl3(${Id} != "<EOF>")}
${__jexl3('${Id}' != '<EOF>')}
I expect the while controller to exit and Application to stop when all the rows from csv file is processed.
You should set CSV Data Set Config parameters Recycle on EOF? as false, and Stop thread on EOF? as false, to get EOF value
When the end of file (EOF) is reached, and the recycle option is true, reading starts again with the first line of the file.
If the recycle option is false, and stopThread is false, then all the variables are set to when the end of file is reached.
I succeeded to make it works following this:
1- in the while condition put this condition:
${__groovy(vars.get('myVar') == null || (vars.get('myVar') as String) !="<EOF>",)}
2- add a if controller inside the while and put the sampler inside it, then in the if condition put this:
${__javaScript(vars.get("myVar")!="<EOF>")}
I received a flat file that cannot be generated in other way. The delimited is a comma and the text qualifier is a double quote. The problem is that sometimes a have a double quote in the value. In example:
"0","12345", "Centre d"edu et de recherche", "B8E7"
Because of the double quote in the value, I received this error:
[Flat File Source [58]] Error: The column delimiter for column "XYZ" was not found.
[Flat File Source [58]] Error: An error occurred while processing file "C:\somefile.csv" on data row 296.
What can I do to process this file?
I use SSIS 2016 with Visual Studio 2015
You can use the Flat File Source error output to redirect bad rows to another flat file and correct values manually while all valid rows will be processed.
There are many links online to learn more about Flat File Source Error Output:
Flat File source Error Output connection in SSIS
How to Avoid Package Design Flaws When Sourcing Data From Flat Files
Flat File Source Editor (Error Output Page)
Update 1 - Workaround using Script Component and conditional split
Since Flat File error output is not working you can use a script component with a conditional split to filter bad rows, the following update is a step by step guide to implement that:
Add a Flat File connection manager, Go To advanced Tab, Delete all columns except one column and change it length to 4000
Add a script component, Go to Input and Output Column Tab, add desired output columns (in this example 4 columns) and add a Flag Column of type DT_BOOL
Inside the Script Component write the following script to check if the number of columns is 4 then Flag = True which means this is a valid row else set Flag as False which mean that this is a bad row:
[Microsoft.SqlServer.Dts.Pipeline.SSISScriptComponentEntryPointAttribute]
public class ScriptMain : UserComponent
{
public override void Input0_ProcessInputRow(Input0Buffer Row)
{
if (!Row.Column0_IsNull && !String.IsNullOrWhiteSpace(Row.Column0))
{
string[] cells = Row.Column0.Split(new string[] { "\",\"" }, StringSplitOptions.None);
if (cells.Length == 4)
{
Row.Col1 = cells[0].TrimStart('\"');
Row.Col2 = cells[1];
Row.Col3 = cells[2];
Row.Col4 = cells[3].TrimEnd('\"');
Row.Flag = true;
}
else
{
bool cancel;
Row.Flag = false;
}
}
else
{
Row.Col1_IsNull = true;
Row.Col2_IsNull = true;
Row.Col3_IsNull = true;
Row.Col4_IsNull = true;
Row.Flag = true;
}
}
}
Add a conditional split to split rows based on Flag column
Map the Valid Rows output to the OLEDB Destination, and the Bad Rows output to another flat file where you only map Column0
I have a hash table here and I have it eventually outputting to an Excel spreadsheet, but the issue appears to be the way the system sorts the hash table by default. I want it to return the machines in the same order that they are inputted, they way it currently works is a box pops up and you paste in all your machine names so they are all in memory prior to the foreach loop. I was previously sorting this by the longest uptime but it now needs to be the same way they are inputted. My initial thought is to create another hash table and capture them in the same order versus the $machineList variable, but that might even leave me in the same position. I tried to search but I couldn't find info on the default way that hash tables sort.
Any ideas?
$machineUptime = #{}
foreach($machine in $machineList){
if(Test-Connection $machine -Count 1 -Quiet){
try{
$logonUser = #gets the logged on user
$systemUptime = #gets the wmi property for uptime
if($logonUser -eq $null){
$logonUser = "No Users Logged on"
}
$machineUptime[$machine] = "$systemUptime - $logonUser"
}
catch{
Write-Error $_
$machineUptime[$machine] = "Error retrieving uptime"
}
}
else{
$machineUptime[$machine] = "Offline"
}
}
Create $machineUptime as an ordered hashtable (provided you have PowerShell v3 or newer):
$machineUptime = [ordered]#{}
I've 10 flat files(.dat format) in a folder which need to be uploaded into database everyday at a scheduled time.
All files related information are present in a database like File name, file path, table name, column names and delimiter.
We need to check each file exists or not, if not, need to log an entry, "File Not Found".
If the file exists, it needs to check for a trailer record(the last record in the file which says, Count=00001000, It has to be the count of number of records in that particular file).
If the trailer record not exists, then need to log an entry "No trailer record found".If the trailer record says zero count, then a log entry has to be made "Zero count" and also, if the counts of the file are not matching a log entry is needed, "Count mismatch".
If all the conditions are satisfied then data need to be loaded into database for each of the file.
Please suggest your ideas to implement the above scenario. Thanks!!!
Following solution may help you to resolve the issue.
Use the For each loop container with "Item" enumerator. Since you have 10 files and if something missing you need raise then you should use this. File enumerator just iterate through the files, not raises any error.
Following are Steps.
Create following SSIS package with variables.
FileFullPath
IsValidated
For each loop enumerator should be configured as following screenshots.
Configuartion in collection:
configuration in Variable section
Inside the container have a script task. you have to mention the FileFullPath as readonly variable and IsValidate as read and write like the following screen.
Click Edit script and insert the following code.
public void Main()
{
Dts.Variables["IsValidated"].Value = true;
string fileFullPath = Dts.Variables["FileFullPath"].Value.ToString();
if (!File.Exists(fileFullPath))
{
var msg = String.Format("File is not available in location : {0}", fileFullPath);
Dts.Events.FireError(0, "Dat file loading", msg, string.Empty, 0);
Dts.TaskResult = (int)ScriptResults.Failure;
}
//Read last line
String lstLine = File.ReadLines(fileFullPath).Last();
int totalCount = 0;
bool talierExists = int.TryParse(lstLine, out totalCount);
if (!talierExists)
{
var msg = String.Format("No tailer row found and last line is : {0}", lstLine);
Dts.Events.FireError(0, "Dat file loading", msg, string.Empty, 0);
Dts.TaskResult = (int)ScriptResults.Failure;
}
//Total count
int fullCount = File.ReadLines(fileFullPath).Count();
if (fullCount != totalCount)
{
var msg = String.Format("No of count is not matching, tailer count = {0} and full count={1}");
Dts.Events.FireError(0, "Dat file loading", msg, string.Empty, 0);
Dts.TaskResult = (int)ScriptResults.Failure;
}
Dts.Variables["IsValidated"].Value = true;
Dts.TaskResult = (int)ScriptResults.Success;
}
After that have your Data flow. Connect the script task with your data flow and right click on the connector and go to edit and configure as follows.
Your SSIS package will looks like follows.
Hope this helps!
I have created an SSIS package which processes .CSV files using a ForEachLoop container.
All the csv files contains "END OF FILE" in the last row.
Only those CSV files will be processed if it contains "END OF FILE" in the last row.
How can it be done. Please help.
Thanks in advance.
Create a variable check
Name DataType Value
check int 0
Let's say you have a package design like the one below
Script task is to check the file which has End of File at the last row
In the Script task add the variable check in ReadWriteVariable section and the output variable from ForEach container (suppose the variable name is LoopFiles) in ReadOnlyVariables
In the script task add the following code to read the file .There are several ways you can read the files here and here
public void Main()
{
int counter = 0;
string loop= Dts.Variables["User::LoopFiles"].Value.ToString();
string line;
using (StreamReader files = new StreamReader(file))
{
while((line = files.ReadLine()) != null)
{
if (line.ToLower() == "End Of File".ToLower())
{
Dts.Variables["User::check"].Value = 1;
}
}
}
Dts.TaskResult = (int)ScriptResults.Success;
}
Double Click the green arrow connection script task and Data Flow Task .A precedence dialog box will open and enter the expression as below
There are a number of ways that this could be done. One way would be:
Create the following variables:
EOF_Found Boolean
Row_Count Integer
Bring the data into a dataflow using the Flat File Source
Use a row count component to add the number of rows to Row_Count, to identify the last row later
Use a script component to loop through the rows, adding 1 to a counter for each row
When your counter equals the value in Row_Count (i.e. you are looking at the last row) check the value in the column that you expect "END OF FILE" to appear (depends on how you set up the flat file connection manager). if it equals "END OF FILE", change the value of EOF_Found to True
After the script component, add a derived column referencing the value in EOF_Found
Use a conditional split, checking the value of the derived column and only process if True
This solution avoids reading the entire file line by line. I have merged Praveen's code here for sake of completeness.
public void Main()
{
string line = ReadLastLine(#"c:\temp\EOF.cs");
if (line.ToUpper() == "END OF FILE")
{
Dts.Variables["User::check"].Value = 1;
}
Dts.TaskResult = (int)ScriptResults.Success;
}
public static string ReadLastLine(string path)
{
StreamReader stream = new StreamReader(path);
string str = stream.ReadToEnd();
int i = str.LastIndexOf('\n');
string lastLine = str.Substring(i + 1);
return lastLine;
}