Get Row Count for Data Transferred using EzAPI SSIS - ssis

I am transferring some data from one table to another using SSIS with EzAPI. How can I get the number of rows that were transferred?
My setup is as follows
EzPackage package = new EzPackage();
EzOleDbConnectionManager srcConn;
EzOleDbSource src;
EzOleDbConnectionManager destConn;
EzOleDbDestination dest;
EzDataFlow dataFlow;
destConn = new EzOleDbConnectionManager(package); //set connection string
srcConn = new EzOleDbConnectionManager(package);
dataFlow = new EzDataFlow(package);
src = Activator.CreateInstance(typeof(EzOleDbSource), new object[] { dataFlow }) as EzOleDbSource;
src.Connection = srcConn;
src.SqlCommand = odbcImport.Query;
dest = Activator.CreateInstance(typeof(EzOleDbDestination), new object[] { dataFlow }) as EzOleDbDestination;
dest.Connection = destConn;
dest.AttachTo(src, 0, 0);
dest.AccessMode = AccessMode.AM_OPENROWSET_FASTLOAD;
DTSExecResult result = package.Execute();
Where in this can I add something to get the number of rows? For all versions of SQL server 2008r2 and up

The quick answer is that the Row Count Transformation isn't included out of the box. I had a brief post about that: Row Count with EzAPI
I downloaded the source project from CodePlex and then edited EzComponents.cs (in EzAPI\src) and added the following code
[CompID("{150E6007-7C6A-4CC3-8FF3-FC73783A972E}")]
public class EzRowCountTransform : EzComponent
{
public EzRowCountTransform(EzDataFlow dataFlow) : base(dataFlow) { }
public EzRowCountTransform(EzDataFlow parent, IDTSComponentMetaData100 meta) : base(parent, meta) { }
public string VariableName
{
get { return (string)Meta.CustomPropertyCollection["VariableName"].Value; }
set { Comp.SetComponentProperty("VariableName", value); }
}
}
The component id above is only for 2008.
For 2012, it's going to be E26997D8C-70DA-42B2-8208-A19CE3A9FE41 I don't have a 2012 installation at the moment to confirm I didn't transpose a value there but drop a Row Count component onto a data flow, right click and look at the properties. The component/class id is what that value needs to be. Similar story if you're dealing with 2005.
So, once you have the ability to use EzRowCountTransform, you can simply patch it into your existing script.
// Create an instance of our transform
EzRowCountTransform newRC = null;
// Create a variable to use it
Variable newRCVariable = null;
newRCVariable = package.Variables.Add("RowCountNew", false, "User", 0);
// ...
src.SqlCommand = odbcImport.Query;
// New code here too
newRC = new EzRowCountTransform(dataFlow);
newRC.AttachTo(src);
newRC.Name = "RC New Rows";
newRC.VariableName = newRCVariable.QualifiedName;
// Continue old code
I have a presentation on various approaches I've used over time and what I like/don't like about them. Type more, click less: a programmatic approach to building SSIS. It contains sample code for creating the EzRowCountTransform and usage.

Related

Retrieve ActiveDirectory from within Azue SSIS IR in Datafactory

I have the script task within SSIS as outlined below. It works on premises, but once deployed to Azure SSIS IR it simply fails with no error messages.
How can I configure Azure SSIS IR differently or how can I change the script below to work within the Azure SSIS IR?
public override void CreateNewOutputRows()
{
// Specify the connnectionstring of your domain
// #<a class="vglnk" href="http://mycompany.com" rel="nofollow"><span>mycompany</span><span>.</span><span>com</span></a> => LDAP://DC=mycompany,dc=com
// Consider using a variable or parameter instead
// of this hardcoded value. On the other hand
// how many times does your domain changes
string domainConnectionString = "LDAP://[complete LDAP connection].com/OU=Staff,DC=PMLLP,DC=com";
using (DirectorySearcher ds = new DirectorySearcher(new DirectoryEntry(domainConnectionString)))
{
ds.Filter = "(&" +
"(objectClass=user)" + // Only users and not groups
")";
// See ds. for more options like PageSize.
//ds.PageSize = 1000;
// Find all persons matching your filter
using (SearchResultCollection results = ds.FindAll())
{
// Loop through all rows of the search results
foreach (SearchResult result in results)
{
// Add a new row to the buffer
Output0Buffer.AddRow();
// Fill all columns with the value from the Active Directory
Output0Buffer.employeeID = GetPropertyValue(result, "employeeID");
Output0Buffer.mail = GetPropertyValue(result, "mail");
Output0Buffer.SamAccountName = GetPropertyValue(result, "SamAccountName");
Output0Buffer.UserPrincipalName = GetPropertyValue(result, "UserPrincipalName");
}
}
}
}

Warning messages with EZAPI EzDerivedColumn and input columns

When adding a derived column to a data flow with ezAPI, I get the following warnings
"Add stuff here.Inputs[Derived Column Input].Columns[ad_zip]" on "Add
stuff here" has usage type READONLY, but is not referenced by an
expression. Remove the column from the list of available input
columns, or reference it in an expression.
I've tried to delete the input columns, but either the method is not working or I'm doing it wrong:
foreach (Microsoft.SqlServer.Dts.Pipeline.Wrapper.IDTSInputColumn100 col in derFull.Meta.InputCollection[0].InputColumnCollection)
{
Console.WriteLine(col.Name);
derFull.DeleteInputColumn(col.Name);
}
I have the following piece of code that fixes the problem.
I got it from a guy called Daniel Otykier. So he is propably the one that should be credited for it... Unlesss he got it from someone else :-)
static public void RemoveUnusedInputColumns(this EzDerivedColumn component)
{
var usedLineageIds = new HashSet<int>();
// Parse all expressions used in new output columns, to determine which input lineage ID's are being used:
foreach (IDTSOutputColumn100 column in component.GetOutputColumns())
{
AddLineageIdsFromExpression(column.CustomPropertyCollection, usedLineageIds);
}
// Parse all expressions in replaced input columns, to determine which input lineage ID's are being used:
foreach (IDTSInputColumn100 column in component.GetInputColumns())
{
AddLineageIdsFromExpression(column.CustomPropertyCollection, usedLineageIds);
}
var inputColumns = component.GetInputColumns();
// Remove all input columns not used in any expressions:
for (var i = inputColumns.Count - 1; i >= 0; i--)
{
if (!usedLineageIds.Contains(inputColumns[i].LineageID))
{
inputColumns.RemoveObjectByIndex(i);
}
}
}
static private void AddLineageIdsFromExpression(IDTSCustomPropertyCollection100 columnProperties, ICollection<int> lineageIds)
{
int lineageId = 1;
var expressionProperty = columnProperties.Cast<IDTSCustomProperty100>().FirstOrDefault(p => p.Name == "Expression");
if (expressionProperty != null)
{
// Input columns used in expressions are always referenced as "#xxx" where xxx is the integer lineage ID.
var expression = expressionProperty.Value.ToString();
var expressionTokens = expression.Split(new[] { ' ', ',', '(', ')' });
foreach (var c in expressionTokens.Where(t => t.Length > 1 && t.StartsWith("#") && int.TryParse(t.Substring(1), out lineageId)))
{
if (!lineageIds.Contains(lineageId)) lineageIds.Add(lineageId);
}
}
}
Simple but not 100% Guaranteed Method
Call ReinitializeMetaData on the base component that EzApi is extending:
dc.Comp.ReinitializeMetaData();
This doesn't always respect some of the customizations and logic checks that EzAPI has, so test it carefully. For most vanilla components, though, this should work fine.
100% Guaranteed Method But Requires A Strategy For Identifying Columns To Ignore
You can set the UsageType property of those VirtualInputColumns to the enumerated value DTSUsageType.UT_IGNORED using EzApi's SetUsageType wrapper method.
But! You have to do this after you're done modifying any of the other metadata of your component (attaching other components, adding new input or output columns, etc.) since each of these triggers the ReinitializeMetaData method on the component, which automatically sets (or resets) all UT_IGNORED VirtualInputColumn's UsageType to UT_READONLY.
So some sample code:
// define EzSourceComponent with SourceColumnToIgnore output column, SomeConnection for destination
EzDerivedColumn dc = new EzDerivedColumn(this);
dc.AttachTo(EzSourceComponent);
dc.Name = "Errors, Go Away";
dc.InsertOutputColumn("NewDerivedColumn");
dc.Expression["NewDerivedColumn"] = "I was inserted!";
// Right here, UsageType is UT_READONLY
Console.WriteLine(dc.VirtualInputCol("SourceColumnToIgnore").UsageType.ToString());
EzOleDbDestination d = new EzOleDbDestination(f);
d.Name = "Destination";
d.Connection = SomeConnection;
d.Table = "dbo.DestinationTable";
d.AccessMode = AccessMode.AM_OPENROWSET_FASTLOAD;
d.AttachTo(dc);
// Now we can set usage type on columns to remove them from the available inputs.
// Note the false boolean at the end.
// That's required to not trigger ReinitializeMetadata for usage type changes.
dc.SetUsageType(0, "SourceColumnToIgnore", DTSUsageType.UT_IGNORED, false);
// Now UsageType is UT_IGNORED and if you saved the package and viewed it,
// you'll see this column has been removed from the available input columns
// ... and the warning for it has gone away!
Console.WriteLine(dc.VirtualInputCol("SourceColumnToIgnore").UsageType.ToString());
I was having exactly your problem and found a way to solve it. The problem is that the EzDerivedColumn has not the PassThrough defined in it's class.
You just need to add this to the class:
private PassThroughIndexer m_passThrough;
public PassThroughIndexer PassThrough
{
get
{
if (m_passThrough == null)
m_passThrough = new PassThroughIndexer(this);
return m_passThrough;
}
}
And alter the ReinitializeMetadataNoCast() to this:
public override void ReinitializeMetaDataNoCast()
{
try
{
if (Meta.InputCollection[0].InputColumnCollection.Count == 0)
{
base.ReinitializeMetaDataNoCast();
LinkAllInputsToOutputs();
return;
}
Dictionary<string, bool> cols = new Dictionary<string, bool>();
foreach (IDTSInputColumn100 c in Meta.InputCollection[0].InputColumnCollection)
cols.Add(c.Name, PassThrough[c.Name]);
base.ReinitializeMetaDataNoCast();
foreach (IDTSInputColumn100 c in Meta.InputCollection[0].InputColumnCollection)
{
if (cols.ContainsKey(c.Name))
SetUsageType(0, c.Name, cols[c.Name] ? DTSUsageType.UT_READONLY : DTSUsageType.UT_IGNORED, false);
else
SetUsageType(0, c.Name, DTSUsageType.UT_IGNORED, false);
}
}
catch { }
}
That is the strategy used by other components. If you want to see all the code you can check my EzApi2016#GitHub. I'm updating the original code from Microsoft to SQL Server 2016.

Create a Non-Database-Driven Lookup

Lots of references for creating lookups out there, but all seem to draw their values from a query.
I want to add a lookup to a field that will add items from a list of values that do not come from a table, query, or any other data source.
Such as from a string: "Bananas, Apples, Oranges"
..or a container ["Bananas", "Apples", "Oranges"]
Assume the string/container is a dynamic object. Drawing from an static enum is not a choice.
Is there a way to create lookups on the fly from something other than a data source?
Example code would be a great help, but I'll take hints as well.
There is the color picker.
Also in the Global you will find pickXxxx such as pickList.
There are others, pickUser, pickUserGroup etc.
Take a look on the implementation. I guess they build a temporary table then displays that. Tables are great!
Update:
To go on you own follow the rules.
For the advanced user, see also: Lookup form returning more than one value.
public void lookup()
{
SysTableLookup sysTableLookup;
TmpTableFieldLookup tmpTableFieldLookup;
Enumerator en;
List entitylist = new list(types::String);
entitylist.addend("Banana");
entitylist.addend("Apple");
en = entityList.getEnumerator();
while (en.moveNext())
{
tmpTableFieldLookup.TableName = en.current();
tmpTableFieldLookup.insert();
}
sysTableLookup = SysTableLookup::newParameters(tableNum(tmpTableFieldLookup), this);
sysTableLookup.addLookupfield(fieldNum(TmpTableFieldLookup, TableName));
//BP Deviation documented
sysTableLookup.parmTmpBuffer(tmpTableFieldLookup);
sysTableLookup.performFormLookup();
}
The above code helps in displaying strings as lookup.
I'm also guessing there's no way to perform a lookup without a table. I say that because a lookup is simply a form with one or more datasources that is displayed in a different way.
I've also blogged about this, so you can get some info on how to perform a lookup, even with a temporary table, here:
http://devexpp.blogspot.com.br/2012/02/dynamics-ax-custom-lookup.html
Example from global::PickEnumValue:
static int pickEnumValue(EnumId _enumId, boolean _omitZero = false)
{
Object formRun;
container names;
container values;
int i,value = -1,valueIndex;
str name;
#ResAppl
DictEnum dictEnum = new DictEnum(_enumId);
;
if (!dictEnum)
return -1;
for (i=1;i<=dictEnum.values();i++)
{
value = dictEnum.index2Value(i);
if (!(_omitZero && (value == 0)))
{
names += dictEnum.index2Label(i);
values += value;
}
}
formRun = classfactory.createPicklist();
formRun.init();
formRun.choices(names, #ImageClass);
formRun.caption(dictEnum.label());
formRun.run();
formRun.wait();
name = formRun.choice();
value = formRun.choiceInt();
if (value>=0) // the picklist form returns -1 if a choice has not been made
{
valueIndex = -1;
for (i=1;i<=conLen(names);i++)
{
if (name == conPeek(names,i))
{
valueIndex = i;
break;
}
}
if (valueIndex>=0)
return conPeek(values,valueIndex);
}
return value;
}
It isn't the most graceful solution, but this does work, and it doesn't override or modify any native AX 2012 objects:
Copy the sysLookup form from AX2009 (rename it) and import it into AX 2012.
We'll call mine myLookupFormCopy.
I did a find/replace of "sysLookup" in the XPO file to rename it.
Create this class method:
public static client void lookupList(FormStringControl _formStringControl, List _valueList, str _columnLabel = '')
{
Args args;
FormRun formRun;
;
if (_formStringControl && _valueList && _valueList.typeId() == Types::String)
{
args = new Args(formstr(myLookupFormCopy));
args.parmObject(_valueList);
args.parm(_columnLabel);
formRun = classFactory.formRunClass(args);
_formStringControl.performFormLookup(formRun);
}
}
In the lookup method for your string control, use:
public void lookup()
{
List valueList = new List(Types::String);
;
...build your valueList here...
MyClass::lookupList(this, valueList, "List Title");
super();
}

How do I retrieve Insert Statements without using SubmitChanges in LINQPad

I have something similar to the code below in LINQPAD using C# Statements. My goal is to get the actual SQL Insert statments not actually update the database.
I can easily delete the data after it inserts with this small sample, but I will need this for a larger push of data. I hope that I have missed something simple either in L2S or LINQPad.
Is there an easier way retrieve the SQL Insert?
var e1 = new MyEntity(){ Text = "First" };
var e2 = new MyEntity(){ Text = "Second" };
MyEntities.InsertOnSubmit(e1);
MyEntities.InsertOnSubmit(e2);
SubmitChanges();
A quick-n-dirty way is to wrap everything in a transaction scope that is never commited:
using(TransactionScope ts = new TransactionScope())
{
var e1 = new MyEntity(){ Text = "First" };
var e2 = new MyEntity(){ Text = "Second" };
MyEntities.InsertOnSubmit(e1);
MyEntities.InsertOnSubmit(e2);
SubmitChanges();
// Deliberately not committing the transaction.
}
This works well for small volumes. If the data volume is large and you have full recovery model on the database the transaction log growth might become a problem.
When we did the samples for "LINQ in Action", we used the following method which gets the scheduled changes from the context:
public String GetChangeText(System.Data.Linq.DataContext context)
{
MethodInfo mi = typeof(DataContext).GetMethod("GetChangeText",
BindingFlags.NonPublic | BindingFlags.Instance);
return mi.Invoke(context, null).ToString();
}
If you want to see this in action, download the samples in LINQPad (see http://www.thinqlinq.com/Default/LINQ-In-Action-Samples-available-in-LINQPad.aspx) and check out chapter 6 example 6.29.

Ways to update only modified columns in table using LINQ

I have 20 fields on form, how to update fields modified by user during runtime and how to check which fields have changed so that i can only update those values in table using LINQ. I am working on windows application using C# and VS2010
Please refer the code (Currently i am passing all values, i knw this is not the correct way)
private void UpdateRecord(string groupBoxname)
{
using (SNTdbEntities1 context = new SNTdbEntities1())
{
{
Vendor_Account va = new Vendor_Account();
var Result = from grd in context.Vendor_Account
where grd.Bill_No == dd_billNo.Text
select grd;
if (Result.Count() > 0)
if ((dd_EditProjectName.Text!= "Select") && (dd_billNo.Text!="Select") && (dd_editVendorName.Text!="Select"))
{
foreach (var item in Result)
{
va.Account_ID = item.Account_ID;
}
va.Amount_After_Retention = Convert.ToDecimal(txt_AD_AfterRet.Text);
va.Balance = Convert.ToDecimal(txt_AD_Balance.Text);
va.Bill_Amount = Convert.ToDecimal(txt_AD_BillAmount.Text);
va.Bill_Date = Convert.ToDateTime(dt_AD_BillDate.Text);
va.Bill_No = dd_billNo.Text;
va.Comments = txt_AD_Comments.Text;
va.Paid_Till_Date = string.IsNullOrEmpty(txt_AD_Paid.Text)?0:Convert.ToDecimal(txt_AD_Paid.Text);
va.Project_Name = dd_EditProjectName.Text;
va.Retention_Perc = Convert.ToDecimal(txt_retPerc.Text);
va.Amount_After_Retention = Convert.ToDecimal(txt_AD_AfterRet.Text);
va.Vendor_Name = dd_editVendorName.Text;
va.Vendor_Code = txt_AD_Code.Text;
context.Vendor_Account.ApplyCurrentValues(va);
//context.Vendor_PersonalInfo.AddObject(vpi);
context.SaveChanges();
MessageBox.Show("Information Updated Sucessfully!");
lbl_Warning.Text = "";
entityDataSource1.Refresh();
}
else
{
MessageBox.Show("Vendor Name,Project Name and Bill No cannot be blank!!");
}
}
}
}
Entity framework will do that task.
Since you didn't provide any code, I cannot be precise about the answer but please check those links:
http://msdn.microsoft.com/en-us/library/aa697427(v=vs.80).aspx, section:Manipulating Data and Persisting Changes
http://www.codeproject.com/KB/database/sample_entity_framework.aspx
Note that the SaveChanges() function will update any modification done the records in EF.
Create some field dublicates, and compare value from the form element with the local dublicate, if it was changed than update it.