I am working with EZApi to assist in creating a package to stage data for transformation. It is working in terms of data movement. When opening the package in the designer however there are warning messages surrounding the Derived Column and the InputColumns being set to read only.
Warning 148 Validation warning. Staging TableName:
{AA700319-FC05-4F06-A877-599E826EA833}: The "Additional
Columns.Inputs[Derived Column Input].Columns[DataSourceID]" on
"Additional Columns" has usage type READONLY, but is not referenced by
an expression. Remove the column from the list of available input
columns, or reference it in an expression. StageFull.dtsx 0 0
I can manually change them in the designer to be Read/Write or unselect them and the warning goes away. I am unable to get this to work programmatically however.
I have tried removing the columns from the metadata which works but doesn't remove them from the component so the columns are still created in the xml.
XML section
<externalMetadataColumn refId="Package\Full\Staging TableName\DestinationStaging TableName.Inputs[OLE DB Destination Input].ExternalColumns[DataSourceID]" dataType="i4" name="DataSourceID" />
When I try to go to the underlying object and delete the column using component.DeleteInput(id) I get an error message stating that the input column cannot be removed.
0xC0208010
-1071611888
DTS_E_CANTDELETEINPUT
An input cannot be deleted from the inputs collection.
Here is the code I am using to create a data flow task with an OLEDB Source, Derived Column, and OLE DB Destination.
Note that the input columns are not present until after the derived column is attached to the Source: dc.AttachTo(source);
public class EzMyDataFlow : EzDataFlow
{
public EzMyDataFlow(EzContainer parent, EzSqlOleDbCM sourceconnection,
EzSqlOleDbCM destinationconnection, string destinationtable, string sourcecomannd, string dataflowname)
: base(parent)
{
Name = dataflowname;
EzOleDbSource source = new EzOleDbSource(this);
source.Connection = sourceconnection;
source.SqlCommand = sourcecomannd;
source.AccessMode = AccessMode.AM_SQLCOMMAND;
source.Name = string.Format("Source_{0}", dataflowname);
EzDerivedColumn dc = new EzDerivedColumn(this);
dc.Name = "Additional Columns";
// Setup DataSourceID
string columnName = DBSchema.ReportFoundationalColumns.DataSourceID;
dc.InsertOutputColumn(columnName);
dc.SetOutputColumnDataTypeProperties(columnName, DataType.DT_I4, 0, 0, 0, 0);
var c = dc.OutputCol(columnName);
var property = c.CustomPropertyCollection["Expression"];
property.Name = "Expression";
property.Value = "#[TM::SourceDatabaseID]";
property = c.CustomPropertyCollection["FriendlyExpression"];
property.Name = "FriendlyExpression";
property.Value = "#[TM::SourceDatabaseID]";
dc.AttachTo(source);
EzOleDbDestination destination = new EzOleDbDestination(this);
destination.Table = destinationtable;
destination.Connection = destinationconnection;
destination.Name = string.Format("Destination{0}", dataflowname);
destination.AttachTo(dc);
}
}
Related
I'm working on a SSIS package where I have a text file with 5 columns. I need to check if all the rows for 5th column are NULL values.
If all the rows in 5th column are NULL then all the data should go for invalid file table.
If any row in 5th column have non NULL value then all the data should go to valid table.
You will need to read the entire file before being able to make the decision of where to write it to, so introduce a third table where you can stage the data first
Next part would be to build the logic that checks the staging table for all NULLS. Below query would return 0 if all was NULL and more than 0 if any record had a value
SELECT COUNT(*) FROM dbo.StagingTable ST WHERE ST.Column5 IS NOT NULL
Once you feed the answer into a variable you can use precedence constraints to fire the dataflow copy [staging to active] if the result was more than 0 or [staging to faulty] if the result was 0
personally if i had to perform this task I would use a script task to do it all:
Load into a data table
Use linq to check column to determine destination .Where(x => x[4]!=null).Count()
Load to destination via bulk Copy
You can check if the file is empty with C# using an OleDbDataAdapter and search the file, then determine where to load the file using SSIS Precedence Constraints. This example uses a CSV file without column names. If the columns do have names add the replacement code noted in the comments below. You will also need the using statements listed.
Add an SSIS Boolean variable. This is IsColumnNull in the following example. Next add a C# Script Task with IsColumnNull variable in the ReadWriteVariables field, and (optionally) a variable holding the file path ReadOnlyVariables pane.
Next set Precedence Constraints to check for both a true condition (has null rows) or false condition (does not have null records). Since the IsColumnNull variable is a Boolean, use just the variable itself as the expression to check for all null rows, but add ! for non-nulls, i.e. !#[User::IsColumnNull].
Connect the appropriate Data Flow Tasks with each destination table to the corresponding Precedence Constraint. For example, add the Data Flow Task with the "invalid file table" as the destination after the Precedence Constraint checking for a true value in the IsColumnNull variable.
Precedence Constraint For Rows with Nulls:
Precedence Constraint for Rows without Nulls:
Script Task Example:
using System;
using System.Data;
using System.IO;
using System.Data.OleDb;
using System.Linq;
using Microsoft.SqlServer.Dts.Runtime;
using System.Windows.Forms;
string fullFilePath = Dts.Variables["User::FilePath"].Value.ToString();
string fileName = Path.GetFileName(fullFilePath);
string filePath = Path.GetDirectoryName(fullFilePath);
string connStr = "Provider=Microsoft.Jet.OLEDB.4.0;Data Source=" + filePath
+ ";Extended Properties=\"text;HDR=No;FMT=Delimited\";";
//add filter for NOT NULL on given column to only return non-nulls
string sql = "SELECT F2 FROM " + fileName + " WHERE F2 IS NOT NULL";
//if file has column names replce "connStr" and "sql" as shown below
/*
string connStr = "Provider=Microsoft.Jet.OLEDB.4.0;Data Source=" + filePath
+ ";Extended Properties=\"text;HDR=Yes;FMT=Delimited\";";
string sql = "SELECT ID FROM " + fileName + " WHERE ID IS NOT NULL";
*/
using (OleDbDataAdapter oleAdpt = new OleDbDataAdapter(sql, connStr))
{
DataTable dt = new DataTable();
oleAdpt.Fill(dt);
//if emtpy set IsColumnNull SSIS variable to true
if (dt.Select().Count() < 1)
{
Dts.Variables["User::IsColumnNull"].Value = true;
}
else
{
Dts.Variables["User::IsColumnNull"].Value = false;
}
}
I received a flat file that cannot be generated in other way. The delimited is a comma and the text qualifier is a double quote. The problem is that sometimes a have a double quote in the value. In example:
"0","12345", "Centre d"edu et de recherche", "B8E7"
Because of the double quote in the value, I received this error:
[Flat File Source [58]] Error: The column delimiter for column "XYZ" was not found.
[Flat File Source [58]] Error: An error occurred while processing file "C:\somefile.csv" on data row 296.
What can I do to process this file?
I use SSIS 2016 with Visual Studio 2015
You can use the Flat File Source error output to redirect bad rows to another flat file and correct values manually while all valid rows will be processed.
There are many links online to learn more about Flat File Source Error Output:
Flat File source Error Output connection in SSIS
How to Avoid Package Design Flaws When Sourcing Data From Flat Files
Flat File Source Editor (Error Output Page)
Update 1 - Workaround using Script Component and conditional split
Since Flat File error output is not working you can use a script component with a conditional split to filter bad rows, the following update is a step by step guide to implement that:
Add a Flat File connection manager, Go To advanced Tab, Delete all columns except one column and change it length to 4000
Add a script component, Go to Input and Output Column Tab, add desired output columns (in this example 4 columns) and add a Flag Column of type DT_BOOL
Inside the Script Component write the following script to check if the number of columns is 4 then Flag = True which means this is a valid row else set Flag as False which mean that this is a bad row:
[Microsoft.SqlServer.Dts.Pipeline.SSISScriptComponentEntryPointAttribute]
public class ScriptMain : UserComponent
{
public override void Input0_ProcessInputRow(Input0Buffer Row)
{
if (!Row.Column0_IsNull && !String.IsNullOrWhiteSpace(Row.Column0))
{
string[] cells = Row.Column0.Split(new string[] { "\",\"" }, StringSplitOptions.None);
if (cells.Length == 4)
{
Row.Col1 = cells[0].TrimStart('\"');
Row.Col2 = cells[1];
Row.Col3 = cells[2];
Row.Col4 = cells[3].TrimEnd('\"');
Row.Flag = true;
}
else
{
bool cancel;
Row.Flag = false;
}
}
else
{
Row.Col1_IsNull = true;
Row.Col2_IsNull = true;
Row.Col3_IsNull = true;
Row.Col4_IsNull = true;
Row.Flag = true;
}
}
}
Add a conditional split to split rows based on Flag column
Map the Valid Rows output to the OLEDB Destination, and the Bad Rows output to another flat file where you only map Column0
Scenario : Upload image using srrs using rs.exe (using code not GUI)
Environment : Sql server 2012, rs.exe , vb.script
'Utility to Publish the Png file ( snippet)
'Sample inputs PublishImageFile("myimagefilename.png", "image/png")
Public Sub PublishImageFile(ByVal imageName As String,ByVal resourceMIME As String)
Try
Dim stream As FileStream = File.OpenRead(filePath + "\" + imageName)
definition = New [Byte](stream.Length - 1) {}
stream.Read(definition, 0, CInt(stream.Length))
stream.Close()
Catch e As IOException
Console.WriteLine(e.Message)
End Try
imageName = imageName.tostring.replace(".png", "")
Console.WriteLine("Attempting to Deploy Resource Name {0}", imageName.tostring)
Dim item As CatalogItem
Dim mimeProperty As New Microsoft.SqlServer.ReportingServices2010.Property
mimeProperty.Name = "MimeType"
mimeProperty.Value = resourceMIME
Dim properties(1) As Microsoft.SqlServer.ReportingServices2010.Property
properties(0) = mimeProperty
Try
item = rs.CreateCatalogItem("Resource", imageName, ReportFolder, True,
definition, properties, warnings) 'Error line
'More code below removed for brevity
Error I receive in the last line above is
The required field Property is missing from the input structure. ---> Microsoft.ReportingServices.Diagnostics.Utilities.MissingElementException: The required field Property is missing from the input structure.
What are the required parameters required in the properties object to fix the issue.
I did not find the same in CreateCatalogitem method in MSDN
https://msdn.microsoft.com/en-us/library/reportservice2010.reportingservice2010.createcatalogitem(v=sql.120).aspx
Nor in the Catalog item description in MSDN
https://msdn.microsoft.com/en-us/library/reportservice2010.catalogitem.aspx
From the error message and investigation it appears ( my assumption) like a required field named 'Property' could missing from the properties array. But what would be its value?
Please share your suggestions & solutions or may be even an code alternative to rs.exe to automate deploy images to ssrs using script.
I found the answer. I just added an additional property named property and name as filename just before the CreateCatalogItem method and it worked!
Dim propertyItem As New Microsoft.SqlServer.ReportingServices2010.Property
propertyItem.Name = "Property"
propertyItem.Value = imageName
properties(1) = propertyItem
Im trying to insert new data into the DB when a user scans a barcode into a field. When I hit save on the screen it says fail to convert circular structure to json.
var report = myapp.activeDataWorkspace.BlanccoData.BMCReports.addNew();
report.c_Date = Date.now();
report.IsScannedReport = true;
if (contentItem.screen.ScanSSN == true) {
report.SSN = contentItem.value;
}
var system = myapp.activeDataWorkspace.BlanccoData.BMCSystemInfo.addNew();
// system.Report = report;
system.Barcode = contentItem.screen.Barcode;
I think the commented line is throwing the exception but I need to reference it.
thanks
Have you considered that you may have a circular relationship in your database? That is reflected in your DataSource?
I have to add new columns in existing table. I can able to successfully add new column, but following exception occur while tying to modify the column attribute to nullable.
Multiple-step OLE DB operation generated errors. Check each OLE DB status value, if available. No work was done
Here my code,
HRESULT hr = S_OK;
ADOX::_CatalogPtr pCatalog = NULL;
ADOX::_TablePtr pTable = NULL;
ADOX::TablesPtr pTables = NULL;
hr = pCatalog.CreateInstance(__uuidof(Catalog));
pCatalog->PutActiveConnection("Provider='Microsoft.JET.OLEDB.4.0';data source='C:\\sample.mdb';");
pTables = pCatalog->GetTables();
pTable = pTables->Item["sampletable"];
hr = pTable->Columns->Append( "age", ADOX::adInteger, 0);
ASSERT(hr == S_OK);
pTable->Columns->Item["age"]->Attributes = ADOX::adColNullable;
The equivalent code in VBA works for me without error (assuming I have translated it faithfully).
Something perhaps to try is to create a Column object, set its properties including NULLable then append it to the Table object's Columns collection e.g. this in VBA:
Set oColumn = New ADOX.Column
oColumn.Name = "age"
oColumn.Type = ADOX.adInteger
oColumn.Attributes = ADOX.adColNullable
oTable.Columns.Append oColumn