I have a C# script in the ssis package as mentioned below
SqlConnection importTab = new SqlConnection(#"Server=ServerName;
Integrated Security=true;user=;pwd=;database=DBname");
I need to pass the database name (DBName) inside a variable...
May be like this
SqlConnection importTab = new SqlConnection(#"Server=ServerName;
Integrated Security=true;user=;pwd=;database="+"User::Variable" +");"
But I know I am wrong...
To use a variable in a script, first ensure that the variable has been added to
either the list contained in the ReadOnlyVariables property or the list contained in
the ReadWriteVariables property of this script task, according to whether or not your
code needs to write to the variable.
//Example of reading from a variable:
DateTime startTime = (DateTime) Dts.Variables["System::StartTime"].Value;
//Example of writing to a variable:
Dts.Variables["User::myStringVariable"].Value = "new value";
//Example of reading from a package parameter:
int batchId = (int) Dts.Variables["$Package::batchId"].Value;
//Example of reading from a project parameter:
int batchId = (int) Dts.Variables["$Project::batchId"].Value;
//Example of reading from a sensitive project parameter:
int batchId = (int) Dts.Variables["$Project::batchId"].GetSensitiveValue();
I do it like this:
When opening the script task properties you have two fields, ReadOnlyVariables and ReadWriteVariables. Write your variable name into the according field based on your needs, in your case User::Variable.
In the code you can use it like this
Dts.Variables["User::Variable"].Value.ToString()
Following code in the Script task may help you
var dbServerName = Dts.Variables["yourVariableName"].Value.ToString();
var sqlConnString = string.Format("Server=ServerName;Integrated Security=true;user=;pwd=;database={0}", dbServerName);
SqlConnection sqlConn = new SqlConnection(sqlConnString);
Related
I am working on a BIML project to generate SSIS packages. I have a separate static class for utility methods.
I am attempting to call GetDropAndCreateDdl() to get the DDL from the souce to dynamically create a table in the destination. This should work in theory as it is referenced in multiple posts: here and here as samples.
When generating the BIML, running the sample code below, I receive an error: Error: 'AstTableNode' does not contain a definition for 'GetDropAndCreateDdl' and no accessible extension method 'GetDropAndCreateDdl' accepting a first argument of type 'AstTableNode' could be found
public static string GetDropAndCreateDDL(string connectionStringSource, string sourceTableName)
{
var sourceConnection = SchemaManager.CreateConnectionNode("Source", connectionStringSource);
var sourceImportResults = sourceConnection.ImportTableNodes(Nomenclature.Schema(sourceTableName),Nomenclature.Table(sourceTableName));
return sourceImportResults.TableNodes.ToList()[0].GetDropAndCreateDdl();
}
(Let's ignore the possibility of getting no table back or multiples for the sake of simplicity)
Looking at the varigence documentation, I don't see any reference to this method. This makes me think that there is a utility library that I am missing in my includes.
using Varigence.Biml.Extensions;
using Varigence.Biml.CoreLowerer.SchemaManagement;
What say you?
Joe
GetDropAndCreateDdl is an extension method in Varigence.Biml.Extensions.SchemaManagement.TableExtensions
ImportTableNodes returns an instance of
Varigence.Biml.CoreLowerer.SchemaManagement.ImportResults and the TableNodes is an IEnumerable of AstTableNodes
So, nothing weird there (like the table nodes in the import results being a different type)
I am not running into an issue if I have the code in-line with BimlExpress.
<Biml xmlns="http://schemas.varigence.com/biml.xsd">
<#
string connectionStringSource = #"Provider=SQLNCLI11;Data Source=localhost\dev2017;Integrated Security=SSPI;Initial Catalog=msdb";
var sourceConnection = SchemaManager.CreateConnectionNode("Source", connectionStringSource);
List<string> schemaList = new List<string>(){"dbo"};
var sourceImportResults = sourceConnection.ImportTableNodes("dbo", "");
WriteLine("<!-- {0} -->", sourceImportResults.TableNodes.Count());
//var sourceImportResults = sourceConnection.ImportTableNodes(schemaList,null);
var x = sourceImportResults.TableNodes.ToList()[0];
var ddl = x.GetDropAndCreateDdl();
WriteLine("<!-- {0} -->", sourceImportResults.TableNodes.FirstOrDefault().GetDropAndCreateDdl());
#>
</Biml>
The above code results in the following expanded Biml
<Biml xmlns="http://schemas.varigence.com/biml.xsd">
<!-- 221 -->
<!-- IF EXISTS (SELECT * from sys.objects WHERE object_id = OBJECT_ID(N'[dbo].[autoadmin_backup_configuration_summary]') AND type IN (N'V'))
DROP VIEW [dbo].[autoadmin_backup_configuration_summary]
GO
CREATE VIEW [dbo].[autoadmin_backup_configuration_summary] AS
SELECT
ManagedBackupVersion,
IsAlwaysOn,
IsDropped,
IsEnabled,
RetentionPeriod,
EncryptionAlgorithm,
SchedulingOption,
DayOfWeek,
COUNT(*) AS DatabaseCount
FROM autoadmin_backup_configurations
GROUP BY
ManagedBackupVersion,
IsAlwaysOn,
IsDropped,
IsEnabled,
RetentionPeriod,
EncryptionAlgorithm,
SchedulingOption,
DayOfWeek
GO
-->
</Biml>
I have some processed and unprocessed files in my Source Folder and the file names
of all the processed files are stored in a table. How can I match the files names of source folder and table prior to ForEach Loop Container and process only unmatched files.
The solution below is a bit elaborate but it's the best I could think of.
STEP 1: Create 2 Variables, both strings.
a)CurrentFile: This will be used for your Foreach Loop Container collection value
b)ToProcess: This will be used to map the result set an Execute SQL Task explained
below
STEP 2: Add an Execute SQL Task into your Foreach Loop Container.
Configure Parameter Mapping as shown below:
Use the script below as your SQL Statement:
DECLARE #ToProcess VARCHAR(1)
IF NOT EXISTS(SELECT [FileNames] FROM [YourFilesTable] WHERE FileNames = ?)
SET #ToProcess = 'Y'
SELECT #ToProcess AS ToProcess
Set ResultSet to Single Row as shown below:
Configure Result Set as shown below:
On the Execute SQL Task, configure the Precedence Constraint as shown below:
Your Foreach Loop Container should look like below:
Before the Foreach Loop, use a Script Task to store the names of unprocessed files in an SSIS object variable, then iterate through this variable to load the new files as you already are. Create an object variable and add this in the ReadWriteVariables field of the Script Task. If you're using an SSIS variable to hold the folder path of the source files as done below, add this in the ReadOnlyVariables field. The Foreach Loop will need to use the Foreach From Variable Enumerator enumerator type. In the Variable field on the Collection page, add the object variable that is populated in the Script Task. As you're probably already doing, add a string variable at Index 0 of the Variable Mapping pane and set this variable as the expression of the ConnectionString property on the connection manager, assuming this is a flat file connection. If this is excel, change the ExcelFilePath property to use this variable as the expression. The example code and referenced namespaces for the Script Task is below and uses C#.
using System.Linq;
using System.Data.SqlClient;
using System.IO;
using System.Collections.Generic;
using System.Data;
string connString = #"Data Source=YourSQLServer;Initial Catalog=YourDatabase;Integrated Security=SSPI;";
string cmdText = #"SELECT DISTINCT ColumnWithFileNames FROM YourDatabase.YourSchema.YourTable";
string sourceFolder = Dts.Variables["User::SourceFilePath"].Value.ToString();
//create DirectoryInfo object from source folder
DirectoryInfo di = new DirectoryInfo(sourceFolder);
List<string> processedFiles = new List<string>();
List<string> newFiles = new List<string>();
//get names of already processed files stored in tavle
using (SqlConnection conn = new SqlConnection(connString))
{
conn.Open();
//data set name does not need to relate to name of table storing processed files
DataSet ds = new DataSet("ProcessedFiles");
SqlDataAdapter da = new SqlDataAdapter(cmdText, conn);
da.Fill(ds, "ProcessedFiles");
foreach (DataRow dr in ds.Tables["ProcessedFiles"].Rows)
{
processedFiles.Add(dr[0].ToString());
}
}
foreach (FileInfo fi in di.EnumerateFiles())
{
//only add files not already processed
if (!processedFiles.Contains(fi.FullName))
{
newFiles.Add(fi.FullName);
}
}
//populate SSIS object variable with unprocessed files
Dts.Variables["User::ObjVar"].Value = newFiles.ToList();
Using the example from the Spring docs, I'm trying to return a value from a mySQL function. I keep getting the error Can't set IN parameter for return value of stored function call;.
I created a mySQL function that works fine (ran in MySQL Workbench). I've written a SimpleJdbcCall statement, set up the parameters as per Spring docs example but consistently get this error. If I turn the function into a procedure, the code works, I just have to retrieve the return value from the result set.
I used https://docs.spring.io/spring/docs/3.0.0.M3/reference/html/ch13s05.html, section 13.5.8 as reference.
CREATE FUNCTION `ScenarioRegistration`(
environment VARCHAR(45),
username VARCHAR(15),
scenario_name VARCHAR(45)) RETURNS int(11)
A couple of SELECT statements followed by an INSERT then
RETURN scenario_id; // The inserted id
Java code:
SimpleJdbcCall simpleJdbcCall = new SimpleJdbcCall(getJdbcTemplate())
.withFunctionName("ScenarioRegistration")
.withoutProcedureColumnMetaDataAccess();
simpleJdbcCall.addDeclaredParameter(new SqlParameter("environment"
,Types.VARCHAR));
simpleJdbcCall.addDeclaredParameter(new SqlParameter("username"
,Types.VARCHAR));
simpleJdbcCall.addDeclaredParameter(new SqlParameter("scenario_name"
,Types.VARCHAR));
SqlParameterSource parameters = new MapSqlParameterSource()
.addValue("environment", environment)
.addValue("username", username)
.addValue("scenario_name", scenario);
simpleJdbcCall.setReturnValueRequired(true);
Integer scenario_id = simpleJdbcCall.executeFunction(
Integer.class, parameters);
All I want the routine to do is give me back the id of the newly inserted scenario.
What I get is:
SQL [{? = call scenarioregistration(?, ?)}]; Can't set IN parameter for return value of stored function call.
I find it interesting that it's taken my THREE input values and changed them to an output and TWO input values.
Anyone enlighten me as to the problem and how to fix it?
Thanks,
Steven.
I would refer to the latest docs here for your answer. It appears Spring is trying to infer the output because you didn't explicity specify one.
Per the docs above there are two valid approaches on calling the desired function with the SimpleJdbcCall:
Inferred Parameters
Because you've specified withoutProcedureColumnMetaDataAccess, Spring isn't going to look and see what the ins/outs are to your function. If you want it easy, just don't specify that and you should be able to do:
SqlParameterSource parameters = new MapSqlParameterSource()
.addValue("environment", environment)
.addValue("username", username)
.addValue("scenario_name", scenario);
Integer scenarioId = new SimpleJdbcCall(getJdbcTemplate())
.withFunctionName("ScenarioRegistration")
.executeFunction(Integer.class, parameters);
Explicit Parameters
If you want to keep withoutProcedureColumnMetaDataAccess turned off for whatever reason, you can do:
Integer scenarioId = new SimpleJdbcCall(getJdbcTemplate)
.withFunctionName("ScenarioRegistration")
.withoutProcedureColumnMetaDataAccess()
.useInParameterNames("environment", "username", "scenario_name")
.declareParameters(
new SqlOutParameter("scenario_id", Types.NUMERIC),
new SqlParameter("environment", Types.VARCHAR),
new SqlParameter("username", Types.VARCHAR),
new SqlParameter("scenario_name", Types.VARCHAR)
).executeFunction(Integer.class, parameters);
Note: It appears that order is critical in this example. The output parameter should be declared first, and the subsequent named IN parameters come last. That is, the order of the parameters ? are ordinal in [{? = call scenarioregistration(?, ?, ?)}])
Alternative NamedParameterJdbcTemplate Solution
Another way to invoke your function is via an actual JDBC call. This could hypothetically save you the grief of using the fine tuning of the SimpleJdbcCall.
Integer scenarioId = namedParameterJdbcTemplate.queryForObject(
"SELECT ScenarioRegistration(:environment, :username, :scenario_name)",
parameters,
Integer.class);
I'm trying to fetch lineageId of output columns in a SSIS Package. I'm traversing SSIS Package as XML Document in C# code.
No issue in traversing and fetching data from the nodes in the package, but facing issue when I'm trying to fetch LineageID.
I many forums and articles said that LineageId generated in SSIS Package is a Unique Integer value for that particular package, so I'm trying to convert LineageID to an integer value. But it seems to be a String value.
Kindly clarify my doubt. Is lineageid STRING OR INT?
If it is Int then please tell me how to find it.
foreach (var x in NodeCollection)
{
var outputCollection = x.Elements("outputColumn");
foreach (var output in outputCollection)
{
string Name = output.Attribute("name").Value;
long LineageId = Convert.ToInt64(output.Attribute("lineageId").Value);
}
}
By debugging with Breakpoint "output.Attribute("lineageId").Value" is returning following value
"Package\DFT\SRC.Outputs[OLE DB Source Output].Columns[TYPE]"
I have performed a very similar exercise and found that the lineage id values were definitely integer. This is an extract from my code:
var nsmgr = new XmlNamespaceManager(doc.NameTable);
nsmgr.AddNamespace("DTS", "www.microsoft.com/SqlServer/Dts");
foreach (XmlNode childnode in doc.SelectNodes("//*[#lineageId != '' and #name != '']"))
{
XmlNode ExecutableNode = childnode.SelectSingleNode("ancestor::DTS:Executable[1]", nsmgr);
var taskName = ExecutableNode.SelectSingleNode("DTS:Property[#DTS:Name='ObjectName']", nsmgr).InnerText;
var targetColumnName = childnode.Attributes["name"].Value;
var lineageID = Convert.ToInt32(childnode.Attributes["lineageId"].Value);
We are using this to map column names to lineageid's so that we can interpret error messages thrown by our SSIS packages.
Hope this helps!
I would like to query a table based on a list of KeyValuePair. With a Model-First approach, I could do the following:
var context = new DataContext();
var whereClause = new StringBuilder();
var objectParameters = new List<ObjectParameter>();
foreach(KeyValuePair<string, object> pair in queryParameters)
{
if (whereClause.Length > 0)
whereClause.Append(" AND ");
whereClause.Append(string.Format("it.[{0}] = #{0}", pair.Key));
parameters.Add(new ObjectParameter(pair.Key, pair.Value));
}
var result = context.Nodes.Where(whereClause.ToString(), parameters.ToArray());
Now I'm using a Code-First approach and this Where method is not available anymore. Fortunately, I saw an article somewhere (I can't remember anymore) which suggested that I could convert the DbContext to a IObjectContextAdapter then call CreateQuery like this:
var result = ((IObjectContextAdapter)context)
.ObjectContext.CreateQuery<Node>(whereClause.ToString(), parameters.ToArray());
Unfortunately, this throws an error:
'{ColumnName}' could not be resolved in the current scope or context. Make sure that all referenced variables are in scope, that required schemas are loaded, and that namespaces are referenced correctly.
Where {ColumnName} is the column specified in the whereClause.
Any ideas how I can dynamically query a DbSet given a list of key/value pairs? All help will be greatly appreciated.
I think your very first problem is that in the first example you are using Where on the entity set but in the second example you are using CreateQuery so you must pass full ESQL query and not only where clause! Try something like:
...
.CreateQuery<Node>("SELECT VALUE it FROM ContextName.Nodes AS it WHERE " + yourWhere)
The most problematic is full entity set name in FROM part. I think it is defined as name of the context class and name of the DbSet exposed on the context. Another way to do it is creating ObjectSet:
...
.ObjectContext.CreateObjectSet<Node>().Where(yourWhere)