I got a valid issue when I run my package. It runs failed in my PC and success in anyone else.
The error is caused by Script Component (turned red), and it is in Post Execute phase, not the post execute in the script componet, but in the runtime of package. The error is:
Information: 0x40043008 at Data Flow Task, SSIS.Pipeline: Post Execute phase is beginning.
Error: 0xC0047062 at Data Flow Task, Script Component [263]: System.InvalidCastException: Unable to cast COM object of type 'System.__ComObject' to interface type 'Microsoft.SqlServer.Dts.Runtime.Wrapper.IDTSVariables100'. This operation failed because the QueryInterface call on the COM component for the interface with IID '{22992C1D-393D-48FB-9A9F-4E4C62441CCA}' failed due to the following error: The application called an interface that was marshalled for a different thread. (Exception from HRESULT: 0x8001010E (RPC_E_WRONG_THREAD)).
I guess the issue is related to variables, because when I remove all the code related to variables, the package run successfully. The code in script component:
private int scheduled550;
private int scheduled620;
private int scheduled720;
private int scheduled820;
public override void PreExecute()
{
base.PreExecute();
scheduled550 = Variables.Count550;
scheduled620 = Variables.Count620;
scheduled720 = Variables.Count720;
scheduled820 = Variables.Count820;
}
public override void PostExecute()
{
base.PostExecute();
}
Did anyone ever encounter the same issue? Can anyone tell me what will POST Execute phase do? Thanks
More info: I have tried to re-install SQL Server, but this is not help. And not all the script component with variables failed running in my SSIS (not in the same package with the error one)
All the tasks/containers in an SSIS have the same lifecycle. You can see some of this by watching the Event Handlers fire. In a script component, inside a Data Flow Task, is going to under go various steps. Part of that is Validation (this contract says I should have a column from this table that is an integer type- can I connect, does it exist, is it the right type, etc).
After validation, tasks will have setup and tear down steps to perform. Since you appear to be working with SSIS Variables in your script, part of that pre/post execute time is spent allowing the translation of Variable (SSIS) to variable (.net) and back again.
Until I see the specific code in your PostExecute method that was causing the failure, I can't state what the code issue might have been.
I cannot recreate your issue. While this is the 2012 release of Integration Services, the Script Component as you are using it will behave the same. I did not send my output to Excel but that should not matter given it's the Script that is failing.
My Script component. I have selected my Variable, User::Count550 as a ReadOnly variable in the menu before editing my code.
public class ScriptMain : UserComponent
{
private int scheduled550;
public override void PreExecute()
{
base.PreExecute();
this.scheduled550 = Variables.Count550;
}
public override void PostExecute()
{
base.PostExecute();
//this.Variables.Count550 = scheduled550;
//this.VariableDispenser.LockForWrite("Count550");
}
public override void Input0_ProcessInputRow(Input0Buffer Row)
{
if (true)
{
this.scheduled550++;
}
}
}
Related
I'm trying to use the WinSCP.NET NuGet to upload some files to an SFTP through a Script Task component in SSIS. While writing the code everything went fine, but if after attempting to build, the WinSCP.NET dll seems to not be picked up breaking all of the references.
I've tried adding WinSCP path to my PATH variable (user). I've tried to add the local version of the WinSCPNET.dll to the GAC. I've tried to reinstall the package through NuGet. I've even tried to change the framework versions.
This is a problem I've had before with the WinSCP.NET DLL. Last time I ended up using a workaround by interfacing with the command line through C#. But I would like to use the DLL, as it's a much simpler implementation.
The code is basically the boilerplate from WinSCP, with some minor changes:
#region Namespaces
using System;
using System.Data;
using Microsoft.SqlServer.Dts.Runtime;
using System.Windows.Forms;
using WinSCP;
#endregion
namespace ST_a1d3d6e0b5d54338bce6c79882c303c6
{
/// <summary>
/// ScriptMain is the entry point class of the script. Do not change the name, attributes,
/// or parent of this class.
/// </summary>
[Microsoft.SqlServer.Dts.Tasks.ScriptTask.SSISScriptTaskEntryPointAttribute]
public partial class ScriptMain : Microsoft.SqlServer.Dts.Tasks.ScriptTask.VSTARTScriptObjectModelBase
{
#region Help: Using Integration Services variables and parameters in a script
/* To use a variable in this script, first ensure that the variable has been added to
* either the list contained in the ReadOnlyVariables property or the list contained in
* the ReadWriteVariables property of this script task, according to whether or not your
* code needs to write to the variable. To add the variable, save this script, close this instance of
* Visual Studio, and update the ReadOnlyVariables and
* ReadWriteVariables properties in the Script Transformation Editor window.
* To use a parameter in this script, follow the same steps. Parameters are always read-only.
*
* Example of reading from a variable:
* DateTime startTime = (DateTime) Dts.Variables["System::StartTime"].Value;
*
* Example of writing to a variable:
* Dts.Variables["User::myStringVariable"].Value = "new value";
*
* Example of reading from a package parameter:
* int batchId = (int) Dts.Variables["$Package::batchId"].Value;
*
* Example of reading from a project parameter:
* int batchId = (int) Dts.Variables["$Project::batchId"].Value;
*
* Example of reading from a sensitive project parameter:
* int batchId = (int) Dts.Variables["$Project::batchId"].GetSensitiveValue();
* */
#endregion
#region Help: Firing Integration Services events from a script
/* This script task can fire events for logging purposes.
*
* Example of firing an error event:
* Dts.Events.FireError(18, "Process Values", "Bad value", "", 0);
*
* Example of firing an information event:
* Dts.Events.FireInformation(3, "Process Values", "Processing has started", "", 0, ref fireAgain)
*
* Example of firing a warning event:
* Dts.Events.FireWarning(14, "Process Values", "No values received for input", "", 0);
* */
#endregion
#region Help: Using Integration Services connection managers in a script
/* Some types of connection managers can be used in this script task. See the topic
* "Working with Connection Managers Programatically" for details.
*
* Example of using an ADO.Net connection manager:
* object rawConnection = Dts.Connections["Sales DB"].AcquireConnection(Dts.Transaction);
* SqlConnection myADONETConnection = (SqlConnection)rawConnection;
* //Use the connection in some code here, then release the connection
* Dts.Connections["Sales DB"].ReleaseConnection(rawConnection);
*
* Example of using a File connection manager
* object rawConnection = Dts.Connections["Prices.zip"].AcquireConnection(Dts.Transaction);
* string filePath = (string)rawConnection;
* //Use the connection in some code here, then release the connection
* Dts.Connections["Prices.zip"].ReleaseConnection(rawConnection);
* */
#endregion
/// <summary>
/// This method is called when this script task executes in the control flow.
/// Before returning from this method, set the value of Dts.TaskResult to indicate success or failure.
/// To open Help, press F1.
/// </summary>
public void Main()
{
// TODO: Add your code here
// User::FileName,$Package::SFTP_HostName,$Package::SFTP_Password,$Package::SFTP_PortNumber,$Package::SFTP_UserName
SessionOptions sessionOptions = new SessionOptions
{
Protocol = Protocol.Sftp,
HostName = (string)Dts.Variables["$Package::SFTP_HostName"].Value,
UserName = (string)Dts.Variables["$Package::SFTP_Password"].Value,
SshHostKeyFingerprint = (string)Dts.Variables["$Package::SFTP_Fingerprint"].Value,
Password = (string)Dts.Variables["$Package::SFTP_Password"].GetSensitiveValue(),
PortNumber = (int) Dts.Variables["$Package::SFTP_PortNumber"].Value,
};
try
{
using (Session session = new Session())
{
// As WinSCP .NET assembly has to be stored in GAC to be used with SSIS,
// you need to set path to WinSCP.exe explicitly,
// if using non-default location.
session.ExecutablePath = (string)Dts.Variables["$Package::WinSCP_Path"].Value;
// Connect
session.Open(sessionOptions);
// Upload files
TransferOptions transferOptions = new TransferOptions();
transferOptions.TransferMode = TransferMode.Binary;
TransferOperationResult transferOperationResult = session.PutFiles(
(string)Dts.Variables["User::FileName"].Value, (string) Dts.Variables["$Package::SFTP_RemoteFileName"].Value,
true, transferOptions);
// Throw on any error
transferOperationResult.Check();
// Print results
bool fireAgain = false;
foreach (TransferEventArgs transferEvent in transferOperationResult.Transfers)
{
Dts.Events.FireInformation(0, null,
string.Format("Upload of {0} succeeded", transferEvent.FileName),
null, 0, ref fireAgain);
}
}
}
catch (Exception e)
{
Dts.Events.FireError(0, null,
string.Format("Error when using WinSCP to upload files: {0}", e),
null, 0);
Dts.TaskResult = (int)DTSExecResult.Failure;
}
Dts.TaskResult = (int)ScriptResults.Success;
}
#region ScriptResults declaration
/// <summary>
/// This enum provides a convenient shorthand within the scope of this class for setting the
/// result of the script.
///
/// This code was generated automatically.
/// </summary>
enum ScriptResults
{
Success = Microsoft.SqlServer.Dts.Runtime.DTSExecResult.Success,
Failure = Microsoft.SqlServer.Dts.Runtime.DTSExecResult.Failure
};
#endregion
}
}
This should compile as is and allow me to run the SSIS, to upload the file. Instead the references break and I receive a lot of missing reference errors:
Error CS0246: The type or namespace name 'WinSCP' could not be found (are you missing a using directive or an assembly reference?)
Error: This project references NuGet package(s) that are missing on this computer. Use NuGet Package Restore to download them. For more information, see http://go.microsoft.com/fwlink/?LinkID=322105. The missing file is ..\packages\WinSCP.5.15.0\build\WinSCP.targets.
I can indeed reproduce your problem, when I use WinSCP NuGet package. It looks like a problem between the NuGet package manager and SQL Server Data Tools. The file the error refers to actually does exist (in a path relative to the script task .csproj file).
Actually, it looks like it's not even recommended to use NuGet in SSIS. You should rather register the assembly to GAC:
How can I use NuGet with SSDT?
Creating a reference to a custom assembly from an SSIS Script Task - vb
SSIS Script Task cant find reference to assembly
And indeed, if I follow the WinSCP instructions for using the assembly from SSIS (using the GAC), it works just fine.
Make sure you have uninstalled the NuGet package.
Install WinSCPnet.dll to GAC or subscribe AppDomain.AssemblyResolve event.
And add WinSCPnet.dll to your script task project.
I have a Spring Batch Task running on our cloud platform that will launch with the provided command line parameters, and then skip over the execution of the first Step with the following error:
[OUT] The job execution id 992 was run within the task execution 1325
[OUT] Step already complete or not restartable, so no action to execute:
StepExecution: id=1071, version=3, name=OFileStep, status=COMPLETED, exitStatus=COMPLETED, readCount=0, filterCount=0, writeCount=0 readSkipCount=0,
writeSkipCount=0, processSkipCount=0, commitCount=1, rollbackCount=0, exitDescription=
I have investigated the metadata tables in the MySQL instance that Spring Batch uses to find that the JOB_INSTANCE_ID is the same between multiple executions, when it should increment by 1 each time.
The #Bean that I have defined for the Job Configuration is:
#Bean
public Job job() {
return jobBuilderFactory.get(OTaskConstants.JOB_NAME)
.listener(listener())
.incrementer(new RunIdIncrementer())
.start(dataTransferTaskStep())
.next(controlMTaskStep())
.build();
}
Is anyone aware of what could be causing this behavior?
Below line clearly says it all.
Step already complete or not restartable, so no action to execute:
Meaning the step/job already complete and can not be restarted. This is the behavior of Spring Batch. In order to by pass this we need to pass an unique argument.
In your case i see you already have RunIdIncrementer. Now question is why it is not working.
Can you see BATCH_JOB_PARMS table to see what arguments are getting passed to the job? May be you are missing something.
You can also use SimpleIncrementor. See below code for explanation.
https://docs.spring.io/spring-batch/docs/current/reference/html/index-single.html#JobParametersIncrementer
Remove #Bean annotation on Job.
It causes the Job to be launched with no parameters every time you launch/start application as spring tries to load the bean and which in-turn launches the batch job.
Remove the annotation and use spring scheduler to schedule the jobs.
I had the same issue. Below code helped me resolve it. By adding params in job launcher a new job_instance_id is created for every run.
#SpringBootApplication
public class App implements CommandLineRunner {
#Autowired
JobLauncher jobLauncher;
#Autowired
Job job;
public static void main(String[] args) {
SpringApplication.run(App.class, args);
}
#Override
public void run(String... args) throws Exception {
JobParameters params = new JobParametersBuilder()
.addString("JobID", String.valueOf(System.currentTimeMillis()))
.toJobParameters();
jobLauncher.run(job, params);
}
}
Solution
Refer error message above “If you want to run this job again, change the parameters.” The formula is JobInstance = JobParameters + Job. If you do not have any parameters for JobParameters, just pass a current time as parameter to create a new JobInstance. For example,
CustomJobLauncher.java
//...
#Component
public class CustomJobLauncher {
#Autowired
JobLauncher jobLauncher;
#Autowired
Job job;
public void run() {
try {
JobParameters jobParameters =
new JobParametersBuilder()
.addLong("time",System.currentTimeMillis()).toJobParameters();
JobExecution execution = jobLauncher.run(job, jobParameters);
System.out.println("Exit Status : " + execution.getStatus());
} catch (Exception e) {
e.printStackTrace();
}
}
}
Source : https://mkyong.com/spring-batch/spring-batch-a-job-instance-already-exists-and-is-complete-for-parameters/
We use Apache Camel in Talend ESB Studio v6.4
In an ESB route, we consume JMS messages, process them then send them to an HTTP server. But that target server is down for maintainance every saturday from 6pm to 10pm.
How can we "pause" message consuming or message processing during that period ? I think quartz only works with file/ftp endpoints.
We could use a Processor component to check in Java if we are in the down period, but what to do after that ?
There are several ways to do this. One camel specific way to do it is through CamelControlBus. It takes in a routeId and performs an action (start/stop/resume etc) on it - Read more here to get an understanding Camel ControlBus
However, there is another approach that you can take. You can create a POJO bean that has 3 methods
shouldRouteStop() : to check the current time and decide if it should stop your route.
startRoute() : Starts a route if it is suspended
stopRoute() : Suspends a route if it is started
A simple implementation can be as follows:
public class ManagementBean {
public boolean shouldRouteStop() {
// Mocking the decision here
return new Random().nextBoolean();
}
public void startRoute(org.apache.camel.CamelContext ctx) throws Exception {
if (ctx.getRouteStatus("GenerateInvoices") == ServiceStatus.Suspended)
// replace the argument with your route Id
ctx.resumeRoute("GenerateInvoices");
}
public void stopRoute(org.apache.camel.CamelContext ctx) throws Exception {
if (ctx.getRouteStatus("GenerateInvoices") == ServiceStatus.Started)
// replace the argument with your route Id
ctx.suspendRoute("GenerateInvoices");
}
}
Make sure that the jms-route that you wish to control has a routeId and add this bean to your base/default CamelContext like this
main.bind("manageRouteBean", new ManagementBean());
Create another timer based route, that checks on each tick, if the route should be stopped or not and then suspends or resumes the route by routeId. This route can be implemented like below:
public class MonitoringRoute extends RouteBuilder {
#Override
public void configure() throws Exception {
onException(Exception.class).log(exceptionMessage().toString());
from("timer:time?period=10000")
.choice()
.when().simple("${bean:manageRouteBean?method=shouldRouteStop}")
.log("Route Should Stop")
.bean(ManagementBean.class, "stopRoute(*)")
.otherwise()
.log("Route Should Start")
.bean(ManagementBean.class, "startRoute(*)")
.end();
}
}
Note that startRoute and stopRoute take the argument as *. This is camel way of automatically binding parameters based on type.
Finally, you can add this route to the main camel context like : main.addRouteBuilder(new MonitoringRoute());
For a complete implementation, have a look at this github repo
I am running into an extremely strange behavior in Groovy. When I throw an exception from a closure in a Script, the end exception that was thrown was different.
Here are the code and the details:
public class TestDelegate {
def method(Closure closure) {
closure.setResolveStrategy(Closure.DELEGATE_FIRST);
closure.delegate = this;
closure.call();
}
public static void main(String[] args) {
// Make Script from File
File dslFile = new File("src/Script.dsl");
GroovyShell shell = new GroovyShell();
Script dslScript = shell.parse(dslFile);
TestDelegate myDelegate = new TestDelegate();
dslScript.metaClass.methodMissing = {
// will run method(closure)
String name, arguments ->
myDelegate.invokeMethod(name, arguments);
}
dslScript.metaClass.propertyMissing = {
String name ->
println "Will throw error now!"
throw new MyOwnException("errrrror");
}
dslScript.run();
}
}
class MyOwnException extends Exception {
public MyOwnException(String message) {
super(message);
}
}
Script.dsl:
method {
println a;
}
So the plan is that when I run the main() method in TestDelegate, it will run the DSL script, which calls for the method method(). Not finding it in the script, it will invoke methodMissing, which then invokes method() from myDelegate, which in turns invoke the closure, setting the delegate to the testDelegate. So far, so good. Then the closure is supposed to try printing out "a", which is not defined and will thus set off propertyMissing, which will will throw MyOwnException.
When I run the code, however, I get the following output:
Will throw error now!
Exception in thread "main" groovy.lang.MissingPropertyException: No such property: a for class: TestDelegate
Now, it must have reached that catch block, since it printed "Will throw error now!" It must have thrown MyOwnException too! But somewhere along the lines, MyOwnException was converted to MissingPropertyException, and I have no idea why. Does anyone have any idea?
P.S. if I remove closure.setResolveStrategy(Closure.DELEGATE_FIRST) from TestDelegate#method(), the code acts as expected and throws MyOwnException. But I really need the setResolveStrategy(Closure.DELEGATE_FIRST) for my DSL project. And I would prefer to know the root cause of this rather than just removing a line or two and see that it works without understanding why.
I think this is what essentially happens: With a delegate-first resolve strategy, the Groovy runtime first tries to access property a on myDelegate, which results in a MissingPropertyException because no such property exists. Then it tries propertyMissing, which causes a MyOwnException to be thrown. Eventually the runtime gives up and rethrows the first exception encountered (a design decision), which happens to be the MissingPropertyException.
With an owner-first resolve strategy, propertyMissing is consulted first, and hence MyOwnException is eventually rethrown.
Looking at the stack trace and source code underneath should provide more evidence.
I start learning Caliburn.Micro and I am little confuse of handling with exception/messange box in view model class.
I found some blogs about, for example:
http://frankmao.com/2010/11/18/handling-messagebox-in-caliburn-micro/
For example some method in view model class which can produce exception.
public void MethodWichCanProduceEx(string arg1, string arg2 )
{
if(arg1==null)
throw new ArgumentNullException("arg1 is null");
if (arg2 == null)
throw new ArgumentNullException("arg2 is null");
try
{
}
catch (Exception exception)
{
throw exception;
//? show message box MessageBox.Shox(exception.Message)
}
}
What is correct handling and showing these exception in view ? It exist any kind of pattern for caliburn.micro?
It possible trace exception as in .NET in text, xml file ?
For example I would like trace exception in xml, text file and in view show only message.box or something message.
Thank for advance, maybe is my question little stupid, sorry I am only learning calibur.micro.
You'll want to always work against abstractions in your view models, in the case of message boxes, you don't want to have to wait for user input when you come to unit test your view models.
The Frank Mao code you linked to uses a delegate to abstract the implementation of the message box from the view model, but I would use an interface here. You can think of a delegate as an interface with a single method, but the advantage of using an interface in this context is that you can have different methods depending on the type of message you wish to show. For example, you could have a ShowMessageError, ShowMessageWarning, ShowMessageInfo etc.
So, define a contract for your message box:
public interface IMessageBox
{
void ShowException(Exception exc);
}
Inject the message box dependency into your view model, e.g. via the constructor
public class MyViewModel
{
private readonly IMessageBox messageBox;
public MyViewModel(IMessageBox messageBox)
{
this.messageBox = messageBox;
}
public void MethodThatCanThrowException()
{
try {}
catch(Exception exc)
{
// log the exception here
...
// show message box
this.messageBox.ShowException(exc);
}
}
}
You can then implement the message box anyway you wish, either using the windows system message box, or nicer still use your own view/viewmodel to display the message, perhaps using the Caliburn.Micro WindowManager.ShowDialog().
An implementation that uses the windows system message box may look like:
public class StandardMessageBox : IMessageBox
{
public void ShowException(Exception exception)
{
MessageBox.Show(exception.ToString(), "Error Occurred");
}
}
In production code, you can register StandardMessageBox against the IMessageBox interface in your IoC container.
In unit test land, you can mock out IMessageBox and have it do nothing, or in the case of methods with a result from the message box, always return a value you wish.
For logging the exception, I would look at a logging framework such as log4net (http://logging.apache.org/log4net/index.html) or NLog (http://nlog-project.org/)