I have struggled and struggled with this issue. I've read a many similar questions concerning this topic, but I'm new at this and I'm reading "Greek to me". Perhaps someone will be willing to help me on an elementary level with this. I created an SSIS package with several steps. The first step is a script task that uses a VB8 script to pull the data from the SFTP server. The script reads like this:
Imports System
Imports System.Data
Imports System.Math
Imports Microsoft.SqlServer.Dts.Runtime
<System.AddIn.AddIn("ScriptMain", Version:="1.0", Publisher:="", Description:="")> _
<System.CLSCompliantAttribute(False)> _
Partial Public Class ScriptMain
Inherits Microsoft.SqlServer.Dts.Tasks.ScriptTask.VSTARTScriptObjectModelBase
Enum ScriptResults
Success = Microsoft.SqlServer.Dts.Runtime.DTSExecResult.Success
Failure = Microsoft.SqlServer.Dts.Runtime.DTSExecResult.Failure
End Enum
Public Sub HealthSmart()
Dim Path As String, Path2 As String ', fileName As String, fo As Object, x As Object, y As Object, z As Object, rs As Object
Path = "\\RENYSQL1\share\Health_Smart\Received_Files\Test"
Path2 = "\\RENYMstr1\shared\CLIENT FOLDERS\HEALTHSMART\Original_PDF_Files\Test"
Shell("""C:\Program Files (x86)\Ipswitch\WS_FTP 12\wsftppro.exe"" -s ""Health_Smart:/Test/AMC_Test/*.pdf"" -d ""local:" & Path & "\"" -quiet", vbMaximizedFocus)
Shell("""C:\Program Files (x86)\Ipswitch\WS_FTP 12\wsftppro.exe"" -s ""Health_Smart:/Test/AMC_Test/*.pdf"" -d ""local:" & Path2 & "\"" -quiet", vbMaximizedFocus)
Dts.TaskResult = ScriptResults.Success
End Sub
End Class
It runs perfectly fine when the step is executed in the SSIS package. It also runs without error when I import the SSIS package into MSDB and select run package. But when I scedule it as a step in a job, it acts as if it ran perfectly. But the first step pulled not data.
I have tried setting the permissions of the SSIS package to a user that has full permissions to everything. I also tried setting the package protection level to DontSaveSensitive with no password.
On the SQL job side, I have tried using Windows authentication in the job as well as the login I mentioned earlier to run the package. Neither worked. Furthermore, I changed it to run on 32 bit - but to no avail.
I understand that the SQL job is run by the SQL agent. Does the agent have a specific login? People in my department say no.
The strange thing is that when I schedule the job to run, it runs and says it is successful. But the first step of getting the SFTP files with the above script runs, but does not pull any data. The Job Activity Monitor says it was Successful.
I have no idea what to do next. Be easy on me. I'm new at this.
While you certainly could have permissions issues if you are not receiving and error it would suggest the code is not executing. Looking at where you placed your code in your script I am thinking it is entirely possible that you did not put a call to your SUB in the Main () Sub. When creating a new script you should see something like:
Public Sub Main()
'
' Add your code here
'
Dts.TaskResult = ScriptResults.Success
End Sub
And basically where it says add your code here is where Microsoft was leading you to add your script. It is completely okay to do it in its own sub as you have, but if you do you need to call your sub within the Main() SUB like so:
Public Sub Main()
'
' Add your code here
'
HealthSmart()
Dts.TaskResult = ScriptResults.Success
End Sub
If you actually do have the Main SUB and calling your code. There could perhaps be an error within wsftppro.exe. Have you run the code outside of the script?
If permissions are causing the issue within the exe and not error you could try running via your dev environment as a user that has access to the locations and if it succeeds for you then you could need permissions for your SQL Agent and/or Service Account.
Related
In my solution I have a VB class library project. This class library has a folder with several html files which are supposed to be email templates. This class library is intended to be included with both a website and a console app to generate customer emails.
What I want to do is read these html templates into a string and replace keywords in the templates with the data from a simple data structure. At present I'm using a dictionary with the key as the keyword and the value as the string to replace it with.
The problem that I am having is that VB doesn't seem to want to find my html files.
Here's the code for my base email class
Imports System.Net.Mail
Imports System.IO
Public MustInherit Class Email
Public Property TheMailMessage As MailMessage
Protected MustOverride Property SendFrom As MailAddress
Protected MessageTemplate As StreamReader
Protected DataModel As Dictionary(Of String, String)
Protected BodyContent As String
Protected Function GenerateMessageBody() As String
BodyContent = MessageTemplate.ReadToEnd
For Each d In DataModel
BodyContent.Replace(d.Key, d.Value)
Next
Return BodyContent
End Function
Protected MustOverride Sub PopulateMailMessage()
Protected MustOverride Sub CreateDataModel()
End Class
Here's the code for the the class inheriting a child of Email that is trying to read the HTML file to for generating the message body content (I didn't include the call between because all it does is set up the from address):
Imports System.IO
Imports System.Net.Mail
Namespace CustomerEmails
Public Class Welcome : Inherits NoReply
Sub New(ByVal Client As NinjaNexus.Model.Client)
MyBase.New(Client)
MessageTemplate = New StreamReader("Welcome.html")
CreateDataModel(Client)
PopulateMailMessage()
End Sub
Protected Overrides Sub CreateDataModel()
Throw New NotImplementedException
End Sub
Protected Overrides Sub PopulateMailMessage()
TheMailMessage.Subject = "Welcome to Company Name"
TheMailMessage.Body = GenerateMessageBody()
End Sub
Protected Overloads Sub CreateDataModel(ByVal Client As NinjaNexus.Model.Client)
DataModel = New Dictionary(Of String, String)
DataModel.Add("{FName}", Client.Name)
DataModel.Add("{Signature}", "Some name here")
End Sub
End Class
End Namespace
When I try and run the code to generate the welcome email I get an error like this:
An exception of type 'System.IO.FileNotFoundException' occurred in
mscorlib.dll but was not handled in user code
Additional information: Could not find file 'C:\Program Files
(x86)\IIS Express\Welcome.html'.
I've tried a few things like GetFullPath and the like, but that hasn't worked. Adding the folder name or ~\ or .\ or anything of that nature does not help. If I use the complete, full absolute path it reads the file. However, this path isn't going to be the same on the machines running the finished applications, so I really need a relative solution.
Does anyone know how to get the StreamReader to read my HTML file correctly? Is there a better approach than using a StreamReader? I want to stress that this library is going to be used for multiple related projects, so ideally I want to keep all the resources it needs with it and not hanging out on some file server somewhere.
It turns out the answer is to set the build action for the files with my templates to "embedded resource." From there I can then use GetManifestResourceStream to get the contents of the file and do what I wish. I also switched the HTML files to TXT files. Though I still feel like there might be a better way to accomplish my goal, this works.
I believe you want to look for file in your local computer.
You can check here...
You may want to use AppDat folder to store your AppData (Recommended).
Check out the snippet below:
Imports System.Environment
Class Sample
Public Shared Sub Main()
' Get the path to the Application Data folder
Dim appData As String = GetFolderPath(SpecialFolder.ApplicationData)
' Display the path
Console.WriteLine("App Data Folder Path: " & appData)
End Sub
End Class
I'm extremely new to vba and feel like I have been falling down at the first hurdle all morning. I'm trying to get the path of my access file by doing the following
Sub getDirectoryPath()
Debug.Print (System.IO.path.GetFullPath())
End Sub
However I get an "Invalid Qualifier" error on System when I try to run it. I've tried adding the 'System' reference but then it says IO is not found. What am I doing wrong?
VBA environment has only access to COM (and COM visible) component.
So forget about importing usual .Net namespaces.
But some Wrappers exists : [https://technet.microsoft.com/en-us/magazine/2007.01.heyscriptingguy.aspx]
For instance this works :
DataList = CreateObject("System.Collections.ArrayList")
BTW, in order to parse file full name in VBA, you can use FileSystemObject.
It seems like a very common issue with SSIS packages is releasing a package to Production that ends up with running the wrong connectionstring parameters. This could happen by making any one of many mistakes or ommisions. As a result, I find it helpful to dump all ConnectionString values to a log file. This helps me understand what connectionstrings were actually applied to the package at run time.
Now, I am considering having my packages check to see if every connnection object in my package had its connectionstring overriden by an entry in the config file and if not, return a warning or even fail the package. This is to allow easier configuration by extracting all environment variables to a config file. If a connectionstring is never overridden, this risks that a package, when run in production, may use development settings or a package, when run in a non production setting when testing, may accidentily be run against production.
I'd like to borrow from anyone who may have tried to do this. I'd also be interested in suggestions on how to accomplish this with minimal work.
Thx
Technical question 1 - what are my connection string
This is an easy question to answer. In your package, add a Script Task and enumerate through the Connections collection. I fire the OnInformation event and if I had this scheduled, I'd be sure to have the /rep iew options in my dtexec to ensure I record Information, Errors and Warnings.
namespace TurnDownForWhat
{
using System;
using System.Data;
using Microsoft.SqlServer.Dts.Runtime;
using System.Windows.Forms;
/// <summary>
/// ScriptMain is the entry point class of the script. Do not change the name, attributes,
/// or parent of this class.
/// </summary>
[Microsoft.SqlServer.Dts.Tasks.ScriptTask.SSISScriptTaskEntryPointAttribute]
public partial class ScriptMain : Microsoft.SqlServer.Dts.Tasks.ScriptTask.VSTARTScriptObjectModelBase
{
public void Main()
{
bool fireAgain = false;
foreach (var item in Dts.Connections)
{
Dts.Events.FireInformation(0, "SCR Enumerate Connections", string.Format("{0}->{1}", item.Name, item.ConnectionString), string.Empty, 0, ref fireAgain);
}
Dts.TaskResult = (int)ScriptResults.Success;
}
enum ScriptResults
{
Success = Microsoft.SqlServer.Dts.Runtime.DTSExecResult.Success,
Failure = Microsoft.SqlServer.Dts.Runtime.DTSExecResult.Failure
};
}
}
Running that on my package, I can see I had two Connection managers, CM_FF and CM_OLE along with their connection strings.
Information: 0x0 at SCR Enum, SCR Enumerate Connections: CM_FF->C:\ssisdata\dba_72929.csv
Information: 0x0 at SCR Enum, SCR Enumerate Connections: CM_OLE->Data Source=localhost\dev2012;Initial Catalog=tempdb;Provider=SQLNCLI11;Integrated Security=SSPI;
Add that to ... your OnPreExecute event for all the packages and no one sees it but every reports back.
Technical question 2 - Missed configurations
I'm not aware of anything that will allow a package to know it's under configuration. I'm sure there's an event as you will see in your Information/Warning messages that a package attempted to apply a configuration, didn't find one and is going to retain it's design time value. Information - I'm configuring X via Y. Warning - tried to configure X but didn't find Y. But how to have a package inspect itself to find that out, I have no idea.
That said, I've seen reference to a property that fails package on missed configuration. I'm not seeing it now, but I'm certain it exists in some crevice. You can supply the /w parameter to dtexec which treats warnings as errors and really, warnings are just errors that haven't grown up yet.
Unspoken issue 1 - Permissions
I had a friend who botched an XML config file as part of their production deploy. Their production server started consuming data from a dev server. Bad things happened. It sounds like you have had a similar situation. The resolution is easy, insulate your environments. Are you using the same service account for your production class SQL Server boxes and dev/test/uat/qa/load/etc? STOP. Make a new one. Don't allow prod to talk to any boxes that aren't in their tier of service. Someone bones a package and doesn't set a configuration? First of all, you'll catch it when it goes from dev to something-before-production because that tier wouldn't be able to talk to anything else that's not that level. But if you're in the ultra cheap shop and you've only got dev and prod, so be it. Non-configured package goes to prod. Prod SQL Agent fires off the package. Package uses default connection manager and fails validation because it can't talk to the dev sales database.
Unspoken issue 2 - template
What's your process when you have a new package to build? Does your team really start from scratch? There are so many ways to solve this problem but the core concept is to define your best practices for Configuration, Logging, Package Protection Level, Transaction levels, etc into some easily consumable form. Maybe that's 3 starter packages: one for raw acquisition, maybe one stages and conforms the data and the last one moves data from conformed into the final destination. Teammates then simply have to pick one to start from and fill in the spots that need it. If they choose to do their own thing, that's the stick you beat them with when their package fails to run in production because they didn't follow the standard path.
There are other approaches here. If you're a strong .NET crew, you can gen your template packages that way. At this point, I create my templates with Biml and use that to drive basic package creation.
If I am understanding you correctly the below solution should work.
My suggestion to you is to turn on the Do not save sensitive option for the ProtectionLevel property at the top level of the package.
This will require you to use package configurations for every connection, otherwise it will not have the credentials to make a connection.
Task: Loop thru these excel files and insert data into SQL table but in the process i get an error and i don't know which it errored on.
My understanding is SSIS doesn't loop thru file in an random order but i get an error about CANNOT ACQUIRE CONNECTION FROM CONNECTIONMANAGER. Excel Source failed validation and returned error code.. I did set 64bitruntime to False. This happened on VS 2008/SQL Server 2008 R2 on Windows 7 OS. Initially i was able to run the whole process successfully on Windows XP- VS2008 /SQL Server 2008 R2.
Problem: How do i know which file system is going to iterate next if i have 70 odd files in a folder. The thing i get an error and i'm not sure which file SSIS is working on. However i do see files are executed and data is in SQL.
Let me know how to find which file SSIS is currently working or the next one it will work on.
You could add a Script Task and log the variable used in the foreach loop.
Add the variable as readonly variable in the script task editor and then add something like this in the main method (C#):
public void Main()
{
bool fireAgain = true;
Dts.Events.FireInformation(0, "Logging FELC variable", "File: " + Dts.Variables["User::FilePath"].Value.ToString(), string.Empty, 0, ref fireAgain);
Add a script task inside your ForEach container, immediately before you do the Excel processing. In the script task, add the variable you configured in your ForEach loop to hold the filename to the Read Only Variables. In the script itself, call the FireInformation event, which will add an informational message to the progress log in SSIS. In the FireInformation call, pass the value of your filename variable as the message argument.
This will let you see each file being processed, and which one it was processing when it failed.
FireInformation help: http://msdn.microsoft.com/en-us/library/microsoft.sqlserver.dts.runtime.idtscomponentevents.fireinformation.aspx
Creating a package that will loop through all the servers in our different environments(DEV, UAT, PROD). We use service accounts and there is one service account for DEV and UAT and another for PROD. I am using a ForEach Loop Container/variables to set the connection string.
The issue: When the data flow in the loop trys to connect to the PROD servers they fail because they are using the DEV/UAT service account which obviously doesnt have access to PROD, which is fine. The problem is this kills the loop. Normally I would just put an event handler on the data flow and set the event handler's System Variable Propagate:OnError = False so that the error doesn't bubble up to the loop and kill it. Well this doesn't work because the OLE DB connection inside the data flow fails during validation(while the package is running) and apparently the Propagate = False setting only keeps an error from bubbling up if it occurs during the execution of a task and not the validation of a task.
I can set the MaximumErrorCount = 0 on everything including the package itself but that is a bit heavy handed and the package would always report that it ran successfully no matter what errors were encountered.
Running SQL Server 2008-R2
Link to an article on how to not kill your loop using the Propagate setting if someone isn't familiar with it.
One suggestion would be to put a Script Task before the Data Flow tasks that checks access to the connection string with a try-catch block and sets a variable upon failure, and then use that variable in a conditional split to determine whether to run the Data Flow or log that the connection string failed.
Alternatively, if you don't care about why it failed (since you already know it's because of permissions) you could just use a Precedence Constraint and only run Data Flows where the connection succeeded.
UPDATE:
Here's some working code for the Script Task:
public void Main() {
string connString = Dts.Variables["ConnectionStringToTest"].Value;
try {
using (OleDbConnection connection = new OleDbConnection()) {
connection.ConnectionString = connString;
connection.Open();
}
Dts.Variables["User::DatabaseCanConnect"].Value = true;
}
catch (Exception ex) {
Dts.Variables["User::DatabaseCanConnect"].Value = false;
}
Dts.TaskResult = (int)ScriptResults.Success;
}
Create a variable called DatabaseCanConnect at the package scope. Set it to Boolean, it'll default to False.
Create a Script Task. Edit the ReadWriteVariables property and add your new variable.
Add to your ReadOnlyVariables whatever variable you're using to build the connection string out of your ForEach loop. I've named mine ConnectionStringToTest.
Add the script code above as your Main() function. Note that this using an OleDbConnection. This should mimic whatever connection you're using for the connection manager you're modifying in your data flow. So if it's a SQL connection, use a SqlConnection instead.
In your code, use the DatabaseCanConnect variable to determine flow from here. You can use it in a Precedence Constraint to prevent flow to the Data Flow, or use it in a Conditional Split (my preference, more visible to other developers) to log connection errors on failure and proceed as normal otherwise.