Reporting services - subscribe to all reports in folder - sql-server-2008

I have a number of reports in a folder, and it will grow in a time. To subscribe to all reports user has to go to each one and create subscription. With about 10 reports it's manageable, but when there are 30 reports and a new user should be added to a subscription it's becoming difficult.
How can I create some kind of batch subscription? I mean - to make easy subscription to all reports in a folder and send it to the user(s) (no matter where - e-mail or file share). Is there some administrative option or should I write some script to achive it?
Environment: SQL Server 2008 R2 + SSRS 2008 R2 (Standard Edition) on W2K8 R2 (Enterprise Edition)

The simplest way to send a subscription to all reports users by e-mail is to set up the subscription on an e-mail group, and add (and remove) users from that e-mail group as desired.
The simplest way to send a subscription to all reports users by file share is to set up the subscription on a file share, and grant (and remove) access to that file share as desired.
I don't know of any way to set up the equivalent of a batch subscription. However, it should be possible to write a stored procedure to update existing schedules or create new schedules in the ReportServer database, based on a user ID passed to the query as a parameter.
You can see examples of queries that access the subscription tables on the ReportServer database here.

I found the solution following this link where rs.exe is used. Esentially it's example from BOL. I've changed it to suit my needs and it works. Usage:
rs.exe -i CreateSubscriptionTest.rss -s http://myreportserveraddresshere/reportserver
' CreateSubscriptionTest.rss
Public Sub Main()
rs.Credentials = System.Net.CredentialCache.DefaultCredentials
Dim desc As String = "Report description"
Dim eventType As String = "TimedSubscription"
Dim scheduleXml As String = "<ScheduleDefinition><StartDateTime>2012-03-22T09:30:00</StartDateTime><WeeklyRecurrence><WeeksInterval>1</WeeksInterval><DaysOfWeek><Thursday>True</Thursday></DaysOfWeek></WeeklyRecurrence></ScheduleDefinition>"
Dim extensionParams(7) As ParameterValue
extensionParams(0) = New ParameterValue()
extensionParams(0).Name = "TO"
extensionParams(0).Value = "my.address#email.com"
extensionParams(1) = New ParameterValue()
extensionParams(1).Name = "ReplyTo"
extensionParams(1).Value = "admin#email.com"
extensionParams(2) = New ParameterValue()
extensionParams(2).Name = "IncludeReport"
extensionParams(2).Value = "True"
extensionParams(3) = New ParameterValue()
extensionParams(3).Name = "RenderFormat"
extensionParams(3).Value = "MHTML"
extensionParams(4) = New ParameterValue()
extensionParams(4).Name = "Subject"
extensionParams(4).Value = "#ReportName was executed at #ExecutionTime"
extensionParams(5) = New ParameterValue()
extensionParams(5).Name = "Comment"
extensionParams(5).Value = "Some HTML code inside email's body<br><br>Go!"
extensionParams(6) = New ParameterValue()
extensionParams(6).Name = "IncludeLink"
extensionParams(6).Value = "True"
extensionParams(7) = New ParameterValue()
extensionParams(7).Name = "Priority"
extensionParams(7).Value = "NORMAL"
Dim parameters() As ParameterValue
' If you need setup parameters
'Dim parameter As New ParameterValue()
'parameter.Name = "EmpID"
'parameter.Value = "288"
'parameters(0) = parameter
'parameter.Name = "ReportMonth"
'parameter.Value = "12"
'parameters(1) = parameter
'parameter.Name = "ReportYear"
'parameter.Value = "2003"
'parameters(2) = parameter
Dim matchData As String = scheduleXml
Dim extSettings As New ExtensionSettings()
extSettings.ParameterValues = extensionParams
extSettings.Extension = "Report Server Email"
Dim returnValue As String
Dim reports() As String = { _
"/MyReports/Executive/SalesYear", _
"/MyReports/Executive/SalesMonth", _
"/MyReports/Executive/SalesWeek"}
For Each report As String In reports
returnValue = rs.CreateSubscription(report, extSettings, desc, eventType, matchData, parameters)
Console.WriteLine(returnValue)
Next
End Sub 'Main
The key part is to define reports() variable with report names. Name is the full report path.
The above example creates new schedule for each subscription. It's getting very messy in SQL Server Agent with a lot of reports schedules, so I changed it to use shared schedule. The difference is in scheduleXml declaration - you use schedule ID instead of XML string:
Dim scheduleXml As String = "924b9bb6-2340-4f5c-a897-465af7ff310e"
How can you obtain the schedule ID:
-- using T-SQL
SELECT
ScheduleID,
Name
FROM ReportServer.dbo.Schedule
WHERE
EventType = 'SharedSchedule'
' using rss file: GetSchedulers.rss
' run with rs.exe -i GetSchedulers.rss -s http://myreportserveraddresshere/reportserver
Public Sub Main()
rs.Credentials = System.Net.CredentialCache.DefaultCredentials
Dim returnValue As Schedule()
returnValue = rs.ListSchedules()
For Each sch As Schedule In returnValue
Console.WriteLine(sch.ScheduleID & " - " & sch.name)
Next
End Sub 'Main

Related

SSRS get list of AD groups for current user but #error

I have a piece of code that allows you to retrieve the AD groups of a user in a SSRS report.
However, I get an #error in my report locally but also when I run it via the SSRS portal. I think it has something to do with additional permissions? Does anyone know how to get it to work?
Public Function GetUserMembership(ByVal strUser As String ) As String
Dim groups As String
Try
Dim obEntry As New System.DirectoryServices.DirectoryEntry("LDAP://DC=aw,DC=com")
Dim srch As New System.DirectoryServices.DirectorySearcher(obEntry, "(sAMAccountName=" & strUser & ")")
Dim res As System.DirectoryServices.SearchResult = srch.FindOne()
If res IsNot Nothing Then
Dim obUser As New System.DirectoryServices.DirectoryEntry(res.Path)
' Invoke Groups method.
Dim obGroups As Object = obUser.Invoke("Groups")
For Each ob As Object In DirectCast(obGroups, System.Collections.IEnumerable)
' Create object for each group.
Dim obGpEntry As New System.DirectoryServices.DirectoryEntry(ob)
groups = groups + obGpEntry.Name + ";"
Next
End If
Catch ex As Exception
groups = ""
End Try
Return groups
End Function
To use it in the report:
=Code.GetUserMembership(Split(User!UserUD, "\")(1))
Additional, I found an assembly reference is required to use the class System.DirectoryServices.DirectoryEntry. In my case I found the .dll (DirectoryServices) at the following location: C:\Windows\Microsoft.NET\Framework\v4.0.30319.
Source: https://social.msdn.microsoft.com/Forums/en-US/6f00f6bc-15dc-4ca4-b0c7-99cabf5abd90/ssrs-2008-r2-how-to-get-the-logged-user-group-information?forum=sqlreportingservices

MS ACCESS VBA module Auto number format

I have a Repoerteq table with a REQ_NUM as ID and another column named "REQ_department".
REQ_department have defult values such as ""Finance".
I want to make VBA looks at the department and then set a prefix formate for REQ_NUM
example is department is finance then it would make id as "FIN 000"
the following code is what i manage fo far but it still not working
Option Compare Database
Function GetData() As String
Dim db As Database
Dim Rrs As DAO.Recordset
Dim RSQL As String
Dim RepData As String
Dim RepDep As String
'TO open connection to current Access DB
Set db = CurrentDb()
'TO create SQL statement and retrieve value from ReportReq table
RSQL = "select * from ReportReq"
Set Rrs = db.OpenRecordset(RSQL)
'Retrieve value if data is found
If Rrs.EOF = False Then
RepData = Rrs("REQ_NUM")
RepDep = Rrs("Req_department")
Else
RepData = "Not found"
RepDep = "Not found"
End If
Lrs.Close
Set Lrs = Nothing
GetData = RepData
If ReqDep = "finance" Then
Range("REQ_NUM") = Format$("FIN", REQ_NUM)
End If
End Function
You'll have to change your call to the Format() function which tries to format a number or date according to the format string. In addition, you're using an undefined variable REQ_NUM.
If ReqDep = "finance" Then
Range("REQ_NUM") = "FIN " & Format$("000", CLng(RepData))
' ^^^^^^ ^^^^^^^^^^^^^^^^^^^^
End If

Programatically fill TFS New Bug workitem from VBA

I am wondering if it is possible to have VBA (Access) open a TFS Bug report webpage and fill in the description ?
While I am able to open the page I have not yet found a way to populate the description and potently other fields.
Perhaps one of the experts knows?
I wouldn't try to do what you are working on.
One, trusting a buggy application to correctly report its own bugs isn't a great idea. But beyond that you will be trying to attach Access to TFS.
That being said you can do this entirely automated. Put in some error trapping and then what you are looking to do is how to call TFS APIs. But you may or may not have to install some third part tools etc.
Starting point TFS API
The way I finally did it is to use a dll to interface between access and tfs...
For anyone trying to do the same here is the code...
Imports Microsoft.TeamFoundation.Client
Imports Microsoft.TeamFoundation.WorkItemTracking.Client
Imports Microsoft.TeamFoundation.WorkItemTracking.Common
Imports System.Runtime.InteropServices
<ComClass(TFSInterOp.ClassId, TFSInterOp.InterfaceId, TFSInterOp.EventsId)> Public Class TFSInterOp
Public Const ClassId As String = "14306fc5-1492-42d6-a032-bc4348508dd3"
Public Const InterfaceId As String = "288339cb-0c2e-45fd-8005-e5fed401f0cc"
Public Const EventsId As String = "723327dc-7777-44e4-b291-9299027665eb"
Public Sub New()
End Sub
Public Function InsertBugWorkItem(Title As String, Desc As String) As String
Dim tfsServer As String = My.Settings("TfsFullPath").ToString ' {YOUR TFS SERVER PATH}
Dim strAssUser As String = My.Settings("AssignTo").ToString
Dim teamFoundationServer1 As Microsoft.TeamFoundation.Client.TfsTeamProjectCollection = Microsoft.TeamFoundation.Client.TfsTeamProjectCollectionFactory.GetTeamProjectCollection(New Uri(tfsServer))
Dim workItemStore1 As New WorkItemStore(teamFoundationServer1)
teamFoundationServer1.Authenticate()
Dim WorkItemStore As WorkItemStore = New WorkItemStore(tfsServer)
Dim tfsProject As Project = WorkItemStore.Projects(0)
Dim wIType As WorkItemType = tfsProject.WorkItemTypes("Bug")
Dim workItem As WorkItem = New WorkItem(wIType)
Dim wiStore = teamFoundationServer1.GetService(Of WorkItemStore)()
Dim Project = wiStore.Projects
Dim Area = Project.Item(0).AreaRootNodes("BITS").Id ' The project to add the work item to
' Prefill items
workItem.Title = Title
workItem.Description = Desc
workItem.AreaId = Project.Item(0).AreaRootNodes("BITS").Id
workItem.Fields("Assigned To").Value = strAssUser
workItem.Fields("System Info").Value = "Access V 1.1.25"
workItem.Fields("Repro Steps").Value = Desc
Dim result As ArrayList = workItem.Validate()
If result.Count > 0 Then
Return (result(0).ToString + "There was an error adding this work item to the work item repository")
Else
workItem.Save()
End If
' Open the new item in explorer
Dim myService = teamFoundationServer1.GetService(Of TswaClientHyperlinkService)()
Dim myUrl = myService.GetWorkItemEditorUrl(workItem.Id)
Dim oProcess As New System.Diagnostics.Process()
oProcess.StartInfo.FileName = myUrl.ToString
oProcess.Start()
Return "OK"
End Function
End Class

Crystal Report 13 prompts Database Login in VS 2010

I am currently having problem displaying the Crystal Report in my VB.Net(VS2010) application. I Googled this issue, found similar problem, but their solution didn't fixed mine.
Everytime I call the CR report from my VB application, it always prompts for a Database Login. I've entered my DB credentials but it fails. I am connecting to a MySQL5.1 via ODBC connector.
In my Crystal Report, I have created a custom Command which accept 2 parameters from my application, e.i, dateFrom and dateTo, which I used to query between the date range.
I've noticed that when I took off my code where it sets the Crystal Report parameter value, the Database Login doesn't appear.
Here's my code snippet:
Dim appPath As String = Path.GetDirectoryName(Application.ExecutablePath)
Dim cryRpt As New ReportDocument
Dim CRTable As CrystalDecisions.CrystalReports.Engine.Table
Dim CRTLI As CrystalDecisions.Shared.TableLogOnInfo
cryRpt.Load(appPath & "\Reports\crDtrLogs.rpt")
frmReportViewer.crReportViewer.Refresh()
For Each CRTable In cryRpt.Database.Tables
CRTLI = CRTable.LogOnInfo
With CRTLI.ConnectionInfo
.ServerName = "dtrsql"
.UserID = "root"
.Password = "root"
.DatabaseName = "dtrsql"
End With
CRTable.ApplyLogOnInfo(CRTLI)
Next CRTable
When I took this out, everything works fine even without adding the table connection info
cryRpt.SetParameterValue("dtpFrom", dtpFrom.Value.Date.ToString("yyyy-MM-dd"))
cryRpt.SetParameterValue("dtpTo", dtpTo.Value.Date.ToString("yyyy-MM-dd"))
cryRpt.SetParameterValue("strDateRange", dtpFrom.Value.Date.ToString("MMMM dd, yyyy") & " - " & dtpTo.Value.Date.ToString("MMMM dd, yyyy"))
frmReportViewer.crReportViewer.ReportSource = cryRpt
frmReportViewer.Show()
As always, any help is greatly appreciated.
eassy and simple solution for all ....
open field explorer---> database field --->Right Click -->current Data source --->reports connection----->report ----->property ----> set Property as---
Data Source: .\Databasename.accdb
and code on viewer form load as
Dim cryRpt As New ReportDocument
Dim Report1 As New rptItemWise
Dim strServerName As String
strServerName = Application.StartupPath
rptItemWise.SetDatabaseLogon("admin", "", strServerName, "dastabasename.accdb", True)
cryRpt.Load(Application.StartupPath + "\rptItemWise.rpt")
also change the report connection same as data source i think that code work for you ..
Check this code. I guess you should access to the database like this:
Dim appPath As String = Path.GetDirectoryName(Application.ExecutablePath)
Dim cryRpt As New ReportDocument
Dim crtableLogoninfos As New TableLogOnInfos
Dim crtableLogoninfo As New TableLogOnInfo
Dim crConnectionInfo As New ConnectionInfo
Dim CrTables As Tables
cryRpt.Load(appPath & "\Reports\crDtrLogs.rpt")
crConnectionInfo.ServerName = "YOUR SERVER NAME"
crConnectionInfo.DatabaseName = "YOUR DATABASE NAME"
crConnectionInfo.UserID = "YOUR DATABASE USERNAME"
crConnectionInfo.Password = "YOUR DATABASE PASSWORD"
CrTables = cryRpt.Database.Tables
For Each CrTable In CrTables
crtableLogoninfo = CrTable.LogOnInfo
crtableLogoninfo.ConnectionInfo = crConnectionInfo
CrTable.ApplyLogOnInfo(crtableLogoninfo)
Next CrTable

SSIS Script Transformation slows down after 400k records

I have an SSIS Transformation Task that I use as my final destination task to insert data into a SQL Server table. The reason I use the transformation task and not an SQL Server Destination task is because I do not know beforehand what the columns will in the table that we will be inserting into.
In a for each loop container, I am looking for access DB's (in 97 format). The rest of the control flow basically creates a new SQL database and also a table. The access files are what we call "minute" databases where they contain minute information gathered by another process. I need to create a new SQL DB named after the 'minute' db and a table called 'MINUTE' with the columns created based on certain info from the access db. For each of our clients, based on the number of parameters they have at their site, determines the number of columns I need to create in the SQL Minute table.
In the data flow I have two key components: The OLE DB source component (Source - Minute Table) and the Script Transformation task (Destination - Minute Table).
The "Source - Minute Table" gets the data from the access database. The "Destination - Minute Table" transforms the data and inserts it into the appropriate DB and table.
Everything works as it should. I tested it on a DB with 491,000+ records and it took 1 minute. However, I'm testing with one of our larger customers that has over 50 parameters and the access database contains 2+ million records. The package flies until I reach around 477,000 records, and then it pretty much comes to a halt. I can wait 10 minutes, and even longer, until the record count updates, and then continue to wait again.
I've done much research and followed all of the recommendations and guidelines that I have found. My datasource is not sorted. I use SQL command instead of Table, etc in the OLE DB Source. I've changed the values of DefaultBufferMaxRows and DefaultBufferSize many times and get the same results.
Code:
Public Class ScriptMain
Inherits UserComponent
Private conn As SqlConnection
Private cmd As SqlCommand
Private DBName As SqlParameter
Private columnsForInsert As SqlParameter
Private tableValues As SqlParameter
Private numberOfParams As Integer
Private db As String
Private folderPath As String
Private dbConn As String
Private folder As String
Private columnParamIndex As Integer
Private columnDate As DateTime
Private columnMinValue As Double
Private columnStatus As String
Private columnCnt1 As Int16
Private dateAdded As Boolean = False
Private columnStatusCnt As String
Private columnsConstructed As Boolean = False
Private buildValues As StringBuilder
Private columnValues As StringBuilder
Private i As Integer = 0
'This method is called once, before rows begin to be processed in the data flow.
'
'You can remove this method if you don't need to do anything here.
Public Overrides Sub PreExecute()
MyBase.PreExecute()
Try
'Dim dbConnection As String = "Server=(local)\SQLExpress;Database=DataConversion;User ID=sa;Password=sa123;"
'conn = New SqlConnection(dbConnection)
'conn.Open()
'cmd = New SqlCommand("dbo.InsertValues", conn) With {.CommandType = CommandType.StoredProcedure}
'columnsForInsert = New SqlParameter("#Columns", SqlDbType.VarChar, -1) With {.Direction = ParameterDirection.Input}
'cmd.Parameters.Add(columnsForInsert)
'DBName = New SqlParameter("#DBName", SqlDbType.VarChar, -1) With {.Direction = ParameterDirection.Input}
'cmd.Parameters.Add(DBName)
'tableValues = New SqlParameter("#Values", SqlDbType.VarChar, -1) With {.Direction = ParameterDirection.Input}
'cmd.Parameters.Add(tableValues)
db = Variables.varMinFileName.ToString
folder = Variables.varMinFolderName.ToString
folderPath = folder & "\" & db & ".mdb"
dbConn = "Provider=Microsoft.Jet.OLEDB.4.0;Data Source=" & folderPath
Using SourceDataAdapter As OleDbDataAdapter = New OleDbDataAdapter("SELECT DISTINCT PARAM_INDEX FROM [MINUTE];", dbConn)
Dim SourceDatatable As New DataTable
SourceDataAdapter.Fill(SourceDatatable)
numberOfParams = SourceDatatable.Rows.Count
End Using
'columnValues.Append("dtmTime, ")
buildValues = New StringBuilder
columnValues = New StringBuilder
columnValues.Append("dtmTime, ")
Catch ex As Exception
Dim writer As New StreamWriter("C:\MinuteLog.log", True, System.Text.Encoding.ASCII)
writer.WriteLine(ex.Message)
writer.Close()
writer.Dispose()
Finally
End Try
End Sub
' This method is called after all the rows have passed through this component.
'
' You can delete this method if you don't need to do anything here.
Public Overrides Sub PostExecute()
MyBase.PostExecute()
'
' Add your code here
'
buildValues = Nothing
columnValues = Nothing
End Sub
Public Overrides Sub Input0_ProcessInput(Buffer As Input0Buffer)
While Buffer.NextRow()
Input0_ProcessInputRow(Buffer)
End While
End Sub
'This method is called once for every row that passes through the component from Input0.
Public Overrides Sub Input0_ProcessInputRow(ByVal Row As Input0Buffer)
Dim column As IDTSInputColumn100
Dim rowType As Type = Row.GetType()
Dim columnValue As PropertyInfo
Dim result As Object
Dim rtnValue As String = Variables.varMinFileName.Replace("_", "")
Dim colName As String
Try
For Each column In Me.ComponentMetaData.InputCollection(0).InputColumnCollection
columnValue = rowType.GetProperty(column.Name)
colName = column.Name.ToString
If Not colName.Contains("NULL") Then
'If Not columnValue Is Nothing Then
Select Case column.Name.ToString
Case "PARAM_INDEX"
'result = columnValue.GetValue(Row, Nothing)
result = Row.PARAMINDEX
columnParamIndex = CType(result, Byte)
If columnsConstructed = False And i <= numberOfParams - 1 Then
columnValues.Append(String.Format("VALUE_{0}, STATUS_{0}, ", columnParamIndex.ToString))
End If
Exit Select
Case "dtmTIME"
'result = columnValue.GetValue(Row, Nothing)
result = Row.dtmTIME
columnDate = CType(result, DateTime)
If dateAdded = False Then ' only need to add once since rows are vertical
buildValues.Append("'" & columnDate & "', ")
dateAdded = True
End If
Exit Select
Case "MIN_VALUE"
'result = columnValue.GetValue(Row, Nothing)
result = Row.MINVALUE
columnMinValue = CType(result, Double)
buildValues.Append(columnMinValue & ", ")
Exit Select
Case "MIN_STATUS"
'result = columnValue.GetValue(Row, Nothing)
result = Row.MINSTATUS
columnStatus = CType(result, String)
Exit Select
Case "MIN_CNT_1"
'result = columnValue.GetValue(Row, Nothing)
result = Row.MINCNT1
columnCnt1 = CType(result, Byte)
columnStatusCnt = columnStatus & "010" & columnCnt1.ToString.PadLeft(5, "0"c) & "-----"
buildValues.Append("'" & columnStatusCnt & "', ")
Case Else
Exit Select
End Select
'End If
End If
Next
If i = numberOfParams - 1 Then
If columnsConstructed = False Then
columnValues.Remove(columnValues.Length - 2, 1)
End If
buildValues.Remove(buildValues.Length - 2, 1)
Dim valueResult As String = buildValues.ToString()
SetStoredProc()
cmd.Parameters("#Columns").Value = columnValues.ToString
cmd.Parameters("#DBName").Value = "[" & rtnValue & "].[dbo].[MINUTE]"
cmd.Parameters("#Values").Value = valueResult
cmd.ExecuteNonQuery()
buildValues.Clear()
columnsConstructed = True
dateAdded = False
columnParamIndex = 0
columnMinValue = 0
columnStatus = String.Empty
columnCnt1 = 0
i = 0
conn.Close()
conn.Dispose()
Else
i += 1
End If
Catch ex As Exception
Dim writer As New StreamWriter("C:\MinuteLog.log", True, System.Text.Encoding.ASCII)
writer.WriteLine(ex.Message)
writer.Close()
writer.Dispose()
Finally
'buildValues = Nothing
'columnValues = Nothing
End Try
End Sub
Private Sub SetStoredProc()
Try
Dim dbConnection As String = "Server=(local)\SQLExpress;Database=DataConversion;User ID=sa;Password=sa123;"
conn = New SqlConnection(dbConnection)
conn.Open()
cmd = New SqlCommand("dbo.InsertValues", conn) With {.CommandType = CommandType.StoredProcedure}
columnsForInsert = New SqlParameter("#Columns", SqlDbType.VarChar, -1) With {.Direction = ParameterDirection.Input}
cmd.Parameters.Add(columnsForInsert)
DBName = New SqlParameter("#DBName", SqlDbType.VarChar, -1) With {.Direction = ParameterDirection.Input}
cmd.Parameters.Add(DBName)
tableValues = New SqlParameter("#Values", SqlDbType.VarChar, -1) With {.Direction = ParameterDirection.Input}
cmd.Parameters.Add(tableValues)
Catch ex As Exception
Dim writer As New StreamWriter("C:\MinuteLog.log", True, System.Text.Encoding.ASCII)
writer.WriteLine(ex.Message)
writer.Close()
writer.Dispose()
End Try
End Sub
End Class
Since I can't upload images yet here I've included a blog link I created that includes ample screen shots to help understand the problem mentioned here:
SSIS slows down during transformation task
Any help in determining why my package slows after 400k records and doesn't process all 2+ million records in a reasonable time is much appreciated!
Thanks,
Jimmy
This probably isn't terribly helpful but my guess is you are running out of memory. If SSIS has to page you've had it in my experience.
Can you batch up the work somehow in several smaller runs perhaps?
Full solution can be viewed here on my blog with screenshots - SSIS slowdown solved
In order to get around SSIS slowing down when a large number of records are being transformed and inserted into SQL Server as my destination, I redesigned my SSIS package. Instead of doing an insert in a data transformation task for every record that comes through the buffer, I’ve eliminated it and have used a stored procedure to do a bulk insert. In order to accomplish this, I read in the data from each access DB into a table called “MINUTE” in my SQL Server instance. This minute table has the same schema as the access DB’s and I let SSIS do the heavy lifting of importing all the data into this table. After the data is imported, I execute my stored procedure which transforms the data in this minute table (horizontal records) and does a bulk insert into my new destination MINUTE SQL table (one vertical record.)
The stored procedure that does the bulk insert and transforms the data looks like this:
PROCEDURE [dbo].[InsertMinuteBulk]
-- Add the parameters for the stored procedure here
(#Columns varchar(MAX), #DBName varchar(4000))
AS
BEGIN
DECLARE #SQL varchar(MAX)
SET #SQL =’;WITH Base AS (
SELECT dtmTime,
param_index,
CONVERT(nvarchar(16), MIN_VALUE) AS [VALUE_],
CONVERT(nvarchar(3), MIN_STATUS) + ”000” + LEFT(replicate(”0”,5) + CONVERT(nvarchar(5), MIN_CNT_1),5) + ”—–” AS [STATUS_]
FROM [DataConversion].[dbo].[MINUTE]
)
,norm AS (
SELECT dtmTime, ColName + CONVERT(varchar, param_index) AS ColName, ColValue
FROM Base
UNPIVOT (ColValue FOR ColName IN ([VALUE_], [STATUS_])) AS pvt
)
INSERT INTO ‘ + #DBName + ‘
SELECT *
FROM norm
PIVOT (MIN(ColValue) FOR ColName IN (‘+#Columns+’)) AS pvt’
EXEC (#SQL);
In the Data Flow task, the “Minute Data Source" is an ADO.NET Data Source and feeds the data into my SQL Server destination – "Minute Data Destination".
In the Control Flow, the final task of "Bulk Insert Minute Data" executes the Bulk Insert stored procedure.
The package now runs uninterrupted and is pretty fast considering the size of data that I’m reading, transforming and inserting.
I’ve ran the package as an SSIS job and it took 38 minutes to complete converting 7 months (or 7 minute access DB’s) worth of minute data with over 2 million rows in each access DB.