How do I execute the SQL script below a SSIS project? I've tried setting up parameters & variables; however.... nothing I do seems to pass the parameters via SSIS
declare #businessunit varchar(255) = 'Test'
declare #advisor varchar(255) = 'Smith'
declare #iuid int =
(
select U.[iuid]
from U
inner join [dbo].A on u.[ipartyid] = A.[iuserid]
inner join [dbo].B on A.[ibusinessunitid] = B.[ipartyid]
inner join [dbo].C on u.[ipartyid] = C.[ipartyid]
inner join [dbo].D on C.[ipartyid] = D.[ipartyid]
where 1 = 1
and B.[name] = #businessunit
and D.[lastname] = #advisor
)
select HHName
,HHID = h.ihhid
,FNAME =
case
when charindex(',', h.vhhname) > 0 and trim(substring(h.vhhname, patindex('% i%', h.vhhname), len(h.vhhname))) in ('i', 'ii', 'iii')
then concat(dbo.Propercase(concat(trim(substring(trim(substring(h.vhhname, charindex(' ', h.vhhname), len(h.vhhname))), 1, charindex(' ', trim(substring(h.vhhname, charindex(' ', h.vhhname), len(h.vhhname)))))), ' ', trim(substring(h.vhhname, 1, (charindex(',', h.vhhname) - 1))))), UPPER(substring(h.vhhname, patindex('% i%', h.vhhname), len(h.vhhname))))
when charindex(',', h.vhhname) > 0 and nullif(h.vdescr, '') is null
then dbo.Propercase(replace(replace(concat(trim(substring(h.vhhname, (charindex(',', h.vhhname) + 1), len(h.vhhname))), ' ', trim(substring(h.vhhname, 1, (charindex(',', h.vhhname) - 1)))), '+', '&'), ' and ', ' & '))
else dbo.Propercase(replace(isnull(nullif(h.vdescr, ''), h.vhhname), ' and ', ' & '))
end
,RepNo = h.planid
from [dbo].[HH] h
inner join
(
select u.[usertype]
,a.[user_planid]
from [dbo].users u
inner join [dbo].user_access a on u.iuid = a.iuid
where 1 = 1
and u.[usertype] <> 'e'
and u.iuid = #iuid
group by u.[usertype],a.[user_planid]
) p
on h.[planid] = p.[user_planid]
I'm going to assume you already have two SSIS variables that correspond to #businessunit and #advisor and they are being populated with the correct values already.
You can use an Execute SQL Task with parameter mapping to run your query. First thing you want to do is open the task editor, and configure your db connection. Next, hit the three dots next to SQLStatement to pull up the query editor window. Now you can start transposing your query, with a few modifications. I find that the Execute SQL Task works best when you separate variable declaration and assignment statements. You can use the following as your query text:
declare #businessunit varchar(255)
declare #advisor varchar(255)
declare #iuid int
SET #businessunit = ?
SET #advisor = ?
SET #iuid =
(
select U.[iuid]
from U
inner join [dbo].A on u.[ipartyid] = A.[iuserid]
inner join [dbo].B on A.[ibusinessunitid] = B.[ipartyid]
inner join [dbo].C on u.[ipartyid] = C.[ipartyid]
inner join [dbo].D on C.[ipartyid] = D.[ipartyid]
where 1 = 1
and B.[name] = #businessunit
and D.[lastname] = #advisor
)
select HHName
,HHID = h.ihhid
,FNAME =
case
when charindex(',', h.vhhname) > 0 and trim(substring(h.vhhname, patindex('% i%', h.vhhname), len(h.vhhname))) in ('i', 'ii', 'iii')
then concat(dbo.Propercase(concat(trim(substring(trim(substring(h.vhhname, charindex(' ', h.vhhname), len(h.vhhname))), 1, charindex(' ', trim(substring(h.vhhname, charindex(' ', h.vhhname), len(h.vhhname)))))), ' ', trim(substring(h.vhhname, 1, (charindex(',', h.vhhname) - 1))))), UPPER(substring(h.vhhname, patindex('% i%', h.vhhname), len(h.vhhname))))
when charindex(',', h.vhhname) > 0 and nullif(h.vdescr, '') is null
then dbo.Propercase(replace(replace(concat(trim(substring(h.vhhname, (charindex(',', h.vhhname) + 1), len(h.vhhname))), ' ', trim(substring(h.vhhname, 1, (charindex(',', h.vhhname) - 1)))), '+', '&'), ' and ', ' & '))
else dbo.Propercase(replace(isnull(nullif(h.vdescr, ''), h.vhhname), ' and ', ' & '))
end
,RepNo = h.planid
from [dbo].[HH] h
inner join
(
select u.[usertype]
,a.[user_planid]
from [dbo].users u
inner join [dbo].user_access a on u.iuid = a.iuid
where 1 = 1
and u.[usertype] <> 'e'
and u.iuid = #iuid
group by u.[usertype],a.[user_planid]
) p
on h.[planid] = p.[user_planid]
Hit OK in the query editor window.
The ? in the SET statements tell the task to pull the values from the Parameter Mapping. So now let's configure the parameter mappings.
In the left pane of the Execute SQL Task Editor, click on Parameter Mapping. If your db connection is OLE or EXCEL, then the Parameter Name will start with 0 and increment by one for each additional parameter. If it's an ODBC connection, you'll start with 1 instead. The parameter names match up with the ordinal position of the ?. So in our example here, #businessunit would be the first parameter mapped and #advisor would be the second. Now you're going to add two parameters. Hit the Add button, then change the Variable Name to your first SSIS variable. Leave Direction set to Input, change Data Type to VARCHAR, set the Parameter Name, then set the Parameter Size to 255. Repeat for the second variable. Your paramter mappings should look something like this:
Make sure you hit OK to save all your changes.
For a script as long as this, one solution that may work for you depending on what you are trying to achieve is to use SSIS script tasks.
Script tasks allow you to use C# (Or Visual Basic) in order to execute SQL via the same System.Data.SqlClient class that you would normally use in other C# programs, such as a console or ASP.NET application.
For your SQL above, put it into a stored procedure, and then execute this stored procedure within the script task. You could then use SqlDataReader or SqlDataAdapter to then read and store the result into a model.
From there, you can choose to manipulate the data within the SSIS script task.
For Example:
SqlConnection connection = new Connection("connection string");
using(SqlCommand command = new SqlCommand("Trans-SQL or stored procedure name", connection)
{
command.CommandType = CommandType.StoredProcedure;
command.Parameters.Add(/*Add your parameters...*/);
connection.Open()
SqlDataReader reader = command.ExecuteReader()
while (reader.Read())
{
//Use reader["name"] in here to read values from response into a model
}
}
Is a completely valid way of querying data within a SSIS task. If you would rather not deal with a Reader, you can use the SqlDataAdapter and use the Fill() method to store the result(s) in a dataset.
Overall, when dealing with complex data (and where efficiency isn't too much of a concern), I find that completing actions within SSIS script tasks that get triggered by the control logic is the easier way to use SSIS.
You may find this Integration Services Programming Overview documentation site useful as a reference for some of the things you can do with SSIS script tasks.
On a final note, please be aware that script tasks in SSIS do have some limitations, a key one is that there is generally worse support for newer C# features, that cause issues such as not being able to hit debug breakpoints.
Related
I have an SSIS package with a variable of data type String.
I have a table with an XML field and I am trying to get data from this field and populate my variable. I have tried numerous ways:
1.-
SELECT ISNULL(STUFF((
SELECT ',' + wrk.CustomFieldsXML.value('(/values/MatterUNO)[1]', 'VARCHAR(MAX)') AS [text()]
FROM Work wrk
WHERE ISNULL(wrk.CustomFieldsXml.value('(values/ReleaseSecureMattertoFoundation)[1]', 'varchar(5)'), 'false') = 'true'
FOR XML path('')), 1, 1, ''), '') AS 'RESULT'
2.-
DECLARE #RESULT VARCHAR(MAX)
SELECT #RESULT = ISNULL(STUFF((
SELECT ',' + wrk.CustomFieldsXML.value('(/values/MatterUNO)[1]', 'VARCHAR(MAX)') AS [text()]
FROM Work wrk
WHERE ISNULL(wrk.CustomFieldsXml.value('(values/ReleaseSecureMattertoFoundation)[1]', 'varchar(5)'), 'false') = 'true'
FOR XML path('')), 1, 1, ''), '')
SELECT ISNULL(#RESULT, '') AS 'RESULT'
3.-
SELECT wrk.CustomFieldsXML.value('(/values/MatterUNO)[1]', 'VARCHAR(MAX)') AS 'RESULT'
FROM Work wrk
WHERE ISNULL(wrk.CustomFieldsXml.value('(values/ReleaseSecureMattertoFoundation)[1]', 'varchar(5)'), 'false') = 'true'
4.-
SELECT (SELECT TOP 1 ISNULL(wrk.CustomFieldsXML.value('(/values/MatterUNO)[1]', 'VARCHAR(MAX)'), '')
FROM Work wrk
WHERE ISNULL(wrk.CustomFieldsXml.value('(values/ReleaseSecureMattertoFoundation)[1]', 'varchar(5)'), 'false') = 'true') AS 'RESULT'
Each of these results in 2 errors:
The value type (_ComObject) can only be converted to variables with type Object.
The type of the value (DBNull) being assigned to variable differs from current type (String).
When I change the SQL to be: SELECT '988776' AS 'RESULT' the variable assignment works fine.
Does anyone know whether this is a limitation with working with XML columns?
Can anyone help?
Thanks
Rob
Value of type Varchar(MAX) from SQL Task cannot be assigned to SSIS variable directly. This accounts for the error
The value type (_ComObject) can only be converted to variables with type Object.
Ways to fix it:
Return a more bounded data type, i.e. Varchar(8000) (no more than 8000) or NVarchar(4000) (no more than 4000), taking the risk of data truncation. This valus can be assigned by SQL Task to your string variable.
Return a Varchar(MAX) value and assign it to an Object type variable. Besides, you have to return Full Result set in properties of SQL Task. Then, in the next step - set a Foreach Loop enumerator, and store 0-th column into your string variable in Variables mappings. Here is a good walkthrough on how to do this.
The second option seems a little bit strange, but it works.
I am currently building a number of logging and analysis tools to keep tabs on our SQL environment. We are currently using SQL Server 2014.
What I want to do is keep check of all the parameters that are passed to our reports during the day. All of the reports are currently using stored procedures so in my table or a select statement based on a table is output the stored procedure with the parameters for every time the report was run.
At the end of the day I would then like to be able to take the outputted statement and run it in SSMS without having to use the report. I have been looking at the ExceutionLogStorage table and the ExecutionLog view's and though it has most of the information that I need, the parameters are not in an easily usable state.
Has anyone done something similar to what I have described?
You need to add logging part in your original SP, for example:
Alter procedure a
(#parameter)
As
Begin
..
..
Insert into loggingTable(col)
Values(#parameter)
..
..
End
Then query directly against that loggingTable for getting the history of used parameters
A Google search around this topic quickly brought up the following blog post already identified by the OP as useful and shown below (this query itself is actually an expansion of work linked to by LONG's answer below)
SELECT TOP 1 ParValue
FROM (
SELECT els.TimeEnd
, IIF(CHARINDEX('&' + 'ParameterName' + '=', ParsString) = 0, 'ParameterName',
SUBSTRING(ParsString
, StartIndex
, CHARINDEX('&', ParsString, StartIndex) - StartIndex)) AS ParValue
FROM (SELECT ReportID, TimeEnd
, '&' + CONVERT(VARCHAR(MAX), Parameters) + '&' AS ParsString
, CHARINDEX('&' + 'ParameterName' + '=', '&' + CONVERT(VARCHAR(MAX), Parameters) + '&')
+ LEN('&' + 'ParameterName' + '=') AS StartIndex
FROM ExecutionLogStorage
WHERE UserName='UserName' -- e.g. DOMAIN\Joe_Smith
) AS els
INNER JOIN [Catalog] AS c ON c.ItemID = els.ReportID
WHERE c.Name = 'ReportName'
UNION ALL
SELECT CAST('2000-01-01' AS DateTime), 'ParameterName'
) i
ORDER BY TimeEnd DESC;
Both these approaches though really only give us a starting point since they (variously) rely upon us knowing in advance the report name and parameter names. Whilst we can quickly make a couple of changes to Ken Bowman's work to get it to run against all executions of all reports, we still have the problem that the query hardcodes the parameter name.
The parameters required to execute a report are stored on the Catalog table in the Parameter column. Although the column has a datatype ntext, it is actually storing an XML string. Meaning we can use an XPath query to get at the parameter names
with
CatalogData as (
select ItemID, [Path], [Name], cast(Parameter as xml) 'ParameterXml'
from Catalog
where [Type] = 2),
ReportParameters as (
select ItemID, [Path], [Name], ParameterXml, p.value('Name[1]', 'nvarchar(256)') 'ParameterName'
from CatalogData
cross apply ParameterXml.nodes('/Parameters/Parameter') as Parameters(p))
select *
from ReportParameters;
Executing this query will list all reports on the server and their parameters. Now we just need to combine this with Ken Bowman's query. I've gone with a CTE approach
with
CatalogData as (
select ItemID, [Path], [Name], cast(Parameter as xml) 'ParameterXml'
from Catalog
where [Type] = 2),
ReportParameters as (
select ItemID, [Path], [Name], p.value('Name[1]', 'nvarchar(256)') 'ParameterName'
from CatalogData
cross apply ParameterXml.nodes('/Parameters/Parameter') as Parameters(p))
select
els.TimeEnd
, c.[Name]
, rp.ParameterName
, iif(
charindex(
'&' + rp.ParameterName + '=', ParametersString) = 0
, rp.ParameterName, substring(ParametersString
, StartIndex, charindex('&', ParametersString, StartIndex) - StartIndex
)) 'ParameterValue'
from (
select
ReportID
, TimeEnd
, rp.ParameterName
, '&' + convert(varchar(max), Parameters) + '&' 'ParametersString'
, charindex(
'&' + rp.ParameterName + '=',
'&' + convert(varchar(max), Parameters) + '&'
) + len('&' + rp.ParameterName + '=') 'StartIndex'
from
ExecutionLogStorage
inner join ReportParameters rp on rp.ItemID = ReportID) AS els
inner join [Catalog] c on c.ItemID = els.ReportID
inner join ReportParameters rp on rp.ItemID = c.ItemID and rp.ParameterName = els.ParameterName;
Note that the parameter values are passed to the report as part of a URL, so you'll still need get rid the literal space encoding and so on. Also, this doesn't (yet...) work for multi-value parameters.
I have a normal SQL statement:
SELECT VALUE_ID, UF_CRM_TASK FROM b_uts_tasks_task
Now this returns a a different field everytime but they take the form of the following:
a:1:{i:0;s:7:"CO_2012";} or a:1:{i:0;s:5:"CO_12";} or a:1:{i:0;s:7:"CO_2017";}
Basically they're different everytime. What I need is to just get the number after the CO_ part. I have tried TRIM but because everything changes in the leading and trailing section I don't think this would work.
I have looked on Stack Overflow for a while and cannot find it. I know how to do it in PHP:
$data = $row['UF_CRM_TASK'];
$companyID = substr($data, strpos($data, "CO_") + 1);
$newCompanyID = preg_replace('/[^0-9.]+/', '', $companyID);
But not SQL. Thanks in advance
In MYSQL is a bit ugly:
/*SUBSTRING_INDEX BASED ON CO_ AND THE LAST " - in 2 SUBSTRINGS*/
SELECT `VALUE_ID`, SUBSTRING_INDEX(SUBSTRING_INDEX(`UF_CRM_TASK`, 'CO_', -1), '"', 1) AS `COMPANY_ID` FROM `b_uts_tasks_task`
In PHP you can just unserialize():
$data = unserialize($row['UF_CRM_TASK']);
$companyID = str_replace('CO_', '', $data[0]);
eg:
$data = unserialize('a:1:{i:0;s:5:"CO_12";}');
echo str_replace('CO_', '', $data[0]);
//==> 12
You need to use CharIndex and SubString (Microsoft SQL) or
This is the sample code I made for my Microsoft SQL server:
declare #companyIdString varchar(50) = 'a:1:{i:0;s:7:"CO_2012";}'
print 'Company ID in a string: ' + #companyIdString
print 'Find first position: ' + Cast(charindex('"CO_', #companyIdString) as varchar(2))
print 'Locate the second position (the last "): ' + Cast(charindex('"', #companyIdString, charindex('"CO_', #companyIdString)+4) as varchar(2))
print 'Extracted Company Id: ' + substring(#companyIdString,charindex('"CO_', #companyIdString)+4, charindex('"', #companyIdString, charindex('"CO_', #companyIdString)+4) - charindex('"CO_', #companyIdString) - 4)
select
#companyIdString as CompanyIdString,
substring(#companyIdString,charindex('"CO_', #companyIdString)+4, charindex('"', #companyIdString, charindex('"CO_', #companyIdString)+4) - charindex('"CO_', #companyIdString) - 4) as CompanyId
I also made the same code on a mySQL server:
set #companyIdString := 'a:1:{i:0;s:7:"CO_2012";}';
select
#companyIdString as CompanyIdString,
substring_index(substring_index(substring_index(#companyIdString, '"', 2), '"', -1), '_', -1) as CompanyId
The substring_index starts by locating the second " (string is now a:1:{i:0;s:7:"CO_2012), then it searches backward with the -1 to locate the first " (string is now CO_2012). And then it searches backward for the underscore (string is now 2012).
I have a query that I am trying to convert to MySQL from MS SQL Server 2008. It runs fine on MSSQL,
I get the error
"Incorrect parameter count in the call to native function 'ISNULL'".
How do I solve this?
SELECT DISTINCT
dbo.`#EIM_PROCESS_DATA`.U_Tax_year,
dbo.`#EIM_PROCESS_DATA`.U_Employee_ID,
CASE
WHEN dbo.`#EIM_PROCESS_DATA`.U_PD_code = 'SYS033' THEN SUM(dbo.`#EIM_PROCESS_DATA`.U_Amount)
END AS PAYE,
CASE
WHEN dbo.`#EIM_PROCESS_DATA`.U_PD_code = 'SYS014' THEN SUM(dbo.`#EIM_PROCESS_DATA`.U_Amount)
END AS TOTALTAXABLE,
dbo.OADM.CompnyName,
dbo.OADM.CompnyAddr,
dbo.OADM.TaxIdNum,
dbo.OHEM.lastName + ', ' + ISNULL(dbo.OHEM.middleName, '') + '' + ISNULL(dbo.OHEM.firstName, '') AS EmployeeName
FROM
dbo.`#EIM_PROCESS_DATA`
INNER JOIN
dbo.OHEM ON dbo.`#EIM_PROCESS_DATA`.U_Employee_ID = dbo.OHEM.empID
CROSS JOIN
dbo.OADM
GROUP BY dbo.`#EIM_PROCESS_DATA`.U_Tax_year , dbo.`#EIM_PROCESS_DATA`.U_Employee_ID , dbo.OADM.CompnyName , dbo.OADM.CompnyAddr , dbo.OADM.TaxIdNum , dbo.OHEM.lastName , dbo.OHEM.firstName , dbo.OHEM.middleName , dbo.`#EIM_PROCESS_DATA`.U_PD_code
MySQL
SELECT DISTINCT
processdata.taxYear, processdata.empID,
CASE WHEN processdata.edCode = 'SYS033' THEN SUM (processdata.amount) END AS PAYE,
CASE WHEN processdata.edCode = 'SYS014' THEN SUM (processdata.amount) END AS TOTALTAXABLE,
company.companyName, company.streetAddress, company.companyPIN, employeemaster.lastName + ', ' + IFNULL(employeemaster.middleName, '')
+ ' ' + IFNULL(employeemaster.firstName, '') AS EmployeeName
FROM
processdata INNER JOIN
employeemaster ON processdata.empID = employeemaster.empID
CROSS JOIN company
GROUP BY processdata.taxYear, processdata.empID, company.companyName, company.streetAddress, company.companyPIN,
employeemaster.lastName, employeemaster.firstName, employeemaster.middleName, processdata.edCode
The MySQL equivalent of ISNULL is IFNULL
If expr1 is not NULL, IFNULL() returns expr1; otherwise it returns
expr2.
Maybe also look at SQL NULL Functions
The ISNULL from MySQL is used to check if a value is null
If expr is NULL, ISNULL() returns 1, otherwise it returns 0.
I would suggest that you switch to the ANSI standard function coalesce():
(dbo.OHEM.lastName + ', ' + coalesce(dbo.OHEM.middleName, '') + '' + coalesce(dbo.OHEM.firstName, '')
) AS EmployeeName
You could also make your query easier to read by including table aliases.
EDIT:
As a note, I seemed to have missed the direction of conversion. The MySQL query would use concat():
CONCAT(OHEM.lastName, ', ', coalesce(OHEM.middleName, ''),
coalesce(concat(' ', OHEM.firstName), '')
) AS EmployeeName
I was getting an error when running JUnit tests against a query which had ISNULL(value) with the error saying ISNULL needed two parameters. I fixed this by changing the query to have value is null and the code works the same while the tests now work.
I have the following mssql query that I found on the net that is supposed to help me with a complex mysql query that I have been struggling with for a few days now.
SELECT
inv.typeID AS typeID,
inv.typeName AS typeName,
invGroups.groupName AS groupName,
inv.published AS published,
inv.description AS description,
rankVal.valueFloat AS rank,
replace (( SELECT skills.attributeName AS [data()]
FROM dgmTypeAttributes tattr -- Link between skillbook and attributes
INNER JOIN dgmAttributeTypes skills ON (skills.attributeID = tattr.valueInt)
WHERE (tattr.typeID = inv.typeID)
AND (tattr.attributeID IN (180, 181)) -- Primary and secondary attributes
ORDER BY inv.typeID FOR xml path('')), ' ', ',') AS prisec,
replace (( SELECT RTRIM(CAST(inv2.typeID AS varchar)) + ',' AS [data()]
FROM (SELECT * FROM dgmTypeAttributes WHERE (attributeID in (182, 183, 184)) -- Pre-req skills 1, 2, and 3
AND (typeID = inv.typeID)) tattr2
INNER JOIN invTypes inv2 ON (tattr2.valueInt = inv2.typeID)
ORDER BY inv.typeID FOR xml path('')), ' ', ' ') AS prereq,
replace (( SELECT RTRIM(CAST(tattr2.valueInt AS varchar)) + ',' AS [data()]
FROM (SELECT * FROM dgmTypeAttributes WHERE (attributeID in (277, 278, 279)) AND (typeID = inv.typeID)) tattr2 -- Link between skillbook and attributes
ORDER BY inv.typeID FOR xml path('')), ' ', ' ') AS prereqlvl
FROM invTypes inv
INNER JOIN invGroups ON (inv.groupID = invGroups.groupID)
INNER JOIN dgmTypeAttributes rankVal ON (inv.typeID = rankVal.typeID)
WHERE invGroups.categoryID = 16 -- Skillbooks category
AND rankVal.attributeID = 275 -- Skill rank attribute
AND inv.published = 1
GROUP BY inv.typeID, inv.typeName, invGroups.groupName, inv.published, inv.description, rankVal.valueFloat
ORDER BY invGroups.groupName, inv.typeName
I am so so with mysql but I know nothing of mssql. Can somebody recommend a good method of converting this query that is low or now cost? I do not expect somebody to convert it for me as that would be asking too much, but some suggestions that would point me in the rite direction (aside from learning mssql lolz) would be very nice. Thank you for your time and patience.
'Recommendation: extract the data out of you MySQL database in a delimited file (csv) using the utf8 (unicode) character set. Import into SQL Server using bcp specifying utf8 with "-Jutf8" parameter and character mode "-c".' See this site. Also, there's a nice tool for this.
Those subqueries with FOR XML PATH('') seem to be used to concatenate strings1. See if you can replace them with GROUP_CONCAT in MySQL. The other bits seem to be standard SQL.