Through my ASP.NET / SQL Server 2008 app I need to do a bulk insert of records from a CSV file (maybe a million records). I will import them into a staging table first, so I can manipulate some of the data before moving it to a permanent table.
This will happen on a regular basis. And multiple imports may happen simultaneously. I also have to tell each import from the others.
My original plan was to use a column that had an Import_ID in it. But I see that Bulk Insert won't allow me to set a field value.
Doing a search, I see that I can do a Bulk Insert into a view. And I'm guessing that the view can have a named parameter (Import_ID). But I haven't really learned setting up parameters yet, so I don't know if this is possible, or how to do it.
Can someone please tell me how to do this, or let me know another solution?
Thanks
You could bulk insert into a temporary staging table, for example since you know your SPID (and assuming you can trust the schema of some static table) you can say something like this, specifying the #filepath for the CSV file and the #ImportID, creating a table with your session id as a suffix, and doing all your work in a single dynamic SQL batch:
DECLARE #sql NVARCHAR(MAX), #spid VARCHAR(12) = RTRIM(##SPID);
SET #sql = N'SELECT * INTO dbo.Stage' + #spid
+ ' FROM dbo.RealStagingTable WHERE 1 = 0;';
SET #sql += N'BULK INSERT dbo.Stage' + #spid + ' FROM ('''
+ #filepath + ''' WITH (options);'
SET #sql += N'INSERT dbo.RealTable(ImportID, other columns)
SELECT ' + RTRIM(#ImportID) + , other columns
FROM dbo.Stage' + #spid + ';';
SET #sql += N'DROP TABLE dbo.Stage' + #spid + ';'
EXEC sp_executesql #sql;
Related
This question already has answers here:
Insert results of a stored procedure into a temporary table
(33 answers)
Closed 9 years ago.
I am using SQL Server 2008 R2 and am trying to get the results of a stored procedure into a temporary table that I can access later on in the calling stored proc. My TSQL is as follows:
CREATE PROCEDURE sp_ToBeCalled AS
(
#SomeParam INT
)
BEGIN
SELECT * FROM tblSomeTable WHERE SomeField = #SomeParam
END
CREATE PROCEDURE sp_CallingProcedure AS
(
#SomeOtherParam INT
)
BEGIN
-- A
SELECT * INTO #MyTempTable FROM sp_ToBeCalled(#SomeOtherParam)
-- B
SELECT * FROM #MyTempTable FOR XML RAW
END
This all compiles fine however when I call sp_CallingProcedure statement -- B returns an error that #MyTempTable.
How can I do "A" so that I can access its results from within a #MyTempTable table without having to declare the structure of #MyTempTable first?
I am looking for a solution that I can use generically. I have a number of existing stored procedures that I need to call from various callers where getting the results queryable is a necessity. I cannot change the existing stored procedures.
I don't want to use
OPENQUOERY() - requires a custom linked server definition
sp_ExecSql() - means I have to build up dynamic SQL which does not give me SP compile time checking.
You are trying to use a Procedure like a tabular function.
Try using
INSERT INTO #MyTempTable (column1, column2...)
exec sp_ToBeCalled(#SomeOtherParam)
A great reference: http://www.sommarskog.se/share_data.html
I managed to partially solve my issue by doing the following:
1) Custom Stored Procedure to select a ROWSET into a global temp table
2) Calling SP calls 1) and then transfers the ##GlobalTempTable into a local #TempTable for processing
This works but has the following "issues":
Potential security risk as "Adhoc Distributed Queries" functionality needs to be turned on
Still requires a Global Temp table that needs to be cleaned up by the caller. Temp table naming is also problematic as multiple 2) will cause an issue.
I include my code below in case it helps someone else. If anyone is able to improve on it please feel free to post.
/* This requires Adhoc Distributed Queries to be turned on:
sp_configure 'Show Advanced Options', 1
GO
RECONFIGURE
GO
sp_configure 'Ad Hoc Distributed Queries', 1
GO
RECONFIGURE
GO
*/
-- Adapted from: http://stackoverflow.com/questions/653714/how-to-select-into-temp-table-from-stored-procedure
CREATE PROCEDURE [dbo].[ExecIntoTable]
(
#tableName NVARCHAR(256),
#storedProcWithParameters NVARCHAR(MAX)
)
AS
BEGIN
DECLARE #driver VARCHAR(10)
DECLARE #connectionString NVARCHAR(600)
DECLARE #sql NVARCHAR(MAX)
DECLARE #rowsetSql NVARCHAR(MAX)
SET #driver = '''SQLNCLI'''
SET #connectionString =
'''server=' +
CAST(SERVERPROPERTY('ServerName') AS NVARCHAR(256)) +
COALESCE('\' + CAST(SERVERPROPERTY('InstanceName') AS NVARCHAR(256)), '') +
';trusted_connection=yes;Database=' + DB_NAME() + ''''
SET #rowsetSql = '''EXEC ' + REPLACE(#storedProcWithParameters, '''', '''''') + ''''
SET #sql = '
SELECT
*
INTO
' + #tableName + '
FROM
OPENROWSET(' + #driver + ',' + #connectionString + ',' + #rowsetSql + ')'
EXEC (#sql)
END
GO
and then to use in another SP as follows:
EXEC ExecIntoTable '##MyGlobalTable', 'sp_MyStoredProc 13, 1'
SELECT *
INTO #MyLocalTable
FROM ##MyGlobalTable
DROP TABLE ##MyGlobalTable
SELECT * FROM #MyLocalTable
I'm working on a stored procedure that will accept a string and return a new string of text. Input parameters are #OrderId and #OrderText which is a string with dollar sign enclosed variables like so... $Order Name$ sent to $Customer$
The valid variables are in a Variables table (values such as Order Name, Customer, a total of 25 of them which should remain fairly static). Variables can only be used once in the string.
The stored procedure needs to return the string but with the variables replaced with their respective values.
Example1
Input: 123, $Order Name$ sent to $Customer$
Returns: Toolkit sent to StackCustomer Inc.
Example2
Input: 456, $Customer$ requests $delivery method$ for $order type$
Returns: ABC Inc requests fast shipping for pallet orders.
Each of the variables can be retrieved using a function.
DECLARE #OrderId int = 123
DECLARE #InputText VARCHAR(500) = '$Order Name$ sent to $Customer$'
select
#InputText = case when #InputText like '%$order name$%'
then replace(#InputText, '$Order Name$', getOrderName(id) else '' end,
#InputText = case when #InputText like '%$customer$'
then replace(#InputText, '$Customer$', getCustomer(id) else '' end
-- repeat 25 times
Is there a better way? My main concern is maintainability - if a variable is added, renamed, or removed, this stored proc will need to be changed (although I'm told it would only happen a couple times a year, if that). Would dynamic sql be able to help in this case?
Personally, I would create a keywords table to maintain it. something like this
CREATE TABLE [keywords] (
key_value VARCHAR(100) NOT NULL,
function_value VARCHAR(100) NOT NULL
)
INSERT INTO [keywords]
VALUES
('$Customer$','getCustomer(id)'),
('$Order Name$' ,'getOrderName(id)'),
('$order type$','getOrderType(id)')
Then use dynamic sql create REPLACE SQL
DECLARE #OrderId int = 123
DECLARE #InputText VARCHAR(500) = '$Order Name$ sent to $Customer$'
DECLARE #sql VARCHAR(8000) = 'SELECT '
SELECT
#sql = #sql +
' #InputText = replace(#InputText, ''' + key_value + ''', ' + function_value + ')'
+ ' ,'
FROM keywords
WHERE #InputText LIKE '%' + key_value + '%'
SELECT #sql = LEFT(#sql, LEN(#sql) -1)
PRINT #sql
EXEC(#sql)
SQLFiddle
I am not really getting why you need to do tokenization if you are going to use input variables for the procedure.
There are token templates already available for SQL Management Studio in the form of:
<(tokenName),(datatype),(defaultvalue)>
You can get their data filled directly with SQL Managment Studio with CTRL + SHIFT + M, or with ALT > Q > S in 2012, or by text finding their values in an environment.
If you are trying to put in an input to be accessed by an outside developing platform that changes the strings like ADO.NET or the Entity Framework in C#/VB.NET I still am not getting why you would not just make more input variables.
You could do this quite easily:
Declare #OrderPlace varchar(128), #Customer varchar(64), #StringCombine varchar(512);
Select
#OrderPlace = 'Place I am at'
, #Customer = 'Mr Customer';
Select #StringCombine = #Customer + ' order at ' + #OrderPlace
Then if a different language accessed your SQL server procedure it would just have to put in two parameters #Customer and #OrderPlace. You could even set #StringCombine to be an output variable. This is much more preferable then text replacing characters and then running a string. This could be able for SQL injection attacks potentially so santizing your inputs is a big part of returning data, especially if you are altering something for SQL to run before it is ran.
You mentioned maintainability and this is much more robust because if I change the logic to the proc but DO NOT change the variables names, I did not have to change anything else. If I have to change an ADO.NET or other library for references and then SQL code, that is lot more work. Generally when working with tokens I strive for reuse of code where one part of it can go down and only hurt that part of it, not take the whole thing down.
I have created a list of databases I want to drop based on a set of conditions and have titled it delete_table in the master database (this already excludes master, tempdb, model and msdb). The only column in this table is name which contain the exact database names that should be deleted.
How would I go about writing the script that would drop the databases on the server based on this list?
Thanks!
If you want a quick way to script out DROP DATABASE commands:
--set database in single user mode, rolling back active transactions...be careful!
SELECT 'ALTER DATABASE ' + QUOTENAME(name) + ' SET SINGLE_USER WITH ROLLBACK IMMEDIATE;' FROM sys.databases WHERE name IN (...)
UNION
--drop database
SELECT 'DROP DATABASE ' + QUOTENAME(name) + ';' FROM sys.databases WHERE name IN (...);
Replace ... with a comma separated list of database names (e.g N'DatabaseOne', 'DatabaseTwo', etc.). Run the select query, then use results for execution.
I don't understand the need for a user table to solve this. As a side note, you should avoid creating user objects in master database. Granted, I can't think of a reason other than one based on aesthetics, but it just seems wrong.
More on user-created objects in system databases...
MSDN states user objects shouldn't be created in master, but I think the reason provided is pretty weak. A more substantive argument involves lack of control over what happens to system database objects during service pack/version upgrades.
I'm not saying the next service pack upgrade will wipe out your user created objects in master, but who knows. Put your utility and administration-type objects in a user created database so there's no confusion.
quick way
DECLARE #SQL VARCHAR(MAX) = ''
SELECT #SQL = #SQL+ 'DROP DATABASE ' + QUOTENAME(name) + ';'
FROM delete_table
EXECUTE (#SQL)
a bit safer way to make sure database exists and each database get scripted only once incase you have dups
DECLARE #SQL VARCHAR(MAX) = ''
SELECT #SQL = #SQL+ 'DROP DATABASE ' + QUOTENAME(name) + ';'
FROM sys.databases WHERE name IN (SELECT name FROM delete_table);
EXECUTE (#SQL)
Edit to add alter database simply add this line
DECLARE #SQL VARCHAR(MAX) = ''
SELECT #SQL = #SQL+
'ALTER DATABASE '+ QUOTENAME(name) + 'SET SINGLE_USER WITH ROLLBACK IMMEDIATE;' +
'DROP DATABASE ' + QUOTENAME(name) + ';'
FROM sys.databases WHERE name IN (SELECT name FROM delete_table);
EXECUTE (#SQL)
How do I copy just the Scalar Functions from one Database to another? I'm not worried about copying any tables or data. I tried performing an Export Task but that seemed to only let me move tables.
These steps were done on SQL Server 2008 R2 in SSMS.
In short, I used Task -> Generate Scripts... instead of Script Database as -> Create To. The latter only returned a SQL script to create the Database (e.g. Create Database, Alter Database, and Filegroups) without creating any other objects in the Database (e.g. Tables, Views or Functions).
Here are the exact steps with screenshots:
Right click on the database with the functions you want and go to Generate Scripts
Click through the first screen of the Wizard
Choose User-Defined Functions
Finish the wizard.
Also, this answer, while it isn't an exact corollary, prompted me to look for the Generate Scripts option.
-- This program copies (CREATE OR ALTER THE FUNCTION) a single Function from one database to another
-- *** Note that all objects mentioned in the function must exist in the target database ***
declare #SourceDatabase nvarchar(50);
declare #SourceSchemaName nvarchar(50)
declare #TargetDatabase nvarchar(50);
declare #FunctionName nvarchar(50);
set #SourceDatabase = N'Northwind' -- The name of the Source database
set #SourceSchemaName = N'dbo' -- The name of the Function SCHEME
set #FunctionName = N'WriteToTextFile' -- The name of the Function
set #TargetDatabase = N'AdventureWorks' -- The name of the Target database
declare #sql nvarchar(max)
-- If the Function SCHEME does not exist, create it
set #sql = ' use [' +#TargetDatabase +'] ' +
' IF NOT EXISTS (SELECT * FROM sys.schemas WHERE lower(name) = lower(''' + #SourceSchemaName + ''')) '+
' BEGIN ' +
' EXEC('' CREATE SCHEMA '+ #SourceSchemaName +''') ' +
' END'
exec (#sql);
-- CREATE OR ALTER THE FUNCTION
set #sql = ''
set #sql = #sql + ' use [' + #TargetDatabase +'] ;' +
' declare #sql2 nvarchar(max) ;' +
' SELECT #sql2 = coalesce(#sql2,'';'' ) + [ROUTINE_DEFINITION] + '' ; '' ' +
' FROM ['+#sourceDatabase+'].[INFORMATION_SCHEMA].[ROUTINES] ' +
' where ROUTINE_TYPE = ''FUNCTION'' and ROUTINE_SCHEMA = ''' +#SourceSchemaName +''' and lower(ROUTINE_NAME) = lower(N''' + #FunctionName + ''') ; ' +
' set #sql2 = replace(#sql2,''CREATE FUNCTION'',''CREATE OR ALTER FUNCTION'')' +
' exec (#sql2)'
exec (#sql)
I have an application that creates a separate database (SQL Server 2008) for each new customer, during testing we end up with a lot of databases called PREFIX.whatever ...
I would love a script that would look for all databases starting with PREFIX. and drop them so we can start a clean test cycle. Any help greatly appreciated.
SELECT ' DROP DATABASE [' + NAME + ']' FROM sys.sysdatabases where name like 'PREFIX%'
Copy the output and execute this to drop Databases in your criteria. You can also schedule this on a daily basis with a little tweaking.
Update:
We ended up expanding the answer from Baaju so I thought I would share it. We call teh following script from MSBuild and it cleans out all of teh existing DB's created during testing:
use master
DECLARE #Name nvarchar(1000);
DECLARE testdb_cursor CURSOR FOR
SELECT 'ALTER DATABASE' + '[' + NAME + ']' + ' SET SINGLE_USER WITH ROLLBACK IMMEDIATE DROP DATABASE ' + '[' + NAME + ']' FROM sys.sysdatabases where name like 'TCM.%'
OPEN testdb_cursor;
-- Perform the first fetch and store the value in a variable.
FETCH NEXT FROM testdb_cursor
INTO #Name;
-- Check ##FETCH_STATUS to see if there are any more rows to fetch.
WHILE ##FETCH_STATUS = 0
BEGIN
-- Concatenate and display the current values in the variables.
exec sp_executesql #Name;
-- This is executed as long as the previous fetch succeeds.
FETCH NEXT FROM testdb_cursor
INTO #Name;
END
CLOSE testdb_cursor;
DEALLOCATE testdb_cursor;
Just ran into this and come up with a slight variation to allow immediate execution without cursors:
DECLARE #SQL NVARCHAR(MAX) = ''
SELECT #SQL = #SQL
+ 'ALTER DATABASE [' + [name] + '] SET SINGLE_USER WITH ROLLBACK IMMEDIATE; '
+ 'DROP DATABASE [' + [name] + ']; '
FROM sys.databases
WHERE [name] like 'temp_%' AND create_date < DATEADD(day,-7,GETDATE())
-- display statements
SELECT #SQL
-- execute (uncomment)
--EXEC sp_executesql #SQL
The above is deleting any databases starting with "temp_" and older than 7 days, but that can be adapted obviously to any situation.
DANGER: Mess up your query, delete some or all of your databases. I left the EXEC statement commented out just to try to avoid someone doing doing this through copy/paste.