Parse MS Excel files dynamically with SSIS - ssis

Question

I have a business requirement that is looking for the ability to have Excel files placed in a network location and the data from these files uploaded to a SQL Server database. The files will contain 1 worksheet of data. The files correspond to a table found within a known database. The files can and will correlate to multiple tables and will be known only when opening up the file, i.e., the name of the worksheet. I'm currently creating multiple SSIS packages for each of these files as they are uploaded to the shared drive but sometimes, I'm not creating the package fast enough.
I guess my question is, is this type of dynamic parsing something that SSIS can accomplish from a Script Task within a Foreach container? or should I look into another option?
So far, I have the following...but as I've researched, I've come across post similar to this: Extracting excel files with SSIS and that is making me slightly concerned regarding the feasiability...
public void Main()
{
// TODO: Add your code here
Dts.TaskResult = (int)ScriptResults.Success;
string NetworkLocation;
//Create database connection
SqlConnection myADONETConnection = new SqlConnection();
myADONETConnection = (SqlConnection)(Dts.Connections["db"].AcquireConnection(Dts.Transaction) as SqlConnection);
//Obtain the location of the file(s)
NetworkLocation = (string)Dts.Variables["User::NetworkLocation"].Value;
string[] dirs = Directory.GetFiles(NetworkLocation, "*.csv");
}
So, any thoughts or ideas or what direction I should look into?

Answer 1

I wrote a SSIS package a few months ago that does exactly what you seek, plus a little more. In my case, several hundred Excel files containing one or more worksheets of differing names needed to be imported into a database as unique staging tables. Also, the column names and number of columns in each worksheet were unknown. Each worksheet became its own table and the table name was a combination of the original Excel file name and the worksheet name (FileName__WorksheetName). I applied two underscores between the file name and worksheet name in case the file and worksheet names contained underscores. There are a few caveats to this process: 1) All of the Excel files must be located in the same folder; 2) The column headers in each worksheet must appear in the first row; and 3) the worksheet names must not contain any special characters (spaces are automatically replaced with an underscore).
Steps:
1) Create a For Each Loop Container. Under Collection, apply a "Foreach File Enumerator" where under Enumerator configuration, list the folder location and the Files. For files you can list . or even *.xlsx or *.xls to filter to specific files. Apply Fully Qualified. For Variable Mappings, apply a string user variable like "ExcelFile" with an index of 0.
2) Add a Script task in the For Each Loop Container. You will send it the ReadOnlyVariable "ExcelFile" and it will write to two new string variables "TableName" and "Worksheets" under ReadWriteVariables. Apply the following C# script. Note, scince the following script will update your Excel files, you should be applying copies of your originals.
#region Namespaces
using System;
using System.Data;
using Microsoft.SqlServer.Dts.Runtime;
using System.Windows.Forms;
using System.IO;
using Excel = Microsoft.Office.Interop.Excel;
using System.Text;
using System.Linq;
using System.Threading.Tasks;
using System.Data.OleDb;
using System.Xml.Serialization;
#endregion
namespace xxxxxxxxx
{
[Microsoft.SqlServer.Dts.Tasks.ScriptTask.SSISScriptTaskEntryPointAttribute]
public partial class ScriptMain : Microsoft.SqlServer.Dts.Tasks.ScriptTask.VSTARTScriptObjectModelBase
{
public void Main()
{
// Includes full path, filename and extension... C:\Documents\ThisExcel.xlsx
string xlFile = Dts.Variables["User::ExcelFile"].Value.ToString();
// Remove path changing value to "ThisExcel.xlsx"
string NoPath = Path.GetFileName(xlFile);
// Remove extension changing value to "ThisExcel".
// This is applied because filename will become part of the name for new database tables
string tableName = Path.GetFileNameWithoutExtension(NoPath);
// Replace any spaces with underscores in tableName (FileName without path and extension)
tableName = tableName.Replace(" ", "_");
Dts.Variables["User::TableName"].Value = tableName;
Excel.Application app = new Excel.Application();
Excel.Workbook excelWorkbook;
try
{
excelWorkbook = app.Workbooks.Open(xlFile);
string tempsheet = " ";
int CountWorksheets = excelWorkbook.Sheets.Count;
//Dts.Variables["User::WorksheetCount"].Value = CountWorksheets;
string[] Excelworksheets;
Excelworksheets = new string[CountWorksheets];
int x = 0;
// Rename worksheets replace empty space with an underscore needed for an SSIS import and
// to avoid empty spaces in final table names.
foreach (Excel.Worksheet sheet in excelWorkbook.Worksheets)
{
tempsheet = sheet.Name;
tempsheet = tempsheet.Replace(" ", "_");
Excelworksheets[x++] = tempsheet.ToString();
sheet.Name = tempsheet;
}
Dts.Variables["User::Worksheets"].Value = Excelworksheets;
excelWorkbook.Save();
excelWorkbook.Close();
}
catch (Exception ex)
{
MessageBox.Show("Excel sheet rename failed for file " + xlFile + " based on " + ex.Message);
}
finally
{
app.Quit();
app = null;
GC.Collect();
GC.WaitForPendingFinalizers();
}
Dts.TaskResult = (int)ScriptResults.Success;
}
#region ScriptResults declaration
enum ScriptResults
{
Success = Microsoft.SqlServer.Dts.Runtime.DTSExecResult.Success,
Failure = Microsoft.SqlServer.Dts.Runtime.DTSExecResult.Failure
};
#endregion
}
}
3) After saving and building the above C# script task, add a For Each Loop Container in the earlier For Each Loop container below the script task just created. This will loop through each worksheet in each Excel file. If you only have one worksheet, that is fine. It will apply an Enumerator of "Foreach From Variable Enumerator", which will be the "Worksheets" string variable created that is populated in the before mentioned script task. It will write to a new user string variable called "Worksheet" with an Index of 0.
4) Within this new nested For Each Loop Container, add script task that will create the database table for each worksheet. The tricky part I had to deal with here was defining the field types, this is not retained from the Excel worksheets or text CSV files. So I made them all nvarchar(255) or, if column headers were something like Remark, Description or something else, I made it nvarchar(max), which is good to 4000 or 4262 characters (I do not recall for certain). Here is the dynamic code I applied stemming from what you began.
#region Namespaces
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Threading.Tasks;
using System.Data;
using System.Data.OleDb;
using System.Xml.Serialization;
using System.IO;
using Microsoft.SqlServer.Dts.Runtime;
using System.Windows.Forms;
#endregion
namespace yyyyyyyyyy
{
[Microsoft.SqlServer.Dts.Tasks.ScriptTask.SSISScriptTaskEntryPointAttribute]
public partial class ScriptMain : Microsoft.SqlServer.Dts.Tasks.ScriptTask.VSTARTScriptObjectModelBase
public void Main()
{
// TODO: Add your code here
string xlFile = Dts.Variables["User::ExcelFile"].Value.ToString(); //Includes full path and filename with extension
//xlFilex = xlFilex.Replace(#"\", #"\\");
string worksheet = Dts.Variables["User::Worksheet"].Value.ToString(); //Worksheet name from Excel file.
string Tablename = Dts.Variables["User::TableName"].Value.ToString(); //Currently file name without path and extension. Spaces replaced by underscores.
string ExcelExtension = Path.GetExtension(xlFile);
string columnName = "";
string columnType = "";
int i = 0;
string worksheet2 = worksheet + "$";
OleDbConnection xl = new OleDbConnection("Provider=Microsoft.ACE.OLEDB.12.0;Data Source=" + xlFile + ";Extended Properties=\"Excel 12.0 Xml;HDR=YES;IMEX=1\"");
xl.Open();
System.Data.DataTable dt = xl.GetOleDbSchemaTable(OleDbSchemaGuid.Columns, new object[] { null, null, worksheet2, null });
List<string> listColumn = new List<string>();
// Create the name of the table that will be created in the SQL Server database, which is
// a concatentation of the root file name and worksheet name separated by two undescores.
Tablename = Tablename + "__" + worksheet;
string CreateTable = "CREATE TABLE " + Tablename + " (";
string InsertTable = "INSERT INTO " + Tablename + " (";
string SelectColumns = "";
// Create the string that will be applied to create the table defining the field types based on the names
foreach (DataRow row in dt.Rows)
{
listColumn.Add(row["Column_name"].ToString());
columnName = listColumn[i].ToString();
if ((columnName == "Remark") || (columnName == "remark") || (columnName == "REMARK") ||
(columnName == "Remarks") || (columnName == "remarks") || (columnName == "REMARKS") ||
(columnName.Contains("Remarks")) || (columnName.Contains("remarks")) || (columnName.Contains("REMARKS")) ||
(columnName.Contains("Remark")) || (columnName.Contains("remark")) || (columnName.Contains("REMARK")) ||
(columnName == "Comment") || (columnName == "comment") || (columnName == "COMMENT") ||
(columnName == "Comments") || (columnName == "comments") || (columnName == "COMMENTS") ||
(columnName == "Description") || (columnName == "description") || (columnName == "DESCRIPTION") ||
(columnName.Contains("Description")) || (columnName.Contains("description")) || (columnName.Contains("DESCRIPTION")) ||
(columnName == "Legal") || (columnName == "legal") || (columnName == "LEGAL") ||
(columnName == "Note") || (columnName == "note") || (columnName == "NOTE") ||
(columnName.Contains("Format")) || (columnName.Contains("format")) || (columnName.Contains("FORMAT")) ||
(columnName == "Notes") || (columnName == "notes") || (columnName == "NOTES")
)
{
columnType = "nvarchar(max),";
}
else
{
columnType = "nvarchar(255),";
}
CreateTable = CreateTable + "[" + columnName + "] " + columnType;
InsertTable = InsertTable + "[" + columnName + "],";
SelectColumns = SelectColumns + "[" + columnName + "],";
//MessageBox.Show(columnName + " " + columnType);
i++;
}
// Remove last comma from CreateTable and add closing
CreateTable = CreateTable.Remove(CreateTable.Length - 1);
CreateTable = CreateTable + ")";
// Removoe last comman from InsertTable and add closing
InsertTable = InsertTable.Remove(InsertTable.Length - 1);
InsertTable = InsertTable + ")";
// Removoe last comman from SelectColumns
SelectColumns = SelectColumns.Remove(SelectColumns.Length - 1);
xl.Close();
string SQL = "";
// Assemble the dynamic SQL that will be applied in the SQL task next to generate and populate a new database table
if (ExcelExtension == ".xlsx")
{
SQL = "IF OBJECT_ID ('dbo." + Tablename + "') IS NOT NULL DROP TABLE dbo." + Tablename +
" " + CreateTable + " " +
InsertTable + " " + "SELECT " + SelectColumns + " FROM OPENROWSET('Microsoft.ACE.OLEDB.12.0', " +
//" INSERT INTO [dbo].[" + Tablename + "] SELECT * FROM OPENROWSET('Microsoft.ACE.OLEDB.12.0', " +
"'Excel 12.0 Xml;HDR=YES;Database=" + xlFile + "', 'SELECT * FROM [" + worksheet + "$]');";
}
else if (ExcelExtension == ".xls")
{
SQL = "IF OBJECT_ID ('dbo." + Tablename + "') IS NOT NULL DROP TABLE dbo." + Tablename +
" " + CreateTable + " " +
" INSERT INTO [dbo].[" + Tablename + "] SELET * FROM OPENROWSET('Microsoft.Jet.OLEDB.4.0', " +
"'Excel 8.0 Xml;HDR=YES;Database=" + xlFile + "', 'SELECT * FROM [" + worksheet + "$]');";
}
//MessageBox.Show(SQL);
Dts.Variables["User::CreateTableSQL"].Value = SQL;
Dts.TaskResult = (int)ScriptResults.Success;
}
#region ScriptResults declaration
enum ScriptResults
{
Success = Microsoft.SqlServer.Dts.Runtime.DTSExecResult.Success,
Failure = Microsoft.SqlServer.Dts.Runtime.DTSExecResult.Failure
};
#endregion
}
}
Looking at the above script you'll notice that the following ReadOnlyVariables will need to be declared: ExelFile, SourceFolder, TableName, tempFileName, and Worksheet. Then the following ReadWriteVariables will need to be declared: ColumnCount, CreateTable, and InsertTableName.
5) Within the nested ForEach Loop Container and just below the above Task script, add an Execute SQL Task that will run the sql contained in the CreateTableSQL variable. Be sure to set the SQLSourceType to "Variable". This will create and populate the table and even overwrite it if it already exists.
When done, you should have something that looks like the following flow:
Hope this helps and let me know if you have any questions. I did not have time to remove the extraneous stuff, but this should put you on the right path. This loop container is for Excel files, but you can add on other loop containers with code modified to handle CSV or other file types. All of this can be enclosed in a single SSIS package.
The final SQL task just runs the following TSQL that looks for field names in your database containing a space between words and replaces that space with an underscore. It is not necessary, but avoids having to apply SQL with columns wrapped with brackets [].
DECLARE My_Cursor Cursor
FOR
SELECT 'sp_rename '''+table_name+'.['+column_name+']'','''+replace(column_name,' ','_')+''',''COLUMN'''
FROM information_schema.columns
WHERE column_name like '% %'
OPEN My_Cursor
DECLARE #SQL NVARCHAR(1000)
FETCH NEXT FROM My_Cursor INTO #SQL
WHILE ##FETCH_STATUS <> -1
BEGIN
EXECUTE sp_executesql #SQL
FETCH NEXT FROM My_Cursor INTO #SQL
END
CLOSE My_Cursor
DEALLOCATE My_Cursor