I am currently working on BI/ETL project in which I have to change the extraction of a dimension table in a very huge SSIS package with lot of dataflow tasks and the data flow tasks names are not in accordance with table name. Is there a way to search for the table in that SSIS package to find its particular data flow task?
Related
We have many SSIS packages that move, import, export around large amount of data. What is the best way to get alerts or notifications if expected amount of data is not processed? or How to get daily report on how different SSIS packages are functioning. Is there a way to write/use a custom component and simply plug it in to SSIS packages instead of writing custom component for each package?
For your first question, we use user variables in SSIS to log the
number of rows processed by each step along with the package name
and execution id. You can then run reports on the history table,
and if any of the executions have a large variance in the rowcounts
processed, you can trigger an event.
Yes. See here, or in the alternative, google "custom ssis
component tutorial".
I'm new to SSIS. I'm running BIDS under SQL Server 2008 R2. I have several text files that I need to import into separate SQL Server destination tables. The tables already exist in the DB. Each file will map to only 1 table. (For example, file_A maps to table_A, and file_B maps to table_B.) My general data flow is as follows:
Flat File Source
Data Conversion (to handle the issue of unicode vs non-unicode strings)
OLE DB Destination (to handle the issue of local server to remote server)
Do I need to create a separate data flow task for each of my text files? If so, my package may be very large.
You can create a single data flow task that has more than one "flow" in it. Hard to describe with words, but you can put a source1 that flows to DataConversion1 that flows to Destination1 and then alongside it (no connections) Source2 flows to DataConversion2 flows to Destination2, and so on.
However, I do agree with #billinkc that using a separate dataflow for each is the better way to go. It will make debugging easier, in addition to the other benefits he mentioned.
This question already has answers here:
ssis best practice to load N tables from Source to Target Server
(3 answers)
Closed 8 years ago.
I am trying to design an SSIS package which copy about 50+ tables from an ODBC DataSource (QuickBooks DB) to an SQL DB.
Should I create 50 Data Flow Task to do this ?
What is the best way to do this ?
Putting DFT inside a Loop, and reading the tables ? Or 50+ Data Flow Tasks ???
You can create 50 Data Flow Tasks, but you don't have to.
It is possible to have multiple independent sources-destinations in the same DFT.
This will be not as flexible, because you can run single DFT separately from the package (while debugging), but you cannot run a piece of DFT without modifying it (as far as I know).
Depending on which option you choose, I see a couple of ways to save yourself from mundane work with 50+ tables:
a) Let SQL Server Import and Export Wizard do the boring work for you.
The best about this tool is that it can create a .dtsx package.
So, with the wizard, you can:
select for importing all 50+ tables from ODBC DataSource
instead of running the wizard till the end, save the result as a .dtsx package.
open the package in Visual Studio with SQL Server Data Tools
modify the package up to your needs (for example logically regrouping the tables in different DFTs, adding any additional transformations).
b) Manually edit the package code (some BIML knowledge might be needed):
In Visual Studio with SQL Server Data Tools, create 1 DFT which will be your sample.
In Solution Exporer, right-click on your package, select View Code.
Either copy/paste the DFT 50+ times, changing the table names, or maybe you will even manage to automate your BIML somehow to avoid copy/paste
I need to read a column from a database table depending upon some parameter. If the database table has two columns, status and ID, then I have to read the ID if the status is true. Then I have to pass this ID to a C# method.
How can I achieve this in SSIS? So basically my database package will read the data from SQL Server and pass it to a C# method.
SSIS is an ETL Tool for moving and transforming large quantities of data. If you need to do a lot of C# work and you only have one record, or a few records, SSIS may not be the right tool for this purpose. You might do better writing an ASP.NET web application or a Windows application. These applications can also use SQL to get data for processing in C#.
If you are determined to do this in SSIS and C#, here are two possible approaches:
You could use an Execute SQL Task to perform your query and save the rowset into a variable. Then you would use a C# Script Task to do something with the contents of the variable.
You could create a Data Flow Task. The dataflow should have the structure Source -> Transformation -> Destination, and can include several transformation components.
You would use, for example, an OLE DB Source Component to perform your query. Then you would use a C# Script Component to transform each record that is returned by the query. Finally, you would use a OLE DB Destination Component to do something with the output for each record.
I am new to SSIS and am not sure if this is possible to upload Excel data into SQL Server without creating table schema, where the Job automatically creates the table schema according to the source file? I used to do this using SAP Data Integrator with Template table component and I am not sure if there is a similar functionality in SSIS
There is no capability in SSIS to interpret the structure from the source file and create the table schema. You can create tables as part of the SSIS control flow, but the data flows that load the data are very fussy about knowing the source and target structure at design time. In other words, it doesn't handle dynamic structures very well.