I wanted to know if there is any system variable which is can use to access package execution start and end times. My requirement is that i need to store this in 2 relevant fields in a table in the database.
Why don't you enable logging via the SSIS menu? You could configure SSIS to log events to a SQL Server.
That way, you will not only have the start and end time it will be much easier to locate any errors.
http://msdn.microsoft.com/en-us/library/ms139845.aspx
There's a variable system::Starttime and why don't you just use a getdate() function when you're saving to the DB
Use the SQL Server 2012 SSISDB execution or execution_component_phases tables
Starting with SQL 2012 you can grab this information directly from catalog.execution or catalog.execution_component_phases tables within the SSISDB database using the start_time and end_time columns. The package is identified by the package_name variable. Knowing this can really come in handy. (Kind of surprised nobody at all mentioned this since OP/June 2017).
https://learn.microsoft.com/en-us/sql/integration-services/system-views/catalog-execution-component-phases
Note:
This should save you the step of having to use a system variable, since the data is already in a table.
Related
I am implementing a SSIS package and currently trying to do the following.
Truncate the destination table
Fetch the data by executing the stored procedure and insert it into the destination table.
I have created an Execute SQL task to address step 1 and dataflow with oledb source and oledb destination to address the second point. It been working successfully so far but isn't working for one my stored procedure that uses temp tables.
When I edit the oledb source and click the preview button, I get the error no column returned
I know that SSIS has an issue with generating column while executing stored procedures that depend on temp tables. I have converted the stored proc to use temporary table variables and its now able to return columns in SSIS when I do a preview. The only downside is that the stored procedure is taking longer time to execute. Its taking 1 hour 15 mins as compared to 15 mins while using temp tables.
I did see a suggestion to use SET FMTONLY before executing the stored procedure as an alternate solution to changing to temp table variables but that didn't seem to work as I am getting syntax or permission denied error.
Could somebody tell me a solution to my problem which does not compromise on the performance.
Sounds like you've already read all the approaches to using Temp tables in SSIS, including the IF 1=0... trick? If you haven't seen that one yet, google it.
You say that using Table Variables causes your stored procedure to take about 5 times longer than using Temp Tables. The most likely reason for that is that you are indexing your temp tables but not your table variables. If you didn't know that table variables can be indexed, they can. You might try that.
Finally, a solution that you haven't mentioned is that you can replace your temporary table with a real table that gets truncated when you're done using it.
Short comment:
Try EXEC WITH RESULT SETS and specify the metadata yourself for a proc with temp tables; or use the Script Component as a source and specify the Output columns yourself.
Long comment:
Technically speaking, it is the driver/database you are using in SSIS that would decide the behavior when working with temp tables.
Metadata is an important factor when using SSIS's pipeline components. By metadata, I mean the names of the columns, their data types etc that a pipeline component uses. When designing a data flow, someone/something should provide this metadata to the components that require it.
In most cases, SSIS automatically retreives the metadata. Components that do not connect to a external data source, like Conditional Split etc, get their metadata from the other components they are connected to. For the pipeline components that connect to a external data source (like Oledb source, oledb destination, Lookup etc.), SSIS provides a mechanism to get this metadata without human involvement. This mechanism involves the driver connecting to the database and retrieving the metadata of the output. If the driver/database is capable of returning the metadata, then that metadata is used. If the driver/database is incapable, then you get the errors you are seeing. The rest of my comments are based on the assumption that you are using a SQL Server database in your question.
When working with a SQL Server database in SSIS, typically, we use the native client drivers provided by Microsoft. When trying to get the metadata, these drivers try to get the metadata without actually executing the SQL Statement (actual execution can have side effects; and also, might take more than a few seconds/minutes/hours; and you dont want side effects and long wait times during package design time.) So to get the metadata, the driver relies on the metadata of the actual objects used in the sql command. If the command uses a physical table or view, SQL Server already has the metadata available and can supply it to the driver. If it is a temp table, SQL Server does not have the metadata until it can create the temp table. If using FMT ONLY option, you can use it in such a way to create the temp tables, but avoid any heavy processing/side affects and thus be able to retrieve metadata without penalties. Post 2012, these native client drivers rely on some newer functionality to retrieve metadata than the drivers before 2012. In 2012 and after, the driver uses the sp_describe_first_result_set proc to retrieve metadata. So, whether you can get metadata or not is determined by the ability of the sp_describe_first_result_set proc.
So while SSIS can automatically get the metadata (because of the driver/database), it does not automatically get the metadata in some cases (again because of the driver/database). In cases involving the second scenario, some other process (typically a human) can help the driver infer metadata or provide the metadata to the component directly.
To help the driver, in case of SQL Server 2012 and after, you can use the WITH RESULTSETS clause to specify the output metadata. When this clause is present, the driver will use it and doesnt try to query the metadata from system objects; and thus avoid the error which you would otherwise get. If you are using the drivers that came with SQL Server 2008, you can use FMT ONLY. This option is at the driver/database level.
Another option could be to use a Script Component as the Source and in the Output columns, you can specify the columns/metadata. SSIS would not try to retrieve metadata from the datasource in this case, but would rely on the definitions you provided in the Output section of the Script Component.
As you can see, both options involve a human (or some other process) specifying the metadata instead of SSIS trying to retrieve the metadata in an automated fashion. I would prefer the first option if working with SQL Server and the second option if working with databases like MySql.
I would like to ask a question for connoisseurs of SQL (MySQL, to be specific).
I have a table reservation schedules. And when a customer makes a reservation there is a time to let the client to use my service. Therefore, the reservation that he did have to leave the table reservations.
Once the time limit of use is reached, there is some method (trigger,
I believe), which automatically erase the record of this book on the
table?
If so, can someone give me some idea of how to start my search for it, or it is also totally welcome some help as some more advanced lines of code.
There is also the possibility that this only be possible to be implemented via Server-Side (PHP, ASP ...), which does not believe is so true because SQL is a language very complete (to my knowledge).
Edit1: The problem is that I believe this is a task of the DBMS, so I wanted to leave this responsibility to the MySQL The problem is: how?
A trigger is triggered by either before or after an insert , update or delete event (at least in MySQL according to the docs)
What you want is some sort of scheduled job either through your application be it php, asp.net, etc.. or cron job that calls some sort of SQL script.
So to answer, it can't be done purely with triggers.
You can use SQL jobs, but if the removal logic is to complex to manage it with queries I suggest you to use a PHP script that does all that work for you.
Just write down the data check/remove logic in PHP and set up a simple cron operation for it.
The advantage of this solution is that you can access to your scripts/classes/db providers and save your time and your can log all the operations separately (instead of logging to MySQL logs, no matter what script language you are relying on).
If you have a full control of your server the scheduled operation will look like this (if you want to check your DB entries every day at 00:01):
cat /etc/cron.d/php5
0 1 * * * php /path/to/your/script.php >> /path/to/your_script.log
..otherwise you will have to check the control panel of your hosting account and figure out how to manage
You can create one more column in your table where you will create the expiration date. Then you can on your sql server create the job that will erase all records that have expiration date less than curent date.
CREATE EVENT db_name
ON SCHEDULE EVERY 10 SECOND
DO
DELETE FROM myschema.mytable WHERE expiration_date < NOW()
I hope that will help.
I have a reporting database and have to transfer data from that to another server where we run some other reports or functions on Data. What is the best way to transfer data periodically like months or by-weekly. I can use SSIS but is there anyway I can put some where clause on what rows should be extracted from the source database? like i only want to extract data for a current month. Please do let me know.
Thanks,
Vivek
For scheduling periodic extractions, I'd leave to that SQL Agent.
As for restricting the results by some condition, that's an easy thing. Instead of this (and you should always use SQL Command or SQL Command From Variable over Table Name/Table Name From Variable as they are faster)
Add a parameter. If you're use OLE DB connection manager, your indicator for a variable is ?. ADO.NET will be #parameterName
Now, wire the filter up by clicking the Parameters... button. With OLE DB, it's ordinal position starting at 0. If you wanted to use the same parameter twice, you will have to list it each time or use the ADO.NET connection manager.
The biggest question you will have to answer is how do I identify what row(s) need to go. Possibilities are endless: query into the target database and find most recent modified date for a table or highest key value. You could create a local table that tracks what's been sent and query that. You could perform an incremental load / ETL Instrumentation to identify new/updated/unchanged rows, etc.
I am writing an SSIS package that has a conditional split from a SQL Server source that splits records to either be updated or inserted into a MYSQL database.
The SQL Server connection has provider .NET Provider for OldDB\SQL Server Native Client 10.0.
The MYSQL connection is a MYSQL ODBC 5.1 ADO.NET connection.
I was thinking about using the OLE DB Command branching off of the conditional split to update records but I connect use this and connect to the MYSQL database.
Does anyone know how to accomplish this task?
I would write to a staging table for updates including the PK and columns to be updated and then execute an UPDATE SQL statement using that table and the table to be updated. The alternative is to use the command for every row and that just doesn't seem to perform that well in my experience - at least compared to a nice fat batch insert and a single update command.
For that matter, I guess you could do without the conditional split altogether, write everything to a staging table and then use an UPDATE and INSERT in SQL back to back.
Probably, the following MSDN blog link might help you. I haven't tried this.
How do I UPDATE and DELETE if I don’t have an OLEDB provider?
The post suggests the following three options.
Script Component
Store the data in a Recordset
Use a custom component (like Merge destination component)
The author also had posted two other articles about MySQL prior to posting the above article.
Connecting to MySQL from SSIS
Writing to a MySQL database from SSIS
Hope that points you in the right direction.
I have 2 Database in my VB.net application. I am using 1st database for daily operations. I would like to send one of the table records to online database. How Can I do that? First database is MSSQL Online database is MYSQL. I have created connections already using MYSQL .net connector.
Any Help will be appreciated.
Regards
Have a look at using a Linked Server instance on SQL Server to write the data to MySQL using the four name notation.
SQL SERVER – Explanation and Example Four Part Name
SQL Server Four-part naming
Ok here is a rough set of steps you need to follow
Query the MSSQL database and retrieve the data you want. Storing it in a DataTable may be the best option starting off.
Loop through the DataTable rows and build an INSERT statement that will be run against the MYSQL database.
Execute the command against the MYSQL db.
This is the basics of what you need to do to get a simple working system. Also take a look at Transactions as a way to manage the rollback of data when something goes wrong.
I'm assuming this is a research project If you are planning on using this code in a production system then i would look into a different alternative such as uploading data files to a service attached to the MYSQL database. This would allow you to batch and retry an import when something goes wrong.