I'm probably going to word this poorly, but here goes.
I've created an SSIS Package with an email proceedure. It is supposed to send three emails, each one based on a SQL query.
The email tasks run sequentially, as part of the reporting requirement (An automated run is required daily, but sometimes one of the three emails needs to be sent manually. In those cases the other two Data Flow Tasks are disabled)
Heres where things get fishy. I can run the task from the editor, and no issues arise. Results: 3 emails, limited latency. These are relatively small queries (~50k records). When the task is run from Windows Task Scheduler, I get two of the three emails (notably the first two in the sequence), and quite a bit of latency (~10 minutes total execution, ~ 3 minutes between emails.) Latency isn't concerning me, but the missing email is.
The task is set to expire if it runs longer than 12 hours, so timeout is unlikely the cause. I'm tearing my hair out trying to figure this out!
Note: to make things more interesting, I recompiled the script executing all three email (script) tasks in one data flow task. Same behaviour there, with a very interesting twist. Every time I complied the bianries with three email tasks I got a two emails
Example:
Compile 1 -> Load into Windows Task Scheduler
Result -> Lab & IT email
Compile 2-> Load into Windows Task Scheduler
Result -> Base & IT email
The heck?
Related
In the investment firm I work for, we have several daily data integration tasks that every night/early morning.
Most if not all are done using SSIS, and they are all scheduled to start at certain times. They work (as in the SSIS packages do their job).
However, we have serious problems when it comes to tackling with dependent processes. For example, if SSIS package is scheduled to run at 3AM does not get the FTP file from a vendor, then dependent processes that are set to run at 3:30AM and 4AM all fail.
Currently we have about 8 different inhouse applications/endpoints, with data coming from around 6 external vendors and the vendor list is beginning to grow and we expect it would around 20 or so in a year.
We don't want to go BizTalk route for financial/complexity/implementation difficulties.
I wanted to create a event-driven approach to this, where the SSIS/nightly processes are oblivious to each other, as they are, yet need subscribe to dependent parent processes to start/stop.
At a glance I feel NServiceBus would give us that flexibility. It lets us keep what we already have and work, but yet give an event driven mechanism.
I need some input here.
I have a weird issue with one of the data driven subscriptions on SSRS.
The subscription is a timed subscription that generates invoices (pdf/excel) and gets triggered by a stored procedure.
The issue we are facing is that the first run always takes 30-60 minutes regardless of how many invoices are being generated. Once the first run has completed the subsequent runs are completed under a minute throughout the day.
There is a second version of the same report that is run manually and it runs fine(ruling out any delays with the data extraction bit).
I have looked at some other questions here but that didnt help identify the problem:
SQL Reporting services: First call is very slow
SSRS report subscription not working sometime
Without knowing more about the query, data, database setup, other process, etc.; it will be quite difficult to say for sure. But if I had to guess, based on your description, it sounds like the query plan cache is lost and is rebuilt on the first run of the day. Without the plan the query can be less efficient. Each subsequent run will use the plan created on the first run, and will therefor run more quickly. There are a number of reasons that could cause the query plan to be wiped from cache. A recompile, other queries using too much memory, not enough system memory to begin with etc.
Hope that helps!
I'm developing a report in ssrs for display in a public location within the company. The report shows up-to-the-minute data on department activity, so is set to refresh every 30 seconds or so.
The problem I'm running into is that once every few hours, the report throws an error (rsErrorReadingNextDataRow) & (rsProcessingAborted). All I have to do is hit refresh in the browser (rendered in a an internet browser) and it reruns and charts along happily for a while.
After looking at the reportserver execution log, it appears that these errors coincide with a phenominal spike in processing time. On average, each time the report refreshes, processing times is 400ms~800ms, but when it spikes it goes to ~90000ms (yes, five zeroes) and then throws this error.
Being new to SSRS I am not sure where to begin looking for the root cause.
Can anyone give me pointers as to where I can start to find out what is causing the processing time to rocket like that? Data retrieval is stable at ~5000ms, and Time rendering is stable at around 200ms. It's only the processing that goes haywire.
Some background on the report and data:
Data is pretty straight forward. It's based on a view that pulls transactions from last 7 weeks. Number or records, therefore ebbs and flows each week ~ 8000 records. When select * from View is run in SSMS, query takes about 5 seconds to run. I'll work on speeding that up after I resolve this processing issue.
No parameters are used, though the view does use getdate() to figure out which records to show from base tables.
No stored procedures are used.
Report itself is comprised of 6 panes within a single tablix, none of which have to draw more than a few dozen marks ('cept one is a map of US states).
The report does have one feature that may be related, though I am not sure how. Report definition filters the dataset based on a mod 3 of the built in execution time variable, rotating the report to show "This Month", "This Week", or "Today" activity. There is an additional mod 3 on execution time which rotates visibility for the panes. The result is that each 30 seconds, a report with different combinations of charts shows up on a rotating time-frame.
Don't know if this could be a cause, but it's the only element of the report that makes it remotely fancy. Everything else about the report is actually rather straightforward/plain.
While I would love to identify and eliminate the spikes, even a mechanism to automatically refresh on error, or refresh on timeout (or something to that effect) would do the trick. I need to be able to launch the report in the morning and have it run unmanned all day without any human interaction, refreshing every 30 seconds for the duration of business hours.
Just a recap of the comments:
it might be possible to push the web page into a frame, then add javascript to the main page to periodically refresh the frame. It wouldn't fix the underlying problem, but it might help.
it might be worth looking at the performance monitoring tools in Sql Server - it might be a database issue (temp table filling up, etc)
You noted that adding a with (nolock) seemed to help locate the problem.
Our shop relies heavily on SSIS to run our back end processes and database tasks. Overall, we have hundreds of jobs, and for the most part, everything runs efficiently and smoothly.
Most of the time, we have a job failure due to an external dependency failing (data not available, files not delivered, etc). Right now, our process is set up to email us every time a job fails. SSIS will generate an email sending us the name of the job and the step it failed on.
I'm looking at creating a dashboard of sorts to monitor this process more efficiently. I know that the same information available in the Job History window from SSIS is also available by querying the msdb database. I want to set up a central location to report failures (probably using SQL Reporting Services), and also a more intelligent email alert system.
Has anyone else dealt with this issue? If so, what kind of processes/reporting did you create around the SSIS procedures to streamline notification of job failures or alerts?
We have a similar setup at our company. We primarily rely on letting the jobs notify us when there is a problem and we have employees who check job statuses at specific times to ensure that everything is working properly and nothing was overlooked.
My team receives a SQL Server Agent Job Activity Report HTML email every morning at 6am and 4pm that lists all failed jobs at the top, running jobs below that, and all other jobs below that grouped into daily, weekly, monthly, quarterly, and other categories. We essentially monitor SQL Server Agent jobs, not the SSIS packages themselves. We rely on job categories and job schedule naming conventions to automate grouping in the report.
We have a similar setup for monitoring our SSRS subscriptions. However, we only monitor this one once a day since most of our subscriptions are triggered around 3am-4am in the morning. The SSRS Subscription Activity Report goes one step further than the SQL Server Agent Job Activity Report in that it has links to the subscription screen for the report and has more exception handling built into it.
Aside from relying on reports, we also have a few jobs that are set to notify the operator via email upon job completion instead of upon job failure. This makes it easy to quickly check if all the major ETL processes have run successfully. It's sort of an early indicator of the health of the system. If we haven't received this email by the time the first team member gets into the office, then we know something is wrong. We also have a series of jobs that will fail with a job error if certain data sources haven't been loaded by a specific time. Before I had someone working an early shift, I use to check my iPhone for the email anytime I woke up in the middle of the night (which happened a lot back then since I had a newborn baby). On the rare occasion that I didn't receive an email indicating everything completed or I received an error regarding a job step, then I logged onto my machine via remote desktop to check the status of the jobs.
I considered having our data center guys check on the status of the servers by running a report each morning around 4am, but in the end I determined this wouldn't be necessary since we have a person who starts work at 6am. The main concern I had about implementing this process is that our ETL changes over time and it would have been necessary for me to maintain documentation on how to check the jobs properly and how to escalate the notifications to my team when a problem was detected. I would be willing to do this if the processes had to run in the middle of the night. However, our ETL runs every hour of the day, so if we had to kick off all our major ETL processes in the early morning, we would still complete loading our data warehouse and publishing reports before anyone made it into the office. Also, our office starts REALLY late for some reason, so people don't normally run our reports interactively until 9am onward.
If you're not looking to do a total custom build out, you can use https://cronitor.io to monitor etl jobs.
Current SSRS Job monitoring Process:
There are no SSRS job monitoring process. If any SSRS job is failed, user creates the incident, then TOPS Reporting and SSRS developer team are started to work on basis of incident. As a result, this process has taken huge turnaround time to resolve this issue.
Proposed SSRS Job monitoring Process:
SSRS subscriptions monitoring job will help to TOPS Reporting and SSRS developer for proactively monitoring the SSRS job. This job will create the report to display the failed list of report along with generic execution log status and subscription errors log status. Initially, developer can understand report failure reason from this report and then developer can start working proactively to resolve the issue.
I use Hudson to automate the testing of a very large important product. I want to have my testing-hosts able to run as many concurrent builds as they will theoretically support with the exception of excel-tests which must only run one per machine at any time. Any number of non-excel tests can run concurrently, however at most one excel test at a time must run per machine.
Background:
Most of my tests are normal unit-tests - the sort of thing that I can easily run in parallel. Unfortunately a substantial and time consuming part of my unit-testing plan consists of tests which have been implemented in Excel.
You might think it crazy to implement a test in Excel - actually there's an important reason: Most of our users access our system via a Excel. Excel has it's own quirky ways of handling data so the only way to guarantee that our stuff works for Excel users is to literally implement our reg-test our application Excel.
I've written a test-runner tool which allows me to easily fire off a group of excel tests: Each test is a single .xls file. Each group is a folder full of excel files. I've got about 30 groups which need to be run for an end-to-end test. My tool converts the result of each of the tests into JUnit style XML which Hudson is able to understand. The tests use the pywin32com library to automate excel. When run on their own they are reliable.
I've got a group of computers which are dedicated to running tests. Each machine is quad-core and can theoretically run quite a lot of stuff at once. Unfortunately I've found that COM cannot be used to safely control more than 1 excel per machine at a time.
That is to say if a 2nd build stars which tries to talk to Excel via COM it might interfere with the one which is already running and cause both tests to fail.
I can run as many other non-excel processes as the machine will allow but I need to find a way so that Hudson does not attempt to launch any more than 1 process which requires excel on any one machine concurrently.
Sounds like the Locks and Latches plugin might help you.
http://hudson.gotdns.com/wiki/display/HUDSON/Locks+and+Latches+plugin
Isn't hudson java?
Since you've tagged this post python, I'll point out that buildbot, has slave locks to limit individual steps on individual slaves (or use them as more coarse locks if you'd like).