Get a warning if an expected schedule report email hasnt arrived [closed] - reporting-services

Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
We don’t allow questions seeking recommendations for books, tools, software libraries, and more. You can edit the question so it can be answered with facts and citations.
Closed 7 years ago.
Improve this question
I (like most tech admins I guess) have quite a lot of status infos from scheduled services in my inbox. However when one service email fails there's obviously no email sent. So I simply want a service looking at my inbox saying "Hey this service did not send an email report yesterday - somethings wrong!".
This one should be solved somewhere I guess. Perhaps Gmail (or some other email provider) has a service of this kind, that would be great.

Wouldn't it be a better option to have a centralized monitoring solution like Nagios that you configure in such way that it only send out notifications when a service misses its heartbeat, reaches highwatermarks, run out of fuel? And then off course of a second monitoring solution that monitors the main monitoring solution....
http://www.nagios.org/documentation
I'm not aware of any service you describe but a manual routine might go like this:
Have a folder/tag structure like this:
Services\Hourly-[NumberOfServices] (or add a folder per service)
Services\Daily-[NumberOfServicves]
Services\Weekly-[NumberOfServicves]
Services\Monthly-[NumberOfServicves]
Have rules for incoming mail to filter each specific service notification and move it to the right folder based on its expected timing.
Wakeup every hour and check if there are unread messages in your Hourly folder. The number of unread should be the same as the NumberOfServices mentioned in the folder. Read/Process them and make sure to all mark them as Read. Any service that didn't e-mailed get's spotted easily.
Wakeup at 0:00 and check if there are unread messages in your Daily folder. etc etc..
Wakeup at 0:00 and Saturday and check if there are unread messages in your Weekly folder. etc.....
Wakeup at 0:00 on the first of the month and check if there are unread messages in your Weekly folder. etc etc etc...
My advice would be to cut down the noise generated by the services.
If you still feel you need a service I can only provide a very very basic .Net implementation roughly based on the above process and works with gmail...
This is also portable to powershell...
static void Main(string[] args)
{
var resolver = new XmlUrlResolver
{
Credentials = new NetworkCredential("yourgoolgeaccount", "yourpassword")
};
var settings = new XmlReaderSettings();
settings.XmlResolver = resolver;
var xr = XmlReader
.Create("https://mail.google.com/mail/feed/atom/[name of your filter]"
, settings);
var navigator = new XPathDocument(xr).CreateNavigator();
var ns = new XmlNamespaceManager(new NameTable());
ns.AddNamespace("fd", "http://purl.org/atom/ns#");
var fullcountNode = navigator.SelectSingleNode(
"/fd:feed/fd:fullcount"
, ns);
Console.WriteLine(fullcountNode.Value);
int fullcount = Int32.Parse(fullcountNode.Value);
int expectCount = 10;
if (expectCount > fullcount)
{
Console.WriteLine("*** NOT EVERY ONE REPORTED BACK");
}
}

You mentioned Gmail, so you may be interested in googlecl, which gives you command-line controls for things like Google Calendar and Docs. Unfortunately they do not yet support Gmail, but if your long-term preference is to use a Gmail account as the hub of your status reports, then googlecl may be your best option.
In the short run, you can try out googlecl right now using the commands for Calendar, Blogger, or Docs, all of which are already supported. For example, these commands add events to Google Calendar:
google calendar add --cal server1 "I'm still alive at 13:45 today"
google calendar add "Server 1 is still alive at 2011-02-08 19:43"
...and these commands query the calendar:
google calendar list --fields title,when,where --cal "commitments"
google calendar list -q party --cal ".*"
Come to think of it, you may even find that Calendar, Blogger, or Docs are a more appropriate place than Gmail for tracking status updates. For example, a spreadsheet or calendar format should make it easier to generate a graphical representation of when a given service was up or down.
You still need to write a little program which uses googlecl to query the calendar (or blog, or docs, or whatever), but once you have simple command lines at your disposal, the rest should be pretty straightforward. Here's a link to further information about googlecl:
http://code.google.com/p/googlecl/
If you really want to use Gmail, and use it right now, they offer an IMAP interface. Using IMAP, you can perform numerous simple operations, such as determining if a message exists which contains a specified subject line. Here's one good place to learn about the details:
http://mail.google.com/support/bin/answer.py?hl=en&answer=75725
Here's a quick example that uses IMAP and Python to list the ten most-recent emails which have a given Gmail "Label":
import getpass, imaplib
# These gmail_* utilties are from https://github.com/drewbuschhorn/gmail_imap
import gmail_mailboxes, gmail_messages, gmail_message
# Update these next lines manually, or turn them into parms or somesuch.
gmail_account_name = "your_user_name#gmail.com" # Your full gmail address.
mailbox_name = "StatusReports" # Use Gmail "labels" to tag the relevant msgs.
class gmail_imap:
def __init__ (self, username, password):
self.imap_server = imaplib.IMAP4_SSL("imap.gmail.com",993)
self.username = username
self.password = password
self.loggedIn = False
self.mailboxes = gmail_mailboxes.gmail_mailboxes(self)
self.messages = gmail_messages.gmail_messages(self)
def login (self):
self.imap_server.login(self.username,self.password)
self.loggedIn = True
def logout (self):
self.imap_server.close()
self.imap_server.logout()
self.loggedIn = False
# Right now this prints a summary of the most-recent ten (or so) messages
# which have been labelled in Gmail with the string found in mailbox_name.
# It won't work unless you've used Gmail settings to allow IMAP access.
if __name__ == '__main__':
gmail = gmail_imap(gmail_account_name,getpass.getpass())
gmail.messages.process(mailbox_name)
for next in gmail.messages:
message = gmail.messages.getMessage(next.uid)
# This is a good point in the code to insert some kind of search
# of gmail.messages. Instead of unconditionally printing every
# entry (which is what the code below does), issue some sort of
# warning if the expected email (message.From and message.Subject)
# did not arrive within the expected time frame (message.date).
print message.date, message.From, message.Subject
gmail.logout()
As noted in the code comments, you could adapt it to issue some sort of warning if the most-recent messages in that mailbox do not contain an expected message. Then just run the Python program once per day (or whatever time period you require) to see if the expected email message was never received.

Related

MediaWiki - Require confirmed emails before allowing read?

I'm trying to setup a MediaWiki for university students. Using the EmailDomainCheck, I prevent anyone except those with a university based email from creating accounts. Using $wgEmailConfirmToEdit, I can require that an email is confirmed before the user can edit files. However, as it is, a user can use a fake email from the correct domain to create an account. With the account they can view all pages (even though they cannot edit them). I do not want to grant them read access unless the email has been confirmed. Is this possible? Note, I want all confirmed emails of the correct domain to be automatically accepted. It should not require manual account creation acceptance.
You could try the following, as outlined in the Documentation
# Disable for everyone.
$wgGroupPermissions['*']['read'] = false;
# Disable for users, too: by default 'user' is allowed to read, even if '*' is not.
$wgGroupPermissions['user']['read'] = false;
# Make it so users with confirmed email addresses are in the group.
$wgAutopromote['emailconfirmed'] = APCOND_EMAILCONFIRMED;
# Hide group from user list.
$wgImplicitGroups[] = 'emailconfirmed';
# Finally, set it to true for the desired group.
$wgGroupPermissions['emailconfirmed']['read'] = true;
As Jenny Shoars has mentioned, you may wish to whitelist some pages such as:
$wgWhitelistRead = array("Main_Page", "Special:CreateAccount", "Special:ConfirmEmail");
So that non registered users can still create accounts and the like.
In theory,
$wgGroupPermissions['*']['read'] = false;
$wgGroupPermissions['emailconfirmed']['read'] = true;
should work. In practice, MediaWiki almost always used with an "everyone can read" or "you can read iff you are logged in" setup and others are not very well tested, so if that wiki had some highly sensitive private information I wouldn't do this, but I imagine for a university website that's not the case.
Alternatively, it should not be too hard to integrate an email confirmation step into account creation, but you'd have to write the code for that. EmailAuth (which does a similar check during login) might give you an idea of how that would look.

Google pubsub into HTTP triggered cloud function?

Is it possible to trigger an HTTP cloud function in response to a pubsub message?
When editing a subscription, google makes it possible to push the message to an HTTPS endpoint, but for abuse reasons one has to be able to prove that you own the domain in order to do this, and of course you can't prove that you own google's own *.cloudfunctions.net domain which is where they get deployed.
The particular topic I'm trying to subscribe to is a public one, projects/pubsub-public-data/topics/taxirides-realtime. The answer might be use a background function rather than HTTP triggered, but that doesn't work for different reasons:
gcloud functions deploy echo --trigger-resource projects/pubsub-public-data/topics/taxirides-realtime --trigger-event google.pubsub.topic.publish
ERROR: gcloud crashed (ArgumentTypeError): Invalid value 'projects/pubsub-public-data/topics/taxirides-realtime': Topic must contain only Latin letters (lower- or upper-case), digits and the characters - + . _ ~ %. It must start with a letter and be from 3 to 255 characters long.
This seems to indicate this is only permitted on topics I own, which is a strange limitation.
It is possible to publish from a pub/sub topic to a cloud function. I was looking for a way to publish messages from a topic in project A to a function in project B. This was not possible with a regular topic trigger, but it is possible with http-trigger. Overall steps to follow:
Creata a http-triggered function in project B.
Create a topic in project A.
Create a push subscription on that topic in project A.
Domain verification
Push subscription
Here we have to fill in three things: the endpoint, the audience and the service account under which the function runs.
Push Endpoint: https://REGION-PROJECT_ID.cloudfunctions.net/FUNC_NAME/ (slash at end)
Audience: https://REGION-PROJECT_ID.cloudfunctions.net/FUNC_NAME (no slash at end)
Service Account: Choose a service account under which you want to send the actual message. Be sure the service account has the "roles/cloudfunctions.invoker" role on the cloud function that you are sending the messages to. Since november 2019, http-triggered functions are automatically secured because AllUsers is not set by default. Do not set this property unless you want your http function to be public!
Domain verification
Now you probably can't save your subscription because of an error, that is because the endpoint is not validated by Google. Therefore you need to whitelist the function URL at: https://console.cloud.google.com/apis/credentials/domainverification?project=PROJECT_NAME.
Following this step will also bring you to the Google Search Console, where you would also need to verify you own the endpoint. Sadly, at the time of writing this process cannot be automated.
Next we need to add something in the lines of the following (python example) to your cloud function to allow google to verify the function:
if request.method == 'GET':
return '''
<html>
<head>
<meta name="google-site-verification" content="{token}" />
</head>
<body>
</body>
</html>
'''.format(token=config.SITE_VERIFICATION_CODE)
Et voila! This should be working now.
Sources:
https://github.com/googleapis/nodejs-pubsub/issues/118#issuecomment-379823198,
https://cloud.google.com/functions/docs/calling/http
Currently, Cloud Functions does not allow one to create a function that receives messages for a topic in a different project. Therefore, specifying the full path including "projects/pubsub-public-data" does not work. The gcloud command to deploy a Cloud Function for a topic expects the topic name only (and not the full resource path). Since the full resource path contains the "/" character, it is not a valid specification and results in the error you see.
The error you are getting seems to be that you are misspelling something in the gcloud command you are issuing.
ERROR: gcloud crashed (ArgumentTypeError): Invalid value 'projects/pubsub-public-data/topics/taxirides-realtime': Topic must contain only Latin letters (lower- or upper-case), digits and the characters - + . _ ~ %. It must start with a letter and be from 3 to 255 characters long
Are you putting a newline character in the middle of the command?

How to restrict fields returned by stackexchange api, and turn off paging?

I'd like to have a list of just the current titles for all questions in one of the smaller (less than 10,000 questions) stackexchange site. I tried the interactive utility here: https://api.stackexchange.com/docs/questions and it both reports the result as a json at the bottom, and produces the requesting url at the top. For example:
https://api.stackexchange.com/2.2/questions?order=desc&sort=activity&tagged=apples&site=cooking
returns this JSON in my browser:
{"items":[{"tags":["apples","crumble"],"owner":{ ...
...
...],"has_more":true,"quota_max":300,"quota_remaining":252}
What is quota? It was 10,000 on one search on one site, but suddenly it's only 300 here.
I won't be doing this very often, what I'd like is the quickest way to edit that (or similar of course) url so I can get a list of all of the titles on a small site. I don't understand how to use paging, and I don't need any of the other fields. I don't care if I get them, but I'm thinking if I exclude them I can have more at once.
If I need to script it, python (2.7) is my preferred (only) language.
quota_max is the number of requests your application is allowed per day. 300 is the default for an unregistered application. This used to be mentioned directly on the page describing throttles, but seems to have been removed. Here is historical information describing the default.
To increase this to 10,000, you need to register an application and then authenticate by passing an access token in your script.
To get all titles on a site, you can use a Python library to help:
StackAPI. The answer below will use this library. DISCLAIMER: I wrote this library
Py-StackExchange
SEAPI
StackPy
Assuming you have registered your application and authenticated we can proceed.
First, install StackAPI (documentation):
pip install stackapi
This code will then grab the 10,000 most recent questions (max_pages * page_size) for the site hardwarerecs. Each page costs you one API hit, so the more items per page, the few API calls.
from stackapi import StackAPI
SITE = StackAPI('hardwarerecs')
SITE.page_size = 100
SITE.max_pages = 100
# Filter to only get question title and link
filter = '!BHMIbze0EQ*ved8LyoO6rNjkuLgHPR'
questions = SITE.fetch('questions', filter=filter)
In the questions variable is a dictionary that looks very similar to the API output, except that the library did all the paging for you. Your data is in questions['data'] and, in this case, contains a list of dictionaries that look like this:
[
...
{u'link': u'http://hardwarerecs.stackexchange.com/questions/29/sound-board-to-replace-a-gl2200-in-a-house-of-worship-foh-setting',
u'title': u'Sound board to replace a GL2200 in a house-of-worship FOH setting?'},
{ u'link': u'http://hardwarerecs.stackexchange.com/questions/31/passive-gps-tracker-logger',
u'title': u'Passive GPS tracker/logger'}
...
]
This result set is limited to only the title and the link because of the filter we applied. You can find the appropriate filter by adjusting what fields you want in the web UI and copying the filter field.
The hardwarerecs parameter that is passed when creating the SITE parameter is the first part of the site's domain URL. Alternatively, you can find it by looking at the api_site_parameter for your site when looking at the /sites end point.

Unable to Change Recurrence Pattern in EWS

I am having a problem that has been reported here and elsewhere before: not being able to change the recurrence pattern on a master via EWS. First, I tried using the old proxy classes against E2010. I have also now tried using the 2.0 Managed API, to no avail. The error FWIW is "Set action is invalid for property". E.g. I want to change the recurrence end date, or the number of recurrences.
In an MSDN post from 2008, Dave Stirling mentions that only the organizer should be able to do this. This is a problem for me because my server application uses a single, full-access id to manage all of the room resource calendars in an enterprise. With this user I can delete appointments on any calendar, regardless of organizer, and I can certainly update a single instance of a recurring series, e.g. changing its start time. I don't understand why manipulating the recurrence pattern would be prohibited because I'm not the organizer while manipulating an instance's spot on the calendar, or deleting the entire series would not be.
I have also tried using impersonation, so that I am (I think) impersonating the Room resource itself, in which case, even though I am not the organizer, I feel I must be the owner of the appointment, and hence entitled to do whatever the heck I want to it. I guess Exchange feels differently. Managed API code below, FWIW.
TIA,
Paul
var Svc = new ExchangeService(ExchangeVersion.Exchange2010, TimeZoneInfo.Local);
Svc.CookieContainer = new CookieContainer();
Svc.Credentials = new WebCredentials(m_SvcUser, m_SvcPswd);
Svc.EnableScpLookup = false;
Svc.ImpersonatedUserId = new ImpersonatedUserId(ConnectingIdType.SmtpAddress, m_RoomMB);
Svc.AutodiscoverUrl(m_RoomMB, RedirectionCallback);
var Master = Appointment.Bind(Svc, new ItemId(args[0]), m_Props);
if (Master.Recurrence.NumberOfOccurrences != null)
Master.Recurrence.NumberOfOccurrences--;
else
Master.Recurrence.EndDate = DateTime.Now;
Master.Update(ConflictResolutionMode.AlwaysOverwrite, SendInvitationsOrCancellationsMode.SendToNone);
To close this out, I will report that I opened this question as a support issue after getting no responses here or on MSDN. The net of it is this: Exchange does not permit an attendee (in this case a conference room I am impersonating in EWS) to change the recurrence "blob." This is by design. The only user who can do so is the organizer. I'm not sure this really makes complete sense to me, but I don't have much say in the matter, now do I?

Retrieving information from a web page

My application is meant to speed up the retrieval of phone call information from our telephone system.
The best way to get this information is to create a new search on the telephone system's web interface and export the results to an Excel spreadsheet which my application then imports into a DataSet.
To get the export, from the login screen, the process goes as follows:
Log in
Navigate to Reports Page
Click "Extension Detail" link
Select "Extensions" CheckBox
Select the extensions (typically all the ones currently being used) from the listbox
Specify date range
Click on Export button
It's not a big job to do it manually every day, but, for reliability, it would be great if I can make my application do this automatically the first time it starts every day.
Since more than 1 person in the company is going to use this application, having a Windows Service do it would be even better.
I don't know if it'll help, but the system is Datatex Topaz Next Generation telephone management system: http://www.datatex.co.za/downloads/index.html#TNG
Can anyone give me a basic idea how to do this?
Also, can anyone post links (in comments if need be) to pages where I can learn more about how to do this?
I have done the something similar to fetch info from a website. I cannot give you a exact answer. But the idea is to send login info to the page with form values. If the site is relying on cookies, you can use this cookie aware WebClient:
public class CookieAwareWebClient : WebClient
{
private CookieContainer cookieContainer = new CookieContainer();
protected override WebRequest GetWebRequest(Uri address)
{
WebRequest request = base.GetWebRequest(address);
if (request is HttpWebRequest)
{
(request as HttpWebRequest).CookieContainer = cookieContainer;
}
return request;
}
}
You should be aware that some sites rely on a session id being passed so the first thing I did was to fetch the session id from the page:
var client = new CookieAwareWebClient();
client.Encoding = Encoding.UTF8;
var indexHtml = client.DownloadString(*index page url*);
string sessionID = fetchSessionID(indexHtml);
Then I had to log in to the page which you can do by uploading values to the page. You can see the specific form elements with "view source" but you have to know a little HTML to do so.
var values = new NameValueCollection();
values.Add("sessionid", sessionID); //Fetched session id
values.Add("brugerid", args[0]); //Username in my case
values.Add("adgangskode", args[1]); //Password in my case
values.Add("login", "Login"); //The login button
//Logging in
client.UploadValues(*url to login*, values); //If all goes perfect, I'm logged in now
And then I could download the page I needed. In your case you may use DownloadFile(...) if the file always have the same url (something like Export.aspx?From=2010-10-10&To=2010-11-11) or UploadValues(...) where you specify the values as before but saves the result.
string html = client.DownloadString(*url*);
It seems you have a lot more steps than I did. But the principle is the same. To see what values your send to the site to login etc. you can use programs such as Fiddler (windows) which can capture the activity going on. Essential you just do exactly the same thing but watch out for session id etc. which is temporary.
The best idea is really to use some native way to fetch data, but if don't got the code, database etc. you have to do it the ugly way. You may also need a HTML parser to fetch the data (ups, you don't because you export to a file). And last but not least, keep in mind that pages can change and there is great potential to fail to login, parse etc.
Please ask for if you are uncertain what is going on.
ADDITION
The CookieAwareWebClient is not my code:
http://code.google.com/p/gardens/source/browse/Montrics/Physical.MyPyramid/CookieAwareWebClient.cs?r=26
Using CookieContainer with WebClient class
I also found some relevant threads:
What's a good tool to screen-scrape with Javascript support?
http://forums.asp.net/t/1475637.aspx
With a HTTP client, you need to do the following:
Log in, using cookies or HTTP authentication
Request a page
Submit form data
This means that you need some class or component in your program that can do HTTP, cookies, authentication and forms. With this, you do the same requests a user would do.