We have ASP.NET applications and supporting web services in production with lot of traffic.
It logs lot of service calls and Exceptions to MS Sql server.
We want to analyze the logs and exceptions, following are some use cases.
Monitor Success and Failures of calls to 3rd party services from Logs.
Generate graphical data from the logs.
e.g AirPrice success 1300 times over a month.
AirBooking failures 200 times over a month.
comparison of above stats over the period of time.
Exceptions statistics :
a. Percent increase in some Type of exceptions.
b. spike in some type of exceptions
c. occurence rate for some type of exceptions over a selectable period of time.
Is there any ready tool or framework which I can use to analyze such logs.
Or do I need to code such a thing myself.
NewRelic (application performance monitoring service) is pretty damn good, though not cheap - check it out at http://newrelic.com/, and they have a free version.
Loggly is a simple logging service with a great and powerful dashboard for analyzing your logs and the price scales nicely with use.
Related
Good morning, everyone,
As part of the migration from a heavy client project to a connected
application to a server (currently under study DataSnap XE10.2) in order to transfer on an ad hoc basis and retrieve information from the server.
We would like to have some feedback on other available technologies,
their durability and ease of adaptation.
Here is the profile of our application
The client connects to a remote server that can be hosted elsewhere.
There can be up to 300 clients connected at the same time over a period of 3 days.
these 300 customers can send on a variable hourly interval (1 to 2 hours and in a different way.
depending on the time of day (different countries).
These connections can transmit up to 5000 data so 300 = 1,500,000 over a period of one month.
For the moment we have chosen the DataSnap solution because it is already used on medical applications.
and especially for its ease of migration from the Delphi heavy client project to this type of architecture.
and also for his perenity with Delphi.
Our questions: what do you think?
What arguments and intermediate or other solutions do you propose? As far as RAD Server is concerned, this has a cost per license, but does it exit it examples of migration from a DataSnap application to RAD Server?
What are your experiences in these different areas? (concrete case in point)
On our side we will launch a simulation of 300 clients transmitting 5000 requests JSON REST to our DATASNAP server which will insert each of these queries into a database.
MySQL of 40GB, the insertion will return an acknowledgement of receipt and a written acknowledgement (simple boolean)
Thank you for your feedback, on our side we will publish the results of our tests
There are several solutions, but I recommend our Open Source mORMot framework.
Its SOA is based on interface type definitions, it is REST/JSON from the ground up, and was reported to have very good performance and stability, especially in respect to DataSnap. It is Open Source and work with both Delphi and FPC (also under Linux) - so could be considered as a safer solution for the middle/long term. DataSnap didn't evolve much since years, and I don't understand the RAD Server "black box" approach.
About migrating an existing database or system, check this blog article which shows some basic steps with mORMot.
You have other bricks available, like an ORM, a MVC layer for dynamic web site generation, logging, interface stubbing, high performance database layer, cross-platform clients, an exhaustive documentation and a lot of other features.
I have a massive table that records events happening on our website. It has tens of millions of rows.
I've already tried adding indexing and other optimizations.
However, it's still very taxing on our server (even though we have quite a powerful one) and takes 20 seconds on some large graph/chart queries. So long in fact that our daemon intervenes to kill the queries often.
Currently we have a Google Compute instance on the frontend and a Google SQL instance on the backend.
So my question is this - is there some better way of storing an querying time series data using the Google Cloud?
I mean, do they have some specialist server or storage engine?
I need something I can connect to my php application.
Elasticsearch is awesome for time series data.
You can run it on compute engine, or they have a hosted version.
It is accessed via an HTTP JSON API, and there are several PHP clients (although I tend to make the API calls directly as i find it better to understand their query language that way).
https://www.elastic.co
They also have an automated graphing interface for time series data. It's called Kibana.
Enjoy!!
Update: I missed the important part of the question "using the Google Cloud?" My answer does not use any specialized GC services or infrastructure.
I have used ElasticSearch for storing events and profiling information from a web site. I even wrote a statsd backend storing stat information in elasticsearch.
After elasticsearch changed kibana from 3 to 4, I found the interface extremely bad for looking at stats. You can only chart 1 metric from each query, so if you want to chart time, average time, and avg time of 90% you must do 3 queries, instead of 1 that returns 3 values. (the same issue existing in 3, just version 4 looked more ugly and was more confusing to my users)
My recommendation is to choose a Time Series Database that is supported by graphana - a time series charting front end. OpenTSDB stores information in a hadoop-like format, so it will be able to scale out massively. Most of the others store events similar to row-based information.
For capturing statistics, you can either use statsd or reimann (or reimann and then statsd). Reimann can add alerting and monitoring before events are sent to your stats database, statsd merely collates, averages, and flushes stats to a DB.
http://docs.grafana.org/
https://github.com/markkimsal/statsd-elasticsearch-backend
https://github.com/etsy/statsd
http://riemann.io/
I've created a nightly sync between two database applications for a small construction company and setup simple notifications using database mail to let a few people know if the load was successful or not. Now that they see this notification is working I've been asked to provide status updates to their clients as employees make changes to the work order throughout the day.
I've done some research and understand DB Mail is not designed for this type of feature but I'm thinking the frequency will be small enough to not be a problem. I'm estimating 50-200 emails per day.
I couldn't find anything on the actual limitations of DB Mail and wondering if anyone has tried something similar in the past or if I could be pushed in the right direction to send these emails using best practice.
If we're talking hundreds here you can definitely go ahead. Take a peak at the Database Mail MSDN page. The current design (i.e. anything post-SQL2000) was specifically designed for large, high-performance enterprise implementations. Built on top of Service Broker (SQL Server's message queuing bus) it offers both asynchronous processing and scalability with process isolation, clustering, and failover. One caveat is increased transaction log pressure as messages, unlike in some other implementations, are ACID-protected by SQL Server which in turn gives you full recoverability of the queues in case of failure.
If you're wondering what Service Broker can handle before migrating to a dedicated solution, there's a great MySpace case study. The most interesting fragment:
We didn’t want to start down the road of using Service Broker unless
we could demonstrate that it could handle the levels of messages that
we needed to support our millions of users across 440 database
servers,” says Stelzmuller. “When we went to the lab we brought our
own workloads to ensure the quality of the testing. We needed to see
if Service Broker could handle loads of 4,000 messages per second. Our
testing found it could handle more than 18,000 messages a second. We
were delighted that we could build our solution using Service Broker,
rather than creating a custom solution on our own.
SQL server deletes all execution plans when SQL service is restarted. Is there any way these can be saved before we restart the service?
No.
The closest thing you can provide is plan guides, but even with plan guides the server will have to recompile every query the first time it is submitted after the restart. Plan guides can theoretically reduce the amount of work required during query compilation.
As Martin suggested, you can pull information out about the query plans, if you want to do analysis, but you can't put that information back in later:
SELECT query_plan FROM sys.dm_exec_cached_plans CROSS APPLY sys.dm_exec_query_plan(plan_handle)
If you are truly concerned, then set up a suite of queries to run after a service restart that will populate the plan cache. But I think that is a waste of time.
The mere fact that the SQL Server service will be down means that users will experience an outage. This already requires planning and communication. Understanding that such a restart's effects have a larger scope than merely the denial of service during the restart time--the first few minutes of usage may be slow--simply changes your planning and communication by a bit. If some kind of theoretical slowness is a big problem, you'll have to plan the downtime more carefully or wait for a more opportune time--2am instead of 2pm perhaps.
I say "theoretical slowness" because no one may experience any true issue with recompiling queries afterward. Queries generally compile very quickly.
I have a client asking this for a requirement and haven't done this before, what does he mean by web service infrastructure?
That phrase encompasses a wide variety of technical aspects. Your infrastructure is all of the components that make up the systems that run a web business or application, including hardware. So it refers to your server and network setup, your bandwidth and connections in and out, your database setup, backup solutions, web server software, code deployment methods, and anything else used to successfully run a web business with high reliability and uptime and low error and bug incidents.
In order to make such a thing scalable, you have to architect all these components together into something that will work smoothly with growth over time. A scalable architecture should be flexible enough to handle sudden traffic spikes.
Methods used to facilitate scalability include replicated databases, clustered web servers, load balancers, RAID disk striping, and network switching. Your code has to take much of this into account.
It's a tough service to provide.
First thing that comes to mind was the Enterprise service bus.
He probably means some sort of "infrastructure" to run a lot of complex interacting web services.
Either an enterprise application that you call via a web service that can run on many instances of a web application server, or run a single instance that are very nicely multithreaded and scale to many CPUs, or deploying loads of different webservices that all talk to each other, often via message queues, until you have something that breaks all the time and requires a huge team of people to maintain. Might as well throw in a load of virtual machines to have a virtualised, scalable, re-deployable web service infrastructure (i.e., loads of tomcats or jbosses in linux VMs ready to deply as needed, one app per VM).
Then there is physical scalability. Is there enough CPU power for your needs? Is there enough bandwidth between physical nodes to send all these messages and SOAP transactions between machines? Is there enough storage? Is the storage available on a fast, low latency interconnect? Is the database nicely fed with CPU power, bandwidth, a disc system that doesn't lag. Is there a database backup. How about when a single machine can't handle the load of a particular function - then you need load balancers, although these are good for redundancy and software updates whilst remaining live as well.
Is there a site backup? Or are you scaling globally - will there be multiple data centres around the globe? Do you have redundant links to the internet from each data centre? What happens when a site goes down? How is data replicated between sites, to reduce inter-site communications, and how do these data caches and pushes work?
And so on and so forth. But your client probably just wants a web service that can be load balanced without thrashing (i.e., two or more instances can share data/sessions/etc, depends on the application really), with easy database configuration and backup. Ease of deployment is desirable, so make the install simple. Or even provide a Linux VM for them to add to their VM infrastructure. Talk to their sysadmin to see what they currently do.
This phrase is often used as a marketing term from companies who sell some part of what they'll call a "scalable web services infrastructure".
Try to find out from the client exactly what they need. Do they have existing web services? Do they have existing business logic they've decided to expose as web services? Do they have customers who are asking to be able to access your client's systems through web services?
Does your client even know what a web service is?