I have a ruby script running a server 7 days a weeks 24 hours. It communicates with a third-party website and locally RabbitMQ and MySql. I want it to reliable, of course. I already have some solution for monitoring the ruby script, whether or not it's up, if it's not then the admin will be sent an email.
I wonder, should I also monitory the availability of RabbitMQ and MySql? Is that a good practice or would it be redundant? It might be redundant because if RabbitMQ and MySql is down, the script will also eventually go down and thus the admin be sent an email anyway.
How is it usually done?
Note that RabbitMQ and MySql are just examples, I might've chosen something else, I just want to know the general approach.
I have been working in 24/7 on-call rotations and I prefer emails that tell me the exact reason why something when down.
An email telling my system is down is okay, but an email
System is down, because mysql doesn't respond.
You might want to check: mysql server at 1.2.3.4 for
is it running,
disk space,
...
is much better and helps a lot to fix a system in short time. Therefore I think, yes an error message should be as precise als possible.
But on the other hand ensure that you do not send many different emails for the same problem that would be counterproductive.
Related
I'm finishing a system at work that makes calls to mysql server. Those calls' arguments reveal information that I need to keep private, like vote(idUser, idCandidate). There's no information in the db that relates those two of course, nor in "the visible part" of the back end, but even though I think this can't be done, I wanted to make sure that it is impossible to trace this sort of calls, with a log or something (calls that were made, or calls being made at the moment), as it is impossible in most languages, unless you specifically "debug" in a certain way, while the system is in production and being used. I hope the questions is clear enough. Thanks.
How do I log thee? Let me count the ways.
MySQL query log. I can enable this per-session and send everything to a log file.
I can set up a slave server and have insertions sent to me by the master. This is a significant intervention and would leave a wide trace.
On the server, unbeknownst to either Web app and MySQL log, I can intercept communications between the two. I need administrative access to the machine, of course.
On the server, again with administrative access, I can both log the query calls and inject a logging instrumentation into the SQL interface (the legitimate one is the MySQL Audit Plugin, but there are several alternatives, developed for various purposes by developers over the years)
What can you do? You can have the applications use a secure protocol, just for starters.
Then, you need to secure your machine so that administrator tricks do not work, and even if the logs are activated, nobody can read them and you can be advised of any new and modified file to delete it promptly.
So Have a web application that has 10-12 pages with many POST/ GET DB Calls. We usually have a apache crash/other problem when site traffic results to 1000 or so (concurrent users) which is very small number, we have updated server with good RAM and resources. When our system admin guy do load testing on blitz and other custom script and is suggesting to move away from Apache. Some things does not make sense to me. Like Apache is not too bad to handle few thousand of concurrent users considering we have cloudflare for caching. Here is what he suggested:
replacement of Apache+mod_fcgi with Nginx+php-fpm which can make the server handle much more users, and then test it.
or
2. For testing: Need 10-20 servers to run a scenario from. Basically, what is needed is a more complex blitz.io analogue. create one server, which takes all those hours, then just clone it in the cloud and pay for about 1 hour of testing multiplied by the number of servers needed.
Once again there are many DB calls anf HT access. ALso what makes Nginx better than apache in this case?
I would check this comparison first. Basically, nginx is event based, so it's able to handle more requests concurrently. However, as the MySQL DB seems to be the choke point here, it's very possible that nginx wouldn't solve all your problems. Perhaps moving to a NoSQL kind of database, that's better at scaling horizontally, would help (if that's feasible).
I've been doing some research of the best way to show an "users online" counter which is updated to the second trying to avoid continuos ajax polling.
Obviously WebSockets seems to be the best option. Since this is an intranet I will make it a requirement to use Chrome or Safari so there shouldn't be compatibility issues.
I've been reading some articles about WebSockets since I'm new to it and I think I pretty much understand how it works.
What I'm not so sure is how to implement it with PHP. Node.js seems the natural choice for this because of it's "always running" nature but that's not an option.
Why I'm most confused about is the fact that PHP runs and when it's done, it ends. If PHP ended, wouldn't the socket connection be lost? Or if the php re-runs it will look back the user by ip? (I don't see that likely)
Then I found this library
http://code.google.com/p/phpwebsocket/
but it seems to be a little old (it mentions only Chrome nightly is compatible with WebSockets)
In one point says "From the command line, run the server.php program to listen for socket connections." which means I need SSH, something many shared hosting plans don't have.
And my other doubt is this other line in the source of that library:
set_time_limit(0);
does that mean that the php file will run continuously? Is that allow in shared hosting? From what I know all hostings kill php after a timeout of 1 o2 minutes.
I have a mysql table with online users and I want to use PHP to broadcast via websocket the amount of logged in users to those online users. Can someone please help me or point me somewhere with better information how this could be achieved?
Thanks
Websockets would require lots of thing even on dedicated hosting, put aside shared hosting.
For your requirement server sent events (sse) is the correct choice, since only the server will be pushing data to the client.
SSE can simply call a server script, very much like ajax, but the client side will receive and be able to process data part by part as it comes in. Dom events would be generated whenever some data comes in.
But IE does not support SSE even in version 10. So for IE you have to use some fallback technique like "foreever iframe".
as far as hosting is concerned, ordinary shared hostings (and those which are not very cheap) would allow php scripts to run for long, as long as they are not seen as a problem.
I have a python application where I want to start doing more work in the background so that it will scale better as it gets busier. In the past I have used Celery for doing normal background tasks, and this has worked well.
The only difference between this application and the others I have done in the past is that I need to guarantee that these messages are processed, they can't be lost.
For this application I'm not too concerned about speed for my message queue, I need reliability and durability first and formost. To be safe I want to have two queue servers, both in different data centers in case something goes wrong, one a backup of the other.
Looking at Celery it looks like it supports a bunch of different backends, some with more features then the others. The two most popular look like redis and RabbitMQ so I took some time to examine them further.
RabbitMQ:
Supports durable queues and clustering, but the problem with the way they have clustering today is that if you lose a node in the cluster, all messages in that node are unavailable until you bring that node back online. It doesn't replicated the messages between the different nodes in the cluster, it just replicates the metadata about the message, and then it goes back to the originating node to get the message, if the node isn't running, you are S.O.L. Not ideal.
The way they recommend to get around this is to setup a second server and replicate the file system using DRBD, and then running something like pacemaker to switch the clients to the backup server when it needs too. This seems pretty complicated, not sure if there is a better way. Anyone know of a better way?
Redis:
Supports a read slave and this would allow me to have a backup in case of emergencies but it doesn't support master-master setup, and I'm not sure if it handles active failover between master and slave. It doesn't have the same features as RabbitMQ, but looks much easier to setup and maintain.
Questions:
What is the best way to setup celery
so that it will guarantee message
processing.
Has anyone done this before? If so,
would be mind sharing what you did?
A lot has changed since the OP! There is now an option for high-availability aka "mirrored" queues. This goes pretty far toward solving the problem you described. See http://www.rabbitmq.com/ha.html.
You might want to check out IronMQ, it covers your requirements (durable, highly available, etc) and is a cloud native solution so zero maintenance. And there's a Celery broker for it: https://github.com/iron-io/iron_celery so you can start using it just by changing your Celery config.
I suspect that Celery bound to existing backends is the wrong solution for the reliability guarantees you need.
Given that you want a distributed queueing system with strong durability and reliability guarantees, I'd start by looking for such a system (they do exist) and then figuring out the best way to bind to it in Python. That may be via Celery & a new backend, or not.
I've used Amazon SQS for this propose and got good results. You will recieve message until you will delete it from queue and it allows to grow you app as high as you will need.
Is using a distributed rendering system an option? Normally reserved for HPC but alot of concepts are the same. Check out Qube or Deadline Render. There are other, open source solutions as well. All have failover in mind given the high degree of complexity and risk of failure in some renders that can take hours per image sequence frame.
We are developing an application which will require to send around 30 outbound emails per second. We have a server running SMTP but this machine in cloud hosted and I do not have any idea what kind of configuration will I require to support such a load. I do not even know if this load is considered to be average or high. Do i need to do anything special for such a load. Do i need a dedicated quad core server for this kind of load or lets say just 1/10th CPU of a quad core server is good enough
Hm
what for?
30 emails per second is nothing. I wrote a server like 10 years ago hitting about 5000 per second (to one other server taking it down in the process - custoemr wanted as fast as possible, i delivered).
Get any little MTA and jst use it. No sense in writing something yourself for that low volume.
Unless you hit the server with a lot of stuff at once (loading it for transfer), a small VPS should be ok.
Seriously, 30 emails per second is what I sometimes send from my dialup account. THis is not even a visible volume for a decent message transfer agent. It is definitely NOT "high volume".
Going to echo TomTom on this one and say just to get one of the many services out there that will help you do this. It's probably far easier to utilize one of their services and not have to worry about reputation monitoring and all the fun stuff of SMTP servers than to create your own solution.
Let me know if you need help finding these services.
(Full Disclosure: I work for PostageApp.com, and we're rolling out a hosted SMTP service soon!)