We are developing an application which will require to send around 30 outbound emails per second. We have a server running SMTP but this machine in cloud hosted and I do not have any idea what kind of configuration will I require to support such a load. I do not even know if this load is considered to be average or high. Do i need to do anything special for such a load. Do i need a dedicated quad core server for this kind of load or lets say just 1/10th CPU of a quad core server is good enough
Hm
what for?
30 emails per second is nothing. I wrote a server like 10 years ago hitting about 5000 per second (to one other server taking it down in the process - custoemr wanted as fast as possible, i delivered).
Get any little MTA and jst use it. No sense in writing something yourself for that low volume.
Unless you hit the server with a lot of stuff at once (loading it for transfer), a small VPS should be ok.
Seriously, 30 emails per second is what I sometimes send from my dialup account. THis is not even a visible volume for a decent message transfer agent. It is definitely NOT "high volume".
Going to echo TomTom on this one and say just to get one of the many services out there that will help you do this. It's probably far easier to utilize one of their services and not have to worry about reputation monitoring and all the fun stuff of SMTP servers than to create your own solution.
Let me know if you need help finding these services.
(Full Disclosure: I work for PostageApp.com, and we're rolling out a hosted SMTP service soon!)
Related
I have a ruby script running a server 7 days a weeks 24 hours. It communicates with a third-party website and locally RabbitMQ and MySql. I want it to reliable, of course. I already have some solution for monitoring the ruby script, whether or not it's up, if it's not then the admin will be sent an email.
I wonder, should I also monitory the availability of RabbitMQ and MySql? Is that a good practice or would it be redundant? It might be redundant because if RabbitMQ and MySql is down, the script will also eventually go down and thus the admin be sent an email anyway.
How is it usually done?
Note that RabbitMQ and MySql are just examples, I might've chosen something else, I just want to know the general approach.
I have been working in 24/7 on-call rotations and I prefer emails that tell me the exact reason why something when down.
An email telling my system is down is okay, but an email
System is down, because mysql doesn't respond.
You might want to check: mysql server at 1.2.3.4 for
is it running,
disk space,
...
is much better and helps a lot to fix a system in short time. Therefore I think, yes an error message should be as precise als possible.
But on the other hand ensure that you do not send many different emails for the same problem that would be counterproductive.
I've created a nightly sync between two database applications for a small construction company and setup simple notifications using database mail to let a few people know if the load was successful or not. Now that they see this notification is working I've been asked to provide status updates to their clients as employees make changes to the work order throughout the day.
I've done some research and understand DB Mail is not designed for this type of feature but I'm thinking the frequency will be small enough to not be a problem. I'm estimating 50-200 emails per day.
I couldn't find anything on the actual limitations of DB Mail and wondering if anyone has tried something similar in the past or if I could be pushed in the right direction to send these emails using best practice.
If we're talking hundreds here you can definitely go ahead. Take a peak at the Database Mail MSDN page. The current design (i.e. anything post-SQL2000) was specifically designed for large, high-performance enterprise implementations. Built on top of Service Broker (SQL Server's message queuing bus) it offers both asynchronous processing and scalability with process isolation, clustering, and failover. One caveat is increased transaction log pressure as messages, unlike in some other implementations, are ACID-protected by SQL Server which in turn gives you full recoverability of the queues in case of failure.
If you're wondering what Service Broker can handle before migrating to a dedicated solution, there's a great MySpace case study. The most interesting fragment:
We didn’t want to start down the road of using Service Broker unless
we could demonstrate that it could handle the levels of messages that
we needed to support our millions of users across 440 database
servers,” says Stelzmuller. “When we went to the lab we brought our
own workloads to ensure the quality of the testing. We needed to see
if Service Broker could handle loads of 4,000 messages per second. Our
testing found it could handle more than 18,000 messages a second. We
were delighted that we could build our solution using Service Broker,
rather than creating a custom solution on our own.
So Have a web application that has 10-12 pages with many POST/ GET DB Calls. We usually have a apache crash/other problem when site traffic results to 1000 or so (concurrent users) which is very small number, we have updated server with good RAM and resources. When our system admin guy do load testing on blitz and other custom script and is suggesting to move away from Apache. Some things does not make sense to me. Like Apache is not too bad to handle few thousand of concurrent users considering we have cloudflare for caching. Here is what he suggested:
replacement of Apache+mod_fcgi with Nginx+php-fpm which can make the server handle much more users, and then test it.
or
2. For testing: Need 10-20 servers to run a scenario from. Basically, what is needed is a more complex blitz.io analogue. create one server, which takes all those hours, then just clone it in the cloud and pay for about 1 hour of testing multiplied by the number of servers needed.
Once again there are many DB calls anf HT access. ALso what makes Nginx better than apache in this case?
I would check this comparison first. Basically, nginx is event based, so it's able to handle more requests concurrently. However, as the MySQL DB seems to be the choke point here, it's very possible that nginx wouldn't solve all your problems. Perhaps moving to a NoSQL kind of database, that's better at scaling horizontally, would help (if that's feasible).
This is not the typical question, but I'm out of ideas and don't know where else to go. If there are better places to ask this, just point me there in the comments. Thanks.
Situation
We have this web application that uses Zend Framework, so runs in PHP on an Apache web server. We use MySQL for data storage and memcached for object caching.
The application has a very unique usage and load pattern. It is a mobile web application where every full hour a cronjob looks through the database for users that have some information waiting or action to do and sends this information to a (external) notification server, that pushes these notifications to them. After the users get these notifications, the go to the app and use it, mostly for a very short time. An hour later, same thing happens.
Problem
In the last few weeks usage of the application really started to grow. In the last few days we encountered very high load and doubling of application response times during and after the sending of these notifications (so basically every hour). The server doesn't crash or stop responding to requests, it just gets slower and slower and often takes 20 minutes to recover - until the same thing starts again at the full hour.
We have extensive monitoring in place (New Relic, collectd) but I can't figure out what's wrong; I can't find the bottlekneck. That's where you come in:
Can you help me figure out what's wrong and maybe how to fix it?
Additional information
The server is a 16 core Intel Xeon (8 cores with hyperthreading, I think) and 12GB RAM running Ubuntu 10.04 (Linux 3.2.4-20120307 x86_64). Apache is 2.2.x and PHP is Version 5.3.2-1ubuntu4.11.
If any configuration information would help analyze the problem, just comment and I will add it.
Graphs
info
phpinfo()
apc status
memcache status
collectd
Processes
CPU
Apache
Load
MySQL
Vmem
Disk
New Relic
Application performance
Server overview
Processes
Network
Disks
(Sorry the graphs are gifs and not the same time period, but I think the most important info is in there)
The problem is almost certainly MySQL based. If you look at the final graph mysql/mysql_threads you can see the number of threads hits 200 (which I assume is your setting for max_connections) at 20:00. Once the max_connections has been hit things do tend to take a while to recover.
Using mtop to monitor MySQL just before the hour will really help you figure out what is going on but if you cannot install this you could just using SHOW PROCESSLIST;. You will need to establish your connection to mysql before the problem hits. You will probably see lots of processes queued with only 1 process currently executing. This will be the most likely culprit.
Having identified the query causing the problems you can attack your code. Without understanding how your application is actually working my best guess would be that using an explicit transaction around the problem query(ies) will probably solve the problem.
Good luck!
I have a new project and need some advice about managing the application’s load.
The application will have over 500 users with about 70% of them logging on between 5 & 7pm each night. The users will pull down about 50 records, update the records and the save the changes back to the data base. Estimate each record will be updated in about a minute and half. The data is all text, no images.
The application will be built in PHP on top of a MySql community edition data base, hosted at Dreamhost.
My question can a site supported on shared hosting company support this kind of traffic? How about MySql ? I know this is a big question but any advice about the app’s architecture ?
Thanks for your advice.
MySql should handle it just fine. I might be a bit worried about the shared hosting setup though. Most shared hosting setups have very restrictive policies on CPU utilization that you might run into.
Rather than letting your app drag down the performance of the server, they will often shut down your app for a few minutes if you exceed a certain CPU utilization threshold over some period of time. Of course they don't advertise what those limits are, but I've often run into this issue with various shared hosting accounts.
You might be better off going with a cheap VPS type setup. That will (generally) get you a dedicated CPU and no artificial limits. Your server might slow down a bit under heavy load, but at least they won't cut your site off entirely.