The overhead for long polling is http connection establishment, also on server side the Java EE server has to hold a lot of suspended requests.
For periodic polls with 10/20 seconds interval, it will work worse than long polling?
It depends on various things:
How expensive is it on your client, server and network to make a new request? In some cases, the server may be able to handle "here's a new result - push it to anyone who cares" (from the backend) more easily than "have there been any results in the last 20 seconds" (from the client). Likewise new connections over a 3G network may be more expensive than keeping an existing one going.
How important is it to get immediate results? (Think about a chat application, for example.)
(And no doubt there are others...)
Related
I've an api, notifyCustomers() implemented on my batch server which gets called from my application server. It can send notification via three channels SMS, Push & Email. I've separate helper classes for each of them and they all execute in async mode.
I've got around 30k users out of which I usually send notification to the particular set of users ranging from 3k to 20k. The issue that I face is whenever I call that api, mysql performance just goes for a toss, particularly CPU. CPU utilisation goes around 100% for a very long period of around 30 mins
I've figured out workaround by doing following things and it's helping me in keeping things under control:
Using projection instead of domain object
Getting data in batch of 500 in each call
Implemented indexing based on the criteria that I need
No database calls from async methods of SMS, Email and Push
Thread.sleep(10 mins) between each subsequent fetch operation of data batches <== This is the dirty hack that's bothering me a lot
If I remove Thread.sleep() then everything goes haywire because batch server just calls async methods and then fires up db call to fetch next batch of 500 users in very quick successions till the time db server stops responding.
I need help with what I shall be doing in order to get rid of 5th point while keeping things under control? I'm running mysql on RDS with 300 IOPS and 4 GB RAM (db.t3.medium)
How to implement dynamically updating vote count similar to quora:- Whenever a user upvotes an answer its reflected automatically for every one who is viewing that page.
I am looking for an answer that address following:
Do we have to keep polling for upvote counts for every answer, If yes
then how to manage the server load arising because of so many users
polling for upvotes.
Or to use websockits/push notifications, how scalable are these?
How to store the upvote/downvote count in databases/inmemory to support this. How do they control the number of read/writes. My backend database is mysql
The answer I am looking for may not be exactly how quora is doing it, but may be how this can be done using available opensource technologies.
It's not the back-end system details that you need to worry about but the front end. Having connection being open all the time is impractical at any real scale. Instead you want the opposite - to be able to serve and close connection from back-end as fast as you can.
Websockets is a sexy technology, but again, in real world there are issues with proxies, if you are developing something that should work on a variety of screens (desktop, tablet, mobile) it might became a concern to you. Even good-old long polls might not work through firewalls and proxies.
Here is a good news: I think
"keep polling for upvote counts for every answer"
is a totally good solution in this case. Consider the following:
your use-case does not need any real real-time updates. There is little harm to see the counter updated a bit later
for very popular topics you would like to squash multiple up-votes/down-votes into one anyway
most of the topics will see no up-vote/down-vote traffic at all for days/weeks, so keeping a connection open, waiting for an event that never comes is a waste
most of the user will never up-vote/down-vote that just came to read a topic, so your read/write ration of topics stats will be greatly skewed toward reads
network latencies varies hugely across clients, you will see horrible transfer rates for a 100B http responses, while this sluggish client is fetching his response byte-by-byte your precious server connection and what is more importantly - thread on a back end server is busy
Here is what I'd start with:
have browsers periodically poll for a new topic stat, after the main page loads
keep your MySQL, keep counters there. Every time there is an up/down vote update the DB
put Memcached in front of the DB as a write-through cache i.e. every time there is an up/down vote update cache, then update DB. Set explicit expire time for a counter there to be 10-15 minutes . Every time counter is updated expire time is prolongated automatically.
design these polling http calls to be cacheable by http proxies, set expire and ttl http headers to be 60 sec
put a reverse proxy(Varnish, nginx) in front of your front end servers, have this proxy do the caching of the said polling calls. These takes care of the second level cache and help free up backend servers threads quicker, see network latencies concern above
set-up your reverse proxy component to talk to memcached servers directly without making a call to the backend server, yes if your can do it with both Varnish and nginx.
there is no fancy schema for storing such data, it's a simple inc()/dec() operation in memcached, note that it's safe from the race condition point of view. It's also a safe atomic operation in MySQL UPDATE table SET field = field + 1 WHERE [...]
Aggressive multi level caching covers your read path: in Memcached and in all http caches along the way, note that these http poll requests will be cached on the edges as well.
To take care of the long tail of unpopular topic - make http ttl for such responses reverse proportional to popularity.
A read request will only infrequently gets to the front end server, when http cache expired and memcached does not have it either. If that is still a problem, add memecached servers and increase expire time in memcached across the board.
After you done with that you have all the reads taken care of. The only problem you might still have, depending on the scale, is high rate of writes i.e. flow of up/down votes. This is where your single MySQL instance might start showing some lags. Fear not - proceed along the old beaten path of sharding your instances, or adding a NoSQL storage just for counters.
Do not use any messaging system unless absolutely necessary or you want an excuse to play with it.
Websockets, Server Sent Events (I think that's what you meant by 'push notifications') and AJAX long polling have the same drawback - they keep underlying TCP connection open for a long time.
So the question is how many open TCP connections can a server handle.
Basically, it depends on its OS, number of file descriptors (a config parameter) and available memory (each open connection reserves a read/write buffers).
Here's more on that.
We once tested a possibility to keep 1 million websocket connections open on a single server (Windows 7 x64 with 16Gb of RAM, JVM 1.7 with 8Gb of heap, using Undertow beta to serve Web requests).
Surprisingly, the hardest part was to generate the load on the server )
It managed to hold 1M. But again the server didn't do something useful, just received requests, went through protocol upgrade and kept those connections open.
There was also some number of lost connections, for whatever reason. We didn't investigate. But in production you would also have to ping the server and handle reconnection.
Apart from that, Websockets seem like an overkill here, SSE still aren't widely adopted.
So I would go with good old AJAX polling, but optimize it as much as possible.
Works everywhere, simple to implement and tweak, no reliance on an external system (I had bad experience with that several times), possibilities for optimization.
For instance, you could group updates for all open articles in a single browser, or adjust update interval according to how popular the article is.
After all it doesn't seem like you need real-time notifications here.
sounds like you might be able to use a messaging system like Kafka, or RabbitMQ, or ActiveMQ. Your front end would sent votes to a message channel and receive them with a listener, and you could have a server side piece persist the votes to the db periodically.
You could also accomplish your task by polling your database, and by incre/decre menting a number related to a post via a stored proc... there are a bunch of options here and it depends on how much concurrency you may be facing.
I have a text box in my application which allows a user to select a location with the help of UI autocomplete. There are around 10,000 valid locations out of which the user must select one. There are two implementations for the autocomplete functionality:
Fetch the list of locations when the page loads for the first time and iterate over the array to find matching items on every keystroke in javascript
Make ajax requests on every keystroke as searching in MySQL(the db being used) is much faster?
Performance wise, which one is better?
An initial test shows that loading the data at once is the better approach from a performance point of view. However, this test was done on a MBP where JavaScipt processing is quite fast. I'm not sure whether this technique is the better one for machines with low processing power like lower end android phones, old systems etc.
Your question revolves around which is quicker, processing over 10,000 rows in the browser, or sending a request to a remote server to return the smaller result set. An interesting problem that depends on context and environment at runtime. Sending to the remote server incurs network delay mostly, with small amounts of server overhead.
So you have two variables in the performance equation, processing speed of the client and network latency. There is also a third variable, volume of data, but this is constant 10k in your question.
If both client browser and network are fast, use whatever you prefer.
If the network is faster, use the remote server approach, although be careful not to overload the server with thousands of little requests.
If the client is faster, probably use the local approach. (see below)
If both are slow, then you probably need to chose either, or spend lots of time and effort optimizing this.
Both clients slow can easily happen, my phone browser on 3G falls into this category, network latency for a random Ajax request is around 200mS, and it performs poorly for some JavaScript too.
As user perceieved performance is all that really matters, you could preload the first N values for each letter as variables in the initial page load, then use these for the first keystroke results, this buys you a few mS.
If you go with the server approach, you can always send requested result AND a few values for each of the next keystroke. This overlaps what users see and makes it appear snappier on slow networks. Eg
Client --> request 'ch'
Server responds with a few result for each potential next letter
'cha' = ...
'chb' = ...
Etc
This of course requires some specialized javascript to alternate between Ajax requests and using cached results from previous requests to prefill the selection.
If you are going with the local client searching through all 10k records, then make sure the server returns the records in sorted order. If your autocomplete scanning is able to use 'starting with' selection rather than 'contains' (eg typing RO will match Rotorua but not Paeroa) then you can greatly reduce processing time by using http://en.wikipedia.org/wiki/Binary_search_algorithm techniques, and I'm sure there are lots of SO answers on this area.
If there is no advantage for querying the backend every time, don't do it.
What could be an advantage of querying the backend all the time? If the amount of returned data for the initial call is to heavy (bandwidth, javascript processing time to prepare it, time at all), the partial request every time could be the smarter option.
I wrote a web application using python and Flask framework, and set it up on Apache with mod_wsgi.
Today I use JMeter to perform some load testing on this application.
For one web URL:
when I set only 1 thread to send request, the response time is 200ms
when I set 20 concurrent threads to send requests, the response time increases to more than 4000ms(4s). THIS IS UNACCEPTABLE!
I am trying to find the problem, so I recorded the time in before_request and teardown_request methods of flask. And it turns out the time taken to process the request is just over 10ms.
In this URL handler, the app just performs some SQL queries (about 10) in Mysql database, nothing special.
To test if the problem is with web server or framework configuration, I wrote another method Hello in the same flask application, which just returns a string. It performs perfectly under load, the response time is 13ms with 20-thread concurrency.
And when doing the load test, I execute 'top' on my server, there are about 10 apache threads, but the CPU is mostly idle.
I am at my wit's end now. Even if the request are performed serially, the performance should not drop so drastically... My guess is that there is some queuing somewhere that I am unaware of, and there must be overhead besides handling the request.
If you have experience in tuning performance of web applications, please help!
EDIT
About apache configuration, I used MPM worker mode, the configuration:
<IfModule mpm_worker_module>
StartServers 4
MinSpareThreads 25
MaxSpareThreads 75
ThreadLimit 64
ThreadsPerChild 50
MaxClients 200
MaxRequestsPerChild 0
</IfModule>
As for mod_wsgi, I tried turning WSGIDaemonProcess on and off (by commenting the following line out), the performance looks the same.
# WSGIDaemonProcess tqt processes=3 threads=15 display-name=TQTSERVER
Congratulations! You found the performance problem - not your users!
Analysing performance problems on web applications is usually hard, because there are so many moving parts, and it's hard to see inside the application while it's running.
The behaviour you describe is usually associated with a bottleneck resource - this happens when there's a particular resource that can't keep up, so queues requests, which tends to lead to a "hockey stick" curve with response times - once you hit the point where this resource can't keep up, the response time goes up very quickly.
20 concurrent threads seems low for that to happen, unless you're doing a lot of very heavy lifting on the page.
First place to start is TOP - while CPU is low, what's memory, disk access etc. doing? Is your database running on the same machine? If not, what does TOP say on the database server?
Assuming it's not some silly hardware thing, the next most likely problem is the database access on that page. It may be that one query is returning literally the entire database when all you want is one record (this is a fairly common anti pattern with ORM solutions); that could lead to the behaviour you describe. I would use the Flask logging framework to record your database calls (start, end, number of records returned), and look for anomalies there.
If the database is performing well under load, it's either the framework or the application code. Again, use logging statements in the code to trace the execution time of individual blocks of code, and keep hunting...
It's not glamorous, and can be really tedious - but it's a lot better that you found this before going live!
Look at using New Relic to identify where the bottleneck is. See overview of it and discussion of identifying bottlenecks in my talk:
http://lanyrd.com/2012/pycon/spcdg/
Also edit your original question and add the mod_wsgi configuration you are using, plus whether you are using Apache prefork or worker MPM as you could be doing something non optimal there.
We are developing an application which will require to send around 30 outbound emails per second. We have a server running SMTP but this machine in cloud hosted and I do not have any idea what kind of configuration will I require to support such a load. I do not even know if this load is considered to be average or high. Do i need to do anything special for such a load. Do i need a dedicated quad core server for this kind of load or lets say just 1/10th CPU of a quad core server is good enough
Hm
what for?
30 emails per second is nothing. I wrote a server like 10 years ago hitting about 5000 per second (to one other server taking it down in the process - custoemr wanted as fast as possible, i delivered).
Get any little MTA and jst use it. No sense in writing something yourself for that low volume.
Unless you hit the server with a lot of stuff at once (loading it for transfer), a small VPS should be ok.
Seriously, 30 emails per second is what I sometimes send from my dialup account. THis is not even a visible volume for a decent message transfer agent. It is definitely NOT "high volume".
Going to echo TomTom on this one and say just to get one of the many services out there that will help you do this. It's probably far easier to utilize one of their services and not have to worry about reputation monitoring and all the fun stuff of SMTP servers than to create your own solution.
Let me know if you need help finding these services.
(Full Disclosure: I work for PostageApp.com, and we're rolling out a hosted SMTP service soon!)