I'm watching this video about Akka.net and the speaker says read after right does not produce consistent results because the order of events is not predictable at the network level. The arhcitecture the presenter is speaking about in this video is as follows:
One Load Balancer
Multiple web servers. Load balancer determines which server to hit.
One database server (SQL Server).
I'm confused to why consistent results are not achieved with a single databse? If a lock is put before data is written wouldn't that bring you back consistent results?
So I'm going to guess you're talking about the scenario Aaron describes about 10 minutes into this video. Here's the scenario:
User is clicking things on a site and we're firing off asynchronous requests to record the clicks.
The not obvious part from the scenario he's describing is that we're not waiting for the previous requests to finish before sending more requests to capture a user's clicks (imagine a single page app where clicks don't cause a full refresh of the page from the server). We want to capture all the clicks.
We have some logic on the server that says, "If the user clicks these 3 things in a row, do some cool reaction..."
We check our condition on the web server ("Has the user clicked these 3 things in a row?") by writing the click event we just got to our DB, then reading to see if they've generated the stream of 3 things clicked to do our cool reaction.
Here's the problem: each request to record a click could be going to a different web server and we're not waiting on the previous one to finish before we send more requests to record clicks. So we have no guarantee that the request to write the first event has completed before we write the second, or the third, etc.
For example, the first request could be delayed (or even fail!) because of a faulty network, so the second request could reach our SQL Server first! And such, when it goes to read the stream of events that's happened, it could not be aware that a request was sent (but hasn't completed) to record that the first event happened.
I think the point he's trying to make is that in the face of multiple clients (in this example, web servers) writing to a database concurrently, you can't count on, "I sent that first so it will be recorded first". This holds true whether you're using DataStax Enterprise, Cassandra, SQL Server, Oracle, or whatever. Hope that helps!
Related
I'm building a mobile app where a pull-down gesture on the UI initiates an update of existing data/posts (also retrieves new posts if there are any, but that's not the point here). The server is stateless meaning there is no sessions.
If the posts have been updated in the database, how do I let the front-end know which posts need to be updated? Only way I could think of is to send a list of ids of all retrieved posts to the server, and have it check if any of the posts have been modified since the time fetched.
This however seems quiet inefficient as the users might have stacked up hundreds of posts in some extreme cases, and it's most likely that only few or none of the posts need to be updated. Issuing hundreds of db requests could be a huge overhead.
There are at least 2 ways of doing this
Long Polling
client requests server for new information.
the server keeps the connection open until there is some new data to send.
once the server gets data, it sends this to the client and connection is closed.
the client then sends a new request for information.
This is a continuous process.
WebSockets
Create a websocket connection, keep it open.
The server pushes any updates as and when they come.
Problems with both situations
May take a significant amount of time to have production ready implementations.
Both them will require the server to be aware of any change in the database. This can be tricky as well
I've built an MMORPG that uses a MySQL database to store player related data when the user logs off.
We built in a auto save timer so that all the data of every logged in user is saved to the database every 3 hours.
In doing so we noticed a fatal flaw....
Due to the fact that all our database transactions are sent to a single DB Thread the thread can become backlogged with requests. This produces a login/saving issue. When this happens players unable to login as the login process requires the use of the DB Thread to confirm login credentials. Similarly all save requests are queued to the back of the DB thread schedule. This produces a backlog of requests...
The only solution that I can think of for this is to introduce multiple threads and have 3-4 threads interacting with the database.
However, this opens up a new issue. Since multiple threads are sent DB requests this means that one thread can receive a save request from a player while another DB thread receives a save request from the same player.
For example....
PlayerA Logs In to the game
3 Hours pass & the auto save happens, playerA's data will now be saved.
PlayerA kills a monster and gains experience.
PlayerA logs off, which adds a save request to a DB thread.
Now we have two different save requests queue'd in the database. Assuming they are both assigned to two different DB threads, this could cause the users data to be saved in the wrong order... For example maybe the the thread handling PlayerA's log out save runs first and then the auto save for PlayerA runs after that on a separate thread.... This would cause loss of data (in this case experience).
How do other MMORPG's handle something like this?
You need a database connection pool if you're not using one already and make sure you're not locking more data than you need. If you are saving how much gold a player has, you don't need to lock the table holding the credentials.
Keeping the order of events in a multi-threaded scenario is not a trivial problem, I suggest using a message queue, a single producer per player and a single consumer per player. This link shows 2 strategies to keep the order.
A queue is actually important for other reasons. If a save request fails, it would remain in the queue to retry later. When dealing with players money and items, you probably want this.
Your autosave is deterministic, meaning that you know exactly when the last one occured and when the next one would occur. I would use that somehow, along with the previously suggested idea to add a timestamp. Actually, it might be better to make the updates represent only the increments/decrements along with a user timestamp and calculate the experience upon request ( maybe cache it then)
To avoid this problem in all cases you must not allow users to continue doing stuff before their last database transaction has been successfully committed. Of course that means that the DB has to be very fast -- if it can't keep the request queue below a couple of seconds worth of transactions at most, you simply have to make it faster. More RAM cache, SSDs, the usual MySQL optimization dance. Adding extra logic in the form of triggers etc. isn't going to help in the long run, especially because they can become really complicated in the case of inventories and the like.
If on average the system is fast enough but struggling in peaks like when everybody logs in during lunch break, adding something like Redis as a fast cache might help. You'd load the data into Redis when a user logs on (or when they first need a certain piece of data) , remove it when they log off or when it expires, and write changes back to the relational DB as fast as it can keep up.
Environment:
Windows Server 2003 - IIS 6.x
ASP.NET 3.5 (C#)
IE 7,8,9
FF (whatever the latest 10 versions are)
User Scenario:
User enters search criteria against large data-set. After initiating the request, they are navigated to a results page, where they wait until the data is loaded and can then refine the data.
Technical Scenario:
After user sends search criteria (via ajax call), UI calls back-end service. Back-end service queries transactional system(s) and puts the resulting data into a db "cache" - a denormalized table, set-up for further refining the of the data (i.e. sorting, filtering). UI waits until the data is cached and then upon getting notified that the process is done, navigates to a resulting page. The resulting page then makes a call to get the data from the denormalized table.
Problem:
The search is relatively slow (15-25 seconds) for large queries that end up having to query many systems based on the criteria entered. It is relatively fast for other queries ( <4 seconds).
Technical Constraints:
We can not entirely re-architect this search / results system. There are way to many complexities here between how the UI and the back-end is tied together. The page is required (because of constraints that can not be solved on StackOverflow) to turn after performing the search criteria.
We also can not ask the organization to denormalize the data prior to searching because the data has to be real-time, i.e. if a user makes a change in other systems, the data has to show up correctly if they do a search afterwards.
Process that I want to follow:
I want to cheat a little. I want to issue the "Cache" request via an async HttpHandler in a fire-forget model.
After issuing the query, I want to transition the page to the resulting page.
On the transition page, I want to poll the "Cache" table to see if the data has been inserted into it yet.
The reason I want to do this transition right away, is that the resulting page is expensive on itself (even without getting the data) - still 2 seconds of load time before even getting to calling the service that gets the data from the cache.
Question:
Will the ASP.NET thread that is called via the async handler reliably continue processing even if I navigate away from the page using a javascript redirect?
Technical Boundaries 2:
Yes, I know... This search process does not sound efficient. There is nothing I can do about that right now. I am trying to do whatever I can to get it to perform a little better while we continue researching how we are going to re-architect it.
If your answer is to: "Throw it away and start over", please do not answer. That is not acceptable.
Yes.
There is the property Response.IsClientConnected which is used to know if a long running process is still connected. The reason for this property is a processes will continue running even if the client becomes disconnected and must be manually detected via the property and manually shut down if a premature disconnect occurs. It is not by default to discontinue a running process on client disconnect.
Reference to this property: http://msdn.microsoft.com/en-us/library/system.web.httpresponse.isclientconnected.aspx
update
FYI this is a very bad property to rely on these days with sockets. I strongly encourage you to do an approach which allows you to quickly complete a request that notes in some database or queue of some long running task to complete, probably use RabbitMQ or something like that, that in turns uses socket.io or similar to update the web page or app once completed.
How about don't do the async operation on an ASP.NET thread at all? Let the ASP.NET code call a service to queue the data search, then return to the browser with a token from the service, where it will then redirect to the result page that awaits the completed result? The result page will poll using the token from the service.
That way, you won't have to worry about whether or not ASP.NET will somehow learn that the browser has moved to a different page.
Another option is to use Threading (System.Threading).
When the user sends the search criteria, the server begins processing the page request, creates a new Thread responsible for executing the search, and finishes the response getting back to the browser and redirecting to the results page while the thread continues to execute on the server background.
The results page would keep verifying on the server if the query execution had finished as the started Thread would share the progress information. When it does finish, the results are returned when the next ajax call is done by the results page.
It could also be considered using WebSockets. In a sense that the Webserver itself could tell the browser when it is done processing the query execution as it offers full-duplex communications channels.
I need to do a live chat system (with multiple user channels, user permissions and must be included in a site and use accounts from that database) so I thought at this solutions: ajax requests at a predefined time like 1 second, long polling or irc.
The advantages and disadvantages would be:
AJAX advantages:
Easy to implement
East to check permissions for the site users, give rights, set channels, access everything I need from the database
Disadvantages:
Inserts lag by default
Kills the poor server
Can be used only in the specified page (no outside site client exists)
Long polling:
Doesn't kill the server
Less lag
Can be used only in the specified page (no outside site client exists)
Harder to implement
IRC:
Doesn't flood the server
No lag
A user can set a client and access chat from outside site
Don't know how to communicate with my database so I can create channels and give permissions according to my data
Since multiple ajax requests flood the server I can't use that. So between long polling and IRC what do you think it should be better to use?
If is long polling can you please point to a good reference (I used ape - ape-project.org in the past but I was disappointed by its stability)?
If is IRC can you please point me to a reference that shows how to create a connection to my database (mysql) and put the new logged user into a desired channel? For example if in my database I have an entry like name: Gogu, occupation: killer; when Gogu connects I need to put him in the "Killers" channel.
On the customizable front page of our web site, we offer users the option of showing modules showing recently updated content, choosing from well over 100 modules.
All of the data is generated by MySQL queries, the results of which are cached via memcached. Our current system works like this: when a user load a page containing modules, module, they are immediately served the data from cache, and the query is added to a queue to be updated by a separate gearman process (so that the page load does not wait for the mysql query). That query is then run once every 15 minutes to refresh the data in cache. The queue of queries itself is periodically purged so that we do not continually refresh data that has not been requested recently.
The problem is what to do when the cache is empty, for some reason. This doesn't happen often, but when it does, the user is currently shown an empty module, and the data is refreshed in the gearman process so that a bit later, when the same (or a different) user reloads the page, there is data to show.
Our traffic is such that, if we were to try to run the query live for the user when the cache is empty, we would have a serious problem with stampeding--we'd be running the same (possibly slow) query many times as many users loaded the page. Is there any way to solve the "blank module" problem without opening up the risk of stampeding?
This is an interesting implementation though varies a bit from the way most typically implement memcached in fronT of MySQL.
In most cases users will set things up to where queries are first evaluated at memcached to see if there is is an available entry. If so they server it from memcached and never query the database at all. If there is a cache miss, then the query is made against the database, the results added to memcached, and the information returned to the caller. This is how you would typically build up your cache for read queries.
In cases where data is being updated, the update would be made against the database, and then the appropriate data in memcached invalidated and/or updated. Similarly for inserts, you could either do nothing regarding the cache (and let the next read on that record populate the cache), or you could actively add the data related to the insert into the cache, depending on your application needs.
In this way you wouldn't need to take the extra step of calling the database to get authoritative data after getting initial data from memcached. The data in memcached would be a copy of the authoritative data which is just updated/invalidated upon updates/inserts.
Based on your comments, one thing you might want to try in order to prevent a number of of queries on your database in case of cache misses is to use a mutex of sorts. For example, when the first client hits memcached and gets a cache miss for that lookup, you could could insert a temporary value in memcached indicating that the data is pending, then make the query against the database, and the update the memcached data with the result.
On the client side, when you get a cache miss or a "pending" result, you could simply initiate a retry for the cache after a certain period of time (which you may want to increase exponentially). So perhaps first hey wait for 1 second, then try back gain in 2 seconds if they still get a "pending" results, then retry in 4 seconds, and so on.
This would amount in possibly more requests against the memcached server, but should resolve any problems on the database layer.