Okay, so I have this small procedural SVG editor in Clojure.
It has a code pane where the user creates code that generates a SVG document, and a preview pane. The preview pane is updated whenever the code changes.
Right now, on a text change event, the code gets recompiled on the UI thread (Ewwwww!) and the preview pane updated. The compilation step should instead happen asynchronously, and agents seem a good answer to that problem: ask an agent to recompile the code on an update, and pass the result to the image pane.
I have not yet used agents, and I do not know whether they work with an implicit queue, but I suspect so. In my case, I have zero interest in computing "intermediate" steps (think about fast keystrokes: if a keystroke happens before a recompilation has been started, I simply want to discard the recompilation) -- ie I want a send to overwrite any pending agent computation.
How do I make that happen? Any hints? Or even a code sample? Is my rambling even making sense?
Thanks!
You describe a problem that has more to deal with execution flow control rather than shared state management. Hence, you might want to leave STM apart for a moment and look into futures: they're still executed in a thread pool as agents, but instead of agents they can be stopped by calling future-cancel, and inspecting their status with future-cancelled?.
There are no strong guarantees that the thread the future is executing can be effectively stopped. Still, your code will be able to try to cancel the future, and move on to schedule the next recompilation.
agents to indeed work on a queue, so each function gets the state of the agent and produces the next state of the agent. Agents track an identity over time. this sounds like a little more than you need, atoms are a slightly better fit for your task and used in a very similar manner.
Related
When using distributed and scalable architecture, eventual consistency is often a requirement.
Graphically, how to deal with this eventual consistency?
Users are used to click save, and see the result instantaneously... with eventual consistency it's not possible.
How to deal with the GUI for such scenarios?
Please note the question applies both for desktop applications and web applications.
PS: I'm working with the Microsoft platform, but I imagine the question applies to any technology...
A Task Based UI fits this model great. You create and execute tasks from the UI. You can also have something like a task status monitor to show the user when a task has executed.
Another option is to use some kind of pooling from the client. You send the command, and pool from the client until the command completed and the new data is available. You will have a delay in some cases from when the user presses save to when he will see the new record, but in most cases it should be almost synchronous.
Another (good?) option is to assume/design commands that don't fail. This is not trivial but you can have a cache on the client and add the data from the command to that cache and display it to the user even before the command has been executed. If the command fails for some unexpected situation, well then just design a good "we are sorry" message for misleading the user for a few seconds.
You can also combine the methods above.
Usually eventual consistency is more of a business/domain problem, and you should have your domain experts handle it.
I think that other answers mix together CQRS in general and eventual consistency in particular. Task-based UI is very suitable for CQRS but it does not resolve the issue with eventually consistent read model.
First, I would like to challenge your statement:
Users are used to click save, and see the result instantaneously... with eventual consistency it's not possible.
What do you by this? Why is it not possible to see the result immediately? I think the issue here is your definition of result.
The result of any action is that that action has been performed. There are numerous of ways to show this! It depends on what kind of action do you want to complete. Examples:
Send an email: if user has entered a correct email address, it is almost guaranteed that the action will complete successfully. To prevent unexpected failures one might use durable queues since this kind of actions do not need to be done synchronously. So you just say "email sent". Typically you see this kind of response when you ask to reset your password.
Update some information in a user profile: after you have validated the new data on the client, most probably the command will succeed too since the only thing that could happen is the database error (if you use database). Again, even this can be mitigated by using durable queues. In this case you just show the updated field in the same form. The good practice for SPA is to have a comprehensive data store on the client side, like Redux does. In this case you can safely update the server by sending a command and also updating the client-side store, which will result in UI to shows the latest data. Disclaimer: some answers refer to this technique as "tricking the user", but I disagree with this definition.
If you have commands that are prone to error, you can use techniques that are already described in other answers like Websockets or Server-side events to communicate errors back. This requires quite a lot of additional work. You can also send a command and wait for reply or execute commands synchronously. Some would say "this is not CQRS" but this would be just another dogma to be challenged. Ensuring the command has completed the execution in combination with the previous point (client-side data store) will be a good solution.
I am not sure if there is any 100% bullet proof technique that allows you to always show non-stale data from the read model. I think it goes against the principles of CQRS. Even with real-time events you will only get events that indicate that you write model has been updated. Still, your projections could have failed and reacting on this is a whole other story.
However, I would not concentrate that much on this issue. The fact is that well-tested projections and almost-guaranteed commands will work very well. For error handling in 90% of situations it is enough to have some manual or half-manual process to recover from those errors. For the last 10% you can combine generic "error" messages pushed from the server saying "sorry, your action XXX has failed to execute" and the top priority actions could have some creative process behind them but in reality those situations would be very very rare.
There are 2 ways:
To trick a user (just to show that things has happened then they
really hasn't happened yet)
Show that system is processing request
and use polling in background (not good) or just timer with value of
your SLA.
I prefer the 1st option.
As someone has already mentioned, task based UI's fit well for this, and what I would do is employ a technique that 'buys you time' for the command to propagate.
For example, imagine we are on a list screen, where the user can perform various actions, one of which being to add a new item to the list. After choosing to add an item you could display a "What would you like to do next?" which could have 'Add another item', 'Do this task', 'Do some other task', 'Go back to list'.
By the time they have clicked on an option, the data would have hopefully been refreshed.
Also, if you're using a task based UI, you can analyse the patterns of task execution and use these "what would you like to do next" screens to streamline the UI. Similar to amazon's "other people also bought these items".
As previously stated, it is fine to tell the user that the request (command) has been acknowledged (successfully issued). In case of some failure, the system should communicate this to the requester, by means of:
email;
SMS;
custom inbox (e.g. like the SO inbox);
whatever.
E.g., mail client / service:
I am sending a mail to a wrong address;
the mail service says: "email sent successfully :)";
after few minutes, I receive a mail from the service: "email could not be delivered".
I believe a great way to inform the user about a recent failure is to present him an error panel while he's navigating through the application. A user gesture might be required in order to dismiss that alert etc.
For example:
I wouldn't go with tricking the user or blocking him from committing some other actions. I would rather go for streaming data toward UI after they are being acknowledged by a read side. Let's consider these two cases:
Users saves data and expects result. Connection is established toward server. After they are being acknowledged by a read side, they are streamed toward UI and UI is being updated.
User saves data and refreshes web page. Upon reload, data are being fetched from data store and connection for streaming is established. If read side didn't update the data store in the meantime, there's still an opened stream and UI should be updated after data reaches the read side.
Why streaming from read side and not directly from write side? Simply, that would be a confirmation that read side has been reached.
From technical aspect, Server-Sent Events could be used.
Disadvantage:
Results will still not be reflected immediately by a read side. But at least, in most cases, user will be able to continue with his work without being blocked by a UI.
There are several ways to handle eventual consistency. All of them are really to occupy the time from the User's action until the backend refresh.
User Reads A given user can only read from the same database node that they write to. Other users read from the replicated nodes. PROS: UI is quick enough, and application stays in sync. CONS: Your service architecture has to track and route Users to specific database nodes.
Disable the UI until the action has completed, and refresh it. Java Server Faces has a classic example of this. One could create a modal with a loading spinner to cover the UI until the refresh was completed. PROS: UI stays in sync with application state. CONS: Most every action creates a blocked UI. Users get very frustrated by the restricted UI, and will complain of application slowness.
Confirmation Immediately thank the user for their submission. Then let them know later (email, SMS, in-app notification) whether or not the action was completed. PROS: It's fast up front. CONS: UI lags behind system until refresh. Even with a notice, the User may get confused that they don't see the updates. It also requires integration of various communication channels. Users won't see their changes right away. If the action fails, they may not know until it's too late.
Fake it Optimistically assume that the action will complete. Show the User the resulting UI (upvote, comment, credit card confirmation, etc) and allow them to continue as if it succeeded. If there were failures, immediately show them as contextual errors: alerts next to the undone upvotes, in-app alert on the post with the failed comment, email for the declined credit card. PROS: UI feels much faster. CONS: UI is temporarily out of sync with application state, and you must resolve that. One case: you might fake creation of content with temp IDs. But after content is created, then the temp IDs will be wrong until the refresh. Second case, you might need to store all state changes on the UI after the action until the refresh. Then you need some Resolver to apply all the local state changes since the action was issued. This is resolution is non-trivial.
Web Sockets Subscribe the UI to an event stream so that when the action is completed on the backend, it is pushed to the front end. Is it one-way or two-way streaming? PROS: UI feels fast, and it's in sync with the application state. CONS: Consistent browser support, need a backend source of streaming events, and socket server scalability.
I am practicing reversing skill using OLLdbg under windows.
there is an interactive window asking you input, let's say "serial number". My question is when user operate on the window, it is hard to locate related data flow within the debugger window. For example, if I click "F9", we can view the instruction flow; but When inputing on the window, I can't know which instructions have been executed.
My target is to find some jump instruction and change it, so that I can bypass the correct input requirement. I think the instruction should be quite close to instruction related to arg#, and related to TEST command.
Looking for hint or trick. Thanks.
One thing you could do is type something in the text field and then use an application such as Cheat Engine to find out where in the memory these characters are stored. Then you can put a memory (on access) breakpoint on the address of the first character in ollydbg. Then press the button that verifies the serial. When an instructions accesses this part of the memory it will break. You're inside a part of the code that verifies your string. Now from here you have to try to understand what the code is doing to find the instruction you want to alter.
Depending on how secure the application is, this will work. With a more secure application this most likely won't work. When your just starting reverse engineering I suggest you find some easy applications made for cracking and work your way to the more secure applications. A site where you can find many of these "crackmes" is crackmes.de. Also i can suggest lene151's tutorials here. Some of the best tutorials I've seen on reverse engineering.
I'm experimenting with P2P on Flash, and I've come across a little hurdle that I'd like to clarify before moving forward. The technology itself (Flash) doesn't matter for this problem, as I think this problem occurs in other languages.
I'm trying to create a document that can be edited "live" by multiple people. Just like Google Docs pretty much. But I'm wondering, how would you suggest synchronizing everyone's text? I mean, should I message everyone with all the text in the text field every time someone makes a change? That seems very inefficient.
I'm thinking there has to be a design pattern that I can learn and implement, but I'm not sure where to start.
Optimally, the application should send the connected clients only the changes that have occurred to the document, and have some sort of buffer or error correction that can be used for retrieving earlier changes that may have been missed. Is there any established design pattern that deals with this type of issue?
Thanks,
Sandro
I think your "Optimally" solution is actually the one you should go for.
each textfield has a model, the model has a history (a FILO storing last, let's say, 10 values).
every time you edit that textfield you push the whole text into the model and send the delta to other connected clients.
as other clients receive the data they just pick the last value from the model and merge it to the received data.
you can refine the mechanism by putting an idle timer in the middle: as a user types something in the textfield you flag that model as "toBeSentThroughTheNet" and you start a timer. as the timer "ticks" (TimerEvent.TIMER) you stop it, collect the flagged data and send it to other clients. just remember to reset the timer everytime the user is actually typing (a semplification coul be keydown = reset, keyup = start).
one more optimization could be send the data packed in a compressed bytearray, but this requires you write your own protocol and may be not so an easy and quick path :)
If the requirement is that everyone can edit the document at the same time and the changes should be propagated to everyone and no changes should be lost, then it is a non-trivial problem. There are few different approaches out there, but one that is quite robust is Operational Transformation. This is the same algorithm that Google Docs uses for collaborative editing.
Understanding and Applying Operational Transformation and the attendant hacker news discussion are probably other good places to start.
The Wave Protocol was released as open source so you can take a look on how it is implemented.
You could of course forgo the tricky synchronization and just allow people to take turns and only one person can edit the document at a time and this person just pushes the changes to the remainder of the group.
I'm working with a time sensitive desktop application that uses p/invoke extensively, and I want to make sure that the code is not wasting a lot of time on CAS stackwalks.
I have used the SuppressUnmanagedCodeSecurity attribute where I think it is necessary, but I might have missed a few places. Does anyone know if there is a way to monitor the number of CAS stackwalks that are occurring, and better yet pinpoint the source of the security demands?
You can use the Process Explorer tool (from Sysinternals) to monitor your process.
Bring up Process Explorer, select your process and right click to show "Properties". Then, on the .NET tab, select the .NET CLR Security object to monitor. Process Explorer will show counters for
Total Runtime Checks
Link Time Checks
% Time in RT Checks
Stack Walk Depth
These are standard security performance counters described here ->
http://msdn.microsoft.com/en-us/library/adcbwb64.aspx
You could also use Perfmon or write your own code to monitor these counters.
As far as I can tell, the only one that is really useful is item 1. You could keep an eye on that while you are debugging to see if it is increasing substantially. If so, you need to examine what is causing the security demands.
I don't know of any other tools that will tell you when a stackwalk is being triggered.
Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 4 years ago.
Improve this question
I'm interested in how the protocols (and game loop) work for these type of games; any pointers or insights are appreciated.
I guess the main loop would have a world state which would be advanced a few "ticks" per second, but how are the commands of the players executed? What kind of data needs to go back and forth?
I can go into a lot of detail about this but first, go read "1500 archers" http://www.gamasutra.com/view/feature/3094/1500_archers_on_a_288_network_.php and this will answer many of your questions. Here's a summary:
First, most games use UDP due to the real-time nature of the game. THe game loop looks something like this:
read network data
do client-side predictions and compare with where the network says
your objects should actually be
mess with physics to fudge what the network says with what your
local game state is
send data back out onto the network based on what you did this
frame (if anything)
render
That's vastly simplified and "messing with physics" could easily be a 200 page book on its own but it involves predicting client-side where something is likely to be, getting data from the server that is old but tells exactly where an object was/should be, and then interpolating those values somehow to make the object appear "close enough" to where it's actually supposed to be that no one notices. This is super-critical in first person shooters but not as much for real-time strategy.
For real-time strategy, what typically happens is a turn-based system where time is divided into discreet chunks called "turns" that happen sequentially and each turn has a number generated by a monotonic function that guarantees ever increasing values in a particular order without duplicates. On any given turn n, each client sends a message to all other clients with their intended action on turn n + m, where m is an arbitrary number that is usually fairly small and can be best determined through trial and error as well as playtesting. Once all the clients have sent their intended action, each client executes all actions that were sent on turn n + m. This introduces a tiny delay in when an action is ordered by the user and when it executes, however this is usually not noticable.
There are several techniques which can be used to fudge the time as well. For example, if you highlite a unit and then tell it to move, it will make a sound and have an animation when it starts moving but won't actually move right away. However, the network message of an intent to move that unit is sent immediately so by the time the screen responds to the player's input, the network messages have already been sent and acknowledged. You can fudge it further by introducing a small delay (100ms or so) between the mouse click and the game object's response. This is usually not noticable by the player but 100ms is an eternity in a LAN game and even with a broadband connection on a home computer the average ping is probably around 15-60ms or so, which gives you ample time to send the packet prior to the move.
As for data to send, there are two types of data in games: deterministic and non-deterministic. deterministic actions are grounded in game physics so that when the action starts, there is a 100% guarantee that I can predict the result of that action. This data never needs to be sent accross the network since I can determine what it will be on the client based on the initial state. Note that using a random number generator with the same seed on every client turns "random" events into deterministic behavior. Non-deterministic data is usually user input but it is possible to predict what a user's input is likely to be in many cases. The way these pair in a real-time strategy game is that the non-deterministic event is some sort of order to one of my game objects. Once the game object has been ordered to move, the way in which it moves is 100% deterministic. Therefore, all you need to send on the network is the ID of the object, the command given (make this an enum to save bandwidth), and the target of the command (if any, so a spell may have no target if it's an area of affet but a move command has an end-destination). If the user clicks like 100 times to make a unit move, there is no need to send a separate move command for each click since they're all in the same general area so be sure to filter this out as well since it will kill your bandwidth.
One final trick for handling a possible delay between a command and its execution is something called a local perception filter. If I get a move order some time t after the order was given, I know when the unit should have started moving and I know its end destination. Rather than teleporting the unit to get it where it's supposed to be, I can start its movement late and then mess with physics to speed it up slightly so that it can catch up to where it's supposed to be, and then slow it back down to put it in the correct place. The exact value you need to speed it up is also relative and playtesting is the only way to determine the correct value because it just has to "feel right" in order for it to be correct. You can do the same thing with firing bullets and missiles as well and it's highly effective for that. The reason this works is that humans aren't horribly good at seeing subtle changes in movement, particularly if an object is heading directly towards them or away from them, so they just don't notice.
The next thing to think about is cutting down on bandwidth. Don't send messages to clients that couldn't possible see or interact with a unit that is moving. Don't send the same message over and over again because the user clicks. Don't send messages immediately for events that have no immediate affect. Finally, don't require an acknowledgement for events that will be stale should they fail to be received. If I don't get a movement update, by the time I re-transmit that update, its value will be so old that it's no longer relevant so it's better to just send another move and use a local perception filter to catch up or use a cubic spline to interpolate the movement so that it looks more correct or something of that nature. However, an event that's critical, such as a "you're dead" or "your flag has been taken" should be acknowledged and re-transmitted if needed. I teach network game programming at Digipen so feel free to ask any other questions about this as I can probably provide you with an answer. Network game programming can be quite complicated but ultimately it's all about making choices in your implementation and understanding the consequences of your choice.
Check out Battle for Wesnoth.
http://www.wesnoth.org/
It's free, open source, and totally awesome. You can learn a lot from digging into its source.
Discussion of Age of Empires network architecture here
IMHO, that style of peer-to-peer duplicated-replay-based architecture is impressive, but a bit of a dead end for anything more than 8 or so players.