I am receiving a stream of time-series data/events from a manufacturing process machine. I need to be able to calculate the start of a cycle and pinpoint this event based on the trough of the data. What would be a recommended way to do this?
Thanks
Edit: I've attached an image of the data graph. Basically I have an ongoing stream of data that I need to be able to effectively identify the troughs in order to establish the start of the next cycle/end of the previous cycle. The only thing I'm guaranteed not to know is the period of the cycle - it will vary.
The only thing I'm guaranteed to know is the initial datapoint value - which could be anywhere in the cycle.
At the moment - I'm kind of thinking I have to do something with a sliding span of time, keeping tracking of the minimum temp seen so far.
Related
I’d like to query an IPFS network for the uptime of its nodes individually. That is, over a duration d I would like to know roughly how much time a node has been participating in the network. Instead of time, I believe it’s also safe to frame this problem in terms of work and query for the number of blocks added within a time period.
Is there a way to do this?
crosspost
The short answer is not really. See the response on discuss.ipfs.io for more detail.
If you are fine with a trustful system, you can potentially expose block add data through the RPC. But, this isn't supported out of the box.
Otherwise, you'll have to resort to some kind of polling and crawling it seems.
Our system requirements say that we need to build a slightly unusual producer-consumer processing system. Imagine we have multiple data streams and we take a snapshot each X seconds and put it into the queue for processing. The streams count is not constant. The more clients we have, the more streams we need to process. At the same time, we don't need to process ALL taken snapshots. If we have too many clients and we are not able to process all items in real-time, we would prefer to skip old snapshots and process only the latest ones.
So as I see, the requirements can be met by keeping only one item in a queue for each stream. If there is a new snapshot, while the previous is still there, we need to REPLACE it using stream id as a key.
Is it possible to implement such behavior by Service Bus queue or something similar? Or maybe it makes sense to look into some other solutions like Redis?
So as I see, the requirements can be met by keeping only one item in a
queue for each stream. If there is a new snapshot, while the previous
is still there, we need to REPLACE it using stream id as a key. Is it
possible to implement such behavior by Service Bus queue or something
similar?
To the best of my knowledge, Azure Service Bus does not support this scenario. Through it's duplicate detection functionality, in fact it supports exact opposite of that. You would need to use some other mechanism (like Redis Cache you mentioned) to accomplish this.
Every month we go thru the same thing. We receive the files from our client, the files need to be decrypted, they need to be loaded, they need to be processed and the output created.
We know that timeliness affects the quantity of our output.
I want to create a regression to see which process we need to be quicker with, which process contributes more to the output.
Let's say we should receive the files from our client on the 6th. I was thinking about creating a dataset of delays of each point from the 6th. However, because when each point starts depends on when the previous one ends, this would introduce covariance. How can I eliminate the covariance? Have each point be the delay from when the previous one ended?
Is a linear model the best approach for this? Are there other models? I've heard of Bayesian, I'm not really familiar with it, but I wasn't sure if it would be helpful.
I have a number of attributes I need for various page loads and other backend tasks, and I'm debating on whether storing these things in a database or calculating them on the fly.
For instance, if there are files that users can upload, and you want to track the size, space taken, format, etc. would it be better to calculate these things once and store them along with the location of the file in the database, or just grab the file each time and get the file attributes manually?
Another use case is shopping cart items. Is it better to calculate the price of an item and store that in a row with the shopping cart table, or calculate the given price each time a page loads. In this case, changes to the price based on site-wide sales, discounts, markups, etc. would not be reflected once the item has been added to the cart unless the prices are updated through another method when sales/discounts/markups are applied. This isn't the best example, but hopefully you understand the idea; maybe you have a better example.
In both of these examples, the source material is available to get the answers from which is key to the question. Obviously, one has a lot overhead for every page load would could be a lot depending on the situation, however the other seems to have less dependence on database integrity in terms of making sure it is always accurate and up-to-date (which I think I prefer). I'm not looking for a specific answer here, because I'm sure it will depend on many variables, but I am looking for a best practice or a method to determine the best solution.
NOTE: This is a similar question but has gotten very little response and no answers.
There can be trade offs between
When a user is waiting, elapsed time is critical.
A user will expect up-to-the-second pricing information.
A user will be frustrated if he orders something, only to find out that you are out-of-stock before his order is submitted.
We cannot say how fast it will be to recalculate things in your system. If you have some SQL code, we can help you speed it up.
Sometimes in Computer Science, it is faster to do the computations when the thing changes; sometimes it is faster to compute when it is requested. It sounds like your case is that it is also slower to compute when it is requested. If it is not "too slow", then that option may be viable.
Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 4 years ago.
Improve this question
I'm interested in how the protocols (and game loop) work for these type of games; any pointers or insights are appreciated.
I guess the main loop would have a world state which would be advanced a few "ticks" per second, but how are the commands of the players executed? What kind of data needs to go back and forth?
I can go into a lot of detail about this but first, go read "1500 archers" http://www.gamasutra.com/view/feature/3094/1500_archers_on_a_288_network_.php and this will answer many of your questions. Here's a summary:
First, most games use UDP due to the real-time nature of the game. THe game loop looks something like this:
read network data
do client-side predictions and compare with where the network says
your objects should actually be
mess with physics to fudge what the network says with what your
local game state is
send data back out onto the network based on what you did this
frame (if anything)
render
That's vastly simplified and "messing with physics" could easily be a 200 page book on its own but it involves predicting client-side where something is likely to be, getting data from the server that is old but tells exactly where an object was/should be, and then interpolating those values somehow to make the object appear "close enough" to where it's actually supposed to be that no one notices. This is super-critical in first person shooters but not as much for real-time strategy.
For real-time strategy, what typically happens is a turn-based system where time is divided into discreet chunks called "turns" that happen sequentially and each turn has a number generated by a monotonic function that guarantees ever increasing values in a particular order without duplicates. On any given turn n, each client sends a message to all other clients with their intended action on turn n + m, where m is an arbitrary number that is usually fairly small and can be best determined through trial and error as well as playtesting. Once all the clients have sent their intended action, each client executes all actions that were sent on turn n + m. This introduces a tiny delay in when an action is ordered by the user and when it executes, however this is usually not noticable.
There are several techniques which can be used to fudge the time as well. For example, if you highlite a unit and then tell it to move, it will make a sound and have an animation when it starts moving but won't actually move right away. However, the network message of an intent to move that unit is sent immediately so by the time the screen responds to the player's input, the network messages have already been sent and acknowledged. You can fudge it further by introducing a small delay (100ms or so) between the mouse click and the game object's response. This is usually not noticable by the player but 100ms is an eternity in a LAN game and even with a broadband connection on a home computer the average ping is probably around 15-60ms or so, which gives you ample time to send the packet prior to the move.
As for data to send, there are two types of data in games: deterministic and non-deterministic. deterministic actions are grounded in game physics so that when the action starts, there is a 100% guarantee that I can predict the result of that action. This data never needs to be sent accross the network since I can determine what it will be on the client based on the initial state. Note that using a random number generator with the same seed on every client turns "random" events into deterministic behavior. Non-deterministic data is usually user input but it is possible to predict what a user's input is likely to be in many cases. The way these pair in a real-time strategy game is that the non-deterministic event is some sort of order to one of my game objects. Once the game object has been ordered to move, the way in which it moves is 100% deterministic. Therefore, all you need to send on the network is the ID of the object, the command given (make this an enum to save bandwidth), and the target of the command (if any, so a spell may have no target if it's an area of affet but a move command has an end-destination). If the user clicks like 100 times to make a unit move, there is no need to send a separate move command for each click since they're all in the same general area so be sure to filter this out as well since it will kill your bandwidth.
One final trick for handling a possible delay between a command and its execution is something called a local perception filter. If I get a move order some time t after the order was given, I know when the unit should have started moving and I know its end destination. Rather than teleporting the unit to get it where it's supposed to be, I can start its movement late and then mess with physics to speed it up slightly so that it can catch up to where it's supposed to be, and then slow it back down to put it in the correct place. The exact value you need to speed it up is also relative and playtesting is the only way to determine the correct value because it just has to "feel right" in order for it to be correct. You can do the same thing with firing bullets and missiles as well and it's highly effective for that. The reason this works is that humans aren't horribly good at seeing subtle changes in movement, particularly if an object is heading directly towards them or away from them, so they just don't notice.
The next thing to think about is cutting down on bandwidth. Don't send messages to clients that couldn't possible see or interact with a unit that is moving. Don't send the same message over and over again because the user clicks. Don't send messages immediately for events that have no immediate affect. Finally, don't require an acknowledgement for events that will be stale should they fail to be received. If I don't get a movement update, by the time I re-transmit that update, its value will be so old that it's no longer relevant so it's better to just send another move and use a local perception filter to catch up or use a cubic spline to interpolate the movement so that it looks more correct or something of that nature. However, an event that's critical, such as a "you're dead" or "your flag has been taken" should be acknowledged and re-transmitted if needed. I teach network game programming at Digipen so feel free to ask any other questions about this as I can probably provide you with an answer. Network game programming can be quite complicated but ultimately it's all about making choices in your implementation and understanding the consequences of your choice.
Check out Battle for Wesnoth.
http://www.wesnoth.org/
It's free, open source, and totally awesome. You can learn a lot from digging into its source.
Discussion of Age of Empires network architecture here
IMHO, that style of peer-to-peer duplicated-replay-based architecture is impressive, but a bit of a dead end for anything more than 8 or so players.