I got a question regarding reinforcement learning. let's say we have a robot that is able to adapt to changing environments. Similar to this paper 1. When there is a change in the environment[light dimming], the robot's performance drops and it needs to explore its new environment by collecting data and running the Q-algorithm again to update its policy to be able to "adapt". The collection of new data and updating of the policy takes about 4/5hrs. I was wondering if I have an army of these robots in the same room, undergoing the same environmental changes, can the data collection be sped up so that the policy can be updated more quickly? so that the policy can be updated in under 1 hour or so, allowing the performance of the robots to increase?
I believe you are talking about scaling learning horizontally as in training multiple agents in parallel.
A3C is one algorithm that does this by training multiple agents in parallel and independently of each other. Each agent has its own environment which allows it to gain a different experience than the rest of the agents, ultimately increasing the breadth of your agents collective experience. Eventually each agent updates a shared network asynchronously and you use this network to drive your main agent.
You mentioned that you wanted to use the same environment for all parallel agents. I can think of this in two ways:
If you are talking about a shared environment among agents, then this could possibly speed things up however you are likely not going to gain much in terms of performance. You are also very likely to face issues in terms of episode completion - if multiple agents are taking steps simultaneously then your transitions will be a mess to say the least. The complexity cost is high and the benefit is negligible.
If you are talking about cloning the same environment for each agent then you end up both gaining speed and a broader experience which translates to performance. This is probably the sane thing to do.
Related
I am trying to understand what is the use case of a queue in distributed system.
And also how it scales and how it makes sure it's not a single point of failure in the system?
Any direct answer or a reference to a document is appreciated.
Use case:
I understand that queue is a messaging system. And it decouples the systems that communicate between each other. But, is that the only point of using a queue?
Scalability:
How does the queue scale for high volumes of data? Both read and write.
Reliability:
How does the queue not becoming a single point of failure in the system? Does the queue do a replication, similar to data-storage?
My question is not specified to any particular queue server like Kafka or JMS. Just in general.
Queue is a mental concept, the implementation decides about 1 + 2 + 3
A1: No, it is not the only role -- a messaging seems to be main one, but a distributed-system signalling is another one, by no means any less important. Hoare's seminal CSP-paper is a flagship in this field. Recent decades gave many more options and "smart-behaviours" to work with in designing a distributed-system signalling / messaging services' infrastructures.
A2: Scaling envelopes depend a lot on implementation. It seems obvious that a broker-less queues can work way faster, that a centralised, broker-based, infrastructure. Transport-classes and transport-links account for additional latency + performance degradation as the data-flow volumes grow. BLOB-handling is another level of a performance cliff, as the inefficiencies are accumulating down the distributed processing chain. Zero-copy ( almost ) zero-latency smart-Queue implementations are still victims of the operating systems and similar resources limitations.
A3: Oh sure it is, if left on its own, the SPOF. However, Theoretical Cybernetics makes us safe, as we can create reliable systems, while still using error-prone components. ( M + N )-failure-resilient schemes are thus achievable, however the budget + creativity + design-discipline is the ceiling any such Project has to survive with.
my take:
I would be careful with "decouple" term - if service A calls api on service B, there is coupling since there is a contract between services; this is true even if the communication is happening over a queue, file or fax. The key with queues is that the communication between services is asynchronous. Which means their runtimes are decoupled - from practical point of view, either of systems may go down without affecting the other.
Queues can scale for large volumes of data by partitioning. From clients point of view, there is one queue, but in reality there are many queues/shards and number of shards helps to support more data. Of course sharding a queue is not "free" - you will lose global ordering of events, which may need to be addressed in you application.
A good queue based solution is reliable based on replication/consensus/etc - depends on set of desired properties. Queues are not very different from databases in this regard.
To give you more direction to dig into:
there an interesting feature of queues: deliver-exactly-once, deliver-at-most-once, etc
may I recommend Enterprise Architecture Patterns - https://www.enterpriseintegrationpatterns.com/patterns/messaging/Messaging.html this is a good "system design" level of information
queues may participate in distributed transactions, e.g. you could build something like delete a record from database and write it into queue, and that will be either done/committed or rolledback - another interesting topic to explore
I am having hard time to grab what etcd (in CoreOS) really does, because all those "distributed key-value storage" thingy seems intangible to me. Further reading into etcd, it delves into into Raft consensus algorithm, and then it becomes really confusing to understand.
Let's say that what happen if a cluster system doesn't have etcd?
Thanks for your time and effort!
As someone with no CoreOS experience building a distributed system using etcd, I think I can shed some light on this.
The idea with etcd is to give some very basic primitives that are applicable for building a wide variety of distributed systems. The reason for this is that distributed systems are fundamentally hard. Most programmers don't really grok the difficulties simply because there are orders of magnitude more opportunity to learn about single-system programs; this has really only started to shift in the last 5 years since cloud computing made distributed systems cheap to build and experiment with. Even so, there's a lot to learn.
One of the biggest problems in distributed systems is consensus. In other words, guaranteeing that all nodes in a system agree on a particular value. Now, if hardware and networks were 100% reliable then it would be easy, but of course that is impossible. Designing an algorithm to provide some meaningful guarantees around consensus is a very difficult problem, and one that a lot of smart people have put a lot of time into. Paxos was the previous state of the art algorithm, but was very difficult to understand. Raft is an attempt to provide similar guarantees but be much more approachable to the average programmer. However, even so, as you have discovered, it is non-trivial to understand it's operational details and applications.
In terms of what etcd is specifically used for in CoreOS I can't tell you. But what I can say with certainty is that any data which needs to be shared and agreed upon by all machines in a cluster should be stored in etcd. Conversely, anything that a node (or subset of nodes) can handle on its own should emphatically not be stored in etcd (because it incurs the overhead of communicating and storing it on all nodes).
With etcd it's possible to have a large number of identical machines automatically coordinate, elect a leader, and guarantee an identical history of data in its key-value store such that:
No etcd node will ever return data which is not agreed upon by the majority of nodes.
For cluster size x any number of machines > x/2 can continue operating and accepting writes even if the others die or lose connectivity.
For any machines losing connectivity (eg. due to a netsplit), they are guaranteed to continue to return correct historical data even though they will fail to write.
The key-value store itself is quite simple and nothing particularly interesting, but these properties allow one to construct distributed systems that resist individual component failure and can provide reasonable guarantees of correctness.
etcd is a reliable system for cluster-wide coordination and state management. It is built on top of Raft.
Raft gives etcd a total ordering of events across a system of distributed etcd nodes. This has many advantages and disadvantages:
Advantages include:
any node may be treated like a master
minimal downtime (a client can try another node if one isn't responding)
avoids split-braining
a reliable way to build distributed locks for cluster-wide coordination
users of etcd can build distributed systems without ad-hoc, buggy, homegrown solutions
For example: You would use etcd to coordinate an automated election of a new Postgres master so that there remains only one master in the cluster.
Disadvantages include:
for safety reasons, it requires a majority of the cluster to commit writes - usually to disk - before replying to a client
requires more network chatter than a single master system
What other stress test cases are there other than finding out the maximum number of users allowed to login into the web application before it slows down the performance and eventually crashing it?
This question is hard to answer thoroughly since it's too broad.
Anyway many stress tests depend on the type and execution flow of your workload. There's an entire subject dedicated (as a graduate course) to queue theory and resources optimization. Most of the things can be summarized as follows:
if you have a resource (be it a gpu, cpu, memory bank, mechanical or
solid state disk, etc..), it can serve a number of users/requests per
second and takes an X amount of time to complete one unit of work.
Make sure you don't exceed its limits.
Some systems can also be studied with a probabilistic approach (Little's Law is one of the most fundamental rules in these cases)
There are a lot of reasons for load/performance testing, many of which may not be important to your project goals. For example:
- What is the performance of a system at a given load? (load test)
- How many users the system can handle and still meet a specific set of performance goals? (load test)
- How does the performance of a system changes over time under a certain load? (soak test)
- When will the system will crash under increasing load? (stress test)
- How does the system respond to hardware or environment failures? (stress test)
I've got a post on some common motivations for performance testing that may be helpful.
You should also check out your web analytics data and see what people are actually doing.
It's not enough to simply simulate X number of users logging in. Find the scenarios that represent the most common user activities (anywhere between 2 to 20 scenarios).
Also, make sure you're not just hitting your cache on reads. Add some randomness / diversity in the requests.
I've seen stress tests where all the users were requesting the same data which won't give you real world results.
What is the difference between coarse-grained and fine-grained?
I have searched these terms on Google, but I couldn't find what they mean.
From Wikipedia (granularity):
Granularity is the extent to which a
system is broken down into small
parts, either the system itself or its
description or observation. It is the
extent to which a larger entity is
subdivided. For example, a yard broken
into inches has finer granularity than
a yard broken into feet.
Coarse-grained systems consist of
fewer, larger components than
fine-grained systems; a coarse-grained
description of a system regards large
subcomponents while a fine-grained
description regards smaller components
of which the larger ones are composed.
In simple terms
Coarse-grained - larger components than fine-grained, large subcomponents. Simply wraps one or more fine-grained services together into a more coarse-grained operation.
Fine-grained - smaller components of which the larger ones are composed, lowerlevel service
It is better to have more coarse-grained service operations, which are composed by fine-grained operations
Coarse-grained: A few ojects hold a lot of related data that's why services have broader scope in functionality. Example: A single "Account" object holds the customer name, address, account balance, opening date, last change date, etc.
Thus: Increased design complexity, smaller number of cells to various operations
Fine-grained: More objects each holding less data that's why services have more narrow scope in functionality. Example: An Account object holds balance, a Customer object holds name and address, a AccountOpenings object holds opening date, etc.
Thus: Decreased design complexity , higher number of cells to various service operations.
These are relationships defined between these objects.
One more way to understand would be to think in terms of communication between processes and threads. Processes communicate with the help of coarse grained communication mechanisms like sockets, signal handlers, shared memory, semaphores and files. Threads, on the other hand, have access to shared memory space that belongs to a process, which allows them to apply finer grain communication mechanisms.
Source: Java concurrency in practice
In the context of services:
http://en.wikipedia.org/wiki/Service_Granularity_Principle
By definition a coarse-grained service operation has broader scope
than a fine-grained service, although the terms are relative. The
former typically requires increased design complexity but can reduce
the number of calls required to complete a task.
A fine grained service interface is about the same like chatty interface.
In term of dataset like a text file ,Coarse-grained meaning we can transform the whole dataset but not an individual element on the dataset While fine-grained means we can transform individual element on the dataset.
Coarse-grained and Fine-grained both think about optimizing a number of servicess. But the difference is in the level. I like to explain with an example, you will understand easily.
Fine-grained: For example, I have 100 services like findbyId, findbyCategry, findbyName...... so on. Instead of that many services why we can not provide find(id, category, name....so on). So this way we can reduce the services. This is just an example, but the goal is how to optimize the number of services.
Coarse-grained: For example, I have 100 clients, each client have their own set of 100 services. So I have to provide 100*100 total services. It is very much difficult. Instead of that what I do is, I identify all common services which apply to most of the clients as one service set and remaining separately. For example in 100 services 50 services are common. So I have to manage 100*50 + 50 only.
Coarse-grained granularity does not always mean bigger components, if you go by literally meaning of the word coarse, it means harsh, or not appropriate. e.g. In software projects management, if you breakdown a small system into few components, which are equal in size, but varies in complexities and features, this could lead to a coarse-grained granularity. In reverse, for a fine-grained breakdown, you would divide the components based on their cohesiveness of the functionalities each component is providing.
coarse grained and fine grained. Both of these modes define how the cores are shared
between multiple Spark tasks. As the name suggests, fine-grained mode is
responsible for sharing the cores at a more granular level. Fine-grained mode has been deprecated by Spark and will soon be removed.
Corse-grained services provides broader functionalities as compared to fine-grained service. Depending on the business domain, a single service can be created to serve a single business unit or specialised multiple fine-grained services can be created if subunits are largely independent of each other.
Coarse grained service may get more difficult may be less adaptable to change due to its size while fine-grained service may introduce additional complexity of managing multiple services.
Granularity has an important application while storing large scale data where space is very important.
The meaning of granularity according to Oxford dictionary is -
"The scale or level of detail in a set of data."
According to Cambridge dictionary -
"A lot of small details included in information, making it possible for you to understand very clearly what is happening"
So from the word specific meaning, it is some kind of partition of data for a continuous process.
Finer granularity consists of small interval partition, so that detailed representation can be achieved.
On the other hand, coarser granularity is larger frame interval, so that it can save storage.
Uses of two types of granularity is application specific.
For example- If we have an application, where recent time information is more important than the older information. For detailed representation of recent data can be found by finer granularity, while for older data representation we can use coarser granularity.
I have recently begun working on a project to establish how best to leverage the processing power available in modern graphics cards for general programming. It seems that the field general purpose GPU programming (GPGPU) has a large bias towards scientific applications with a lot of heavy math as this fits well with the GPU computational model. This is all good and well, but most people don't spend all their time running simulation software and the like so we figured it might be possible to create a common foundation for easily building GPU-enabled software for the masses.
This leads to the question I would like to pose; What are the most common types of work performed by programs? It is not a requirement that the work translates extremely well to GPU programming as we are willing to accept modest performance improvements (Better little than nothing, right?).
There are a couple of subjects we have in mind already:
Data management - Manipulation of large amounts of data from databases
and otherwise.
Spreadsheet type programs (Is somewhat related to the above).
GUI programming (Though it might be impossible to get access to the
relevant code).
Common algorithms like sorting and searching.
Common collections (And integrating them with data manipulation
algorithms)
Which other coding tasks are very common? I suspect a lot of the code being written is of the category of inventory management and otherwise tracking of real 'objects'.
As I have no industry experience I figured there might be a number of basic types of code which is done more often than I realize but which just doesn't materialize as external products.
Both high level programming tasks as well as specific low level operations will be appreciated.
General programming translates terribly to GPUs. GPUs are dedicated to performing fairly simple tasks on streams of data at a massive rate, with massive parallelism. They do not deal well with the rich data and control structures of general programming, and there's no point trying to shoehorn that into them.
General programming translates terribly to GPUs. GPUs are dedicated to performing fairly simple tasks on streams of data at a massive rate, with massive parallelism. They do not deal well with the rich data and control structures of general programming, and there's no point trying to shoehorn that into them.
This isn't too far away from my impression of the situation but at this point we are not concerning ourselves too much with that. We are starting out by getting a broad picture of which options we have to focus on. After that is done we will analyse them a bit deeper and find out which, if any, are plausible options. If we end up determining that it is impossible to do anything within the field, and we are only increasing everybody's electricity bill then that is a valid result as well.
Things that modern computers do a lot of, where a little benefit could go a long way? Let's see...
Data management: relational database management could benefit from faster relational joins (especially joins involving a large number of relations). Involves massive homogeneous data sets.
Tokenising, lexing, parsing text.
Compilation, code generation.
Optimisation (of queries, graphs, etc).
Encryption, decryption, key generation.
Page layout, typesetting.
Full text indexing.
Garbage collection.
I do a lot of simplifying of configuration. That is I wrap the generation/management of configuration values inside a UI. The primary benefit is I can control work flow and presentation to make it simpler for non-techie users to configure apps/sites/services.
The other thing to consider when using a GPU is the bus speed, Most Graphics cards are designed to have a higher bandwidth when transferring data from the CPU out to the GPU as that's what they do most of the time. The bandwidth from the GPU back up to the CPU, which is needed to return results etc, isn't as fast. So they work best in a pipelined mode.
You might want to take a look at the March/April issue of ACM's Queue magazine, which has several articles on GPUs and how best to use them (besides doing graphics, of course).