We have a on-premise setup of the free version of the ELK stack. Actually Elasticsearch cluster and some Kibana nodes (no Logstash).
On the application servers we have installed filebeat 7.9.0 which ships the logs to the Elasticsearch ingest nodes, and there is very minimal processing done by the filebeat on the log events before sending (e.g. multiline=true, dissect, drop_fields and json_decode).
As of today, there are only 3 application servers on the production set-up, but it might scale to more number of machines (application servers) going forward.
I understand that, the central management of the filebeat configuration is possible (which is also coming to its end of life) with a license version of ELK stack.
I want to know what are the alternatives available to manage the filebeat configuration apart from the central management through Kibana.
The goal is in future if number of application servers grow to lets say 20, and the filebeat configuration has to undergo a change, changing the configuration on each of the servers shall be manual activity with its own risks associated. i.e. change the configuration at one location and somehow it is updated on filebeat on all application servers.
Please let me know, if this can be achieved ..
Any pointers / thoughts towards the solution let me know
Note: We do not have infrastructure as a code in the organization yet, so this may not be a suitable solution.
Thanks in advance ..
The replacement of Central Management is Elastic Fleet: Installing a single agent on a server, the rest can be done from Kibana. https://www.elastic.co/blog/introducing-elastic-agent-and-ingest-manager gives a better overview of the features and current screenshots.
Most parts of Fleet are also available for free.
Related
Just for fun, I'm designing a few web applications using a microservices architecture. I'm trying to determine the best way to do configuration management, and I'm worried that my approach for configuration may have some enormous pitfalls and/or something better exists.
To frame the problem, let's say I have an authentication service written in c++, an identity service written in rust, an analytics services written in haskell, some middletier written in scala, and a frontend written in javascript. There would also be the corresponding identity DB, auth DB, analytics DB, (maybe a redis cache for sessions), etc... I'm deploying all of these apps using docker swarm.
Whenever one of these apps is deployed, it necessarily has to discover all the other applications. Since I use docker swarm, discovery isn't an issue as long all the nodes share the requisite overlay network.
However, each application still needs the upstream services host_addr, maybe a port, the credentials for some DB or sealed service, etc...
I know docker has secrets which enable apps to read the configuration from the container, but I would then need to write some configuration parser in each language for each service. This seems messy.
What I would rather do is have a configuration service, which maintains knowledge about how to configure all other services. So, each application would start with some RPC call designed to get the configuration for the application at runtime. Something like
int main() {
AppConfig cfg = configClient.getConfiguration("APP_NAME");
// do application things... and pass around cfg
return 0;
}
The AppConfig would be defined in an IDL, so the class would be instantly available and language agnostic.
This seems like a good solution, but maybe I'm really missing the point here. Even at scale, tens of thousands of nodes can be served easily by a few configuration services, so I don't forsee any scaling issues. Again, it's just a hobby project, but I like thinking about the "what-if" scenarios :)
How are configuration schemes handled in microservices architecture? Does this seem like a reasonable approach? What do the major players like Facebook, Google, LinkedIn, AWS, etc... do?
Instead of building a custom configuration management solution, I would use one of these existing ones:
Spring Cloud Config
Spring Cloud Config is a config server written in Java offering an HTTP API to retrieve the configuration parameters of applications. Obviously, it ships with a Java client and a nice Spring integration, but as the server is just a HTTP API, you may use it with any language you like. The config server also features symmetric / asymmetric encryption of configuration values.
Configuration Source: The externalized configuration is stored in a GIT repository which must be made accessible to the Spring Cloud Config server. The properties in that repository are then accessible through the HTTP API, so you can even consider implementing an update process for configuration properties.
Server location: Ideally, you make your config server accessible through a domain (e.g. config.myapp.io), so you can implement load-balancing and fail-over scenarios as needed. Also, all you need to provide to all your services then is just that exact location (and some authentication / decryption info).
Getting started: You may have a look at this getting started guide for centralized configuration on the Spring docs or read through this Quick Intro to Spring Cloud Config.
Netflix Archaius
Netflix Archaius is part of the Netflix OSS stack and "is a Java library that provides APIs to access and utilize properties that can change dynamically at runtime".
While limited to Java (which does not quite match the context you have asked), the library is capable of using a database as source for the configuration properties.
confd
confd keeps local configuration files up-to-date using data stored in external sources (etcd, consul, dynamodb, redis, vault, ...). After configuration changes, confd restarts the application so that it can pick up the updated configuration file.
In the context of your question, this might be worthwhile to try as confd makes no assumption about the application and requires no special client code. Most languages and frameworks support file-based configuration so confd should be fairly easy to add on top of existing microservices that currently use env variables and did not anticipate decentralized configuration management.
I don't have a good solution for you, but I can point out some issues for you to consider.
First, your applications will presumably need some bootstrap configuration that enables them to locate and connect to the configuration service. For example, you mentioned defining the configuration service API with IDL for a middleware system that supports remote procedure calls. I assume you mean something like CORBA IDL. This means your bootstrap configuration will not be just the endpoint to connect to (specified perhaps as a stringified IOR or a path/in/naming/service), but also a configuration file for the CORBA product you are using. You can't download that CORBA product's configuration file from the configuration service, because that would be a chicken-and-egg situation. So, instead, you end up with having to manually maintain a separate copy of the CORBA product's configuration file for each application instance.
Second, your pseudo-code example suggests that you will use a single RPC invocation to retrieve all the configuration for an application in a single go. This coarse level of granularity is good. If, instead, an application used a separate RPC call to retrieve each name=value pair, then you could suffer major scalability problems. To illustrate, let's assume an application has 100 name=value pairs in its configuration, so it needs to make 100 RPC calls to retrieve its configuration data. I can foresee the following scalability problems:
Each RPC might take, say, 1 millisecond round-trip time if the application and the configuration server are on the same local area network, so your application's start-up time is 1 millisecond for each of 100 RPC calls = 100 milliseconds = 0.1 second. That might seem acceptable. But if you now deploy another application instance on another continent with, say, a 50 millisecond round-trip latency, then the start-up time for that new application instance will be 100 RPC calls at 50 milliseconds latency per call = 5 seconds. Ouch!
The need to make only 100 RPC calls to retrieve configuration data assumes that the application will retrieve each name=value pair once and cache that information in, say, an instance variable of an object, and then later on access the name=value pair via that local cache. However, sooner or later somebody will call x = cfg.lookup("variable-name") from inside a for-loop, and this means the application will be making a RPC every time around the loop. Obviously, this will slow down that application instance, but if you end up with dozens or hundreds of application instances doing that, then your configuration service will be swamped with hundreds or thousands of requests per second, and it will become a centralised performance bottleneck.
You might start off writing long-lived applications that do 100 RPCs at start-up to retrieve configuration data, and then run for hours or days before terminating. Let's assume those applications are CORBA servers that other applications can communicate with via RPC. Sooner or later you might decide to write some command-line utilities to do things like: "ping" an application instance to see if it is running; "query" an application instance to get some status details; ask an application instance to gracefully terminate; and so on. Each of those command-line utilities is short-lived; when they start-up, they use RPCs to obtain their configuration data, then do the "real" work by making a single RPC to a server process to ping/query/kill it, and then they terminate. Now somebody will write a UNIX shell script that calls those ping and query commands once per second for each of your dozens or hundreds of application instances. This seemingly innocuous shell script will be responsible for creating dozens or hundreds of short-lived processes per second, and of those short-lived processes will make numerous RPC calls to the centralised configuration server to retrieve name=value pairs one at a time. That sort of shell script can pu a massive load on your centralised configuration server.
I am not trying to discourage you from designing a centralised configuration server. The above points are just warning about scalability issues you need to consider. Your plan for an application to retrieve all its configuration data via one coarse-granularity RPC call will certainly help you to avoid the kinds of scalability problems I mentioned above.
To provide some food for thought, you might want to consider a different approach. You could store each application's configuration files on a web sever. A shell start script "wrapper" for an application can do the following:
Use wget or curl to download "template" configuration files from the web server and store the files on the local file system. A "template" configuration file is a normal configuration file but with some placeholders for values. A placeholder might look like ${host_name}.
Also use wget or curl to download a file containing search-and-replace pairs, such as ${host_name}=host42.pizza.com.
Perform a global search-and-replace of those search-and-replace terms on all the downloaded template configuration files to produce the configuration files that are ready to use. You might use UNIX shell tools like sed or a scripting language to perform this global search-and-replace. Alternatively, you could use a templating engine like Apache Velocity.
Execute the actual application, using a command-line argument to specify the path/to/downloaded/config/files.
State: We are sharing zookeeper with kafka and several different services, which are using zookeeper for coordination. They are nicely operating on zookeeper subcontext. Looks like this:
/
/service1/...
/service2/...
/brokers/...
/consumers/...
My question is.. Is it possible to setup kafka to use subcontext? So, the other services can't eventually modify other services subcontext. It would be:
/
/service1/...
/service2/...
/kafka/brokers/...
/kafka/consumers/...
I saw this syntax in other projects:
zk://10.0.0.1,10.0.0.2/kafka
lets say. So, kafka would see only the brokers and consumers paths and there would be no way to mess up with other subcontext.
I'm afraid kafka is just not supported this format at the time. Other question is, is there a workaround? Like wrap up zookeeper somehow? Any ideas? Or kafka is supposed to use zookeeper exclusively. Is it best practice and we should spawn zookeeper for each project, which is overkill thus zookeeper need ensemble consists atleast of 3 nodes.
Thanks for your answer!
You can use the zk chroot syntax with Kafka, as detailed in the Kafka configuration documentation.
Zookeeper also allows you to add a "chroot" path which will make all kafka data for this cluster appear under a particular path. This is a way to setup multiple Kafka clusters or other applications on the same zookeeper cluster. To do this give a connection string in the form hostname1:port1,hostname2:port2,hostname3:port3/chroot/path which would put all this cluster's data under the path /chroot/path. Note that you must create this path yourself prior to starting the broker and consumers must use the same connection string.
The best practice is to maintain a single ZooKeeper cluster (at least that is what I've seen). Otherwise you are creating more operational workload for maintaining a good ZK ensemble.
The Kafka documentation on operationalizing ZK sort of recommends having multiple ZKs though:
Application segregation: Unless you really understand the application patterns of other apps that you want to install on the same box, it can be a good idea to run ZooKeeper in isolation (though this can be a balancing act with the capabilities of the hardware).
Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 1 year ago.
Improve this question
In Kafka, I would like to use only a single broker, single topic and a single partition having one producer and multiple consumers (each consumer getting its own copy of data from the broker). Given this, I do not want the overhead of using Zookeeper; Can I not just use the broker only? Why is a Zookeeper must?
Yes, Zookeeper is required for running Kafka. From the Kafka Getting Started documentation:
Step 2: Start the server
Kafka uses zookeeper so you need to first start a zookeeper server if
you don't already have one. You can use the convenience script
packaged with kafka to get a quick-and-dirty single-node zookeeper
instance.
As to why, well people long ago discovered that you need to have some way to coordinating tasks, state management, configuration, etc across a distributed system. Some projects have built their own mechanisms (think of the configuration server in a MongoDB sharded cluster, or a Master node in an Elasticsearch cluster). Others have chosen to take advantage of Zookeeper as a general purpose distributed process coordination system. So Kafka, Storm, HBase, SolrCloud to just name a few all use Zookeeper to help manage and coordinate.
Kafka is a distributed system and is built to use Zookeeper. The fact that you are not using any of the distributed features of Kafka does not change how it was built. In any event there should not be much overhead from using Zookeeper. A bigger question is why you would use this particular design pattern -- a single broker implementation of Kafka misses out on all of the reliability features of a multi-broker cluster along with its ability to scale.
As explained by others, Kafka (even in most recent version) will not work without Zookeeper.
Kafka uses Zookeeper for the following:
Electing a controller. The controller is one of the brokers and is responsible for maintaining the leader/follower relationship for all the partitions. When a node shuts down, it is the controller that tells other replicas to become partition leaders to replace the partition leaders on the node that is going away. Zookeeper is used to elect a controller, make sure there is only one and elect a new one it if it crashes.
Cluster membership - which brokers are alive and part of the cluster? this is also managed through ZooKeeper.
Topic configuration - which topics exist, how many partitions each has, where are the replicas, who is the preferred leader, what configuration overrides are set for each topic
(0.9.0) - Quotas - how much data is each client allowed to read and write
(0.9.0) - ACLs - who is allowed to read and write to which topic
(old high level consumer) - Which consumer groups exist, who are their members and what is the latest offset each group got from each partition.
[from https://www.quora.com/What-is-the-actual-role-of-ZooKeeper-in-Kafka/answer/Gwen-Shapira]
Regarding your scenario, only one broker instance and one producer with multiple consumer, u can use pusher to create a channel, and push event to that channel that consumer can subscribe to and hand those events.
https://pusher.com/
Important update - August 2019:
ZooKeeper dependency will be removed from Apache Kafka. See the high-level discussion in KIP-500: Replace ZooKeeper with a Self-Managed Metadata Quorum.
These efforts will take a few Kafka releases and additional KIPs. Kafka Controllers will take over the tasks of current ZooKeeper tasks. The Controllers will leverage the benefits of the Event Log which is a core concept of Kafka.
Some benefits of the new Kafka architecture are a simpler architecture, ease of operations, and better scalability e.g. allow "unlimited partitions".
Updated on Oct 2022
For new clusters in the 3.3 release you can use Apache Kafka without ZooKeeper (in new mode, called KRaft mode) in production.
Apache Kafka Raft (KRaft) is the consensus protocol that was introduced to remove Apache Kafka’s dependency on ZooKeeper for metadata management. The development progress is tracked in KIP-500.
KRaft mode was released in early access in Kafka 2.8. It was not suitable for production before 3.3 version (see details in KIP-833: Mark KRaft as Production Ready)
1. Benefits of Kafka’s new quorum controller
Enables Kafka clusters to scale to millions of partitions through improved control plane performance with the new metadata management
Improves stability, simplifies the software, and makes it easier to monitor, administer, and support Kafka.
Allows Kafka to have a single security model for the whole system
Provides a lightweight, single process way to get started with Kafka
Makes controller failover near-instantaneous
2. Timeline
Note: this timeline is very rough and subject to change.
2022/10: KRaft mode declared production-ready in Kafka 3.3
2023/02: Upgrade from ZK mode supported in Kafka 3.4 as early access.
2023/04: Kafka 3.5 released with both KRaft and ZK support. Upgrade from ZK goes production. ZooKeeper mode deprecated.
2023/10: Kafka 4.0 released with only KRaft mode supported.
References:
KIP-500: Replace ZooKeeper with a Self-Managed Metadata Quorum
Apache Kafka Needs No Keeper: Removing the Apache ZooKeeper Dependency
Preparing Your Clients and Tools for KIP-500: ZooKeeper Removal from Apache Kafka
KRaft: Apache Kafka Without ZooKeeper
Kafka is built to use Zookeeper. There is no escaping from that.
Kafka is a distributed system and uses Zookeeper to track status of kafka cluster nodes. It also keeps track of Kafka topics, partitions etc.
Looking at your question, it seems you do not need Kafka. You can use any application that supports pub-sub such as Redis, Rabbit MQ or hosted solutions such as Pub-nub.
IMHO Zookeeper is not an overhead but makes your life a lot easier.
It is basically used to maintain co-ordination between different nodes in a cluster. One of the most important things for Kafka is it uses zookeeper to periodically commit offsets so that in case of node failure it can resume from the previously committed offset (imagine yourself taking care of all this by your own).
Zookeeper also plays a vital role for serving many other purposes, such as leader detection, configuration management, synchronization, detecting when a new node joins or leaves the cluster, etc.
Future Kafka releases are planning to remove the zookeeper dependency but as of now it is an integral part of it.
Here are a few lines taken from their FAQ page:
Once the Zookeeper quorum is down, brokers could result in a bad state and could not normally serve client requests, etc. Although when Zookeeper quorum recovers, the Kafka brokers should be able to resume to normal state automatically, there are still a few corner cases the they cannot and a hard kill-and-recovery is required to bring it back to normal. Hence it is recommended to closely monitor your zookeeper cluster and provision it so that it is performant.
For more details check here
Zookeeper is centralizing and management system for any kind of distributed systems. Distributed system is different software modules running on different nodes/clusters (might be on geographically distant locations) but running as one system. Zookeeper facilitates communication between the nodes, sharing configurations among the nodes, it keeps track of which node is leader, which node joins/leaves, etc. Zookeeper is the one who keeps distributed systems sane and maintains consistency. Zookeeper basically is an orchestration platform.
Kafka is a distributed system. And hence it needs some kind of orchestration for its nodes that might be geographically distant (or not).
Apache Kafka v2.8.0 gives you early access to KIP-500 that removes the Zookeeper dependency on Kafka which means it no longer requires Apache Zookeeper.
Instead, Kafka can now run in Kafka Raft metadata mode (KRaft mode) which enables an internal Raft quorum. When Kafka runs in KRaft mode its metadata is no longer stored on ZooKeeper but on this internal quorum of controller nodes instead. This means that you don't even have to run ZooKeeper at all any longer.
Note however that v2.8.0 is currently early access and you should not use Zookeeper-less Kafka in production for the time being.
A few benefits of removing ZooKeeper dependency and replacing it with an internal quorum:
More efficient as controllers no longer need to communicate with ZooKeeper to fetch cluster state metadata every time the cluster is starting up or when a controller election is being made
More scalable as the new implementation will be able to support many more topics and partitions in KRaft mode
Easier cluster management and configuration as you don't have to manage two distinct services any longer
Single process Kafka Cluster
For more details you can read the article Kafka No Longer Requires ZooKeeper
Yes, Zookeeper is must by design for Kafka. Because Zookeeper has the responsibility a kind of managing Kafka cluster. It has list of all Kafka brokers with it. It notifies Kafka, if any broker goes down, or partition goes down or new broker is up or partition is up. In short ZK keeps every Kafka broker updated about current state of the Kafka cluster.
Then every Kafka client(producer/consumer) all need to do is connect with any single broker and that broker has all metadata updated by Zookeeper, so client need not to bother about broker discovery headache.
Other than the usual payload message transfer, there are many other communications that happens in kafka, like
Events related to brokers requesting the cluster membership.
Events related to Brokers becoming available.
Getting bootstrap config setups.
Events related to controller and leader updates.
Help status updates like Heartbeat updates.
Zookeeper itself is a distributed system consisting of multiple nodes in an ensemble. Zookeeper is centralised service for maintaining such metadata.
This article explains the role of Zookeeper in Kafka. It explains how kafka is stateless and how zookeper plays an important role in distributed nature of kafka (and many more distributed systems).
The request to run Kafka without Zookeeper seems to be quite common. The library Charlatan addresses this.
According to the description is Charlatan more or less a mock for Zookeeper, providing the Zookeeper services either backed up by other tools or by a database.
I encountered that library when dealing with the main product of the authors for the Charlatan library; there it works fine …
Firstly
Apache ZooKeeper is a distributed store which is used to provide configuration and synchronization services in a high available way.
In more recent versions of Kafka, work was done in order for the client consumers to not store information about how far it had consumed messages (called offsets) into ZooKeeper.This reduced usage did not get rid of the need for consensus and coordination in distributed systems however. While Kafka provides fault-tolerance and resilience, something is needed in order to provide the coordination needed and ZooKeeper enables that piece of the overall system.
Secondly
Agreeing on who the leader of a partition is, is one example of the practical application of ZooKeeper within the Kafka ecosystem.
Zookeeper would work if there was even a single broker.
These are from Kafka In Action book.
Image is from this course
I'm planning to make a system that will utilize a central corporate database and several local in-house database.
Connection between the central database and local databases would depend on the local's availability (The central database is always online). So I'm thinking of synchronizing them by "pushing" updates from the local database to the central database using a website user interface and vice versa.
The central database is expected to have decent and stable internet connection. So it just sit there waiting for updates from local databases. Also local systems should have the capability to download updates from some tables that are only updated from the central database. The local system should only contain local information and general settings set by the central system. Local systems have no capability to see other local information in other branches(This capability is only for the central system).
So basically, information that are pushed to the central database are only modified in the local system and the data downloaded from the central database are only updated from the central database. So the local system can live and operate on its own, and the only true purpose of the central database is to create an overview of all the updates.
I've already checked Microsoft Sync Framework and it looks promising. I just can't find any tutorial that could demonstrate it completely. I'm hoping for a solution that can be implemented using a website interface. Just a nice button in my local system's page.
If anyone would be able to point me to good source or starting point, it would be really helpful.
I have just finished my Microsoft Sync Framework research. In fact, MSDN provided all the solutions that you need. In my opinion, you don't have to pay for any commercial sync solution as you could get it done by yourself. Basically, you have to understand the overall structure of Microsoft Sync Framework: Concept of Microsoft Sync Framework. Besides that, there are two type of sync module available which is 2-Tier and N-Tier Synchronization. Please study 2-Tier and N-Tier if you doesn't know about it. However, here is a good starting point for you to begin the N-Tier Synchronization N-Tier Sync. Meanwhile, you can set your synchronization direction to match with various kind of situation Sync Direction. In addition, data conflict could be occur in certain situation which should be handled Data Conflict Handling.
I would like to know how we are supposed to do integration between different Perforce servers/depots.
I'm looking for a solution that would allow us to do both-ways integrations.
This Using Remote Depots article describes how to map the remote depot as read only. Is this the only solution to do mappings on both servers? If so, this means that I could not use a single branch spec to do both ways integrations.
From reading the Perforce knowledge base, I believe the preferred/suggested solution is for each server to do the integrate from the read-only remote depot.
This is a by-design limitation of Perforce because the meta-data is only available to the local server, e.g. serverA:1666 does not know commands performed by a user on serverB:1666 (as explained in the case-study at the bottom of this article).
Also the point regarding performance is absolutely true; our server was hammered this afternoon during a code drop from a remote depot. All we could do was wait until the integrate/diff was complete.
To find out what is happening on your server, use the command p4 monitor show to show what the current workload on your server.