I am looking for a way to collect Java exceptions thrown by containers. I know the function from the logging system of GKE/GCP and would like to implement a similar logging system in our self-hosted cluster.
I am using Prometheus and Grafana for monitoring metrics.
You need a centralized logging solution. There are some common solutions out there. One of them is the ELK Stack (now named Elastic stack).
It has 3 main components:
Elasticsearch: To store the logs, index them, make them searchable etc.
Logstash: To collect the logs from various sources (containers in your case), parse/filter them and push them to other systems. In ELK's case, push them to Elasticsearch.
Kibana: A web GUI to visualize the data in Elasticsearch, allows searching, creating visual graphs and so on.
See the official page of Elastic stack for more information.
You can also use Fluentd or Fluent Bit instead of Logstash, so it'll be an EFK stack. I personally had pretty good experience with an EFK stack with Fluent Bit.
For another, lighter alternative, you can check out Grafana Loki, which is kind of a logging extension to the popular monitoring setup of Prometheus+Grafana.
If you are already familiar with the Stackdriver solution from GKE, I'd bet your best choice would be to stick with it and install Stackdriver on your self managed Kubernetes cluster as well:
https://kubernetes.io/docs/tasks/debug-application-cluster/logging-stackdriver/
Related
The 3rd factor of "The twelve-factor app", states that the application configuration must be stored in the environment.
For the image of a Quarkus application deployed on OpenShift, what is the best solution: where can I put the application configuration?
Thanks a lot.
Kind regards.
ConfigMaps are designed for this exact purpose. (The opening line of the documentation is "Config maps allow you to decouple configuration artifacts from image content to keep containerized applications portable.") Secrets are the equivalent functionality for sensitive information.
ConfigMaps generally the I recommend for most situations. Because, as you say, this is a universal solution that can be used for all kinds of applications, no matter how they are implemented. Pods have many ways of accessing ConfigMaps , all OpenShift and Kubernetes tools will understand how to work with ConfigMaps (icluding things like CI/CD), and ConfigMaps are native API resources. Because they are so pervasive, many frameworks also have native support for them. Including Quarkus which was your specific example.)
That said, you also ask in your comments about whether you can use Consul. Absolutely you can, Consul is certified on OpenShift. There are LOTS of third party solutions. And there are reasons that people choose to use them.
But if you ask the generic question "How do you do this in OpenShift?" I will point you to ConfigMaps. If you want to use something else (like Consul) then how you access and manage that config really has to be a question for that product. It looks like there is a Quarkus extension for Consul, but I don't know anything about it.
I want to build my own pass platform based on cloudfoundry and openshift. I want to use some of the functions of these two platforms, and I don't want to deploy them all in the environment. Is this feasible? What similar open source projects can learn from?
Let me produce some contents about OpenShift for you as follows.
OpenShift Online : Free plan is enough to your first training.
OpenShift HandsOn training : Awesome practical training, it need not to prepare your env.
OpenShift Documentation - Enterprise and OpenShift OpenSource AKA OKD - Documentation
If you'd like to deploy to your on-premise as open source project of OpenShift, you can review/test/operate the OKD (former name: OpenShift Origin).
I hope if help you. :^)
In regards to Cloud Foundry, it is just a collection of services. We use Bosh to deploy Cloud Foundry, which knows how to deploy all the services so that they can talk to each other & function cohesively. There's nothing that would prevent you from using a different Bosh configuration (or even totally different tool) to deploy these services in a different way.
You can run projects like Gorouter, UAA, Cloud Controller and Garden stand-alone. The individual project sites typically have instructions for doing this.
Ex:
https://github.com/cloudfoundry/gorouter#start
https://github.com/cloudfoundry/uaa#quick-start
Other components might be a little trickier as they depend on each other. Diego, for example, depends on Garden and is built to send logs through Loggregator. In these cases, you might need to do a little work if you didn't want to use one of the dependent components.
https://github.com/cloudfoundry/diego-design-notes#what-are-all-these-repos-and-what-do-they-do
I would disagree with your comment about these systems being bloated, and say that depends on your perspective. If you don't need a lot of the features, then I could see why you might think that. I'd say overkill might be a better way to put it though.
If you don't need all the functionality that PaaS platforms provide, you could look at other options: Dokku, Kubernetes, Knative, etc... You don't get all the features of CF, but the systems have smaller footprints. If you can live without the extra features, then these might be better options for you.
Hope that helps!
I am asking a very basic question here.
Question is
I am using Apache Sling , Apache Jackrabbit, Apache Felix in my project as said by my instructor. I am trying to understand why these software is developed by Apache. I tried a lot on the internet,, but I didn't find any blog or wordpress blog, or any useful youtube video that explain all these projects. Can you explain me about these projects.
Why these projects developed?
What they do ?
and more questions like this
Previously I found the same doubt with Apache Hadoop, but all the material that I found on net is sufficient for me to get a feel of this project. This time I am struggling with Sling, Felix, Jackrabbit.
I will be very thankful for you. Waiting for your kind response.
The combination of Apache Jackrabbit, Apache Sling, and Apache Felix allows you to build web application.
Apache Jackrabbit is the reference implementation of the JCR API. The JCR API is to manage content repositories; to manage, for example, web content. A content repository is a mix between file system and a database.
The JCR API is specially made to deal with web content. Why use the JCR API, and why not use a relational database API? URLs are hierarchical, as in a file system. Relational databases don't easily support hierarchical access. Why not use a file system API? Because the JCR supports transactions, versioning, and a lot of other features that file system APIs don't support.
Apache Sling is a web framework based on the JCR API, and taking advantage of the features provided by the JCR API (15 Minute introduction).
Apache Felix is an OSGi container. It allows to seamlessly start, stop, and replace components of a web application (jar files, in a sense), while the web server is running. That means it allows you to change the application without having to restart the server.
Sling in very simple terms could be described as a REST API for JCR. you can use http requests to manage content inside the repository.
Additionally, Sling provides a mechanism to render that content in different ways for web consumption. you can use scripts (JSP for example) and the java code (servlets, pojos, etc) in the Felix container to process requests and deliver a request.
When a request is made for a particular node, Sling looks up for a property called sling:resourceType, this is a lookup key for rendering scripts. Then the appropiate script is executed using the node as input.
You could write different kinds for renderers and then use it to display your content in different ways.
For example, you could write two scripts full.json.jsp and short.json.jsp and then use them to render the same node in two different ways:
/content/app/node.full.json
OR
/content/app/node.short.json.
Sling basically matches tokens in the request URL to select an appropriate script.
They have a really nice cheat sheet that explains how request resolution and rendering works
it is a bit more complex than this, since everything is organized in resources and components. you want to check their site for more info
I had the same doubts. The best response I was able to find is in the official Sling page (https://sling.apache.org/)
(What is) Apache Sling, in a hundred words:
Apache Sling is a web framework that uses a Java Content Repository, such as Apache Jackrabbit, to store and manage content.
Sling applications use either scripts or Java servlets, selected based on simple name conventions, to process HTTP requests in a RESTful way.
The embedded Apache Felix OSGi framework and console provide a dynamic runtime environment, where code and content bundles can be loaded, unloaded and reconfigured at runtime.
So, resuming it:
Sling is a web framework --> using jackrabbit --> based/supported on JCR API.
You can see Apache Felix like a container with its manager.
Note that Sling started as an internal project at Day Software. It's the reason why some bundles/libraries are named like com.day, but in the ends they are two names for the same.
Also, if you want to be clear about Jackrabbit and JCR API you can visit the Jackrabbit's official page http://jackrabbit.apache.org/jcr/jackrabbit-architecture.html
I am starting research on how to implement Node.js SOA (service oriented architecture) with JSON web-services.
As a small sub-question, I need an approach/framework/system to make universal configuration center for all companies web-services. So that we don't configure every application with exact address of other application, but just link to some central server to get that information.
(This should be very well worked-out topic for XML-based services, so some terminology/approaches/etc could/should be borrowed.)
Related to
RESTful JSON based SOA Registry
Service Oriented Architecture suggestions
UPDATE: This questions is about web-services configuration & orchestration.
GO for an active(having activity happening off late) framework with lean architecture.There's one called Geddy and another called Restify. If in doubt, Express can also be used for building webservices with JSON.
You can work on reading the centrally stored config from different app codebse when you use any of these.
Time and again I am faced with the issue of having multiple environments that must be configured individually for an application that would run in all of them (e.g. QA, regional production env's, dev, staging, etc.) and I am wondering what would be the best way to organize different configurations?
Would it be in the database? Different configuration files per environment? Or maybe the same file with different sections/xml tags? How would these be then deployed? Embedded within the app? Or put manually in after installation to be modified in-place?
This question is not technology-specific - I've worked with .net and Java, web-apps and desktop apps and this issue comes up time and again. I'm looking to learn different approaches to maybe adapt a hybrid to address this.
EDIT: There's one caveat that I must point out - when configuration is part of the deployed solution, it is generally installed under root user on the host. In large organizations developers usually don't have a root access to production hosts so any changes to the configuration require a new build and deployment. Needless to say this isn't the nicest approach - especially at organizations that have a very strict release process involving multiple teams and approval levels... (sigh I know!)
Borrowed from Jez Humble and David Farley's book "Continuous Delivery (page 41)", you can:
Your build scripts can pull configuration in and incorporate it into your binaries at build time.
Your packaging software can inject configuration at packaging time, such as when creating assemblies, ears, or gems.
Your deployment scripts or installers can fetch the necessary information or ask the user for it and pass it to your application at
deployment time as part of the installation process.
Your application itself can fetch configuration at startup time or run time.
It is considered bad practice by them to inject configuration files in build and compile times, because you should be able to deploy the same binary file to every environments.
My experience was that you could bake all configuration files for every environments (except sensitive information) to your deployment file (war, jar, zip, etc). And you design your application to take in an extra parameter when starts, to pickup the right sets of configuration files (from your extracted deployment file, or from local/remote file system if they are sensitive, or from a database) during application's startup time.
The question is difficult to answer because it's somewhat vague. There is no technology-agnostic approach to configuration as far as I know. Exactly how configuration is set up will depend on the language/technology in question.
I'm not familiar with .net but with java a popular approach is to have a maven build set up with different profiles. Each profile is specific to an environment. You can then define different properties files that have environment-specific values, an example from the above link is:
environment.properties - This is the default configuration and will be packaged in the artifact by default.
environment.test.properties - This is the variant for the test environment.
environment.prod.properties - This is basically the same as the test variant and will be used in the production environment.
You can then build your project as follows:
mvn -Pprod package
I have good news and bad news.
The good news is that Config4* (of which I am the maintainer) neatly addresses this issue with its support for adaptive configuration. Basically, this is the ability for a configuration file to adapt itself to the environment (including hostname, username, environment variables, and command-line options) in which it is running. Read Chapter 2 of the "Getting Started" manual for details. Don't worry: it is a short chapter.
The bad news is that, currently, Config4* implementations exist only for C++ and Java, so your .Net applications are out of luck. And even with C++ and Java applications, it won't make pragmatic sense to retrofit Config4* into an existing application. Because of this, I'd advise trying to use Config4* only in new applications.
Despite the bad news, I think it is worth your while to read the above-mentioned chapter of the Config4* documentation, because doing so may provide you with ideas that you can adapt to fit your needs.