I'm have successfully deployed Nexus3 on openshift cluster via Nexus3 Helm Chart.
In order to feed the nexus container logs in the EFK stack I want nexus container to output logs in JSON format.
I'm unable to find some documentation to change the logging format for Nexus3.
How can I modify the configurations in image so that the std output is in JSON and easily parsed?
Related
electron-builder generates "latest.yml" blockmap and exe for windows. But in production environment yml is not accepted. Need to change "latest.yml" to "latest.json". What are the configuration require to change "latest.yml" to "latest.json"?
electron-builder#^22.9.1
We tried it , there are no configuration options to change to json.We converted from yml to json at jenkins build . Electron-builder is using js-yaml node module to parse the yml response, which will accept both json and yml. If you send json instead of yml present version of electron-updater will accept and works fine.
Is there a way to convert the error.log in Nginx to json? I need to ship the logs to an external log viewer, to do that I need to convert the error.log to json.
Using filebeat may be an alternative depending on were you want to ship your logs (see https://www.elastic.co/guide/en/beats/filebeat/current/configuring-output.html)
I have been trying to set up keycloak logging to be scraped by fluentd to be used in elasticsearch. So far I have used the provided CLI string to use in my helm values script.
cli:
# Custom CLI script
custom: |
/subsystem=logging/json-formatter=json:add(exception-output-type=formatted, pretty-print=true, meta-data={label=value})
/subsystem=logging/console-handler=CONSOLE:write-attribute(name=named-formatter, value=json)
However, as you can see in the picture provided, the logs that are generated seem to be completely json apart from the core of the log, the message field. Currently the message field is provided as comma separated key-value pairs. Is there any way to tell keycloak, jboss or wildfly that it needs to provide the message in JSON too? This allows me to efficiently search through the data in elastic.
Check this project on GitHub: keycloak_jsonlog_eventlistener: Outputs Keycloak events as JSON into the server log.
Keycloak JSON Log Eventlistener
Primarily written for the Jboss Keycloak docker image, it will output Keycloak events as JSON into the keycloak server log.
The idea is to parse logs once they get to logstash via journalbeat.
Tested with Keycloak version 8.0.1
I have Keycloak deployed in Kubernetes using the official codecentric chart. Now I want to make Keycloak logs into json format in order to export them to Kibana.
A comment to the original reply pointed to a cli command to do this.
cli:
# Custom CLI script
custom: |
/subsystem=logging/json-formatter=json:add(exception-output-type=formatted, pretty-print=false, meta-data={label=value})
/subsystem=logging/console-handler=CONSOLE:write-attribute(name=named-formatter, value=json)
It is a Java application that is running on Wildfly. If you check the main process that is running inside the pod, you will see something like:
/usr/lib/jvm/java/bin/java -D[Standalone] -server -Xms64m -Xmx512m -XX:MetaspaceSize=96M -XX:MaxMetaspaceSize=256m -Djava.net.preferIPv4Stack=true -Djboss.modules.system.pkgs=org.jboss.byteman -Djava.awt.headless=true -Dorg.jboss.boot.log.file=/opt/jboss/keycloak/standalone/log/server.log -Dlogging.configuration=file:/opt/jboss/keycloak/standalone/configuration/logging.properties -jar /opt/jboss/keycloak/jboss-modules.jar -mp /opt/jboss/keycloak/modules org.jboss.as.standalone -Djboss.home.dir=/opt/jboss/keycloak -Djboss.server.base.dir=/opt/jboss/keycloak/standalone -Djboss.bind.address=10.217.0.231 -Djboss.bind.address.private=10.217.0.231 -b 0.0.0.0 -c standalone.xml
Important part here is the following:
-Dlogging.configuration=file:/opt/jboss/keycloak/standalone/configuration/logging.properties
So, the logging configuration is passed to the Java process as a JVM option, and read from the file on the path /opt/jboss/keycloak/standalone/configuration/logging.properties.
If you check the content of the file, it has a section like the following:
...
handler.CONSOLE=org.jboss.logmanager.handlers.ConsoleHandler
handler.CONSOLE.level=INFO
handler.CONSOLE.formatter=COLOR-PATTERN
handler.CONSOLE.properties=autoFlush,target,enabled
handler.CONSOLE.autoFlush=true
handler.CONSOLE.target=SYSTEM_OUT
handler.CONSOLE.enabled=true
...
You need to figure out what to change in this logging configuration to meet your JSON requirements. An example would be:
formatter.json=org.jboss.logmanager.formatters.JsonFormatter
formatter.json.properties=keyOverrides,exceptionOutputType,metaData,prettyPrint,printDetails,recordDelimiter
formatter.json.constructorProperties=keyOverrides
formatter.json.keyOverrides=timestamp\=#timestamp
formatter.json.exceptionOutputType=FORMATTED
formatter.json.metaData=#version\=1
formatter.json.prettyPrint=false
formatter.json.printDetails=false
formatter.json.recordDelimiter=\n
Then, in Kubernetes you can create a ConfigMap with the logging config that you want, define it as a volume in your pod/deployment, and mount it as a file to that exact path in the pod/deployment definition. If you do all steps correctly, you should be able to customize the logging format as you need.
I tried to set-up a elasticsearch on my Windows 7 OS PC. Installed elasticsearch and curl and it's working as the loacahost:9200 is working fine.
Now I am strugging to search in a file located at c:\user\rajesh\raj.txt.
My doubt is, Where do mention that I have tos search in this file? elasticsearch.yml? Which parameter I need to set to point this text file?
Indexing is working with curl but mapping gives nullpointer exception? Do I need to install something else?
I tried to install sense plugin for chrome but says moved to marvel, and from there unable to install marvel!
From what I can tell, you've installed Elasticsearch and you're now expecting to be able to search within files on your local file system. This isn't how ES works. You need to create a mapping for an index and then populate that index with the content you want to search in. If you're looking to index files on your local file system rather than data you have pulled from a database you should look in to the File system River Plugin for Elasticsearch, http://www.pilato.fr/fsriver/. This deals with all of the indexing of file system based documents automatically, once you've got it set up correctly.
EDIT:
I also see you're trying to set up Kibana and Marvel/Sense. To set up Kibana just follow the instructions here: http://www.elasticsearch.org/overview/kibana/installation/
To set up Marvel open powershell, CD to C:\elasticsearch\bin then run plugin.bat -i elasticsearch/marvel/latest then you'll need to restart your cluster. Once you've done that if you go to http://localhost:9200/_plugin/marvel/ you'll see your marvel dashboard. You'll also see a tab for "Sense" which is the other plugin you referred to.
If you are using elastic search for retrieving data from any DB like PostgreSQL, then go to folder bin/rivers.bat and edit as
curl -XPUT localhost:9200/_river/actor_jdbc_river/_meta -d "{\"type\":\"jdbc\",\"jdbc\":{\"strategy\":\"simple\",\"poll\":\"1h\",\"driver\":\"org.postgresql.Driver\",\"url\":\"jdbc:postgresql://10.5.2.132:5432/prodDB\",\"user\":\"UserName\",\"password\":\"Password\",\"sql\":\"select t.id as _id,t.name from topic as t \",\"digesting\" : true},\"index\":{\"index\":\"jdbc\",\"type\":\"actor_jdbc_river1\"}}"
Then create a client in Java side to access data in river.
Here cluster name is same as that mention in folder config/elasticsearch.yml (testDBsearch)
private static Client createClient() {
//Create Client
Settings settings = ImmutableSettings.settingsBuilder().put("cluster.name", "testDBsearch").build();
TransportClient transportClient = new TransportClient(settings);
transportClient = transportClient.addTransportAddress(new InetSocketTransportAddress("10.5.2.132", 9300));
return (Client) transportClient;
}
public static void main(String[] args) {
Client client = createClient();
String queryString = "python";
search(client, 100, queryString);
}
public static void search(Client client,int size, String queryString) {
queryString=queryString +"*";
try{
SearchResponse responseActor;
responseActor = client.prepareSearch("jdbc").setTypes("actor_jdbc_river1").setSearchType(SearchType.DEFAULT)
.setQuery(QueryBuilders.queryString(queryString)
.field("designation",new Float(2.0)).field("name", new Float(5.0)).field("email") .defaultOperator(Operator.OR)).setFrom(0).setSize(size).setExplain(true).execute().actionGet();
for(SearchHit hit:responseActor.getHits()) {
System.out.println(hit.getSourceAsString());
System.out.println(hit.getScore());
System.out.println("---------------------------");
}
}catch(Exception e){
System.out.println("Error in elastic search "+queryString+" Error :"+e);
}
}
clear installation of elasticsearch in windows:
1) check whether your system has latest java version
2) download and extract elasticsearch from "download.elastic.co/elasticsearch/release/org/elasticsearch/distribution/zip/elasticsearch/2.3.3/elasticsearch-2.3.3.zip"
3) set JAVA_HOME environment variable "C:\Program Files (x86)\Java\jdk1.8.0_91"
4) check JAVA_HOME environment variable using command "service" in bin directry of elasticsearch shown in below figure checking whether JAVA_HOME is set properly or not
5) install service.bat using command service.bat install
6) uncomment network.host and give value as localhost in configuration file of elasticsearch
network.host= localhost in elasticsearch.yml (config file)
7)run elasticsearch "C:\elasticsearch-2.3.3\bin\elasticsearch"
if you get error while running elastic search saying update JVM to latest version means java in your system is not containing latest version (install and run latest java version)
8)install elasticsearch-head plugin to visualize things in elasticsearch
run command "plugin install elasticsearch-head"
if its failed to install elasticsearch-head then use command-
plugin install "github.com/mobz/elasticsearch-head/archive/master.zip"
9)open elasticsearch in browser using link "localhost:9200/_plugin/head/"
elasticsearch visual interface