I'm trying to load csv file in Elasticsearch using filebeat.
Here is my filebeat.yml
filebeat.inputs:
#- type: log
- type: stdin
setup.template.overwrite: true
enabled: true
close_eof: true
paths:
- /usr/share/filebeat/dockerlogs/*.csv
processors:
- decode_csv_fields:
fields:
message: "message"
separator: ","
ignore_missing: false
overwrite_keys: true
trim_leading_space: true
fail_on_error: true
- drop_fields:
fields: [ "log", "host", "ecs", "input", "agent" ]
- extract_array:
field: message
mappings:
sr: 0
Identifiant PSI: 1
libellé PSI: 2
Identifiant PdR: 3
T3 Date Prévisionnelle: 4
DS Reporting PdR: 5
Status PSI: 6
Type PdR: 7
- drop_fields:
fields: ["message","sr"]
#index: rapport_g035_prov_1
filebeat.registry.path: /usr/share/filebeat/data/registry/filebeat/filebeat
output:
elasticsearch:
enabled: true
hosts: ["IPAdress:8081"]
indices:
- index: "rapport_g035_prov"
#- index: "filebeat-%{[agent.version]}-%{+yyyy.MM.dd}"
#- index: "filebeat-7.7.0"
#setup.dashboards.kibana_index: file-*
seccomp.enabled: false
logging.metrics.enabled: false
But when I check the index in kibana I found that the index can't read column names
https://i.stack.imgur.com/renYt.png
I tried to process the csv if filebeat.yml in another way
filebeat.inputs:
- type: log
setup.template.overwrite: true
enabled: true
close_eof: true
paths:
- /usr/share/filebeat/dockerlogs/*.csv
processors:
- decode_csv_fields:
fields:
message: decoded.csv
separator: ","
ignore_missing: false
overwrite_keys: true
trim_leading_space: false
fail_on_error: true
#index: rapport_g035_prov_1
filebeat.registry.path: /usr/share/filebeat/data/registry/filebeat/filebeat
output:
elasticsearch:
enabled: true
hosts: ["IPAdress:8081"]
indices:
- index: "rapport_g035_prov"
#- index: "filebeat-%{[agent.version]}-%{+yyyy.MM.dd}"
#- index: "filebeat-7.7.0"
#setup.dashboards.kibana_index: file-*
seccomp.enabled: false
logging.metrics.enabled: false
But I got the same error it can't map the index correctly I know there is a problem in the processing of csv in the filebeat.yml but I don't know what is it
Related
I am trying to get switch user feature in spring security to work. I am using grails 4.0.10 and mysql 8.
I created a sample hello world grails app and followed the switch user guide from the documentation. https://grails.github.io/grails-spring-security-core/4.0.x/index.html#switchUser
If i use the default h2 database then it works but if i switch to the mysql 8 database it is throwing page not found 404 error and it is not switching.
i have published the code in github. here is the link.
https://github.com/sanjaygir/switching
I have created a simple page in secure controller. The page is index.gsp that has a form to switch to another user. Logged in user should be displayed at the top of this page. In bootstrap file i have created two users. one admin and another regular user.
i have a local database with this configuration
dataSource:
dbCreate: create
url: jdbc:mysql://localhost:3307/switch?useUnicode=yes&characterEncoding=UTF-8
username: root
password: password
In order to run this app you need a mysql 8 db running. please change the mysql db name and username and password in the above section in application.yml.
After the app fires please go directly to http://localhost:8080/secure/index and then enter in the textbox "user" and click on the button switch. It will throw an error page not found and if you go back to http://localhost:8080/secure/index you can not see at the top loggedin user name. That means the switch was not successful.
here is the simple code for secure/index.gsp
<%# page contentType="text/html;charset=UTF-8" %>
<html>
<head>
<title></title>
</head>
<body>
<sec:ifLoggedIn>
Logged in as <sec:username/>
</sec:ifLoggedIn>
<form action='${request.contextPath}/login/impersonate' method='POST'>
Switch to user: <input type='text' name='username'/> <br/>
<input type='submit' value='Switch'/>
</form>
</body>
</html>
i hope i have made it clear. this is a simple hello world app created to see switch user feature i n action. I am puzzled why switch user works with default h2 db but not when connected to mysql 8. if anyone have any idea i appreciate your help. Thanks
UPDATE:
Today i switched the database to mysql version 5 and it works.
I changed the following configuration in application.yml
hibernate:
cache:
queries: false
use_second_level_cache: false
use_query_cache: false
dataSource:
pooled: true
jmxExport: true
driverClassName: com.mysql.jdbc.Driver
dialect: org.hibernate.dialect.MySQL5InnoDBDialect
username: root
password: 'password'
environments:
development:
dataSource:
dbCreate: create-drop
url: jdbc:mysql://localhost:3306/switch?useUnicode=yes&characterEncoding=UTF-8
in build.gradle i used
runtime 'mysql:mysql-connector-java:5.1.19'
still i am not sure why it doesnt work in mysql 8.
i finally found the bug. i cannot believe what caused the 404 not found issue.it was a single line in the configuration file.
before the application.yml looked like this
---
grails:
profile: web
codegen:
defaultPackage: rcroadraceweb4
gorm:
reactor:
# Whether to translate GORM events into Reactor events
# Disabled by default for performance reasons
events: false
info:
app:
name: '#info.app.name#'
version: '#info.app.version#'
grailsVersion: '#info.app.grailsVersion#'
spring:
jmx:
unique-names: true
main:
banner-mode: "off"
groovy:
template:
check-template-location: false
devtools:
restart:
additional-exclude:
- '*.gsp'
- '**/*.gsp'
- '*.gson'
- '**/*.gson'
- 'logback.groovy'
- '*.properties'
management:
endpoints:
enabled-by-default: false
server:
servlet:
context-path: '/roadrace'
---
hibernate:
cache:
queries: false
use_second_level_cache: false
use_query_cache: false
grails:
plugin:
databasemigration:
updateOnStart: true
updateOnStartFileName: changelog.groovy
controllers:
upload:
maxFileSize: 2000000
maxRequestSize: 2000000
mail:
host: "localhost"
port: 25
default:
to: 'root#localhost'
from: 'noreply#runnercard.com'
dataSource:
type: com.zaxxer.hikari.HikariDataSource
pooled: true
driverClassName: com.mysql.cj.jdbc.Driver
dialect: org.hibernate.dialect.MySQL8Dialect
dbCreate: none
properties:
minimumIdle: 5
maximumPoolSize: 10
poolName: main-db
cachePrepStmts: true
prepStmtCacheSize: 250
prepStmtCacheSqlLimit: 2048
useServerPrepStmts: true
useLocalSessionState: true
rewriteBatchedStatements: true
cacheResultSetMetadata: true
cacheServerConfiguration: true
elideSetAutoCommits: true
maintainTimeStats: false
dataSources:
logging:
# This is not used unless `useJdbcAccessLogger` or `useJdbcLogger` is set to `true`
# This does not need to be setup unless it is in use.
type: com.zaxxer.hikari.HikariDataSource
pooled: true
driverClassName: com.mysql.cj.jdbc.Driver
properties:
minimumIdle: 2
maximumPoolSize: 5
poolName: logging-db
cachePrepStmts: true
prepStmtCacheSize: 250
prepStmtCacheSqlLimit: 2048
useServerPrepStmts: true
useLocalSessionState: true
rewriteBatchedStatements: true
cacheResultSetMetadata: true
cacheServerConfiguration: true
elideSetAutoCommits: true
maintainTimeStats: false
environments:
development:
dataSource:
dbCreate: none
url: jdbc:mysql://localhost:3307/dev2?useUnicode=yes&characterEncoding=UTF-8
username: root
password: password
grails:
# mail:
# host: "smtp.gmail.com"
# port: 465
# username: "justforstackoverflow123#gmail.com"
# password: "1asdfqwef1"
# props:
# "mail.smtp.auth": "true"
# "mail.smtp.socketFactory.port": "465"
# "mail.smtp.socketFactory.class": "javax.net.ssl.SSLSocketFactory"
# "mail.smtp.socketFactory.fallback": "false"
test:
dataSource:
# dialect: org.hibernate.dialect.MySQL5InnoDBDialect
dbCreate: none
url: jdbc:mysql://localhost:3307/test?useUnicode=yes&characterEncoding=UTF-8
username: root
password: password
production:
---
logging:
level:
root: INFO
org.springframework: WARN
grails.plugin.springsecurity.web.access.intercept.AnnotationFilterInvocationDefinition: ERROR
grails.plugins.DefaultGrailsPluginManager: WARN
org.hibernate: ERROR # TODO: we need to lower this, and fix the warnings this is talking about.
rcroadraceweb4: DEBUG
com.runnercard: DEBUG
liquibase.ext.hibernate.snapshot.HibernateSnapshotGenerator: ERROR
---
#debug: true
#useJdbcSessionStore: true
---
environments:
nateDeploy:
behindLoadBalancer: true
grails:
insecureServerURL: 'https://nate-dev.nate-palmer.com/roadrace'
serverURL: 'https://nate-dev.nate-palmer.com/roadrace'
dataSource:
url: 'jdbc:mysql://10.1.10.240:3306/rcroadwebDEV?serverTimezone=America/Denver'
after the fix it looks like this
---
grails:
profile: web
codegen:
defaultPackage: rcroadraceweb4
gorm:
reactor:
# Whether to translate GORM events into Reactor events
# Disabled by default for performance reasons
events: false
info:
app:
name: '#info.app.name#'
version: '#info.app.version#'
grailsVersion: '#info.app.grailsVersion#'
spring:
jmx:
unique-names: true
main:
banner-mode: "off"
groovy:
template:
check-template-location: false
devtools:
restart:
additional-exclude:
- '*.gsp'
- '**/*.gsp'
- '*.gson'
- '**/*.gson'
- 'logback.groovy'
- '*.properties'
management:
endpoints:
enabled-by-default: false
server:
servlet:
context-path: '/roadrace'
---
hibernate:
cache:
queries: false
use_second_level_cache: false
use_query_cache: false
grails:
plugin:
databasemigration:
updateOnStart: true
updateOnStartFileName: changelog.groovy
controllers:
upload:
maxFileSize: 2000000
maxRequestSize: 2000000
mail:
host: "localhost"
port: 25
default:
to: 'root#localhost'
from: 'noreply#runnercard.com'
dataSource:
type: com.zaxxer.hikari.HikariDataSource
pooled: true
driverClassName: com.mysql.cj.jdbc.Driver
dialect: org.hibernate.dialect.MySQL8Dialect
dbCreate: none
properties:
minimumIdle: 5
maximumPoolSize: 10
poolName: main-db
cachePrepStmts: true
prepStmtCacheSize: 250
prepStmtCacheSqlLimit: 2048
useServerPrepStmts: true
useLocalSessionState: true
rewriteBatchedStatements: true
cacheResultSetMetadata: true
cacheServerConfiguration: true
elideSetAutoCommits: true
maintainTimeStats: false
dataSources:
logging:
# This is not used unless `useJdbcAccessLogger` or `useJdbcLogger` is set to `true`
# This does not need to be setup unless it is in use.
type: com.zaxxer.hikari.HikariDataSource
pooled: true
driverClassName: com.mysql.cj.jdbc.Driver
properties:
minimumIdle: 2
maximumPoolSize: 5
poolName: logging-db
cachePrepStmts: true
prepStmtCacheSize: 250
prepStmtCacheSqlLimit: 2048
useServerPrepStmts: true
useLocalSessionState: true
rewriteBatchedStatements: true
cacheResultSetMetadata: true
cacheServerConfiguration: true
elideSetAutoCommits: true
maintainTimeStats: false
environments:
development:
dataSource:
dbCreate: none
url: jdbc:mysql://localhost:3307/dev2?useUnicode=yes&characterEncoding=UTF-8
username: root
password: password
# grails:
# mail:
# host: "smtp.gmail.com"
# port: 465
# username: "justforstackoverflow123#gmail.com"
# password: "1asdfqwef1"
# props:
# "mail.smtp.auth": "true"
# "mail.smtp.socketFactory.port": "465"
# "mail.smtp.socketFactory.class": "javax.net.ssl.SSLSocketFactory"
# "mail.smtp.socketFactory.fallback": "false"
test:
dataSource:
# dialect: org.hibernate.dialect.MySQL5InnoDBDialect
dbCreate: none
url: jdbc:mysql://localhost:3307/test?useUnicode=yes&characterEncoding=UTF-8
username: root
password: password
production:
---
logging:
level:
root: INFO
org.springframework: WARN
grails.plugin.springsecurity.web.access.intercept.AnnotationFilterInvocationDefinition: ERROR
grails.plugins.DefaultGrailsPluginManager: WARN
org.hibernate: ERROR # TODO: we need to lower this, and fix the warnings this is talking about.
rcroadraceweb4: DEBUG
com.runnercard: DEBUG
liquibase.ext.hibernate.snapshot.HibernateSnapshotGenerator: ERROR
---
#debug: true
#useJdbcSessionStore: true
---
environments:
nateDeploy:
behindLoadBalancer: true
grails:
insecureServerURL: 'https://nate-dev.nate-palmer.com/roadrace'
serverURL: 'https://nate-dev.nate-palmer.com/roadrace'
dataSource:
url: 'jdbc:mysql://10.1.10.240:3306/rcroadwebDEV?serverTimezone=America/Denver'
it was this line in environments > development block
# grails:
it worked after commenting the grails line.
but all the contents of grails block is commented so i am still confused why having grails uncommented would have this big issue. anyways solved after days of hard search!
I'm working with Filebeat 7.9.3 as a daemonset on k8s.
I'm not able to parse docker container logs of a Springboot app that writes logs to stdout in json.
The fact is that the every row of the Springboot app logs is written in this way:
{ "#timestamp": "2020-11-16T13:39:57.760Z", "log.level": "INFO", "message": "Checking comment 'se' done = true", "service.name": "conduit-be-moderator", "event.dataset": "conduit-be-moderator.log", "process.thread.name": "http-nio-8081-exec-2", "log.logger": "it.koopa.app.ModeratorController", "transaction.id": "1ed5c62964ff0cc2", "trace.id": "20b4b28a3817c9494a91de8720522972"}
But the corresponding docker log file under /var/log/containers/ writes log in this way:
{
"log": "{\"#timestamp\":\"2020-11-16T11:27:32.273Z\", \"log.level\": \"INFO\", \"message\":\"Checking comment 'a'\", \"service.name\":\"conduit-be-moderator\",\"event.dataset\":\"conduit-be-moderator.log\",\"process.thread.name\":\"http-nio-8081-exec-4\",\"log.logger\":\"it.koopa.app.ModeratorController\",\"transaction.id\":\"9d3ad972dba65117\",\"trace.id\":\"8373edba92808d5e838e07c7f34af6c7\"}\n",
"stream": "stdout",
"time": "2020-11-16T11:27:32.274816903Z"
}
I always receive this on filebeat logs
Error decoding JSON: json: cannot unmarshal number into Go value of type map[string]interface {}
This is my filebeat config that tries to parse json log message from docker logs where I'm using decode_json_fields to try to catch Elasticsearch standard fields (I'm using co.elastic.logging.logback.EcsEncoder)
filebeat.yml: |-
filebeat.inputs:
- type: container
#json.keys_under_root: true
json.overwrite_keys: true
json.add_error_key: true
json.message_key: log
paths:
- /var/log/containers/*.log
include_lines: "conduit-be-moderator"
processors:
- decode_json_fields:
fields: ["log"]
overwrite_keys: true
- add_kubernetes_metadata:
host: ${NODE_NAME}
in_cluster: true
matchers:
- logs_path:
logs_path: "/var/log/containers/"
processors:
- add_cloud_metadata:
- add_host_metadata:
How can I do this???
As processors are applied before the JSON parser of the input, you will need to first configure the decode_json_fields processors which will allow you to decode your json.log field. You will then be able to apply the json configuration fo the inputs on the message fields. Something like:
filebeat.yml: |-
filebeat.inputs:
- type: container
json.keys_under_root: true
json.overwrite_keys: true
json.add_error_key: true
json.message_key: message
paths:
- /var/log/containers/*.log
include_lines: "conduit-be-moderator"
processors:
- decode_json_fields:
fields: ['log']
expand_keys: true
- add_kubernetes_metadata:
host: ${NODE_NAME}
in_cluster: true
matchers:
- logs_path:
logs_path: "/var/log/containers/"
processors:
- add_cloud_metadata:
- add_host_metadata:
This configuration assumes that all your logs use JSON format. Else you will probably need to add an exclude or include regex pattern.
I've been following https://quarkus.io/guides/kubernetes-config in order to create a configMap and test my Quarkus service into my CDK v3.5.0-1 before to push it to OpenShift 3.11 but KubernetesConfigSourceProvider is not happy:
Using:
Quarkus 1.8.1.Final
Java 11
CDK 3.5
Here is the yaml file I want to convert into a configMap doing: oc create configmap quarkus-service-configmap --from-file=application.yml
jaeger_endpoint: http://192.168.56.100:14268/api/traces
jaeger_sampler_manager_host_port: 192.168.56.100:14250
sql_debug: false
quarkus:
datasource:
db-kind: h2
jdbc:
detect-statement-leaks: true
driver: io.opentracing.contrib.jdbc.TracingDriver
enable-metrics: true
url: jdbc:tracing:h2:./db;AUTO_SERVER=TRUE
max-size: 13
metrics:
enabled: false
password: sa
username: sa
flyway:
locations: db/prod/migration
migrate-at-start: true
hibernate-orm:
database:
charset: UTF-8
generation: none
dialect: org.hibernate.dialect.H2Dialect
http:
port: 6280
jaeger:
enabled: true
endpoint: ${jaeger_endpoint}
sampler-manager-host-port: ${jaeger_sampler_manager_host_port}
sampler-param: 1
sampler-type: const
resteasy:
gzip:
enabled: true
max-input: 10M
smallrye-health:
ui:
always-include: true
swagger-ui:
always-include: true
Here is the generated configMap:
apiVersion: v1
data:
application.yml: |
jaeger_endpoint: http://192.168.56.100:14268/api/traces
jaeger_sampler_manager_host_port: 192.168.56.100:14250
sql_debug: false
quarkus:
datasource:
db-kind: h2
jdbc:
detect-statement-leaks: true
driver: io.opentracing.contrib.jdbc.TracingDriver
enable-metrics: true
url: jdbc:tracing:h2:./db;AUTO_SERVER=TRUE
max-size: 13
metrics:
enabled: false
password: sa
username: sa
flyway:
locations: db/prod/migration
migrate-at-start: true
hibernate-orm:
database:
charset: UTF-8
generation: none
dialect: org.hibernate.dialect.H2Dialect
http:
port: 6280
jaeger:
enabled: true
endpoint: ${jaeger_endpoint}
sampler-manager-host-port: ${jaeger_sampler_manager_host_port}
sampler-param: 1
sampler-type: const
resteasy:
gzip:
enabled: true
max-input: 10M
smallrye-health:
ui:
always-include: true
swagger-ui:
always-include: true
kind: ConfigMap
metadata:
creationTimestamp: '2020-09-21T17:56:40Z'
name: quarkus-service-configmap
namespace: dci
resourceVersion: '9572968'
selfLink: >-
/api/v1/namespaces/dci/configmaps/quarkus-service-configmap
uid: cd4570ff-fc33-11ea-bff0-080027af1c97
Here my quarkus-service/src/main/resources/application.yml:
quarkus:
application:
name: quarkus-service
kubernetes-config: # https://quarkus.io/guides/kubernetes-config
enabled: true
fail-on-missing-config: true
config-maps: quarkus-service-configmap
# secrets: quarkus-service-secrets
jaeger:
service-name: ${quarkus.application.name}
http:
port: 6280
log:
category:
"io.quarkus.kubernetes.client":
level: DEBUG
"io.fabric8.kubernetes.client":
level: DEBUG
console:
format: '%d{HH:mm:ss} %-5p traceId=%X{traceId}, spanId=%X{spanId}, sampled=%X{sampled} [%c{2.}] (%t) %s%e%n'
native:
additional-build-args: -H:ReflectionConfigurationFiles=reflection-config.json
'%minishift':
quarkus:
kubernetes: # https://quarkus.io/guides/deploying-to-openshift / https://quarkus.io/guides/kubernetes
container-image:
group: dci
registry: image-registry.openshift-image-registry.svc:5000
deploy: true
expose: true
The command I run: mvn clean package -Dquarkus.profile=minishift
The result I get:
WARN: Unrecognized configuration key "%s" was provided; it will be ignored; verify that the dependency extension for this configuration is set or you did not make a typo
Sep 21, 2020 6:36:15 PM io.quarkus.config
WARN: Unrecognized configuration key "%s" was provided; it will be ignored; verify that the dependency extension for this configuration is set or you did not make a typo
Sep 21, 2020 6:36:15 PM org.hibernate.validator.internal.util.Version
INFO: HV000001: Hibernate Validator %s
Sep 21, 2020 6:36:27 PM io.quarkus.application
ERROR: Failed to start application (with profile minishift)
java.lang.RuntimeException: Unable to obtain configuration for ConfigMap objects from Kubernetes API Server at: https://172.30.0.1:443/
at io.quarkus.kubernetes.client.runtime.KubernetesConfigSourceProvider.getConfigMapConfigSources(KubernetesConfigSourceProvider.java:85)
at io.quarkus.kubernetes.client.runtime.KubernetesConfigSourceProvider.getConfigSources(KubernetesConfigSourceProvider.java:45)
at io.quarkus.runtime.configuration.ConfigUtils.addSourceProvider(ConfigUtils.java:107)
at io.quarkus.runtime.configuration.ConfigUtils.addSourceProviders(ConfigUtils.java:121)
at io.quarkus.runtime.generated.Config.readConfig(Config.zig:2060)
at io.quarkus.deployment.steps.RuntimeConfigSetup.deploy(RuntimeConfigSetup.zig:60)
at io.quarkus.runner.ApplicationImpl.doStart(ApplicationImpl.zig:509)
at io.quarkus.runtime.Application.start(Application.java:90)
at io.quarkus.runtime.ApplicationLifecycleManager.run(ApplicationLifecycleManager.java:91)
at io.quarkus.runtime.Quarkus.run(Quarkus.java:61)
at io.quarkus.runtime.Quarkus.run(Quarkus.java:38)
at io.quarkus.runtime.Quarkus.run(Quarkus.java:106)
at io.quarkus.runner.GeneratedMain.main(GeneratedMain.zig:29)
Caused by: io.fabric8.kubernetes.client.KubernetesClientException: Operation: [get] for kind: [ConfigMap] with name: [quarkus-service-configmap] in namespace: [dci] failed.
at io.fabric8.kubernetes.client.KubernetesClientException.launderThrowable(KubernetesClientException.java:64)
at io.fabric8.kubernetes.client.KubernetesClientException.launderThrowable(KubernetesClientException.java:72)
at io.fabric8.kubernetes.client.dsl.base.BaseOperation.getMandatory(BaseOperation.java:244)
at io.fabric8.kubernetes.client.dsl.base.BaseOperation.get(BaseOperation.java:187)
at io.fabric8.kubernetes.client.dsl.base.BaseOperation.get(BaseOperation.java:79)
at io.quarkus.kubernetes.client.runtime.KubernetesConfigSourceProvider.getConfigMapConfigSources(KubernetesConfigSourceProvider.java:69)
... 12 more
Caused by: java.net.SocketTimeoutException: Read timed out
at java.base/java.net.SocketInputStream.socketRead0(Native Method)
at java.base/java.net.SocketInputStream.socketRead(SocketInputStream.java:115)
at java.base/java.net.SocketInputStream.read(SocketInputStream.java:168)
at java.base/java.net.SocketInputStream.read(SocketInputStream.java:140)
at java.base/sun.security.ssl.SSLSocketInputRecord.read(SSLSocketInputRecord.java:467)
at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(SSLSocketInputRecord.java:461)
at java.base/sun.security.ssl.SSLSocketInputRecord.decode(SSLSocketInputRecord.java:160)
at java.base/sun.security.ssl.SSLTransport.decode(SSLTransport.java:110)
at java.base/sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1403)
at java.base/sun.security.ssl.SSLSocketImpl.readHandshakeRecord(SSLSocketImpl.java:1309)
at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:440)
at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:411)
at okhttp3.internal.connection.RealConnection.connectTls(RealConnection.java:336)
at okhttp3.internal.connection.RealConnection.establishProtocol(RealConnection.java:300)
at okhttp3.internal.connection.RealConnection.connect(RealConnection.java:185)
at okhttp3.internal.connection.ExchangeFinder.findConnection(ExchangeFinder.java:224)
at okhttp3.internal.connection.ExchangeFinder.findHealthyConnection(ExchangeFinder.java:108)
at okhttp3.internal.connection.ExchangeFinder.find(ExchangeFinder.java:88)
at okhttp3.internal.connection.Transmitter.newExchange(Transmitter.java:169)
at okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.java:41)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:117)
at okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.java:94)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:117)
at okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.java:93)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142)
at okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.java:88)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:117)
at io.fabric8.kubernetes.client.utils.BackwardsCompatibilityInterceptor.intercept(BackwardsCompatibilityInterceptor.java:134)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:117)
at io.fabric8.kubernetes.client.utils.ImpersonatorInterceptor.intercept(ImpersonatorInterceptor.java:68)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:117)
at io.fabric8.kubernetes.client.utils.HttpClientUtils.lambda$createHttpClient$3(HttpClientUtils.java:147)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:117)
at okhttp3.RealCall.getResponseWithInterceptorChain(RealCall.java:229)
at okhttp3.RealCall.execute(RealCall.java:81)
at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:490)
at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:451)
at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleGet(OperationSupport.java:416)
at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleGet(OperationSupport.java:397)
at io.fabric8.kubernetes.client.dsl.base.BaseOperation.handleGet(BaseOperation.java:890)
at io.fabric8.kubernetes.client.dsl.base.BaseOperation.getMandatory(BaseOperation.java:233)
... 15 more
I've setup a sample Kubernetes cluster using minikube with Elasticsearch and Kibana 6.8.6, and Filebeat 7.5.1.
My application generate log messages in json format {"#timestamp":"2019-12-30T21:59:48+0000","message":"example","data":"data-462"}
I can see the log message in Kibana, but my json log is embedded inside "message" atribute as a string:
I configured json.keys_under_root: true to no effect (as stated in documentation: https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-input-log.html#filebeat-input-log-config-json)
My configuration:
filebeat.yml: |-
migration.6_to_7.enabled: true
filebeat.config:
modules:
path: ${path.config}/modules.d/*.yml
reload.enabled: false
filebeat.autodiscover:
providers:
- type: kubernetes
hints.enabled: true
hints.default_config.enabled: false
json.keys_under_root: true
json.add_error_key: true
output.elasticsearch:
hosts: ['${ELASTICSEARCH_HOST:elasticsearch}:${ELASTICSEARCH_PORT:9200}']
username: ${ELASTICSEARCH_USERNAME}
password: ${ELASTICSEARCH_PASSWORD}
kubernetes.yml: |-
- type: docker
containers.ids:
- "*"
processors:
- add_kubernetes_metadata:
in_cluster: true
I need the "message" and "data" fields as separate fields in Kibana.
What I'm missing?
Try adding json.message_key: message in your filebeat configuration
I'm trying to incorporate a JSON into a YAML file.
The YAML looks like this:
filebeat.inputs:
- type: log
<incorporate here with a single level indent>
enabled: true
paths:
Suppose you have the following variable:
a = { processors: { drop_event: { when: { or: [ {equals: { status: 500 }},{equals: { status: -1 }}]}}}}
I want to incorporate it into an existing YAML.
I've tried to use:
JSON.parse((a).to_json).to_yaml
After applying this, I got a valid YAML but without indentation (all lines have to be indented) and with a "---" which is Ruby's new document in YAML.
The result:
filebeat.inputs:
- type: log
---
processors:
drop_event:
when:
or:
- equals:
status: 500
- equals:
status: -1
enabled: true
The result I'm looking for:
filebeat.inputs:
- type: log
processors:
drop_event:
when:
or:
- equals:
status: 500
- equals:
status: -1
enabled: true```
It’s easier to produce a valid ruby object by merging hashes and then serialize the result to YAML than vice versa.
puts(yaml.map do |hash|
hash.each_with_object({}) do |(k, v), acc|
# the trick: we insert before "enabled" key
acc.merge!(JSON.parse(a.to_json)) if k == "enabled"
# regular assignment for all hash elements
acc[k] = v
end
end.to_yaml)
Results in:
---
- type: log
processors:
drop_event:
when:
or:
- equals:
status: 500
- equals:
status: -1
enabled: true
JSON.parse(a.to_json) basically converts symbols to strings.
In order to do that first you need to convert your original YAML into JSON
original = YAML.load(File.read(File.join('...', 'filebeat.inputs')))
# => [
{
"type": "log",
"enabled": true,
"paths": null
}
]
Then you have to merge your JSON into this original variable
original[0].merge!(a.stringify_keys)
original.to_yaml
# =>
---
-
type: log
enabled: true
paths:
processors:
drop_event:
when:
or:
- equals:
status: 500
- equals:
status: -1