Possible to set multiple slow log and error log on mysql module filebeat? - mysql

Question

I have one development server and already installed
elasticsearch
kibana
filebeat
docker
on docker already running 2 container mariadb database.
I already set filebeat for 1 mariadb database.
with config on /etc/filebeat/modules.d/mysql.yml like this
- module: mysql
# Error logs
error:
enabled: true
var.paths: ["/media/dbdev1/data/mysql_error.log"]
# Slow logs
slowlog:
enabled: true
var.paths: ["/media/dbdev1/data/mysql_slow.log"]
if I need more error log and slow log from other container mariadb database is just change /etc/filebeat/modules.d/mysql.yml like this
- module: mysql
# Error logs
error:
enabled: true
var.paths: ["/media/dbdev1/data/mysql_error.log","/media/dbdev2/data/mysql_error.log"]
# Slow logs
slowlog:
enabled: true
var.paths: ["/media/dbdev1/data/mysql_slow.log","/media/dbdev2/data/mysql_slow.log"]
my expectation filebeat can pull mysql_error.log from 2 different mariadb container with different path also
some filebeat setup log
2021-10-25T10:43:25.616+0700 INFO instance/beat.go:665 Home path: [/usr/share/filebeat] Config path: [/etc/filebeat] Data path: [/var/lib/filebeat] Logs path: [/var/log/filebeat]
2021-10-25T10:43:25.619+0700 INFO instance/beat.go:673 Beat ID: cb340c7a-15b4-44f7-8a66-06f6850c1c0f
2021-10-25T10:43:26.499+0700 INFO [beat] instance/beat.go:1014 Beat info {"system_info": {"beat": {"path": {"config": "/etc/filebeat", "data": "/var/lib/filebeat", "home": "/usr/share/filebeat", "logs": "/var/log/filebeat"}, "type": "filebeat", "uuid": "cb340c7a-15b4-44f7-8a66-06f6850c1c0f"}}}
2021-10-25T10:43:26.501+0700 INFO [beat] instance/beat.go:1023 Build info {"system_info": {"build": {"commit": "5ae799cb1c3c490c9a27b14cb463dc23696bc7d3", "libbeat": "7.15.1", "time": "2021-10-07T22:06:49.000Z", "version": "7.15.1"}}}
2021-10-25T10:43:26.501+0700 INFO [beat] instance/beat.go:1026 Go runtime info {"system_info": {"go": {"os":"linux","arch":"amd64","max_procs":2,"version":"go1.16.6"}}}
2021-10-25T10:43:26.503+0700 INFO [beat] instance/beat.go:1030 Host info {"system_info": {"host": {"architecture":"x86_64","boot_time":"2021-10-25T09:45:23+07:00","containerized":false,"name":"localhost.localdomain","ip":["127.0.0.1/8","::1/128","10.0.2.20/24","fe80::a00:27ff:fe8c:82d0/64","192.168.131.5/24","fe80::a00:27ff:fedd:bb9e/64","172.17.0.1/16","172.18.0.1/16","fe80::42:89ff:fe04:e2cb/64","fe80::2c35:92ff:fe88:4daf/64","fe80::38b2:66ff:fe52:b1ec/64"],"kernel_version":"4.18.0-305.19.1.el8_4.x86_64","mac":["08:00:27:8c:82:d0","08:00:27:dd:bb:9e","02:42:ad:3d:07:6b","02:42:89:04:e2:cb","2e:35:92:88:4d:af","3a:b2:66:52:b1:ec"],"os":{"type":"linux","family":"redhat","platform":"centos","name":"CentOS Linux","version":"8","major":8,"minor":4,"patch":2105},"timezone":"WIB","timezone_offset_sec":25200,"id":"b14f68ad4b8c4732a4cfe379692179ec"}}}
2021-10-25T10:43:26.503+0700 INFO [beat] instance/beat.go:1059 Process info {"system_info": {"process": {"capabilities": {"inheritable":null,"permitted":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend","audit_read","38","39"],"effective":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend","audit_read","38","39"],"bounding":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend","audit_read","38","39"],"ambient":null}, "cwd": "/media/dbdev1", "exe": "/usr/share/filebeat/bin/filebeat", "name": "filebeat", "pid": 3503, "ppid": 1941, "seccomp": {"mode":"disabled","no_new_privs":false}, "start_time": "2021-10-25T10:43:23.920+0700"}}}
2021-10-25T10:43:26.503+0700 INFO instance/beat.go:309 Setup Beat: filebeat; Version: 7.15.1
2021-10-25T10:43:26.504+0700 INFO [index-management] idxmgmt/std.go:184 Set output.elasticsearch.index to 'filebeat-7.15.1' as ILM is enabled.
2021-10-25T10:43:26.517+0700 INFO [esclientleg] eslegclient/connection.go:100 elasticsearch url: http://localhost:9200
2021-10-25T10:43:26.521+0700 INFO [publisher] pipeline/module.go:113 Beat name: localhost.localdomain
2021-10-25T10:43:26.585+0700 INFO [esclientleg] eslegclient/connection.go:100 elasticsearch url: http://localhost:9200
2021-10-25T10:43:26.820+0700 INFO [esclientleg] eslegclient/connection.go:273 Attempting to connect to Elasticsearch version 7.15.0
2021-10-25T10:43:26.895+0700 INFO [index-management] idxmgmt/std.go:261 Auto ILM enable success.
2021-10-25T10:43:26.929+0700 INFO [index-management.ilm] ilm/std.go:170 ILM policy filebeat exists already.
2021-10-25T10:43:26.929+0700 INFO [index-management] idxmgmt/std.go:401 Set setup.template.name to '{filebeat-7.15.1 {now/d}-000001}' as ILM is enabled.
2021-10-25T10:43:26.929+0700 INFO [index-management] idxmgmt/std.go:406 Set setup.template.pattern to 'filebeat-7.15.1-*' as ILM is enabled.
2021-10-25T10:43:26.929+0700 INFO [index-management] idxmgmt/std.go:440 Set settings.index.lifecycle.rollover_alias in template to {filebeat-7.15.1 {now/d}-000001} as ILM is enabled.
2021-10-25T10:43:26.929+0700 INFO [index-management] idxmgmt/std.go:444 Set settings.index.lifecycle.name in template to {filebeat {"policy":{"phases":{"hot":{"actions":{"rollover":{"max_age":"30d","max_size":"50gb"}}}}}}} as ILM is enabled.
2021-10-25T10:43:26.974+0700 INFO template/load.go:229 Existing template will be overwritten, as overwrite is enabled.
2021-10-25T10:43:28.637+0700 INFO [add_cloud_metadata] add_cloud_metadata/add_cloud_metadata.go:101 add_cloud_metadata: hosting provider type not detected.
2021-10-25T10:43:31.539+0700 INFO template/load.go:132 Try loading template filebeat-7.15.1 to Elasticsearch
2021-10-25T10:43:32.442+0700 INFO template/load.go:124 Template with name "filebeat-7.15.1" loaded.
2021-10-25T10:43:32.442+0700 INFO [index-management] idxmgmt/std.go:297 Loaded index template.
2021-10-25T10:43:32.475+0700 INFO [index-management.ilm] ilm/std.go:126 Index Alias filebeat-7.15.1 exists already.
2021-10-25T10:43:32.476+0700 INFO kibana/client.go:167 Kibana url: http://localhost:5601
2021-10-25T10:43:38.391+0700 INFO kibana/client.go:167 Kibana url: http://localhost:5601
2021-10-25T10:44:58.953+0700 INFO instance/beat.go:848 Kibana dashboards successfully loaded.
2021-10-25T10:44:58.976+0700 WARN [cfgwarn] instance/beat.go:574 DEPRECATED: Setting up ML using Filebeat is going to be removed. Please use the ML app to setup jobs. Will be removed in version: 8.0.0
2021-10-25T10:44:58.993+0700 INFO [esclientleg] eslegclient/connection.go:100 elasticsearch url: http://localhost:9200
2021-10-25T10:44:59.006+0700 INFO [esclientleg] eslegclient/connection.go:273 Attempting to connect to Elasticsearch version 7.15.0
2021-10-25T10:44:59.006+0700 INFO kibana/client.go:167 Kibana url: http://localhost:5601
2021-10-25T10:44:59.098+0700 WARN fileset/modules.go:425 X-Pack Machine Learning is not enabled
2021-10-25T10:44:59.207+0700 WARN fileset/modules.go:425 X-Pack Machine Learning is not enabled
2021-10-25T10:44:59.207+0700 INFO [esclientleg] eslegclient/connection.go:100 elasticsearch url: http://localhost:9200
2021-10-25T10:44:59.212+0700 INFO [esclientleg] eslegclient/connection.go:273 Attempting to connect to Elasticsearch version 7.15.0
2021-10-25T10:44:59.214+0700 INFO [esclientleg] eslegclient/connection.go:100 elasticsearch url: http://localhost:9200
2021-10-25T10:44:59.219+0700 INFO [esclientleg] eslegclient/connection.go:273 Attempting to connect to Elasticsearch version 7.15.0
2021-10-25T10:44:59.351+0700 INFO [modules] fileset/pipelines.go:133 Elasticsearch pipeline loaded. {"pipeline": "filebeat-7.15.1-mysql-error-pipeline"}
2021-10-25T10:44:59.480+0700 INFO [modules] fileset/pipelines.go:133 Elasticsearch pipeline loaded. {"pipeline": "filebeat-7.15.1-mysql-slowlog-pipeline"}
2021-10-25T10:44:59.480+0700 INFO cfgfile/reload.go:262 Loading of config files completed.
2021-10-25T10:44:59.481+0700 INFO [load] cfgfile/list.go:129 Stopping 1 runners ...