I've been trying to figure this out myself for awhile now, and I'm having difficulty figuring out the syntax for using the logstash filter. In my program I log the user's IP address and other values into keyvalue pairs for json format.
I have a standard setup for the elastic stack that follows:
.net program -> log.json -> filebeat -> logstash -> elasticsearch -> kibana
logstash.conf:
input {
beats {
port => 5044
}
}
filter {
json {
source => "message"
}
geoip {
source => "Portal-IPAddress"
}
mutate {
remove_field => ["Timestamp","message","MessageTemplate","log.offset","#version"]
}
}
output {
elasticsearch {
hosts => ["http://localhost:9200"]
index => "%{[#metadata][beat]}-%{+YYYY.MM}"
}
stdout { codec => rubydebug }
}
My questions being how can I:
1.Get all key value pairs from "Properties" into their own respective fields
2.Use the Portal-IPAddress field to lookup & set a geoip field (currently lookup fails, even when placing a known working value into that field & I suspect that I'm not locating the proper value with source).
for extra info here is my log structure
{"Timestamp":"2022-11-01T16:36:24.4924785-07:00","Level":"Information","MessageTemplate":"\"Portal-Module\":\"{Portal-Module}\",\"Portal-SubModule\":\"{Portal-SubModule}\",\"Portal-Action\":\"{Portal-Action}\",\"Portal-Result\":\"{Portal-Result}\",\"Portal-User\":\"{Portal-User}\",\"Portal-Object\":\"{Portal-Object}\",\"Portal-Message\":\"{Portal-Message}\",\"Portal-URL\":\"{Portal-URL}\",\"Portal-Milliseconds\":{Portal-Milliseconds},\"Portal-IPAddress\":\"{Portal-IPAddress}\",\"Portal-LogType\":\"{Portal-LogType}\",","Properties":{"Portal-Module":"Reporting","Portal-SubModule":"Availability","Portal-Action":"Generate","Portal-Result":"Succeeded","Portal-User":"local","Portal-Object":"Run on Large Screen","Portal-Message":"","Portal-URL":"https://localhost:5001/reports/availability?readydate=20230630&seasonids=112&pricelevelid=37","Portal-Milliseconds":5636,"Portal-IPAddress":"::1","Portal-LogType":"FrontEnd","SourceContext":"Portal.Web.Services.PortalLogger","TransportConnectionId":"caBKjkS1R-Rzkz4auPIAcA","RequestId":"0HMLSB0UJLBU9:00000002","RequestPath":"/_blazor","ConnectionId":"0HMLSB0UJLBU9","MachineName":"WS-56","ClientIp":"::1","ClientAgent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/105.0.0.0 Safari/537.36 OPR/91.0.4516.95","Email":"local","IsImpersonating":false,"Environment":"Development"}}
logstash output
{
"Properties" => {
"SourceContext" => "Portal.Web.Services.PortalLogger",
"RequestPath" => "/_blazor",
"MachineName" => "WS-56",
"Portal-URL" => "https://localhost:5001/reports/availability?readydate=20230630&seasonids=112&pricelevelid=37",
"Email" => "local",
"Portal-Result" => "Succeeded",
"ConnectionId" => "0HMLSB0UJLBU9",
"Portal-Object" => "Run on Large Screen",
"Portal-Milliseconds" => 5636,
"TransportConnectionId" => "caBKjkS1R-Rzkz4auPIAcA",
"Environment" => "Development",
"RequestId" => "0HMLSB0UJLBU9:00000002",
"Portal-LogType" => "FrontEnd",
"Portal-SubModule" => "Availability",
"Portal-Message" => "",
"Portal-IPAddress" => "x",
"ClientIp" => "x",
"IsImpersonating" => false,
"ClientAgent" => "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/105.0.0.0 Safari/537.36 OPR/91.0.4516.95",
"Portal-Action" => "Generate",
"Portal-Module" => "Reporting",
"Portal-User" => "local"
},
"host" => {
"hostname" => "WS-56",
"os" => {
"build" => "19044.2130",
"name" => "Windows 10 Pro",
"platform" => "windows",
"type" => "windows",
"family" => "windows",
"version" => "10.0",
"kernel" => "10.0.19041.2130 (WinBuild.160101.0800)"
},
"name" => "WS-56",
"ip" => [
[0] "x",
[1] "x",
[2] "x",
[3] "x",
[4] "x",
[5] "x",
[6] "x"
],
"mac" => [
[0] "x",
[1] "x"
],
"architecture" => "x86_64",
"id" => "72eeb3df-a50c-4be6-8c13-ca7a860e61e0"
},
"agent" => {
"hostname" => "WS-56",
"name" => "WS-56",
"type" => "filebeat",
"ephemeral_id" => "ed62668d-e91c-47e0-9b41-dd0854f13c06",
"version" => "7.14.1",
"id" => "c8494d5f-c460-4977-849e-c61c4fd1726f"
},
"log" => {
"offset" => 89193,
"file" => {
"path" => "C:\\ProgramData\\filebeat\\datalogs\\log-11-2022-20221101.json"
}
},
"input" => {
"type" => "log"
},
"tags" => [
[0] "beats_input_codec_plain_applied"
],
"#timestamp" => 2022-11-01T23:36:33.462Z,
"Level" => "Information",
"ecs" => {
"version" => "1.10.0"
}
}
Every sensitive value has been replaced with "x", but ip & mac addresses are generating properly
So I was able to answer my own question by editing how my logger outputs the json file.
By creating a custom json formatter for use in the rolling file Serilog sink, I was able format all properties into individual fields instead of nesting all of them in the "Properties" field that the default Json formatter used.
Related
I feed JSON to some webhook to trigger a notification (M$ Teams). This works well. However, I want to extend my Perl script: I need to add a new node to my "messagecard" construct on a certain condition.
E.g. I defined this:
my $payload={};
$payload = {
'#type' => 'MessageCard',
'#context' => 'http://schema.org/extensions',
themeColor => $event{COLOR},
text => $event{SERVICEOUTPUT},
sections => [{
facts => [{
name => 'Type',
value => "$event{NOTIFICATIONTYPE} $event{ADDITIONALINFO}"
},
]
}],
potentialAction => [{
'#type' => "OpenUri",
name => "View Monitoring",
targets => [{
os => "default",
uri => $naemon_url
}]
}]
};
$ua = LWP::UserAgent->new;
my $req = POST($opt_webhook
, 'Content-Type' => 'application/json; charset=UTF-8'
, 'Content' => encode_json($payload)
);
my $resp = $ua->request($req);
And if (conditon), I want to extend this as follows (order is important):
$payload = {
'#type' => 'MessageCard',
'#context' => 'http://schema.org/extensions',
themeColor => $event{COLOR},
text => $event{SERVICEOUTPUT},
sections => [{
facts => [{
name => 'Type',
value => "$event{NOTIFICATIONTYPE} $event{ADDITIONALINFO}"
},
]
}],
potentialAction => [{
'#type' => "OpenUri",
name => "View Monitoring",
targets => [{
os => "default",
uri => $naemon_url
}]
},
{
'#type' => "OpenUri",
name => "Notes (Logs, Docs,..)",
targets => [{
os => "default",
uri => $event{SERVICENOTESURL}
}]
}]
};
I am unsure how this can be achieved. Can anyone please provide wisdom how to tackle this?
You can push into the array reference that you've got inside your potentialAction key. In order to do that, you need to dereference it as an array.
my $payload = {
'#type' => 'MessageCard',
potentialAction => [{
name => "View Monitoring",
targets => [{
os => "default",
}]
}]
};
if ($maybe) {
push #{ $payload->{potentialAction} }, {
name => "Notes (Logs, Docs,..)",
targets => [{
os => "default",
}]
};
}
If your Perl version is 5.24 or newer you can also use postfix dereferencing, which some people find easier to read.
push $payload->{potentialAction}->#*, ...
See perlref and perlreftut for more information.
I have this config file (logstash):
input {
beats {
port => 5044
host => "0.0.0.0"
}
}
filter {
json {
source => "message"
target => "log"
}
}
output {
elasticsearch {
hosts => ["http://localhost:9200"]
index => "logs-%{+YYYY.MM.dd}"
}
jdbc {
driver_jar_path => "/etc/logstash/mysql-connector-java-8.0.11.jar"
driver_class => "com.mysql.cj.jdbc.Driver"
connection_string => "jdbc:mysql://localhost:3306/cste?user=master&password=testets!123"
statement => ["INSERT INTO cste_log (ip, log, event, created, inserted) VALUES(?,?,?,?,?)", "log.userid", "log", "log.event", "#timestamp", "#timestamp"]
}
stdout {
codec => "rubydebug"
}
for save data as mySQL Database. but it doesn't work with an error message (column 'ip', 'event' cannot be null)
I think the grammar of 'jdbc.statement' is wrong, and I'm trying to fix it.
'output.elasticsearch' works very nice.
{
"agent" => {
"version" => "7.10.0",
"name" => "DESKTOP-GEB1AGR",
"id" => "7e109ece-5874-4149-9842-21acb86c9da0",
"type" => "filebeat",
"hostname" => "DESKTOP-GEB1AGR",
"ephemeral_id" => "0730755e-f234-48c4-b7f1-2d2339df0e86"
},
"#version" => "1",
"#timestamp" => 2020-11-23T06:31:59.005Z,
"log" => {
"userid" => "192.111.11.111",
"writetime" => "2020/11/23 15:31:51",
"target" => "crackme.exe - PID: 5528 - Module: ntdll.dll - Thread: Main Thread 3240 (switched from 19C0)",
"event" => "dbgRestart"
},
"input" => {
"type" => "log"
},
"ecs" => {
"version" => "1.6.0"
},
"message" => "{\"writetime\": \"2020/11/23 15:31:51\", \"userid\": \"111.111.111.111\", \"target\": \"crackme.exe - PID: 5528 - Module: ntdll.dll - Thread: Main Thread 3240 (switched from 19C0)\", \"event\": \"dbgRestart\"} ",
"host" => {
"name" => "DESKTOP-GEB1AGR",
"architecture" => "x86_64",
"os" => {
"version" => "10.0",
"name" => "Windows 10 Home",
"build" => "16299.1087",
"family" => "windows",
"platform" => "windows",
"kernel" => "10.0.16299.1087 (WinBuild.160101.0800)"
},
"id" => "659f1b29-3-2cb22793a39c",
"ip" => [
[0] "fe80::adb9:b",
[1] "192.168.43.",
[2] "2001:0:348b:",
[3] "fe80::180947e"
],
"hostname" => "DESKTOP-GEB1AGR",
"mac" => [
[0] "00:0c:6c:d7",
[1] "00:00:00:e0"
]
},
"tags" => [
[0] "beats_input_codec_plain_applied"
]
}
How can I use the value 'writetime' and 'event'?
Please give me some advice.
"log" => {
"userid" => "192.168.43.129",
"writetime" => "2020/11/23 15:31:51",
"target" => "crackme.exe - PID: 5528 - Module: ntdll.dll - Thread: Main Thread 3240 (switched from 19C0)",
"event" => "dbgRestart"},
If event is a field inside the log object then in logstash you refer to that as "[log][event]". [log.event] refers to a field that has a period in its name. Similarly for "[log][userid]".
I am posting a json from an application to logstash wanting to get the location of an IP-adress with logstashes geoip plugin. However i get a _geoip_lookup_failure.
this is my logstash config
http {
port => "4200"
codec => json
}
}
filter{
geoip {
source => "clientip"
}
}
output {
elasticsearch { hosts => ["localhost:9200"] }
stdout { codec => rubydebug }
}
this is what I post to the port :
{'used_credentials': [
{'username': 'l', 'number_of_usages': 1, 'used_commands': {},
'get_access': 'false',
'timestamps': {'1': '04/15/2019, 21:08:54'}, 'password': 'l'}],
'clientip': '192.168.xxx.xx',
'unsuccessfull_logins': 1}
and this is what i get in logstash:
{
"unsuccessfull_logins" => 1,
"#version" => "1",
"used_credentials" => [
[0] {
"username" => "l",
"used_commands" => {},
"password" => "l",
"timestamps" => {
"1" => "04/15/2019, 21:08:54"
},
"number_of_usages" => 1,
"get_access" => "false"
}
],
"clientip" => "192.168.xxx.xx",
"#timestamp" => 2019-04-15T19:08:57.147Z,
"host" => "127.0.0.1",
"headers" => {
"request_path" => "/telnet",
"connection" => "keep-alive",
"accept_encoding" => "gzip, deflate",
"http_version" => "HTTP/1.1",
"content_length" => "227",
"http_user_agent" => "python-requests/2.21.0",
"request_method" => "POST",
"http_accept" => "*/*",
"content_type" => "application/json",
"http_host" => "127.0.0.1:4200"
},
"geoip" => {},
"tags" => [
[0] "_geoip_lookup_failure"
]
}
I don't understand why the input is recognized corectly but goeip does not find it
The problem is that your clientip is in the 192.168.0.0/16 network, which is a private network reserved for local use only, it is not present on the database used by the geoip filter.
The geoip filter will only work with public IP addresses.
I'm trying to send logstash outputs to csv, but the columns are not being written in the file.
This is my logstash configuration:
input
{
http
{
host => "0.0.0.0"
port => 31311
}
}
filter
{
grok {
match => { "id" => "%{URIPARAM:id}?" }
}
kv
{
field_split => "&?"
source => "[headers][request_uri]"
}
}
output
{
stdout { codec => rubydebug }
csv
{
fields => ["de,cd,dl,message,bn,ua"]
path => "/tmp/logstash-bq/text.csv"
flush_interval => 0
csv_options => {"col_sep" => ";" "row_sep" => "\r\n"}
}
}
This is my input:
curl -X POST 'http://localhost:31311/?id=9decaf95-20a5-428e-a3ca-50485edb9f9f&uid=1-fg4fuqed-j0hzl5q2&ev=pageview&ed=&v=1&dl=http://dev.xxx.com.br/&rl=http://dev.xxxx.com.br/&ts=1491758180677&de=UTF-8&sr=1600x900...
This is logstash answer:
{
"headers" => {
"http_accept" => "*/*",
"request_path" => "/",
"http_version" => "HTTP/1.1",
"request_method" => "POST",
"http_host" => "localhost:31311",
"request_uri" => "/?id=xxx...",
"http_user_agent" => "curl/7.47.1"
},
"de" => "UTF-8",
"cd" => "24",
"dl" => "http://dev.xxx.com.br/",
"message" => "",
"bn" => "Chrome%2057",
"ua" => "Mozilla/5.0%20(Macintosh;%20Intel%20Mac%20OS%20X%2010_11_3)%20AppleWebKit/537.36%20(KHTML,%20like%20Gecko)%20Chrome/57.0.2987.133%20Safari/537.36",
"dt" => "xxxx",
"uid" => "1-fg4fuqed-j0hzl5q2",
"ev" => "pageview",
"#timestamp" => 2017-04-09T17:41:03.083Z,
"v" => "1",
"md" => "false",
"#version" => "1",
"host" => "0:0:0:0:0:0:0:1",
"rl" => "http://dev.xxx.com.br/",
"vp" => "1600x236",
"id" => "9decaf95-20a5-428e-a3ca-50485edb9f9f",
"ts" => "1491758180677",
"sr" => "1600x900"
}
[2017-04-09T14:41:03,137][INFO ][logstash.outputs.csv ] Opening file {:path=>"/tmp/logstash-bq/text.csv"}
But when I open /tmp/logstash-bq/text.csv I see this:
2017-04-09T16:26:17.464Z 127.0.0.1 abc2017-04-09T17:19:19.690Z 0:0:0:0:0:0:0:1 2017-04-09T17:23:12.117Z 0:0:0:0:0:0:0:1 2017-04-09T17:24:08.067Z 0:0:0:0:0:0:0:1 2017-04-09T17:31:39.269Z 0:0:0:0: 0:0:0:1 2017-04-09T17:38:02.624Z 0:0:0:0:0:0:0:1 2017-04-09T17:41:03.083Z 0:0:0:0:0:0:0:1
CSV output is bugged for logstash 5.x. I had to install logstash 2.4.1.
I have a Problem with accessing a nested JSON field in logstash (latest version).
My config file is the following:
input {
http {
port => 5001
codec => "json"
}
}
filter {
mutate {
add_field => {"es_index" => "%{[statements][authority][name]}"}
}
mutate {
gsub => [
"es_index", " ", "_"
]
}
mutate {
lowercase => ["es_index"]
}
ruby {
init => "
def remove_dots hash
new = Hash.new
hash.each { |k,v|
if v.is_a? Hash
v = remove_dots(v)
end
new[ k.gsub('.','_') ] = v
if v.is_a? Array
v.each { |elem|
if elem.is_a? Hash
elem = remove_dots(elem)
end
new[ k.gsub('.','_') ] = elem
} unless v.nil?
end
} unless hash.nil?
return new
end
"
code => "
event.instance_variable_set(:#data,remove_dots(event.to_hash))
"
}
}
output {
stdout {
codec => rubydebug
}
elasticsearch {
hosts => "elasticsearch:9200"
index => "golab-%{+YYYY.MM.dd}"
}
}
I have a filter with mutate. I want to add a field that I can use as a part of the index name. When I use this "%{[statements][authority][name]}" the content in the brackets is used as string.%{[statements][authority][name]} is saved in the es_indexfield. Logstash seems to think this is a string, but why?
I've also tried to use this expression: "%{statements}". It's working like expected. Everything in the field statements is passed to es_index. If I use "%{[statements][authority]}" strange things happen. es_index is filled with the exact same output that "%{statements}" produces. What am I missing?
Logstash Output with "%{[statements][authority]}":
{
"statements" => {
"verb" => {
"id" => "http://adlnet.gov/expapi/verbs/answered",
"display" => {
"en-US" => "answered"
}
},
"version" => "1.0.1",
"timestamp" => "2016-07-21T07:41:18.013880+00:00",
"object" => {
"definition" => {
"name" => {
"en-US" => "Example Activity"
},
"description" => {
"en-US" => "Example activity description"
}
},
"id" => "http://adlnet.gov/expapi/activities/example"
},
"actor" => {
"account" => {
"homePage" => "http://example.com",
"name" => "xapiguy"
},
"objectType" => "Agent"
},
"stored" => "2016-07-21T07:41:18.013880+00:00",
"authority" => {
"mbox" => "mailto:info#golab.eu",
"name" => "GoLab",
"objectType" => "Agent"
},
"id" => "0771b9bc-b1b8-4cb7-898e-93e8e5a9c550"
},
"id" => "a7e31874-780e-438a-874c-964373d219af",
"#version" => "1",
"#timestamp" => "2016-07-21T07:41:19.061Z",
"host" => "172.23.0.3",
"headers" => {
"request_method" => "POST",
"request_path" => "/",
"request_uri" => "/",
"http_version" => "HTTP/1.1",
"http_host" => "logstasher:5001",
"content_length" => "709",
"http_accept_encoding" => "gzip, deflate",
"http_accept" => "*/*",
"http_user_agent" => "python-requests/2.9.1",
"http_connection" => "close",
"content_type" => "application/json"
},
"es_index" => "{\"verb\":{\"id\":\"http://adlnet.gov/expapi/verbs/answered\",\"display\":{\"en-us\":\"answered\"}},\"version\":\"1.0.1\",\"timestamp\":\"2016-07-21t07:41:18.013880+00:00\",\"object\":{\"definition\":{\"name\":{\"en-us\":\"example_activity\"},\"description\":{\"en-us\":\"example_activity_description\"}},\"id\":\"http://adlnet.gov/expapi/activities/example\",\"objecttype\":\"activity\"},\"actor\":{\"account\":{\"homepage\":\"http://example.com\",\"name\":\"xapiguy\"},\"objecttype\":\"agent\"},\"stored\":\"2016-07-21t07:41:18.013880+00:00\",\"authority\":{\"mbox\":\"mailto:info#golab.eu\",\"name\":\"golab\",\"objecttype\":\"agent\"},\"id\":\"0771b9bc-b1b8-4cb7-898e-93e8e5a9c550\"}"
}
You can see that authority is part of es_index. So it was not chosen as a field.
Many thanks in advance
I found a solution. Credits go to jpcarey (Elasticsearch Forum)
I had to remove codec => "json". That leads to another data structure. statements is now an array and not an object. So I needed to change %{[statements][authority][name]} to %{[statements][0][authority][name]}. That works without problems.
If you follow the given link you'll also find an better implementation of my mutate filters.