I have a problem in ELK stack and I am not sure what is the cause of that.
A lot of fields in my index have multiple values.
For example:
records.Type Event, Event, Event, Event
records.EventCategory 1, 1, 1, 1, 1, 1, 1
records.EventLevelName Success, Success, Success, Success, Success, Success, Success
The implementation is quite easy and there's no fancy parsing in place.
I am pulling logs with Logstash from EventHub and store them in Elastic.
The only filter which is in place in Logstash is json filter:
filter {
json {
add_tag => [ "EventHub" ]
source => "message"
remove_field => [ "message" ]
}
}
That's all.
Nothing useful in logs either.
Does anyone faced this problem? I couldn't find any helpful information.
Related
I have a question about filtering entries in Logstash. I have two different logs coming into Logstash. One log is just a std format with a timestamp and message, but the other comes in as JSON.
I use an if statement to test for a certain host and if that host is present, then I use the JSON filter to apply to the message... the problem is that when it encounters the non-JSON stdout message it can't parse it and throws exceptions.
Does anyone know how to test to see if an entry is JSON coming in apply the filter and if not, just ignore it?
thanks
if [agent][hostname] == "some host"
# if an entry is not in json format how to ignore?
{
json {
source => "message"
target => "gpfs"
}
}
You can try with a grok filter as a first step.
grok {
match => {
"message" => [
"{%{GREEDYDATA:json_message}}",
"%{GREEDYDATA:std_out}"
]
}
}
if [json_message]
{
mutate {
replace => { "json_message" => "{%{json_message}}"}
}
json {
source => "json_message"
target => "gpfs"
}
}
Probably there is a more cleaner solution then this, but it will do the job.
Given the following multiline log
{
"code" : 429
}
And the following pipeline logstash.conf
filter {
grok {
match =>
{
"message" =>
[
"%{GREEDYDATA:json}"
]
}
}
json {
source => "json"
target => "json"
}
}
When Log is send into logstash through filebeat
Then Logstash returns
[2018-08-07T10:48:41,067][WARN ][logstash.outputs.elasticsearch] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"filebeat-to-logstash", :_type=>"doc", :_routing=>nil}, #<LogStash::Event:0x2bf7b08d>], :response=>{"index"=>{"_index"=>"filebeat-to-logstash", "_type"=>"doc", "_id"=>"trAAFGUBnhQ5nUWmyzVg", "status"=>400, "error"=>{"type"=>"mapper_parsing_exception", "reason"=>"failed to parse [json]", "caused_by"=>{"type"=>"illegal_state_exception", "reason"=>"Can't get text on a START_OBJECT at 1:3846"}}}}}
This is incorrect behavior as the JSON is perfectly valid, how should this be solved?
I found out that in Logstash 6.3.0 this problem occurs when one tries to serialize JSON on the "json" field. Changing this field name to anything else solves this issue.
Since Elastic JSON filter plugin documentation does not mention anything about this behaviour and the error message is inaccurate it can be assumed this is a bug.
Bug report has been send: https://github.com/elastic/logstash/issues/9876
Assuming one were to post multiple data sets of one model at the time through JSON, it is possible to insert these using Eloquent's Model::create() function. However in my case I'll also need to validate this data.
The Validator only takes a Request object as input, and as far as I've seen I can't create a new Request instance with only one model.
Assuming this would be the input data (JSON), and index is the value for the browser to know what data belongs to an what item (as they have no unique ID assigned at the point of creation)
[
{
"index" : 1,
"name" : "Item 1",
"value" : "Some description"
},
{
"index" : 2,
"name" : "Item 2",
"value" : "Something to describe item 2"
},
(and so on)
]
Every object in the root array needs to be ran through the same validator. The rules of it are defined in Model::$rules (public static array).
Would there be a way to run the validator against every item, and possibly capture the errors per item?
You can utilize Validator for manual validation:
...
use Validator;
...
$validator = Validator::make(
json_decode($data, true), // where $data contains your JSON data string
[
// List your rules here using wildcard syntax.
'*.index' => 'required|integer',
'*.name' => 'required|min:2',
...
],
[
// Array of messages for validation errors.
...
],
[
// Array of attribute titles for validation errors.
...
]
);
if ($validator->fails()) {
// Validation failed.
// $validator->errors() will return MessageBag with what went wrong.
...
}
You can read more about validating arrays here.
I'm using Laravel 5.4 and trying to validate JSON in my POST request however the validator fails stating that the JSON isn't valid, even though it is. I'm assuming I'm not understanding the validation rules correctly and my implementation is wrong, rather than a bug or something else.
I have a simple POST endpoint which has both the Accept and Content-Type headers set to application/json.
In my POST request (testing using Postman) I'm supplying RAW data.
{
"only_this_key": { "one": "two" }
}
In my controller method I have the following:
// I'm using intersect to remove any other parameters that may have been supplied as this endpoint only requires one
$requestData = $request->intersect(['only_this_key']);
$messages = [
'only_this_key.required' => 'The :attribute is required',
'only_this_key.json' => 'The :attribute field must be valid JSON',
];
$validator = \Validator::make($requestData, [
'only_this_key' => 'required|json',
], $messages);
if ($validator->fails()) {
return new APIErrorValidationResponse($request, $validator);
}
return response()->json(['all good' => 'here']);
The error I get back is The inventory field must be valid JSON even though it is!
Passing in the raw data using Postman
{
"only-this-key": {
"item-one": "one",
"item-two": "two",
"item-three": "three"
},
"not": "wanted"
}
When I use dd($request->all()); within the method
array:2 [
"what-i-want" => array:3 [
"item-one" => "one"
"item-two" => "two"
"item-three" => "three"
]
"not" => "wanted"
]
The problem is with how Laravel is interpreting the raw data in the request. If you run dd($request->all()) in your controller you will see this output:
array:1 [
"{"only_this_key":{"one":"two"}}" => ""
]
Your entire JSON string is getting set as a key with a value of an empty string. If you absolutely must send it as raw data, then you're going to have to grab that key value and save it to an array with the key that you want. This should work (instead of the intersect line).
$requestData = ['only_this_key' => key($request->all())];
Alternatively, you can just send the body as x-www-form-urlencoded with your entire JSON string as the only value for one key.
I have a block of code which needs to loop through a JSON array which is obtained from response of a REST service. (Full gist available here.)
.exec(http("Request_1")
.post("/endPoint")
.headers(headers_1)
.body(StringBody("""REQUEST_BODY""")).asJSON
.check(jsonPath("$.result").is("SUCCESS"))
.check(jsonPath("$.data[*]").findAll.saveAs("pList")))
.exec(session => {
println(session)
session
})
.foreach("${pList}", "player"){
exec(session => {
val playerId = JsonPath.query("$.playerId", "${player}")
session.set("playerId", playerId)
})
.exec(http("Request_1")
.post("/endPoint")
.headers(headers_1)
.body(StringBody("""{"playerId":"${playerId}"}""")).asJSON
.check(jsonPath("$.result").is("SUCCESS")))
}
The response format of the first request was
{
"result": "SUCCESS",
"data": [
{
"playerId": 2
},
{
"playerId": 3
},
{
"playerId": 4
}
]
}
And playerId shows up in the session as
pList -> Vector({playerId=2, score=200}, {playerId=3, score=200}
I am seeing in the second request the body is
{"playerId":"Right(empty iterator)}
Expected : 3 requests with body as
{"playerId":1}
{"playerId":2}
{"playerId":3}
I can loop over the resulting array successfully if I save just the playerIds:
.check(jsonPath("$.data[*].playerId").findAll.saveAs("pList")))
I managed to get the requests you're looking for sent out (although still getting a 404, but that might be server-side or the request your gist is sending might be missing something). The trick was to give up on JsonPath entirely:
.exec(http("Request_10")
.get("gatling1")
.headers(headers_10)
.check(jsonPath("$.result").is("SUCCESS"),
jsonPath("$.data[*]").ofType[Map[String,Any]].findAll.saveAs("pList")))
.foreach("${pList}", "player") {
exec(session => {
val playerMap = session("player").as[Map[String,Any]]
val playerId = playerMap("playerId")
session.set("playerId", playerId)
})
Here, the jsonPath check can automatically store your JSON object as a map, and then you can access the player ID by key. The value type doesn't have to be Any, you could use Int or Long if all your values are numbers. If you want more info on what went wrong with JsonPath, read on.
Your first problem is that JsonPath.query() doesn't just return the value you're looking for. From the JsonPath readme:
JsonPath.query("$.a", jsonSample) gives you Right(non-empty iterator). This will allow you to iterate over all possible solutions to the query.
Now, when it says Right(non-empty iterator), I assumed that meant the iterator was not empty. However, if you try this:
val playerId = JsonPath.query("$.playerId", session("player").as[String]).right.get
println(playerId)
...it prints "empty iterator". I'm not sure whether it's a problem with JsonPath, the jsonPath check, or usage somewhere in between, but there's not quite enough documentation for me to want to dig into it.