Using Loki and Promtail and I have some JSON which I am looking to parse with a pipeline stage. I have defined expressions and created a new JSON stage for every nested object/array, however the logs that I am parsing do not have a definite number of objects in an array. It could be 2, it could be 5.
Here is some example JSON
{
"level": "info",
"msg": "This is the message",
"stats": {
"bytes": 12345,
"checks": 1234,
"transferring": [
{
"eta": 1234,
"name": "path/to/file/1"
},
{
"eta": 1234,
"name": "path/to/file/2"
},
{
"eta": 1234,
"name": "path/to/file/3"
},
{
"eta": 1234,
"name": "path/to/file/4"
},
{
"eta": 1234,
"name": "path/to/file/5"
}
],
"transfers": 28
},
"time": "2022-07-28T13:55:47.251259+01:00"
}
Here is my pipeline_stage:
pipeline_stages:
- json:
expressions:
level: level
msg: msg
timestamp: time
stats:
- json:
expressions:
bytes: bytes
checks: checks
transferring:
source: stats
- json:
expressions:
transferring_eta: eta
transferring_name: name
source: transferring
- labels:
level:
msg:
timestamp:
stats:
bytes:
checks:
transferring:
transferring_eta:
transferring_name:
The "bytes" and "checks" key-value pairs get created, as they are nested inside the "stats" value, however I can not create the "transferring_eta" or "transferring_name" key-value pairs.
I need to use filebeat to push my json data into elastic search, but I'm having trouble decoding my json fields into separate fields extracted from the message field.
Filebeat version : 7.16.2
Filebeat.yml
filebeat.inputs:
- type: log
enabled: true
paths:
- /logs/*.json
multiline.pattern: '^{'
multiline.negate: true
multiline.match: after
processors:
- decode_json_fields:
fields: ["message"]
process_array: false
max_depth: "2"
target: ""
overwrite_keys: true
add_error_key: false
output.elasticsearch:
# Boolean flag to enable or disable the output module.
enabled: true
hosts: ["http://localhost:9200"]
Json Input :
{
"Source": [
{
"date": "28-09-2021",
"language": " C++",
"comment": 11,
"code": 150325
},
{
"date": "28-09-2021",
"language": " C++",
"comment": 11,
"code": 106026
}
]
}
Current Output:
Expected Output:
separate fields
_source: {
#timestamp: "2022-01-12T09:12:36.904Z",
"date": "28-09-2021",
"language": " C++",
"comment": 11,
"code": 106026
input: {
type: "log"
},
Please suggest to decode the multiline json in filebeat?
I have to create an API that can return JSON object in two different formats based on the input parameter. Suppose if we pass "report=monthly" in query string the output will be like:
[{
"Month": "Jan",
"Details": [{
"userId": 12345,
"userName": "ABC"
}]
}, {
"Month": "Feb",
"Details": [{
"userId": 12346,
"userName": "ABD"
}]
}]
If the above parameter is not passed the output will be like:
{
"Details": [{
"userId": 12345,
"userName": "ABC"
}, {
"userId": 12346,
"userName": "ABD"
}]
}
How can I define schema for a single API to return JSON in the above formats? I can not create 2 endpoints.
The easier way is to define Month as an optional property.
paths:
/report:
parameters:
...
get:
responses:
200:
description: OK
schema:
type: array
items:
$ref: '#/definitions/ReportEntry'
definitions:
ReportEntry:
type: object
properties:
Month:
type: string
example: Jan
Details:
type: array
items:
$ref: '#/definitions/UserInfo'
required:
- Details # <---- "Details" is required, "Month" is optional
UserInfo:
type: object
properties:
userId:
type: integer
example: 12345
userName:
type: string
example: ABC
required:
- userId
- userName
I have a requirement like convert JSON data into string can you please help me on this.
My JSON data from service:
{
"chathistory": [
{
"broadcastId": "123",
"caption": "",
"chattype": "CHAT_SINGLE",
"delivered": "2015-03-23T10:20",
"duration": "0"
},
{
"broadcastId": "124",
"caption": "",
"chattype": "CHAT_SINGLE",
"delivered": "2015-03-23T10:20",
"duration": "5"
}
],
"status": "success"
}
I need to convert it like the below format
chathistory: [
{
broadcastId: '123',
caption: '',
chattype: 'CHAT_SINGLE',
delivered: '2015-03-23T10: 20',
duration: '0'
},
{
broadcastId: '124',
caption: '',
chattype: 'CHAT_SINGLE',
delivered: '2015-03-23T10: 30',
duration: '5'
}
]
Thanks,
Sridhar
I've got some JSON that looks like this:
[
{
_id: ObjectId("544809736654daf1ea897ca"),
project: "demo",
tools: ['ajax', 'javascript', 'html'],
},
{
_id: ObjectId("322148965654daf1ea81ca"),
project: "trial",
tools: ['haskell'],
}
]
I've saved it in a file called items.
I'm trying to import it to my Angular project with this code:
app.service("getItemsService",
function($http, $q){
return {
getItems: function getItems(){
return $http.get('data/_items').success(function(data){
return data;
});
}
}
}
);
but when I do, I get an error saying:
SyntaxError: Unexpected token _
at Object.parse (native)
I've tried everything I can think of to fix this - i.e. I've named the file items.json, I changed _id to id, I've tried adding {'Content-Type': 'application/json'} to get() function as a parameter to indicate its JSON.
Nothing is working! Any tips?
Your angular service seems good. It is your "JSON" content file that is not formated as a regular JSON string:
properties as to be strings double quoted
values as to be strings double quoted or object {} or arrays []
functions couldn't be used in json strings (your ObjectId() function)
Last elements in arrays can't finish by a comma as the previous elements
Try this instead :
[
{
"_id": "544809736654daf1ea897ca",
"project": "demo",
"tools": ["ajax", "javascript", "html"]
},
{
"_id": "322148965654daf1ea81ca",
"project": "trial",
"tools": ["haskell"]
}
]
Always try your JSON in a linter in that kind of problem like http://jsonlint.com/
It would have tell you
Parse error on line 2:
[ { _id: ObjectId("54480
--------------^
Expecting 'STRING', '}'
That is a little less cryptic : it tells you that _id is not a String => you'r missing the "
After that it will tell you that :
Parse error on line 3:
... { "_id": ObjectId("5448097366
----------------------^
Expecting 'STRING', 'NUMBER', 'NULL', 'TRUE', 'FALSE', '{', '['
ObjectId is not valid for JSON, you have to find somehting else, like only the id, or all in a String :
[
{
"_id": "544809736654daf1ea897ca",
"project": "demo",
"tools": [
'ajax',
'javascript',
'html'
],
},
{
"_id": "322148965654daf1ea81ca",
"project": "trial",
"tools": [
'haskell'
],
}
]
But hey, it's not finish :
Parse error on line 5:
...ols": [ 'ajax',
----------------------^
Expecting 'STRING', 'NUMBER', 'NULL', 'TRUE', 'FALSE', '{', '[', ']'
Well yeah, ' is not valid, some parser actualy fail on single quote.
[
{
"_id": "544809736654daf1ea897ca",
"project": "demo",
"tools": [
"ajax",
"javascript",
"html"
],
},
{
"_id": "322148965654daf1ea81ca",
"project": "trial",
"tools": [
"haskell"
],
}
]
Still not there :
Parse error on line 9:
... ], }, { "_i
---------------------^
Expecting 'STRING'
it seems that there is a trouble around line 9, and if you look closely you will see a , with nothing behind, let's remove the useless ,
[
{
"_id": "544809736654daf1ea897ca",
"project": "demo",
"tools": [
"ajax",
"javascript",
"html"
]
},
{
"_id": "322148965654daf1ea81ca",
"project": "trial",
"tools": [
"haskell"
]
}
]
And here it is !