I used the code to convert json data from Yarn REST API to Prometheus data type:
https://github.com/prometheus-community/json_exporter.
However, it printed errors:
level=error ts=2021-07-08T06:31:03.712Z caller=collector.go:83 msg="Failed to extract value for metric" path={.capacity} err="capacity is not found" metric="Desc{fqName: "queues_capacity", help: "information on queues", constLabels: {}, variableLabels: [type]}"
I was wondering if there is some wrong in my configuration of YAML file (such as in terms of nested json) or just the reason about the code.
my yaml config is:
metrics:
- name: queues
path: "{ .scheduler.schedulerInfo.queues.queue }"
help: information on queues
type: object
labels:
type: '{.type}'
values:
capacity: '{.capacity}'
and part of the json file is:
{
"scheduler": {
"schedulerInfo": {
"type": "capacityScheduler",
"capacity": 100,
"usedCapacity": 1.0526316,
"maxCapacity": 100,
"queueName": "root",
"queues": {
"queue": [
{
"type": "capacitySchedulerLeafQueueInfo",
"capacity": 10,
"usedCapacity": 10.526316,
"maxCapacity": 100,
use [*] to get all object first, so it should be:
path: "{ .scheduler.schedulerInfo.queues.queue[*] }"
according to this: https://kubernetes.io/docs/reference/kubectl/jsonpath/
Related
I am using log4js to log data to a file in my app. I want to display some of this data in my Grafana dashboard and for that I am using Promtail to read logs from the file, pre-process it and send it to Loki. In Loki, I want to filter the data based on the parsed values.
Here is an example of my logs:
[2023-02-12T04:01:23.587] [DEBUG] default - {
"message_id": 123,
"from": {
"id": 123,
"is_bot": false,
"first_name": "XXX",
"last_name": "XXXXX",
"username": "XXXXX",
"is_premium": true
},
"chat": {
"id": 123,
"title": "XXXXX",
"username": "XXXXX",
"type": "supergroup"
},
"date": 123,
"message_thread_id": 123,
"text": "XXX XXXXX XXXXX"
}
Here is my current Promtail configuration:
server:
http_listen_port: 80
grpc_listen_port: 9095
log_level: debug
positions:
filename: /tmp/positions.yaml
clients:
- url: http://192.168.1.64:3100/loki/api/v1/push
scrape_configs:
- job_name: patriotbot
pipeline_stages:
- multiline:
firstline: \[\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}\.\d{3}\]
- regex:
expression: '^\[.*\]\s\w*\s-\s(\{.*\})'
name: log_entry
- json:
source: log_entry
name: only_json
expressions:
from_id: 'from.id'
is_bot: 'from.is_bot'
first_name: 'from.first_name'
last_name: 'from.last_name'
username: 'from.username'
chat_id: 'chat.id'
chat_title: 'chat.title'
chat_type: 'chat.type'
- output:
source: only_json
static_configs:
- targets:
- localhost
labels:
job: patriot
type: all
__path__: /logs/all.log
I have two issues with my current configuration:
My logs are saved as multiple lines, making them difficult to parse. I have attempted to fix this issue with the multiline stage, but which is displayed properly in Loki, though maybe it won't work for following parsing.
The timestamp and metadata in front of the JSON object is preventing it from being parsed properly. Should I get rid it before sending it to the pipe and parse?
So can somebody suggest changes to my configuration that would allow me to properly parse these multiline logs and extract the relevant data?
I am able to get a single JSON object in Kibana:
By having this in the filebeat.yml file:
output.elasticsearch:
hosts: ["localhost:9200"]
How can I get the individual elements in the JSON string. So say if I wanted to compare all the "pseudorange" fields of all my JSON objects. How would I:
Select "pseudorange" field from all my JSON messages to compare them.
Compare them visually in kibana. At the moment I can't even find the message let alone the individual fields in the visualisation tab...
I have heard of people using logstash to parse the string somehow but is there no way of doing this simply with filebeat? If there isn't then what do I do with logstash to help filter the individual fields in the json instead of have my message just one big json string that I cannot interact with?
I get the following output from output.console, note I am putting some information in <> to hide it:
"#timestamp": "2021-03-23T09:37:21.941Z",
"#metadata": {
"beat": "filebeat",
"type": "doc",
"version": "6.8.14",
"truncated": false
},
"message": "{\n\t\"Signal_data\" : \n\t{\n\t\t\"antenna type:\" : \"GPS\",\n\t\t\"frequency type:\" : \"GPS\",\n\t\t\"position x:\" : 0.0,\n\t\t\"position y:\" : 0.0,\n\t\t\"position z:\" : 0.0,\n\t\t\"pseudorange:\" : 20280317.359730639,\n\t\t\"pseudorange_error:\" : 0.0,\n\t\t\"pseudorange_rate:\" : -152.02620448094211,\n\t\t\"svid\" : 18\n\t}\n}\u0000",
"source": <ip address>,
"log": {
"source": {
"address": <ip address>
}
},
"input": {
"type": "udp"
},
"prospector": {
"type": "udp"
},
"beat": {
"name": <ip address>,
"hostname": "ip-<ip address>",
"version": "6.8.14"
},
"host": {
"name": "ip-<ip address>",
"os": {
<ubuntu info>
},
"id": <id>,
"containerized": false,
"architecture": "x86_64"
},
"meta": {
"cloud": {
<cloud info>
}
}
}
In Filebeat, you can leverage the decode_json_fields processor in order to decode a JSON string and add the decoded fields into the root obejct:
processors:
- decode_json_fields:
fields: ["message"]
process_array: false
max_depth: 2
target: ""
overwrite_keys: true
add_error_key: false
Credit to Val for this. His answer worked however as he suggested my JSON string had a \000 at the end which stops it being JSON and prevented the decode_json_fields processor from working as it should...
Upgrading to version 7.12 of Filebeat (also ensure version 7.12 of Elasticsearch and Kibana because mismatched versions between them can cause issues) allows us to use the script processor: https://www.elastic.co/guide/en/beats/filebeat/current/processor-script.html.
Credit to Val here again, this script removed the null terminator:
- script:
lang: javascript
id: trim
source: >
function process(event) {
event.Put("message", event.Get("message").trim());
}
After the null terminator was removed the decode_json_fields processor did its job as Val suggested and I was able to extract the individual elements of the JSON field which allowed Kibana visualisation to look at the elements I wanted!
Let's say we have an example json swagger spec:
{
"swagger": "2.0",
"info": {
"version": "1.0.0",
"title": "Some API"
},
"basePath": "/api/v1",
"consumes": [
"application/json"
],
"produces": [
"application/json",
"text/csv"
],
"paths": {
"/some/endpoint": {
"get": {
"parameters": [
{
"in": "body",
"name": "body",
"required": false,
"schema": {
"$ref": "#/definitions/BodyParamsDefinition"
}
}
],
"responses": {
"200": { ?? } ...
There are two content types that can be produced:
application/json
text/csv
Default response for GET /some/endpoint is a csv file, but if the format query param is used like this /some/endpoint?format=json, the response would be in json format.
I have trouble finding how should I finish my specification with proper responses.
When I use this approach: https://swagger.io/docs/specification/describing-responses/ i get a validation error: ...get.responses['200'] should NOT have additional properties
You are almost there, you just need to define a schema for the response. This schema defines the response structure for all content types associated with this status code.
For example, if the operation returns this JSON:
[
{
"petType": "dog",
"name": "Fluffy"
},
{
"petType": "cat",
"name": "Crookshanks"
}
]
and this CSV:
petType,name
dog,Fluffy
cat,Crookshanks
you would use:
# YAML
responses:
200:
description: OK
schema:
type: array
items:
type: object
properties:
petType:
type: string
name:
type: string
More info: Describing Responses
In OpenAPI 3.0, content type definitions were improved and schemas can vary by content type:
openapi: 3.0.0
...
paths:
/some/endpoint:
get:
responses:
'200':
description: OK
content:
# JSON data is an object
application/json:
schema:
type: object
properties:
message:
type: string
# CSV data is a string of text
text/csv:
schema:
type: string
Default response for GET /some/endpoint is a csv file, but if the format query param is used like this /some/endpoint?format=json, the response would be in json format.
There's currently no way to map specific responses to specific operation parameters, but there are several related proposals in the OpenAPI Specification repository:
Accommodate legacy APIs by allowing query parameters in the path
Querystring in Path Specification
Support an operation to have multiple specifications per path
Overloading
I'm new to OpenAPI and I need some help to create a basic swagger file for PayPal's payment API to create a payment from our platform. Note: OAuth is already configured.
Below is a basic swagger file but I don't know where to add the paymet request information (i.e. intent, payer, transactions etc.) into:
{
"swagger": "2.0",
"info": {
"description": "this is a payment request to through PayPal",
"title": "Swagger PayPal Payment",
"version": "1.0.0"
},
"host": "api.sandbox.paypal.com",
"basePath": "/v1/payments", //
"schemes": [ "https" ],
"paths": {
"/payment":
{
"post": {
"summary": "Creates a payment"
"description": "Creates a payment request to Paypal",
"parameters": {
},
//"intent": "sale",
//"payer":
//{
// "payment_method": "paypal"
//},
//"transactions": [
// {
// "amount": {
// "total": "9.00",
// "currency": "EUR"
// }
// }
//],
"responses": {
"200": {
"description": "OK"
}
}
}
}
}
}
Testing the file on editor.swagger, I get an "OBJECT_ADDITIONAL_PROPERTIES" error on transactions, payer, and intent.
JSON payload is defined as a body parameter (parameter with in: body), and this parameter needs a schema that defines the JSON object properties.
You would typically define object schemas in the global definitions section and reference them using $ref.
Here is the YAML version for readability. To convert it to JSON, paste it into http://editor.swagger.io and use File > Download JSON.
swagger: "2.0"
info:
description: this is a payment request to through PayPal
title: Swagger PayPal Payment
version: "1.0.0"
host: api.sandbox.paypal.com
basePath: /v1/payments
schemes: [ https ]
paths:
/payment:
post:
summary: Creates a payment
description: Creates a payment request to Paypal
parameters:
- in: body
name: payment
required: true
schema:
$ref: "#/definitions/Payment" # <--------
responses:
"200":
description: OK
definitions:
# Request body object
Payment:
type: object
properties:
intent:
type: string
payer:
$ref: "#/definitions/Payer"
transactions:
type: array
items:
$ref: "#/definitions/Transaction"
Payer:
type: object
properties:
payment_method:
type: string
example: paypal
Transaction:
type: object
properties:
... # TODO
I have the following files
user.json
"user": {
"id": 1,
"name": "nameuser",
"online": true,
"profile": {
"photo": "",
"validated": true,
"popular": true,
"suspect": false,
"moderator": false,
"age": "22 ani",
"gender_id": "M"
}
}
profile.raml
displayName: Profile
get:
description: Get profile data
queryParameters:
userId:
description: The user id for which we are requesting the profile data
type: integer
required: true
responses:
200:
body:
application/json:
example: |
{
"user": !include user.json,
"details": {
"friend": true
}
}
The user json is present in more examples and I want to reuse it.
I'm using raml2html and it compiles it to
so how do I do this ?
I have used parameters successfully in the past. You will not be able to put a parameter inside an included file because RAML views all included files as strings. But you can do something like this in your profile.raml:
example: |
{
"user": <<userItem>>,
"details": {
"friend": true
}
}
The RAML 200 Tutorial has a good explanation and code examples (see snippets below) on how to declare parameters and them pass them in. I highly recommend reading the entire tutorial though.
resourceTypes:
- collection:
description: Collection of available <<resourcePathName>> in Jukebox.
get:
description: Get a list of <<resourcePathName>>.
responses:
200:
body:
application/json:
example: |
<<exampleCollection>>
/songs:
type:
collection:
exampleCollection: !include jukebox-include-songs.sample