I'm trying to convert a map in jenkinsfile to json for which I'm using
def pullRequestData = [
title: "Deployment-to-SIT",
description: "Pull request from",
state: "OPEN",
open: "true",
closed: "false",
fromRef: [
id: "refs/heads/${branchjsonObj.displayId}",
repository: [
slug: "${reposlug}",
name: "null",
project: [
key: "${projectkey}"
]
]
]
]
def jsonmap = JsonOutput.toJson(pullRequestData)
echo ${jsonmap}
gives the output as json formatted but without strings like,
{title:Deployment-to-SIT,description:Pull request from,state:OPEN,open:true,closed:false,fromRef:{id:refs/heads/deployment-to-SIT,repository:{slug:argocd-sample-chart,name:null,project:{key:TD}}}}
but the output i need is
{"title":"Deployment-to-SIT","description":"Pull request from","state":"OPEN","open":"true","closed":"false","fromRef":{"id":"refs/heads/deployment-to-SIT","repository":{"slug":"argocd-sample-chart","name":"null","project":{"key":"TD"}}}
Also tried,
def jsonpullRequestData = this.steps.writeJSON(file: 'jsonmap.json', json: pullRequestData)
But the behavior is same,o/p is without quotes.Any help will be much appreciated.
Instead of using json groovy lib.. as per your requirement you can directly use $pullRequestData.toString() method to get in string format with quotes.
Related
I am able to get a single JSON object in Kibana:
By having this in the filebeat.yml file:
output.elasticsearch:
hosts: ["localhost:9200"]
How can I get the individual elements in the JSON string. So say if I wanted to compare all the "pseudorange" fields of all my JSON objects. How would I:
Select "pseudorange" field from all my JSON messages to compare them.
Compare them visually in kibana. At the moment I can't even find the message let alone the individual fields in the visualisation tab...
I have heard of people using logstash to parse the string somehow but is there no way of doing this simply with filebeat? If there isn't then what do I do with logstash to help filter the individual fields in the json instead of have my message just one big json string that I cannot interact with?
I get the following output from output.console, note I am putting some information in <> to hide it:
"#timestamp": "2021-03-23T09:37:21.941Z",
"#metadata": {
"beat": "filebeat",
"type": "doc",
"version": "6.8.14",
"truncated": false
},
"message": "{\n\t\"Signal_data\" : \n\t{\n\t\t\"antenna type:\" : \"GPS\",\n\t\t\"frequency type:\" : \"GPS\",\n\t\t\"position x:\" : 0.0,\n\t\t\"position y:\" : 0.0,\n\t\t\"position z:\" : 0.0,\n\t\t\"pseudorange:\" : 20280317.359730639,\n\t\t\"pseudorange_error:\" : 0.0,\n\t\t\"pseudorange_rate:\" : -152.02620448094211,\n\t\t\"svid\" : 18\n\t}\n}\u0000",
"source": <ip address>,
"log": {
"source": {
"address": <ip address>
}
},
"input": {
"type": "udp"
},
"prospector": {
"type": "udp"
},
"beat": {
"name": <ip address>,
"hostname": "ip-<ip address>",
"version": "6.8.14"
},
"host": {
"name": "ip-<ip address>",
"os": {
<ubuntu info>
},
"id": <id>,
"containerized": false,
"architecture": "x86_64"
},
"meta": {
"cloud": {
<cloud info>
}
}
}
In Filebeat, you can leverage the decode_json_fields processor in order to decode a JSON string and add the decoded fields into the root obejct:
processors:
- decode_json_fields:
fields: ["message"]
process_array: false
max_depth: 2
target: ""
overwrite_keys: true
add_error_key: false
Credit to Val for this. His answer worked however as he suggested my JSON string had a \000 at the end which stops it being JSON and prevented the decode_json_fields processor from working as it should...
Upgrading to version 7.12 of Filebeat (also ensure version 7.12 of Elasticsearch and Kibana because mismatched versions between them can cause issues) allows us to use the script processor: https://www.elastic.co/guide/en/beats/filebeat/current/processor-script.html.
Credit to Val here again, this script removed the null terminator:
- script:
lang: javascript
id: trim
source: >
function process(event) {
event.Put("message", event.Get("message").trim());
}
After the null terminator was removed the decode_json_fields processor did its job as Val suggested and I was able to extract the individual elements of the JSON field which allowed Kibana visualisation to look at the elements I wanted!
I'm trying to send a custom JSON payload from my pipeline on Jenkins after the last successful stage, like this:
post {
success {
script {
def payload = """
{
"type": "AdaptiveCard",
"body": [
{
"type": "TextBlock",
"size": "Medium",
"weight": "Bolder",
"text": "SonarQube report from Jenkins Pipeline"
},
{
"type": "TextBlock",
"text": "Code was analyzed was successfully.",
"wrap": true,
"color": "Good",
"weight": "Bolder"
}
],
"$schema": "http://adaptivecards.io/schemas/adaptive-card.json",
"version": "1.3"
}"""
httpRequest httpMode: 'POST',
acceptType: 'APPLICATION_JSON',
contentType: 'APPLICATION_JSON',
url: "URL",
requestBody: payload
}
}
}
}
But I get an error
Error when executing success post condition:
groovy.lang.MissingPropertyException: No such property: schema for class: groovy.lang.Binding
at groovy.lang.Binding.getVariable(Binding.java:63)
I'm using the HTTP Request plugin available for Jenkins and the format of the JSON payload is correct for MS Teams.
The issue is actually a groovy syntax error. You can easily check this in something like https://groovy-playground.appspot.com/ by adding your def payload = ... statement.
There are multiple ways to get multiline strings in groovy:
triple single quoted string
triple double quoted string
slashy string
dollary slashy string
Apart from the single quoted string, they also have a secondary property which is interpolation
Notice how in the initial JSON payload, there's a "$schema" key? Using triple double quoted strings makes groovy want to find a schema variable and use it's value to construct that payload variable.
You have two separate solutions:
Use triple single quoted string - just update """ to '''
Escape the variable - just update "$schema" to "\$schema" (making $ a literal $ instead of it being used as an interpolation prefix)
Im using the following JSON and query to calculate the array length in the JMeter json extractor.
{
"data": {
"modal": "HJ",
"technicalid": "e492fc62-a886-67a461b76de8",
"viewModel": {
"series": [
{
"name": "H_0_G_0_R_0",
"UID": "J_0_G_0_R_0",
"description": "Test1",
"type": "series",
"groups": [
{
"name": "H_0_G_0",
"UID": "G_0_G_0",
"description": "Group 1",
"type": "group"
}
],
"postProcessing": null
}
]
},
"status": "success"
},
"success": true,
"statusCode": 200,
"errorMessage": ""
}
Here is the query.
data.Model.series[0].groups.length
This is working fine in the online jsonquerytool. When I use this query in the JMeter json extractor, it is returning null. I assume this is because it is returning an integer because other similar queries which are returning strings are working fine with json extractor . How to find the array length in JMeter json extractor?
Why JSON extractor to calculate the length? You could use a post processer. Like JSR223 post processer using groovy script.
import groovy.json.*
def response = prev.responseDataAsString ;
def json = new JsonSlurper().parseText(response) ;
def sizeResultPractitioners = json.data.viewModel.series[0].groups.size();
log.info("---------->"+sizeResultPractitioners);
I tried with your JSON response payload and also tried with modified response payload,
With modified response payload,
With JSON Extractor you can provide "Match No." as -1:
and the number of matches will be available as foo_matchNr JMeter Variable:
Alternative option is going for JSON JMESPath Extractor which provides length() function so you can get the size of the array as:
length(data.viewModel.series[0].groups)
or if you prefer pipe expressions
data.viewModel.series[0].groups | length(#)
Let's say we have an example json swagger spec:
{
"swagger": "2.0",
"info": {
"version": "1.0.0",
"title": "Some API"
},
"basePath": "/api/v1",
"consumes": [
"application/json"
],
"produces": [
"application/json",
"text/csv"
],
"paths": {
"/some/endpoint": {
"get": {
"parameters": [
{
"in": "body",
"name": "body",
"required": false,
"schema": {
"$ref": "#/definitions/BodyParamsDefinition"
}
}
],
"responses": {
"200": { ?? } ...
There are two content types that can be produced:
application/json
text/csv
Default response for GET /some/endpoint is a csv file, but if the format query param is used like this /some/endpoint?format=json, the response would be in json format.
I have trouble finding how should I finish my specification with proper responses.
When I use this approach: https://swagger.io/docs/specification/describing-responses/ i get a validation error: ...get.responses['200'] should NOT have additional properties
You are almost there, you just need to define a schema for the response. This schema defines the response structure for all content types associated with this status code.
For example, if the operation returns this JSON:
[
{
"petType": "dog",
"name": "Fluffy"
},
{
"petType": "cat",
"name": "Crookshanks"
}
]
and this CSV:
petType,name
dog,Fluffy
cat,Crookshanks
you would use:
# YAML
responses:
200:
description: OK
schema:
type: array
items:
type: object
properties:
petType:
type: string
name:
type: string
More info: Describing Responses
In OpenAPI 3.0, content type definitions were improved and schemas can vary by content type:
openapi: 3.0.0
...
paths:
/some/endpoint:
get:
responses:
'200':
description: OK
content:
# JSON data is an object
application/json:
schema:
type: object
properties:
message:
type: string
# CSV data is a string of text
text/csv:
schema:
type: string
Default response for GET /some/endpoint is a csv file, but if the format query param is used like this /some/endpoint?format=json, the response would be in json format.
There's currently no way to map specific responses to specific operation parameters, but there are several related proposals in the OpenAPI Specification repository:
Accommodate legacy APIs by allowing query parameters in the path
Querystring in Path Specification
Support an operation to have multiple specifications per path
Overloading
I've been given some JSON files generated by a REST API with plenty of properties.
I've created a Swagger 2.0 definition for this API and need to give it the corresponding schema for the response.
The main problem: this JSON file has loads of properties. It would take so much time and I would make many mistakes if I write the schema manually. And it’s not the only API I need to describe.
I know there are some tools to convert JSON to JSON schemas but, if I’m not mistaken, Swagger only has $refs to other objects definitions thus only has one level whereas the tools I’ve found only produce tree structured schemas.
My question: is there any tool to convert a JSON (or JSON Schema) to a Swagger 2.0 compatible one ?
Note: I'm working in YAML but I wouldn't be an issue, would it ?
For example, what I need:
List of Movements:
type: "array"
items:
$ref: "#/definitions/Movement"
Movement:
properties:
dateKey:
type: "string"
movement:
$ref: "#/definitions/Stock"
additionalProperties: false
Stock:
properties:
stkUnitQty:
type: "string"
stkDateTime:
type: "string"
stkUnitType:
type: "string"
stkOpKey:
type: "string"
additionalProperties: false
For my JSON document:
[
{
"dateKey": "20161110",
"stkLvls": [
{
"stkOpKey": "0",
"stkUnitType": "U",
"stkDateTime": "20161110T235010.240+0100",
"stkUnitQty": 30
}
]
},
{
"dateKey": "20161111",
"stkLvls": [
{
"stkOpKey": "0",
"stkUnitType": "U",
"stkDateTime": "20161111T231245.087+0100",
"stkUnitQty": 21
}
]
}
]
But, what http://jsonschema.net/#/ gives me:
---
"$schema": http://json-schema.org/draft-04/schema#
type: array
items:
type: object
properties:
dateKey:
type: string
stkLvls:
type: array
items:
type: object
properties:
stkOpKey:
type: string
stkUnitType:
type: string
stkDateTime:
type: string
stkUnitQty:
type: integer
required:
- stkOpKey
- stkUnitType
- stkDateTime
- stkUnitQty
required:
- dateKey
- stkLvls
I'm new to that, but curious, don't hesitate to explain deeply.
Thank you in advance for your help !
I also needed a converter tool and came across this. So far it seems to work pretty well. It does both JSON and YAML formats.
https://swagger-toolbox.firebaseapp.com/
Given this JSON (their sample):
{
"id": 1,
"name": "A green door",
"price": 12,
"testBool": false,
"tags": [
"home",
"green"
]
}
it generated this:
{
"required": [
"id",
"name",
"price",
"testBool",
"tags"
],
"properties": {
"id": {
"type": "number"
},
"name": {
"type": "string"
},
"price": {
"type": "number"
},
"testBool": {
"type": "boolean"
},
"tags": {
"type": "array",
"items": {
"type": "string"
}
}
}
}
I know there are some tools to convert JSON to JSON schemas but, if
I’m not mistaken, Swagger only has $refs to other objects definitions
thus only has one level
You are mistaken. Swagger will respect any valid v4 JSON schema, as long as it only uses the supported subset.
The Schema Object...is based on the JSON Schema Specification Draft 4 and
uses a predefined subset of it. On top of this subset, there are extensions provided by this specification to allow for more complete
documentation.
It goes on to list the parts of JSON schema which are supported, and the bits which are not, and the bits which are extended by swagger.
You can directly goto https://bikcrum.github.io/Swagger-JSON-Schema-In-YAML_webversion/ for online conversion.
I wrote following python script to generate JSON schema in YAML format (preserving key order) that is used in Swagger.
import json
# input file containing json file
with open('data.json') as f:
json_data = json.load(f)
# json schema in yaml format
out = open('out.yaml','w')
def gettype(type):
for i in ['string','boolean','integer']:
if type in i:
return i
return type
def write(string):
print(string)
out.write(string+'\n')
out.flush()
def parser(json_data,indent):
if type(json_data) is dict:
write(indent + 'type: object')
if len(json_data) > 0:
write(indent + 'properties:')
for key in json_data:
write(indent + ' %s:' % key)
parser(json_data[key], indent+' ')
elif type(json_data) is list:
write(indent + 'type: array')
write(indent + 'items:')
if len(json_data) != 0:
parser(json_data[0], indent+' ')
else:
write(indent + ' type: object')
else:
write(indent + 'type: %s' % gettype(type(json_data).__name__))
parser(json_data,'')
Update: If you want YAML with sorted keys (which is by default) use YAML library
import json
import yaml
# input file containing json file
with open('data.json') as f:
json_data = json.load(f)
# json schema in yaml format
def gettype(type):
for i in ['string','boolean','integer']:
if type in i:
return i
return type
def parser(json_data):
d = {}
if type(json_data) is dict:
d['type'] = 'object'
for key in json_data:
d[key] = parser(json_data[key])
return d
elif type(json_data) is list:
d['type'] = 'array'
if len(json_data) != 0:
d['items'] = parser(json_data[0])
else:
d['items'] = 'object'
return d
else:
d['type'] = gettype(type(json_data).__name__)
return d
p = parser(json_data)
with open('out.yaml','w') as outfile:
yaml.dump(p,outfile, default_flow_style=False)