Convert JSON to JSON Schema draft 4 compatible with Swagger 2.0 - json

I've been given some JSON files generated by a REST API with plenty of properties.
I've created a Swagger 2.0 definition for this API and need to give it the corresponding schema for the response.
The main problem: this JSON file has loads of properties. It would take so much time and I would make many mistakes if I write the schema manually. And it’s not the only API I need to describe.
I know there are some tools to convert JSON to JSON schemas but, if I’m not mistaken, Swagger only has $refs to other objects definitions thus only has one level whereas the tools I’ve found only produce tree structured schemas.
My question: is there any tool to convert a JSON (or JSON Schema) to a Swagger 2.0 compatible one ?
Note: I'm working in YAML but I wouldn't be an issue, would it ?
For example, what I need:
List of Movements:
type: "array"
items:
$ref: "#/definitions/Movement"
Movement:
properties:
dateKey:
type: "string"
movement:
$ref: "#/definitions/Stock"
additionalProperties: false
Stock:
properties:
stkUnitQty:
type: "string"
stkDateTime:
type: "string"
stkUnitType:
type: "string"
stkOpKey:
type: "string"
additionalProperties: false
For my JSON document:
[
{
"dateKey": "20161110",
"stkLvls": [
{
"stkOpKey": "0",
"stkUnitType": "U",
"stkDateTime": "20161110T235010.240+0100",
"stkUnitQty": 30
}
]
},
{
"dateKey": "20161111",
"stkLvls": [
{
"stkOpKey": "0",
"stkUnitType": "U",
"stkDateTime": "20161111T231245.087+0100",
"stkUnitQty": 21
}
]
}
]
But, what http://jsonschema.net/#/ gives me:
---
"$schema": http://json-schema.org/draft-04/schema#
type: array
items:
type: object
properties:
dateKey:
type: string
stkLvls:
type: array
items:
type: object
properties:
stkOpKey:
type: string
stkUnitType:
type: string
stkDateTime:
type: string
stkUnitQty:
type: integer
required:
- stkOpKey
- stkUnitType
- stkDateTime
- stkUnitQty
required:
- dateKey
- stkLvls
I'm new to that, but curious, don't hesitate to explain deeply.
Thank you in advance for your help !

I also needed a converter tool and came across this. So far it seems to work pretty well. It does both JSON and YAML formats.
https://swagger-toolbox.firebaseapp.com/
Given this JSON (their sample):
{
"id": 1,
"name": "A green door",
"price": 12,
"testBool": false,
"tags": [
"home",
"green"
]
}
it generated this:
{
"required": [
"id",
"name",
"price",
"testBool",
"tags"
],
"properties": {
"id": {
"type": "number"
},
"name": {
"type": "string"
},
"price": {
"type": "number"
},
"testBool": {
"type": "boolean"
},
"tags": {
"type": "array",
"items": {
"type": "string"
}
}
}
}

I know there are some tools to convert JSON to JSON schemas but, if
I’m not mistaken, Swagger only has $refs to other objects definitions
thus only has one level
You are mistaken. Swagger will respect any valid v4 JSON schema, as long as it only uses the supported subset.
The Schema Object...is based on the JSON Schema Specification Draft 4 and
uses a predefined subset of it. On top of this subset, there are extensions provided by this specification to allow for more complete
documentation.
It goes on to list the parts of JSON schema which are supported, and the bits which are not, and the bits which are extended by swagger.

You can directly goto https://bikcrum.github.io/Swagger-JSON-Schema-In-YAML_webversion/ for online conversion.
I wrote following python script to generate JSON schema in YAML format (preserving key order) that is used in Swagger.
import json
# input file containing json file
with open('data.json') as f:
json_data = json.load(f)
# json schema in yaml format
out = open('out.yaml','w')
def gettype(type):
for i in ['string','boolean','integer']:
if type in i:
return i
return type
def write(string):
print(string)
out.write(string+'\n')
out.flush()
def parser(json_data,indent):
if type(json_data) is dict:
write(indent + 'type: object')
if len(json_data) > 0:
write(indent + 'properties:')
for key in json_data:
write(indent + ' %s:' % key)
parser(json_data[key], indent+' ')
elif type(json_data) is list:
write(indent + 'type: array')
write(indent + 'items:')
if len(json_data) != 0:
parser(json_data[0], indent+' ')
else:
write(indent + ' type: object')
else:
write(indent + 'type: %s' % gettype(type(json_data).__name__))
parser(json_data,'')
Update: If you want YAML with sorted keys (which is by default) use YAML library
import json
import yaml
# input file containing json file
with open('data.json') as f:
json_data = json.load(f)
# json schema in yaml format
def gettype(type):
for i in ['string','boolean','integer']:
if type in i:
return i
return type
def parser(json_data):
d = {}
if type(json_data) is dict:
d['type'] = 'object'
for key in json_data:
d[key] = parser(json_data[key])
return d
elif type(json_data) is list:
d['type'] = 'array'
if len(json_data) != 0:
d['items'] = parser(json_data[0])
else:
d['items'] = 'object'
return d
else:
d['type'] = gettype(type(json_data).__name__)
return d
p = parser(json_data)
with open('out.yaml','w') as outfile:
yaml.dump(p,outfile, default_flow_style=False)

Related

Can Filebeat parse JSON fields instead of the whole JSON object into kibana?

I am able to get a single JSON object in Kibana:
By having this in the filebeat.yml file:
output.elasticsearch:
hosts: ["localhost:9200"]
How can I get the individual elements in the JSON string. So say if I wanted to compare all the "pseudorange" fields of all my JSON objects. How would I:
Select "pseudorange" field from all my JSON messages to compare them.
Compare them visually in kibana. At the moment I can't even find the message let alone the individual fields in the visualisation tab...
I have heard of people using logstash to parse the string somehow but is there no way of doing this simply with filebeat? If there isn't then what do I do with logstash to help filter the individual fields in the json instead of have my message just one big json string that I cannot interact with?
I get the following output from output.console, note I am putting some information in <> to hide it:
"#timestamp": "2021-03-23T09:37:21.941Z",
"#metadata": {
"beat": "filebeat",
"type": "doc",
"version": "6.8.14",
"truncated": false
},
"message": "{\n\t\"Signal_data\" : \n\t{\n\t\t\"antenna type:\" : \"GPS\",\n\t\t\"frequency type:\" : \"GPS\",\n\t\t\"position x:\" : 0.0,\n\t\t\"position y:\" : 0.0,\n\t\t\"position z:\" : 0.0,\n\t\t\"pseudorange:\" : 20280317.359730639,\n\t\t\"pseudorange_error:\" : 0.0,\n\t\t\"pseudorange_rate:\" : -152.02620448094211,\n\t\t\"svid\" : 18\n\t}\n}\u0000",
"source": <ip address>,
"log": {
"source": {
"address": <ip address>
}
},
"input": {
"type": "udp"
},
"prospector": {
"type": "udp"
},
"beat": {
"name": <ip address>,
"hostname": "ip-<ip address>",
"version": "6.8.14"
},
"host": {
"name": "ip-<ip address>",
"os": {
<ubuntu info>
},
"id": <id>,
"containerized": false,
"architecture": "x86_64"
},
"meta": {
"cloud": {
<cloud info>
}
}
}
In Filebeat, you can leverage the decode_json_fields processor in order to decode a JSON string and add the decoded fields into the root obejct:
processors:
- decode_json_fields:
fields: ["message"]
process_array: false
max_depth: 2
target: ""
overwrite_keys: true
add_error_key: false
Credit to Val for this. His answer worked however as he suggested my JSON string had a \000 at the end which stops it being JSON and prevented the decode_json_fields processor from working as it should...
Upgrading to version 7.12 of Filebeat (also ensure version 7.12 of Elasticsearch and Kibana because mismatched versions between them can cause issues) allows us to use the script processor: https://www.elastic.co/guide/en/beats/filebeat/current/processor-script.html.
Credit to Val here again, this script removed the null terminator:
- script:
lang: javascript
id: trim
source: >
function process(event) {
event.Put("message", event.Get("message").trim());
}
After the null terminator was removed the decode_json_fields processor did its job as Val suggested and I was able to extract the individual elements of the JSON field which allowed Kibana visualisation to look at the elements I wanted!

Jenkins Pipeline Error while trying to send JSON Payload after post success

I'm trying to send a custom JSON payload from my pipeline on Jenkins after the last successful stage, like this:
post {
success {
script {
def payload = """
{
"type": "AdaptiveCard",
"body": [
{
"type": "TextBlock",
"size": "Medium",
"weight": "Bolder",
"text": "SonarQube report from Jenkins Pipeline"
},
{
"type": "TextBlock",
"text": "Code was analyzed was successfully.",
"wrap": true,
"color": "Good",
"weight": "Bolder"
}
],
"$schema": "http://adaptivecards.io/schemas/adaptive-card.json",
"version": "1.3"
}"""
httpRequest httpMode: 'POST',
acceptType: 'APPLICATION_JSON',
contentType: 'APPLICATION_JSON',
url: "URL",
requestBody: payload
}
}
}
}
But I get an error
Error when executing success post condition:
groovy.lang.MissingPropertyException: No such property: schema for class: groovy.lang.Binding
at groovy.lang.Binding.getVariable(Binding.java:63)
I'm using the HTTP Request plugin available for Jenkins and the format of the JSON payload is correct for MS Teams.
The issue is actually a groovy syntax error. You can easily check this in something like https://groovy-playground.appspot.com/ by adding your def payload = ... statement.
There are multiple ways to get multiline strings in groovy:
triple single quoted string
triple double quoted string
slashy string
dollary slashy string
Apart from the single quoted string, they also have a secondary property which is interpolation
Notice how in the initial JSON payload, there's a "$schema" key? Using triple double quoted strings makes groovy want to find a schema variable and use it's value to construct that payload variable.
You have two separate solutions:
Use triple single quoted string - just update """ to '''
Escape the variable - just update "$schema" to "\$schema" (making $ a literal $ instead of it being used as an interpolation prefix)

Neither JsonOutput.toJson or writeJSON is producing strings with quotes(JSON)

I'm trying to convert a map in jenkinsfile to json for which I'm using
def pullRequestData = [
title: "Deployment-to-SIT",
description: "Pull request from",
state: "OPEN",
open: "true",
closed: "false",
fromRef: [
id: "refs/heads/${branchjsonObj.displayId}",
repository: [
slug: "${reposlug}",
name: "null",
project: [
key: "${projectkey}"
]
]
]
]
def jsonmap = JsonOutput.toJson(pullRequestData)
echo ${jsonmap}
gives the output as json formatted but without strings like,
{title:Deployment-to-SIT,description:Pull request from,state:OPEN,open:true,closed:false,fromRef:{id:refs/heads/deployment-to-SIT,repository:{slug:argocd-sample-chart,name:null,project:{key:TD}}}}
but the output i need is
{"title":"Deployment-to-SIT","description":"Pull request from","state":"OPEN","open":"true","closed":"false","fromRef":{"id":"refs/heads/deployment-to-SIT","repository":{"slug":"argocd-sample-chart","name":"null","project":{"key":"TD"}}}
Also tried,
def jsonpullRequestData = this.steps.writeJSON(file: 'jsonmap.json', json: pullRequestData)
But the behavior is same,o/p is without quotes.Any help will be much appreciated.
Instead of using json groovy lib.. as per your requirement you can directly use $pullRequestData.toString() method to get in string format with quotes.

how to nest an array inside an object in yaml?

suppose you have a Map<String, Object> called "something" in YAML
something:
and the corresponding JSON should look like this:
json
"something": {
"else": "then",
"array": [
"element in array"
]
}
so for this yaml spec might be:
something:
else: then
array:
- element in array
but since something is a Map it does not let me do
array:
- element in array
or this
array: ['element in array']
so the question is what should be the yaml to get the above mentioned JSON considering something is a Map<String, Object> is it possible?
This is regarding the defining of the ServiceCatalogDefinition for the implementation of OpenServiceBroker API.
OSB Catalog using Yaml
OSB Catalog json looks like this
I am trying to make the "properties" mentioned in schemas in the above link as required.
for that I need to make it return the json like this:
"properties" : {
"someProperty" : {
"description": "description",
"type": "string"
},
"required": [
"someProperty"
]
}
And the yaml does validation in my application.yml throwing the error mentioned in comment
There is two things you need to do:
make the JSON valid, e.g. by inserting a comma (as #flyx suggests) and adding curly braces around the root level object:
{
"something": {
"else": "then",
"array": [
"element in array"
]
}
}
change the plain scalar (i.e. without quotes) mapping key something, to a double quoted scalar:
{
"something": {
"else": "then",
"array": [
"element in array"
]
}
}
Since YAML has, for all practical purposes, effectively been a superset of JSON (since YAML 1.2 from 2009), you don't need to do anything else. And of course you can read the above with both a YAML loader, as well as with a JSON parser.
Using the site json2yaml, you get YAML :
---
something:
else: then
array:
- element in array
from the json :
{
"something": {
"else": "then",
"array": [
"element in array"
]
}
}
Compare to you, I think it's your "-" must be to the same level as "array".

How to document multiple content types in successful GET response in swagger

Let's say we have an example json swagger spec:
{
"swagger": "2.0",
"info": {
"version": "1.0.0",
"title": "Some API"
},
"basePath": "/api/v1",
"consumes": [
"application/json"
],
"produces": [
"application/json",
"text/csv"
],
"paths": {
"/some/endpoint": {
"get": {
"parameters": [
{
"in": "body",
"name": "body",
"required": false,
"schema": {
"$ref": "#/definitions/BodyParamsDefinition"
}
}
],
"responses": {
"200": { ?? } ...
There are two content types that can be produced:
application/json
text/csv
Default response for GET /some/endpoint is a csv file, but if the format query param is used like this /some/endpoint?format=json, the response would be in json format.
I have trouble finding how should I finish my specification with proper responses.
When I use this approach: https://swagger.io/docs/specification/describing-responses/ i get a validation error: ...get.responses['200'] should NOT have additional properties
You are almost there, you just need to define a schema for the response. This schema defines the response structure for all content types associated with this status code.
For example, if the operation returns this JSON:
[
{
"petType": "dog",
"name": "Fluffy"
},
{
"petType": "cat",
"name": "Crookshanks"
}
]
and this CSV:
petType,name
dog,Fluffy
cat,Crookshanks
you would use:
# YAML
responses:
200:
description: OK
schema:
type: array
items:
type: object
properties:
petType:
type: string
name:
type: string
More info: Describing Responses
In OpenAPI 3.0, content type definitions were improved and schemas can vary by content type:
openapi: 3.0.0
...
paths:
/some/endpoint:
get:
responses:
'200':
description: OK
content:
# JSON data is an object
application/json:
schema:
type: object
properties:
message:
type: string
# CSV data is a string of text
text/csv:
schema:
type: string
Default response for GET /some/endpoint is a csv file, but if the format query param is used like this /some/endpoint?format=json, the response would be in json format.
There's currently no way to map specific responses to specific operation parameters, but there are several related proposals in the OpenAPI Specification repository:
Accommodate legacy APIs by allowing query parameters in the path
Querystring in Path Specification
Support an operation to have multiple specifications per path
Overloading