this is a simple question,I am new to nifi and jolt. I just wanted to know,how to travserse the jolt spec while using wild card characters.For example,this is an example in jolt demo site,
input is
{
"data": {
"1234": {
"clientId": "12",
"hidden": true
},
"1235": {
"clientId": "35",
"hidden": false
}
}
}
Spec is
[
{
"operation": "shift",
"spec": {
"data": {
"*": {
"hidden": {
"true": {
// if hidden is true, then write the value disabled to the RHS output path
// Also #(3,clientId) means lookup the tree 3 levels, to the "1234" or "1235" level,
// and then come back down down the tree and grabe the value of "clientId"
"#disabled": "clients.#(3,clientId)"
},
"false": {
"#enabled": "clients.#(3,clientId)"
}
}
}
}
}
}
]
and output is
{
"clients" : {
"12" : "disabled",
"35" : "enabled"
}
}
How did we get the above output? like what #(3,clientsid).As far as I understand, it goes 3 levels up.But 3 levels with respect to what..the spec or the input? Either way,how to move 3 levels up,can you please define which are the levels here?
Thanks in advance
Just count the number of each opening curly-braces({) or colons(:) in the backward direction. Whenever they're not independent such as :{, then count this as only one in order to reach to the target key which is "*" wildcard just under "data" key in this case, and for #(3,clientId); first level is crossing the colon(:) next to "#disabled" or "#enabled", second level is crossing opening curly-braces next to those boolean keys for each, and then the third level is crossing opening curly-brace just after "hidden" key to reach the indexes the object with the "data" key.
Related
Value input
"ItemInternalId" : [ "01", "02", "011055000000345" ]
Value output expected
"ItemInternalId" : [ "01", "16", "011055000000345" ]
Jolt only has two functions, "lastElement" and "firstElement", so I couldn't find anything on the internet that modified the second value
No need to use a modify transformation which has those functions, but a shift transformation spec along with index of the respective component such as
[
{
"operation": "shift",
"spec": {
"*": { //stands for "ItemInternalId"
"1": { // the level of the second component of the array in which index starts from zero
"#16": "&2" // the fixed value might be assigned by prepending the value with "#" on the left-hand-side, &2 represents going the tree two levels up to grab the tag of the array
},
"*": "&1"//else case. The &1 on the right-hand-side represents going the tree one level up to grab the tag of the array
}
}
}
]
assuming your input is
{
"ItemInternalId": [
"01",
"02",
"011055000000345"
]
}
the demo on the site http://jolt-demo.appspot.com/ is
I am struggling to understand if it's possible to write a json schema that requires certain properties, but also allows those properties to be in different areas of the json file (e.g. a property value can be in the main top-level object OR it can be in an array - it just needs to be somewhere).
For example, I have some devices that collect multiple temperature records over the course of a few hours and send the records in batches. However, some of the devices send the software version once in the main object, while others send the software version along with each hourly temperature record (inside a "records" array).
Example 1 (swversion sent once in main object):
{
"name": "device1",
"swversion": "1.3.abc2",
"records": [
{
"time": "10am",
"temp": 2
},
{
"time": "11am",
"temp": 4
}
]
}
Example 2 (swversion sent inside "records" array):
{
"name": "device1",
"records": [
{
"time": "10am",
"temp": 2,
"swversion": "1.3.abc2"
},
{
"time": "11am",
"temp": 4,
"swversion": "1.3.abc2"
}
]
}
Using these examples, I would like to write my schema definition as follows (the first two bullets are easy, the last one is where I'm struggling):
Main object requires name property and records array
records array can contain objects where time and temp would be required
swversion is required somewhere (could be in the main object or inside records array)
Is there a feature I'm missing in json-schema that enforces required properties, yet allows the flexibility for said properties to be anywhere (e.g. within an object OR an array), as long as they are present somewhere?
The anyOf keyword is a boolean OR operation. At least one of the schemas must pass for the keyword to pass. The first schema requires that the "swversion" property is present at the top level. The second schema requires that the "swversion" property is required in each of the items in the "record" array.
{
... define the easy stuff here, then ...,
"anyOf": [
{ "required": ["swversion"] },
{
"properties": {
"records": {
{ "items": { "required": ["swversion"] } }
}
}
}
]
}
In this example, "swversion" could appear in both places. If you only want to ensure that it only appear in one place (top level or items), you can use oneOf instead of anyOf.
Is there a feature .. that allows the flexibility for said properties to be anywhere
Not directly, but it's not difficult to express this. You can define the structure of "swversion" itself in a definition that is re-used via a reference.
In pseudocode, that would be:
any of:
the main object contains a "swversion" property,
all the items under "records" contain a "swversion" property
In code:
{
"$defs": {
"swversion": {
"type": "string",
.. other constraints?
}
},
"type": "object",
"properties": {
... other property definitions ...,
"records": {
"items": {
"type": "object",
... other definitions for the mandatory portion of records ...
}
}
},
"anyOf": [
{
"$comment": "swversion is a member of the main object",
"required": [ "swversion" ],
"properties": {
"swversion": {
"$ref": "#/$defs/swversion"
}
},
{
"$comment": "swversion is a member of all the items under the records property",
"properties": {
"records": {
"items": {
"type": "object",
"required": [ "swversion" ],
"properties": {
"swversion": {
"$ref": "#/$defs/swversion"
}
}
}
}
}
}
],
}
Note that if you are using JSON Schema version draft7 or earlier, change $defs to definitions.
I've a JSON blob in the following form:
{
"allEntries": [
{
"property": {
"type": "ab"
}, "values": {
"score": 10.40
}
},
{
"property": {
"type": "ty"
}, "values": {
"score": 90.45
}
}
]
}
I want to just check if the score of property type ab is less than 10. However $min(allEntries.values.score) goes through all the properties and does not filter out types which I'm not interested in.
I tried using the parent property '%.' however that doesn't work either (The object representing the 'parent' cannot be derived from this expression)
You haven't said what the desired output is, but you can filter the list of array entries using the following expression:
allEntries[property.type='ab' and values.score < 10]
See https://try.jsonata.org/BzJKGrIIG
I'm a newbie on Apache Nifi and have the following Problem: I would like to transform a json file as follows:
From:
{
"Property1": "x1",
"Property2": "Tag_**2ABC**",
"Property3": "x3",
"Property4": "x4"
}
to:
{
"**2ABC**_Property1": "x1",
"**2ABC**_Property3": "x3",
"**2ABC**_Property4": "x4"
},
it means: taking the value from a certain Attribute to update all other attributes.
I could find examples using JoltTransformer-Processor that works well when the update is only adding a string. But not for my case
What I've done so far: I have set each Attribute using evaluateJSONPath processor. But I just tried quite a lot of possibilities to use the update Attribute processor to do it without success. All my possible tests looked like (within UpdateAttribute):
Property1 --> ${'Property2':substring(4,6)}"_"${'Property1'}
Using Jolt:
[
{"operation": "modify-overwrite-beta",
"spec": {
"Property1": "${'Property2':substring(4,6)}_${'Property1'}"
}
}
]
Which point am I missing here? Thanks in advance!
I don't know about Nifi, but here is how you can do it in Jolt.
Spec
[
{
"operation": "shift",
"spec": {
// match Property2
"Property2": {
"Tag_*": { // capture the nasty "**2ABC**" part to reference later
// go back up the tree to the root
"#2": {
// match and ignore Property2
"Property2": null,
//
// match Property* and use it and the captured
// "prefix" to create the output key
// &(2,1) references the Tag_*, and pull off the "**2ABC**" part
"Property*": "&(2,1)_&"
}
}
}
}
}
]
I have a JSON input that I am transform using JOLT shift. My problem is I want to use the value of an input key, as a new key in the output data, and in parallel add another value into that new outputted key. Here is my input:
"Description": {
"Name": "John",
"KeyNameId": "John123",
"Description": "John's description"
}
And I want my output to be:
"Description": {
"John123": "John's description"
}
Anyway to do this without using two shift operations?
Or with two shifts if one isn't possible?
Yes, it can be done in a single shift using the "#(Number,words)" operator.
Input - slightly modified for clarity
{
"Top": {
"Name": "John",
"KeyNameId": "John123",
"Description": "John's description"
}
}
Spec
[
{
"operation": "shift",
"spec": {
"Top": {
// match the key "Description" and copy it's value to the Output.
// The Output path being defined by #(1,KeyNameId), which means
// go back up the tree 2 levels (0,1) and lookup the value of
// "KeyNameId"
"Description": "#(1,KeyNameId)"
}
}
}
]
To be more precise,
[
{
"operation": "shift",
"spec": {
"Description": {
"#Description": "Description.#KeyNameId"
}
}
}
]