I came across this question regarding replacing a non-existing field with another field's value. It explains the usage of modify-default-beta operation. Example mentioned in the post:
Spec
[
{
"operation": "modify-default-beta",
"spec": {
"shipping_address": {
"address": "#(2,payment_address.address)"
}
}
}
]
Input A, where there is not shipping address
{
"payment_address": {
"address": "some address"
},
"shipping_address": {}
}
Produces output A, where the billing address is copied over
{
"payment_address" : {
"address" : "some address"
},
"shipping_address" : {
"address" : "some address"
}
}
What is the meaning of the "2" in #(2,payment_address.address). I tried this example here and it works even if I replace the "2" with a "3".
The Jolt operations do a parallel tree walk of the input JSON and the Spec. It starts at the root the Spec & input JSON and then does a depth first traversal.
While it is is doing it's depth first traversal it maintains a "stack" of the data / nodes that it has matched.
Thus in this spec, when you "match" down to "address"
"operation": "modify-default-beta",
"spec": {
"shipping_address": {
"address": "#(2,payment_address.address)"
}
}
The stack looks like :
Stack "pointer" Matched value Pointer to Input
0 "address" Value of Address : String if it exists
1 "shipping_address" Value of "shipping_address" : Map if it exists
2 "_root_" A made up entry to point to Input Json (Map or List)
3 "_root_" Another "root" that points to a Map
WorkAround to deal with top level List JSON input
So the "2" gets you back up to the "top level" of the input Json, so that you can "navigate" down "payment_address.address".
"3" works because you are now in the wrapper that makes it so that all "top level" input to the Transforms is a Map, to workaround the fact that a top level list / "[]" if valid JSON. This wrapper is special cased to have the same reference as the "2".
"4" does not not exist in the stack, and so it doesn't do anything.
Related
I am trying to create an azure policy that audits vms. Conditions I want to satisfy is that the vm has all of the tags specified by parameter and that all of those corresponding tags contain a value. The first condition I have working with below. However for determining if they are blank or not is a bit more challenging as looks like you can not use current() in the field key.
{
"parameters": {
"requiredTags": {
"type": "Array",
"metadata": {
"displayName": "Required Tags",
"description": "The list of tags that should exist on the virtual machine"
}
}
},
"policyRule": {
"if": {
"allof": [
{
"field": "type",
"equals": "Microsoft.Compute/VirtualMachines"
},
{
"count": {
"value": "[parameters('requiredTags')]",
"where": {
"field": "tags",
"containsKey": "[current()]"
}
},
"notEquals": "[length(parameters('requiredTags'))]"
},
{
"count": {
"value": "[parameters('requiredTags')]",
"where": {
"field": "[concat('tags[', current(), ']')]",
"notEquals": ""
}
},
"notEquals": "[length(parameters('requiredTags'))]"
}
]
},
"then": {
"effect": "audit"
}
}
}
This was very tricky to say at least, and it seems like no such policy exists out there. Nevertheless, I believe that the 2 options below will do the trick - at least it did so when I tested it.
Option 1:
{
"not": {
"value": "[contains(string(field('tags')), '\"\"')]",
"equals": true
}
}
Option 2:
{
"value": "[indexOf(string(field('tags')), '\"\"')]",
"greaterOrEquals": 0
}
Description:
Option 1:
Use contains to check wheather an object contains a key or a string contains a substring.
The container contains nested parameters.
string converts the specified value to a string. In this case, the specified value is the field = tags, which are objects, not an array. In this case, the specified value is the field = tags, which are objects, not an array.
Example of 2 tags, "tagnumber1" with the value "value1" and "tagnumber2" with an empty value:
"{\"tagnumber1\":\"value1\",\"tagnumber2\":\"\"}"
Note that the empty value is \"\" - this is our itemToFind.
Option 2:
Use the indexOf to return the first position of a value within a string.
The stringToSearch contains nested parameters.
The stringToFind is empty.
string converts the specified value to a string. In this case, the specified value is the field = tags, which are objects, not an array.
Example of 2 tags, "tagnumber1" with the value "value1" and "tagnumber2" with an empty value:
"{\"tagnumber1\":\"value1\",\"tagnumber2\":\"\"}"
Note that the empty value is \"\".
Therefore, we must search for that \"\" as this represents the empty value in the object.
The index is zero-based. If the item is not found, -1 is returned. An integer represents the first index of the item, so by looking at "greaterOrEquals": 0 it will only return that is the item is found - meaning a tag value is empty.
Links:
https://learn.microsoft.com/en-us/azure/governance/policy/concepts/definition-structure#fields
https://learn.microsoft.com/en-us/azure/azure-resource-manager/templates/template-functions
https://learn.microsoft.com/en-us/azure/azure-resource-manager/templates/template-expressions#escape-characters
https://learn.microsoft.com/en-us/azure/templates/microsoft.compute/virtualmachines?pivots=deployment-language-arm-template#resource-format-1
Value input
"ItemInternalId" : [ "01", "02", "011055000000345" ]
Value output expected
"ItemInternalId" : [ "01", "16", "011055000000345" ]
Jolt only has two functions, "lastElement" and "firstElement", so I couldn't find anything on the internet that modified the second value
No need to use a modify transformation which has those functions, but a shift transformation spec along with index of the respective component such as
[
{
"operation": "shift",
"spec": {
"*": { //stands for "ItemInternalId"
"1": { // the level of the second component of the array in which index starts from zero
"#16": "&2" // the fixed value might be assigned by prepending the value with "#" on the left-hand-side, &2 represents going the tree two levels up to grab the tag of the array
},
"*": "&1"//else case. The &1 on the right-hand-side represents going the tree one level up to grab the tag of the array
}
}
}
]
assuming your input is
{
"ItemInternalId": [
"01",
"02",
"011055000000345"
]
}
the demo on the site http://jolt-demo.appspot.com/ is
this is a simple question,I am new to nifi and jolt. I just wanted to know,how to travserse the jolt spec while using wild card characters.For example,this is an example in jolt demo site,
input is
{
"data": {
"1234": {
"clientId": "12",
"hidden": true
},
"1235": {
"clientId": "35",
"hidden": false
}
}
}
Spec is
[
{
"operation": "shift",
"spec": {
"data": {
"*": {
"hidden": {
"true": {
// if hidden is true, then write the value disabled to the RHS output path
// Also #(3,clientId) means lookup the tree 3 levels, to the "1234" or "1235" level,
// and then come back down down the tree and grabe the value of "clientId"
"#disabled": "clients.#(3,clientId)"
},
"false": {
"#enabled": "clients.#(3,clientId)"
}
}
}
}
}
}
]
and output is
{
"clients" : {
"12" : "disabled",
"35" : "enabled"
}
}
How did we get the above output? like what #(3,clientsid).As far as I understand, it goes 3 levels up.But 3 levels with respect to what..the spec or the input? Either way,how to move 3 levels up,can you please define which are the levels here?
Thanks in advance
Just count the number of each opening curly-braces({) or colons(:) in the backward direction. Whenever they're not independent such as :{, then count this as only one in order to reach to the target key which is "*" wildcard just under "data" key in this case, and for #(3,clientId); first level is crossing the colon(:) next to "#disabled" or "#enabled", second level is crossing opening curly-braces next to those boolean keys for each, and then the third level is crossing opening curly-brace just after "hidden" key to reach the indexes the object with the "data" key.
I have a JSON input that I am transform using JOLT shift. My problem is I want to use the value of an input key, as a new key in the output data, and in parallel add another value into that new outputted key. Here is my input:
"Description": {
"Name": "John",
"KeyNameId": "John123",
"Description": "John's description"
}
And I want my output to be:
"Description": {
"John123": "John's description"
}
Anyway to do this without using two shift operations?
Or with two shifts if one isn't possible?
Yes, it can be done in a single shift using the "#(Number,words)" operator.
Input - slightly modified for clarity
{
"Top": {
"Name": "John",
"KeyNameId": "John123",
"Description": "John's description"
}
}
Spec
[
{
"operation": "shift",
"spec": {
"Top": {
// match the key "Description" and copy it's value to the Output.
// The Output path being defined by #(1,KeyNameId), which means
// go back up the tree 2 levels (0,1) and lookup the value of
// "KeyNameId"
"Description": "#(1,KeyNameId)"
}
}
}
]
To be more precise,
[
{
"operation": "shift",
"spec": {
"Description": {
"#Description": "Description.#KeyNameId"
}
}
}
]
I have a REST service that allows partial updates of a record using PATCH. For an example, let's say the REST service operates on a name record, where name has three fields - first, middle, and last name. A record for my user "Jane Q. User" looks like this in JSON:
{
"first" : "Jane",
"middle" : "Q.",
"last" : "User"
}
Now Jane Q. User has legally changed her last name to "Admin" and wants my service to quit displaying her middle name. Her new name is "Jane Admin".
Should she submit a PATCH request with the new last name and the middle is set to the empty string to clear out her middle name like this?
PATCH /myservice/users/1
{
"middle" : "",
"last" : "Admin"
}
I do not think there is a general best practice out there yet, but several ideas have been proposed. In this post I present three ideas (one of which is my own), but there may be many more.
JSON Patch
Website: http://jsonpatch.com/
RFC 6902: https://datatracker.ietf.org/doc/html/rfc6902
Example obtained from the website:
The original document
{
"baz": "qux",
"foo": "bar"
}
The patch
[
{ "op": "replace", "path": "/baz", "value": "boo" },
{ "op": "add", "path": "/hello", "value": ["world"] },
{ "op": "remove", "path": "/foo" }
]
The result
{
"baz": "boo",
"hello": ["world"]
}
This approach is very verbose and I suggest to use one of the libraries linked on website. It's not what you should implement over night yourself, imho, but it looks very powerful.
JSON Merge Patch
RFC 7396: https://datatracker.ietf.org/doc/html/rfc7396
Example from the spec:
{
"a": "b",
"c": {
"d": "e",
"f": "g"
}
}
patched with
{
"a":"z",
"c": {
"f": null
}
}
will change the value of "a" to "z" and remove "f". However, many JSON deserialization libraries (e.g. for JVM based languages) do not distinguish between null and undefined, so this does not really solve the original problem, unless you are writing your own deserializer to parse that JSON in a way to preserve the information about the explicit null.
More discussion about these two can be found here:
https://erosb.github.io/post/json-patch-vs-merge-patch/
Clear Fields
This is what I came up with by myself to keep it very simple. It is a non-standard custom solution.
A document like
{
"a": "foo",
"b": "bar"
}
can be patched with
{
"a": "baz",
"_clear:" {
"b:" true
}
}
The main idea is to include boolean fields in a special "_clear" object for only the optional fields. That makes it explicit (in an auto generated documentation based on the DTO classes) which fields can be removed and which can not. Omitting a field or writing null or undefined are all treated as "do not modify this field".
The syntax (underscored field name) is inspired by HAL+JSON, but afaik that never exited draft status.
Mmmmm I'd not do it quite that way. If possible I'd recommend instead ask folks to issue a PUT
PUT /myservice/users/1
{
"first" : "Jane",
"last" : "User"
}
And this would replace the user record.
However, depending on the potential size of the record, a PATCH is easier and so if you wanted to also/or use PATCH to update I'd recommend a null value.
PATCH /myservice/users/1
{
"middle" : null
}
However... if your API returns empty values in responses (which personally I don't like), then for consistency an empty value would probably make a tad more sense..
PATCH /myservice/users/1
{
"middle" : ""
}
However you end up doing it, just make sure you're consistent and when documenting your API, call out this functionality.
Cheers.