How to pass only integer value in key value in nifi - json

I have a use case, where i wants to pass only the integer value in the key datasum, my input data is randomly generates with both the integer and string, whenever the datasum contain the string as below it should replace with 0
input JSON
[
{
"hostname": "ip-192-168-1-1.demo-sample.com",
"detail": "Host name",
"datasum": "ip-192-168-1-1.demo-sample.com",
"toolname": "nifi",
"datacount": "1"
},
{
"hostname": "demo-env",
"detail": "System information",
"datasum": "Windows DEMO-ENV 10.0.19 Microsoft Windows Server 2016",
"toolname": "nifi",
"datacount": "1"
}
]
in above two records datasum having the string value and we need to replace it by 0
expected output
[
{
"hostname": "ip-192-168-1-1.demo-sample.com",
"parametername": "Host name",
"datasum": 0,
"toolname": "nifi",
"datacount": "1",
"severity": "1",
"createdby": "ApacheNifi"
},
{
"hostname": "demo-env",
"parametername": "System information",
"datasum": 0,
"toolname": "nifi",
"datacount": "1",
"severity": "1",
"createdby": "ApacheNifi"
}
]
please suggest.

You can use toInteger function within a modify transformation spec along with a second argument equals to zero such as
[
{
"operation": "modify-overwrite-beta",
"spec": {
"*": {
"datasum": ["=toInteger", 0]
}
}
}
]
In this case, the function would silently stop whenever the value of datasum is string, eg not convertible to integer
the demo on the site http://jolt-demo.appspot.com/ is

Related

Convert Nested JSON to Flat JSON using jolt Spec based on multiple inputs (if else)

I need to convert the nested json to flat json by given type of input, if its flat json or nested json gives output as flat json respectively. I haven't seen any correct resource related to the this topic. I given the inputs and outputs below.
I am having some trouble with converting the nested JSON to flat JSON, and didn't get any closer as to what is mentioned problem. I need to transform a JSON structure by using a JOLT spec. I use https://jolt-demo.appspot.com to test the following below. Help me, how can i get the expected out, with this code
Input-1:
{
"MessageType": "CREATION",
"Number": "123",
"Status": "created sucessfully",
"StopSequence": "1",
"Code": [
{
"CodeName": "ABC",
"ShortDescription": "short description about ABC",
"TimeImpact": 234,
"Rank": 1
},
{
"ReasonCodeName": "XYZ",
"ShortDescription": "short description about ABC",
"TimeImpact": 123,
"Rank": 2
}
]
}
Input-2:
{
"MessageType": "UPDATE",
"Number": "345",
"PNumber": "P123",
"Status": "updated sucessfully",
"StopSequence": "2",
"Id": 1234,
"LNumber": "34565",
"DeviceID": "7645235",
"Timestamp": "2015-10-01T16:00:00-05:00",
"Timezone": "US/New_York",
"TimezoneShortName": "EST",
"Unlocode": "XXXX",
}
Expected Output-1 if input 1 passes:
[
{
"MessageType": "CREATION",
"Number": "123",
"Status": "created sucessfully",
"StopSequence": "1",
"Code": "1",
"CodeName": "ABC",
"ShortDescription": "short description about ABC",
"TimeImpact": 234,
"CodeRank": 1
},
{
"MessageType": "CREATION",
"Number": "123",
"Status": "created sucessfully",
"StopSequence": "1",
"Code": "2",
"ReasonCodeName": "XYZ",
"ShortDescription": "short description about ABC",
"TimeImpact": 123,
"Rank": 2
}
]
Expected Output-2 if input 2 passes:
{
"MessageType": "UPDATE",
"PNumber": "P123",
"Status": "updated sucessfully",
"StopSequence": "2",
"Id": 1234,
"Timestamp": "2015-10-01T16:00:00-05:00",
"Timezone": "US/New_York",
}
What you need is to repeat as much as the number of objects of the Code array. So, walk through by them as picking the related values from the out of the array such as
[
{
"operation": "shift",
"spec": {
"Code": {
"*": {
"#(2,MessageType)": "[&1].MessageType", // "#(2,MessageType)" means going up the tree 2 levels to reach the position of "MessageType" , [&1] means combining the values as array of objects depending on the indexes of the array
"#(2,Number)": "[&1].Number",
"#(2,Status)": "[&1].Status",
"#(2,StopSequence)": "[&1].StopSequence",
"#(0,Rank)": "[&1].Code", // to copy the values of the "Rank" to "Code"
"*": "[&1].&"
}
},
"MessageT*|PNu*|St*|Id|Times*|Timezone": "&"
}
}
]

How to get JSON data from attributes that start with # or # in Typescript

I am working with a specific API that returns a JSON that looks like the below sample.
I want to get both values that contain the #text and #attr but I get error messages in typescript when I try to get the values.
try using,
album[0]["#attr"]
album[0]["artist"]["#text"]
Hey for JSON you can use get details by its attribute name in it and it's the same for all-weather it starts with # or # it will be the same.
See below code to get the value of your specified key:
Sample JSON:
{
"weeklyalbumchart": {
"album": [
{
"artist": {
"mbid": "data",
"#text": "Flying Lotus"
},
"mbid": "data",
"url": "",
"name": "",
"#attr": {
"rank": "1"
},
"playcount": "21"
},
{
"artist": {
"mbid": "data",
"#text": "Flying Lotus"
},
"mbid": "data",
"url": "",
"name": "",
"#attr": {
"rank": "1"
},
"playcount": "21"
}
]
}
}
Read JSON:
#attr ===> json["weeklyalbumchart"]["album"][0]["#attr"]
#text ===> json["weeklyalbumchart"]["album"][0]["artist"]["#text"]
Hope this will help you to understand it.

flatten a structure where the elements can be an object or a nested array

I have a JSON document that I would like to flatten where I get a resultant object resemling this:
[
{
"name": "Form/Field/Tokens.php",
"line": "62",
"severity": "info",
"message": "Expected #param annotation",
"source": "PhanCommentParamOutOfOrder"
}
]
In order to do this, I would need to flatten the file object but different ones represent the nested error node as an object or an array. I'm not sure how to flatten it.
Here's the given document.
{
"version": "6.5",
"file": [
{
"name": "Form/Field/Tokens.php",
"error": {
"line": "62",
"severity": "info",
"message": "Expected #param annotation",
"source": "PhanCommentParamOutOfOrder"
}
},
{
"name": "Console/Command/Reference.php",
"error": [
{
"line": "40",
"severity": "warning",
"message": "Possibly zero references to use statement",
"source": "PhanUnreferencedUseNormal"
},
{
"line": "54",
"severity": "warning",
"message": "Property has undeclared type",
"source": "PhanUndeclaredTypeProperty"
}
]
}
]
}
Given JQ's type builtin, you just need an if-then-else expression to determine if .file.error is an array or an object.
.file | map({name} + (.error |
if type == "array" then .[] else . end))
Online demo

Transform JSON file to add keys to array values

I am using Apache NiFi and receive from an embedded micro over a TCP/IP socket a JSON file of the form:
{
"id": 123456,
"ip": "192.168.1.1",
"t": -12.9,
"T": -23.8,
"variables": [
"user1",
0,
-12.97,
23.87
]
}
and would like to transform it such that the keys of the variables are added as they are known to me as follows:
{
"id": 123456,
"ip": "192.168.1.1",
"t": -12.9,
"T": -23.8,
"variables": [
"username" : "user1",
"valid" : 0,
"temperature 1" : -12.97,
"temperature 2" : 23.87
]
}
and then to be able to access a key value pair such as variables.username.
I have tried using JoltTransformJSON but don't know how to write the spec correctly if it can do it!! I have written the following jolt spec:
[
{
"operation": "shift",
"spec": {
"id": "id",
"ip": "ip",
"t": "t",
"T": "T",
"variables": {
"username": "",
"valid": "",
"temperature 1": "",
"temperature 2": ""
}
}
}
]
I also have tried using UpdateRecord with Record Reader/Writer but passing the correct schema causes an error on the first variable as it can't match "user1" with a key from the initial JSON.
I have managed to work out the Jolt Spec as follows:
[
{
"operation": "shift",
"spec": {
"id": "id",
"ip": "ip",
"t": "t",
"T": "T",
"variables": {
"0": "username",
"1": "valid",
"2": "temperature 1",
"3": "temperature 2"
}
}
}
]
Thank you daggett for your contribution.

Error running template Expected a hash, but this has evaluated to a string (wrapper: f.t.SimpleScalar):

When tried reading a hash using Freemarker, it is reading the string variable having same name as that of hash prior to that, even though the string variable exists at different sub-level.
The JSON file is in below format.
"Record": {
"#id": "Vehicle#1234#",
"Color": {
"#id": "MasterCode#5678#",
"Code": "RED",
"CodeClass": "",
"CreatedBy": "system",
"CreatedOn": "2015-09-10T00:00:00.0",
"CustomFields": {
},
"DisplayOrder": "999",
"Name": "RED",
"NibrsType": "",
"Recnum": "5678",
"Type": "XXXX"
},
"Type": {
"#id": "MasterCode#9012#",
"Code": "MPVH",
"CodeClass": "",
"CreatedBy": "system",
"CreatedOn": "2015-09-10T00:00:00.0",
"CustomFields": {
},
"DisplayOrder": "999",
"Name": "SUV TYPE VEHICLE",
"NibrsType": "",
"Recnum": "9012",
"Type": "YYYY"
}
}
when I tried reading payload.Record.Type hash, freemarker is reading the string under payload.Record.Color.Type and gives following error.
Error running template Expected a hash, but this has evaluated to a string (wrapper: f.t.SimpleScalar):
Looking for some suggestions to get rid of this.