I am sending messages in JSON format to an ActiveMQ server. I am trying to use JMS Transformation to transform the JSON encoded object into a true Java Object in hopes of being able to use selectors on the data inside.
Here is a link to the documentation on Stomp and Message Transformation.
Here is a link to a discussion on the patch where someone shows an example of a legal JSON object
The format of the JSON objects I am sending (in pretty print) are similar to this:
{
"msg": {
"flag1" : "value1",
"flag2" : "value2"
}
}
The messages arrive in the message queue, but with the transformation-error property set to 'msg : msg'.
The only format accepted by the transformation jms-map-json or jms-object-json is a simple Map format, which in JSON is:
{"map" :
{"entry" :
[
{ "string1": [ "key1", "value1" ] },
{ "string2": [ "key2", "value2" ] }
]
}
}
This is the same format shown in the discussion forum. This format represents a name/value pair map object in java.
Selectors are only usable on Properties and Headers.
you can use any JSON notation for your jms-object-json transformations as long as XStream can handle it. You can take a look at test cases for some examples. There, we use SamplePojo class:
https://svn.apache.org/repos/asf/activemq/trunk/activemq-stomp/src/test/java/org/apache/activemq/transport/stomp/SamplePojo.java
which is properly annotated so it can be represented with the following JSON
{"pojo":{
"name":"Dejan",
"city":"Belgrade"
}}
You can try using the same approach for your classes.
Hope this helps,
Dejan
It should be mentioned that ActiveMQ version must at least 5.8, because with 5.6 version I had problem when transformation just did not work.
Related
I have a scenario where I can have multiple different types of json objects coming into my system. I do not know the object type ahead of time and based upon object type, will route to a different processor in my flow
{
"book": {
"id": "1234",
"name": "book1"
}
}
or
{
"video": {
"id": "3214",
"name": "video1"
}
}
or
{
"magazine": {
"id": "3233",
"name": "magazine1"
}
}
how can I evaluate if the object is a book, or a video, or a magazine to route to the correct processor
I've tried using evaluatejsonpath using the ~ but it just outputs the entire json object
Current flow :
One way is to extract all top level fields using EvaluateJsonPath processor, set the extracted field values to dynamic properties, and use the properties in RouteOnAttribute processor to route the flow to correct downstream processor.
EvaluateJsonPath:
Please don't forget to set
'Destination' to 'flowfile-attribute' and
'Return Type' to 'json'
If EvaluateJsonPath processor could not find the field or element, then the value of dynamic property will be set to empty string.
All we need to do is to use the dynamic properties in RouteOnAttribute processor.
RouteOnAttribute:
Using equals() and not()
or using isEmpty() and not()
Please don't forget to set
'Routing Strategy' to 'Route to Property Name'.
Apache NiFi expression language guide
Example Flow:
I am using PutFile processor as a downstream processor, for an example. It could be any processor.
I'm trying to convert a NiFi flow file containing JSON to an AVRO record.
The problem I have is that I don't know how to deal with a fixed type in AVRO, i.e. how to specifiy the proper JSON for converting to fixed?
Currently I'm using the ConvertJsonToAvro-processor.
The AVRO output schema:
{
"type" : "record",
"name" : "Message",
"namespace" : "com.example",
"fields" : [ {
"name" : "MAC",
"type" : {
"type" : "fixed",
"name" : "MY_FIXED_TYPE",
"size" : 6
}
}]
}
The input JSON-forms I tried are
{ "MAC": [ 0, 1, 2, 3, 4, 5] }
{ "MAC": "012345" }
{"MAC":"\u0000\u0001\u0002\u0003\u0004\u0005"}
{"MAC":{"MY_FIXED_TYPE": "\u0000\u0001\u0002\u0003\u0004\u0005"}}
Unfortunately none of them worked for me.
I also tried the ConvertRecord-processor instead of the ConvertJsonToAvro-processor. Also without any luck.
Any ideas?
After some further investigation, it looks like the ConvertJsonToAvro processor can't be used to generate Avro FIXED or BYTES datum. This is likely a bug with NiFi and how the processor uses Avro.
If I'm not mistaken:
The NiFi ConvertJsonToAvro uses the KiteSDK to interpret JSON into Avro data. This JSON-to-Avro conversion is not the same as Avro JSON encoding from the specification.
This processor reads the incoming string into a jackson JsonNode.
FIXED and BYTES types need to correspond to a JsonNode where isBinary() is true.
As far as I can tell, parsing a JSON string with Jackson never generates such a JSON node.
I would raise a NiFi JIRA about this, or an issue on the KiteSDK.
Note: this answer does not apply to NiFi JSON to Avro conversion. My apologies for the mistaken assumption! I'm unsure of the best practice for an answer known to be wrong.
An example of a "correct" Avro JSON encoding for the bytes type is given in the spec. I think you're looking for:
{"MAC":"\u0000\u0001\u0002\u0003\u0004\u0005"}
Or alternatively (for a fixed schema in a union):
{"MAC":{"MY_FIXED_TYPE": "\u0000\u0001\u0002\u0003\u0004\u0005"}}
You can correctly parse this input string using your given Schema.
I have a JSON response like below
{
"queryStartDate": "20170523134739822",
"queryEndDate": "20170623134739822",
"Rows": [
{
"hasScdHistoryOnly": false,
"Values": [
"1",
"53265",
"CO"
]
},
{
"hasScdHistoryOnly": false,
"Values": [
"1",
"137382",
"CO"
]
},
{
"hasScdHistoryOnly": false,
"Values": [
"1",
"310824",
"CO"
]
}
]
}
I am using Jmeter's JSON Extractor post-processor to receive the second value from the last of the 'Values' list. i.e. 53265, 137382, 310824.
I've tried to use $.Rows[*].Values[-2:-1], and $.Rows[*].Values[(#.length-2)], according to Stefan's introduction: http://goessner.net/articles/JsonPath/index.html#e2, but neither of them are working. Would you please help me out?
I believe JMeter is using JayWay JSON Path library, so you should be looking for the documentation here instead.
In general I would recommend using JSR223 PostProcessor as an alternative to JSON Path Extractors, both are applicable for basic scenarios only, when it comes to advanced queries and operators their behaviour is flaky.
Add JSR223 PostProcessor as a child of the request which returns above JSON
Make sure you have "groovy" selected in the "Language" drop down and "Cache compiled script if available" box is ticked
Put the following code into "Script" area
def values = com.jayway.jsonpath.JsonPath.parse(prev.getResponseDataAsString()).read('$..Values')
values.eachWithIndex { val, idx ->
vars.put('yourVar_' + (idx + 1), val.get(val.size()-2))
}
It should generate the following JMeter Variables:
yourVar_1=53265
yourVar_2=137382
yourVar_3=310824
which seem to be something you're looking for.
References:
Groovy: Parsing and producing JSON
Apache Groovy - Why and How You Should Use It
Using View Results tree's JSon Path Tester I could see that the following expression you used for extracting the values were not correct (correct for online JSONPath Online Evaluator but not working for JMeter)
Used Expression: $.Rows[*].Values[-2:-1]
Output from JSon Path Tester: No Match Found.
Used Expression: $.Rows[*].Values[(#.length-2)]
Output from JSon Path Tester: Exception: Could not parse token starting at position 16. Expected ?, ', 0-9, *
If the expression $.Rows[*].Values[1] is used it extracts the desired responses.
Used Expression: $.Rows[*].Values[1]
Output from JSon Path Tester:
Result[0]=53265
Result[1]=137382
Result[2]=310824
I've got a task to write a JSON parser in java with a little help.
I'm already able to parse this:
{
"ArrayWithOneString" : [ "ArrayContent" ],
"Array" : [
{
"ArrayinArray" : [
{
"NumberInArray" : 1337,
"StringInArray" : "String"
}
]
}
]
}
I've got only one last problem:
"string" : { // The bracket
"string" : "valueString"
},
My problem is that I expect a value and not another object for this opening bracket ({).
I wanted to ask if this is valid json before trying to parse it.
Yes it is valid. Well without your highlighting attempts, and assuming it is part of a parent object.
Just because you have a property called "string" doesn't mean it has to be a string value. I suggest perhaps whoever made it just isn't being very consistent, but it is still valid.
The question is, why are you expecting a value? Either the person who constructed the JSON has not done it to specification, or it is you that is not understanding the specification.
Also, you can easily validate JSON here.
Look here for specifications. Your example is valid according to this.
Yes it is valid JSON. You can now parse in your code.
You can check Valid JSON
See below screenshot.
Is there another way to track what is unmarshalled than write own reverter for each field?
I'm updating my local data based on json message and my problem is (simplified):
I'm expecting json like
{ "items": [ { "id":1, "name":"foobar", "price":"12.34" } ] }
which is then unmarshaled to class TItems by
UnMarshaller.TryCreateObject( TItems, TJsonObject( OneJsonElement ), TargetItem )
My problem is that I can't make difference between
{ "items": [ { "id":1, "name":"", "price":"12.34" } ] }
and
{ "items": [ { "id":1, "price":"12.34" } ] }
In both cases name is blank and i'd like to update only those fields that are passed on json message. Of course I could create a reverted for each field, but there are plenty of fields and messages so it's quite huge.
I tried to look REST.Jsonreflect.pas source, but couldn't make sense.
I'm using delphi 10.
In Rest.Json unit there is a TJson class defined that offers several convenience methods like converting objects to JSON and vice versa. Specifically, it has a class function JsonToObject where you can specify options like for example ignore empty strings or ignore empty arrays. I think the TJson class can serve you. For unmarshalling complex business objects you have to write custom converters though.
Actually, my problem was finally simple to solve.
Instead of using TJSONUnMarshal.tryCreateObject I use now TJSONUnMarshal.CreateObject. First one has object parameters declared with out modifier, but CreateObject has Object parameter var modifier, so I was able to
create object, initalize it from database and pass it to CreateObject which only modifies fields in json message.